OPATCHAUTO Bug on 19.3 GI/RDBMS when Patching to 19.6

OPATCHAUTO Bug on 19.3 GI/RDBMS when Patching to 19.6

The other day I was doing a fresh install of Oracle 19.3, using the binaries from edelivery , and doing some testing in that version and then patch it to 19.6. The installation and the tests went fine. What I did next is proceed with the analyze of the patch 30501910 and saw that everything was ok . 

## Note.- I trimmed the log for readability
[root@node2 ~]$ . oraenv
ORACLE_SID = [root] ? DB193H12
The Oracle base has been set to /u01/app/oracle
[root@node2 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH
[root@node2 ~]$ opatchauto apply /u01/patches/30501910 -analyze
OPatchauto session is initiated at Tue Oct  6 13:23:11 2020
...
Host:node2
CRS Home:/u01/app/19.3.0.0/grid
Version:19.0.0.0.0

==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /u01/patches/30501910/30489227
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log

Patch: /u01/patches/30501910/30489632
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log

Patch: /u01/patches/30501910/30655595
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log

Patch: /u01/patches/30501910/30557433
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log

OPatchauto session completed at Tue Oct  6 11:42:46 2020

Time taken to complete the session 2 minutes, 41 seconds


So I proceeded with the patch  with the command below 
opatchauto apply /u01/patches/30501910

Everything went fine in node 1, the patching completed successfully in GI and RDBMS, but the patch on node 2 failed on the RDBMS with the error below 

## Note.- I trimmed the log for readability
[root@node2 ~]$ opatchauto apply /u01/patches/30501910
...
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.3.0.0/dbhome_1, host: node2.
Command failed:  /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/opatchauto  apply /u01/patches/30501910 -oh /u01/app/oracle/product/19.3.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.3.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_node2_rac_2.ser -analyzedresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_node2_rac_2.ser
Command failure output: 
==Following patches FAILED in apply:

Patch: /u01/patches/30501910/30489227
Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-05_17-25-22PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' 

After fixing the cause of failure Run opatchauto resume
...

After looking for information, I found MOS note 2582139.1 in which it basically states that I need to copy the oui-patch.xml from node 1 to node2. 

[grid@node1 ~]$ cd /u01/app/oraInventory/ContentsXML/
[grid@node1 ContentsXML]$ scp oui-patch.xml node2:/u01/app/oraInventory/ContentsXML/
oui-patch.xml   
 
[root@node2 ContentsXML]# ls -ltr
total 20
-rw-rw----. 1 grid   oinstall 565 Oct  6 11:13 inventory.xml
-rw-r--r--. 1 oracle oinstall 174 Oct  6 12:43 oui-patch.xml.back
-rw-rw----. 1 grid   oinstall 300 Oct  6 12:43 comps.xml
-rw-rw----. 1 grid   oinstall 292 Oct  6 12:43 libs.xml
-rw-rw----. 1 grid   oinstall 174 Oct  6 14:02 oui-patch.xml                                                                        

So after doing this , I reran the opatchauto and now faced a the error Unable to create patchObject

## Note.- I trimmed the log for readability
[root@node2 ~]$ opatchauto apply /u01/patches/30501910
...
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.3.0.0/dbhome_1, host: node2.
Command failed:  /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/opatchauto  apply /u01/patches/30501910 -oh /u01/app/oracle/product/19.3.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.3.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_node2_rac_2.ser -analyzedresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_node2_rac_2.ser
Command failure output: 
==Following patches FAILED in apply:

Patch: /u01/patches/30501910/30489227
Log: 
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/30489227 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19.3.0.0/dbhome_1/inventory/oneoffs/30489227/etc/config/actions" or "/u01/app/oracle/product/19.3.0.0/dbhome_1/inventory/oneoffs/30489227/etc/config/inventory" does not exist. 

After fixing the cause of failure Run opatchauto resume
...

For this I found another note 2286025.1 in which it is saying to do any of the following :

  1. Opatch lsinventory
  2. opatch -analyze
  3. Restore Database home backup for Hostname2.

But after doing points 1 and 2, I kept getting the same error, so what I proceeded to do is patch the GI on node2

## Note.- I trimmed the log for readability
[root@node2 ~]$ . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/grid
[root@node2 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH
[root@node2 ~]$ opatchauto apply /u01/patches/30501910 -oh $ORACLE_HOME
OPatchauto session is initiated at Tue Oct  6 12:28:25 2020

...

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:node2
CRS Home:/u01/app/19.3.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/patches/30501910/30489227
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log

Patch: /u01/patches/30501910/30489632
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log

Patch: /u01/patches/30501910/30557433
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log

Patch: /u01/patches/30501910/30655595
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log

OPatchauto session completed at Tue Oct  6 12:49:02 2020
Time taken to complete the session 20 minutes, 38 seconds

For the RDBMS I followed my own blog post (How to clone an oracle home in 19c) and cloned the Oracle Home from node1 , as this one was already patched correctly with 30501910

[oracle@node1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
[oracle@node1 ~]$ $ORACLE_HOME/runInstaller -silent  -createGoldImage -destinationLocation /u01/patches
Launching Oracle Database Setup Wizard...

Successfully Setup Software.
Gold Image location: /u01/patches/db_home_2020-10-06_02-11-59PM.zip

[oracle@node1 ~]$ unset ORACLE_HOME
[oracle@node1 ~]$ unset ORACLE_SID
[oracle@node1 ~]$ unset ORACLE_RSID
[oracle@node1 ~]$ unset ORACLE_UNQNAME
[oracle@node1 ~]$ unset ORACLE_BASE

[oracle@node1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_2
[oracle@node1 ~]$ export ORACLE_HOSTNAME=node1
[oracle@node1 ~]$ export ORA_INVENTORY=/u01/app/oraInventory
[oracle@node1 ~]$ export NODE1_HOSTNAME=node1
[oracle@node1 ~]$ export NODE2_HOSTNAME=node2
[oracle@node1 ~]$ export ORACLE_BASE=/u01/app/oracle

[oracle@node1 ~]$ unzip -q /u01/patches/db_home_2020-10-06_02-11-59PM.zip -d $ORACLE_HOME
 
[oracle@node1 ~]$ env | egrep "ORA|NODE"
ORACLE_BASE=/u01/app/oracle
NODE2_HOSTNAME=node2
ORACLE_HOSTNAME=node1
NODE1_HOSTNAME=node1
ORA_INVENTORY=/u01/app/oraInventory
ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_2

[oracle@node1 ~]$ cd $ORACLE_HOME
[oracle@node1 dbhome_2]$ ${ORACLE_HOME}/runInstaller -ignorePrereq -waitforcompletion -silent \
> -responseFile ${ORACLE_HOME}/install/response/db_install.rsp \
> oracle.install.option=INSTALL_DB_SWONLY \
> ORACLE_HOSTNAME=${ORACLE_HOSTNAME} \
> UNIX_GROUP_NAME=oinstall \
> INVENTORY_LOCATION=${ORA_INVENTORY} \
> SELECTED_LANGUAGES=en \
> ORACLE_HOME=${ORACLE_HOME} \
> ORACLE_BASE=${ORACLE_BASE} \
> oracle.install.db.InstallEdition=EE \
> oracle.install.db.OSDBA_GROUP=dba \
> oracle.install.db.OSOPER_GROUP=dba \
> oracle.install.db.OSBACKUPDBA_GROUP=dba \
> oracle.install.db.OSDGDBA_GROUP=dba \
> oracle.install.db.OSKMDBA_GROUP=dba \
> oracle.install.db.OSRACDBA_GROUP=dba \
> oracle.install.db.CLUSTER_NODES=${NODE1_HOSTNAME},${NODE2_HOSTNAME} \
> oracle.install.db.isRACOneInstall=false \
> oracle.install.db.rac.serverpoolCardinality=0 \
> oracle.install.db.config.starterdb.type=GENERAL_PURPOSE \
> oracle.install.db.ConfigureAsContainerDB=false \
> SECURITY_UPDATES_VIA_MYORACLESUPPORT=false \
> DECLINE_SECURITY_UPDATES=true
Launching Oracle Database Setup Wizard...
...
The response file for this session can be found at:
 /u01/app/oracle/product/19.3.0.0/dbhome_2/install/response/db_2020-10-06_03-06-10PM.rsp

You can find the log of this install session at:
 /u01/app/oraInventory/logs/InstallActions2020-10-06_03-06-10PM/installActions2020-10-06_03-06-10PM.log

As a root user, execute the following script(s):
	1. /u01/app/oracle/product/19.3.0.0/dbhome_2/root.sh

Execute /u01/app/oracle/product/19.3.0.0/dbhome_2/root.sh on the following nodes: 
[node1, node2]

After doing this, the only thing I needed to do was verify that the Oracle Home is patched correctly and switch the database to the new Oracle Home

[oracle@node1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
30557433;Database Release Update : 19.6.0.0.200114 (30557433)
30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)

OPatch succeeded.

[oracle@node2 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
30557433;Database Release Update : 19.6.0.0.200114 (30557433)
30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)

OPatch succeeded.

[oracle@node1 ~]$ srvctl modify database -db test -oraclehome /u01/app/oracle/product/19.3.0.0/dbhome_2
[oracle@node1 ~]$ srvctl config database -db test -a
Database unique name: test
Database name: test
Oracle home: /u01/app/oracle/product/19.3.0.0/dbhome_2
Oracle user: oracle
Spfile: +DATA/test/PARAMETERFILE/spfile.270.1053084685
Password file: +DATA/test/PASSWORD/pwdtest.258.1053083711
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: 
Disk Groups: RECO,DATA
Mount point paths: 
Services: 
Type: RAC
Start concurrency: 
Stop concurrency: 
OSDBA group: dba
OSOPER group: oper
Database instances: test1,test2
Configured nodes: node1,node2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services: 
Database is administrator managed

I will try again this exercise by doing the copy of the oui-patch.xml before patching and see if that fixes this error, but hopefully this blog post helps you in case you face these bugs, 

Tags:
Rene Antunez
[email protected]
No Comments

Sorry, the comment form is closed at this time.