06 Oct OPATCHAUTO Bug on 19.3 GI/RDBMS when Patching to 19.6
The other day I was doing a fresh install of Oracle 19.3, using the binaries from edelivery , and doing some testing in that version and then patch it to 19.6. The installation and the tests went fine. What I did next is proceed with the analyze of the patch 30501910 and saw that everything was ok .
## Note.- I trimmed the log for readability [root@node2 ~]$ . oraenv ORACLE_SID = [root] ? DB193H12 The Oracle base has been set to /u01/app/oracle [root@node2 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH [root@node2 ~]$ opatchauto apply /u01/patches/30501910 -analyze OPatchauto session is initiated at Tue Oct 6 13:23:11 2020 ... Host:node2 CRS Home:/u01/app/19.3.0.0/grid Version:19.0.0.0.0 ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /u01/patches/30501910/30489227 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log Patch: /u01/patches/30501910/30489632 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log Patch: /u01/patches/30501910/30655595 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log Patch: /u01/patches/30501910/30557433 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_11-41-25AM_1.log OPatchauto session completed at Tue Oct 6 11:42:46 2020 Time taken to complete the session 2 minutes, 41 seconds
So I proceeded with the patch with the command below opatchauto apply /u01/patches/30501910
Everything went fine in node 1, the patching completed successfully in GI and RDBMS, but the patch on node 2 failed on the RDBMS with the error below
## Note.- I trimmed the log for readability [root@node2 ~]$ opatchauto apply /u01/patches/30501910 ... ---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /u01/app/oracle/product/19.3.0.0/dbhome_1, host: node2. Command failed: /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/opatchauto apply /u01/patches/30501910 -oh /u01/app/oracle/product/19.3.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.3.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_node2_rac_2.ser -analyzedresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_node2_rac_2.ser Command failure output: ==Following patches FAILED in apply: Patch: /u01/patches/30501910/30489227 Log: /u01/app/oracle/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-05_17-25-22PM_1.log Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' After fixing the cause of failure Run opatchauto resume ...
After looking for information, I found MOS note 2582139.1 in which it basically states that I need to copy the oui-patch.xml from node 1 to node2.
[grid@node1 ~]$ cd /u01/app/oraInventory/ContentsXML/ [grid@node1 ContentsXML]$ scp oui-patch.xml node2:/u01/app/oraInventory/ContentsXML/ oui-patch.xml [root@node2 ContentsXML]# ls -ltr total 20 -rw-rw----. 1 grid oinstall 565 Oct 6 11:13 inventory.xml -rw-r--r--. 1 oracle oinstall 174 Oct 6 12:43 oui-patch.xml.back -rw-rw----. 1 grid oinstall 300 Oct 6 12:43 comps.xml -rw-rw----. 1 grid oinstall 292 Oct 6 12:43 libs.xml -rw-rw----. 1 grid oinstall 174 Oct 6 14:02 oui-patch.xml
So after doing this , I reran the opatchauto and now faced a the error Unable to create patchObject
## Note.- I trimmed the log for readability [root@node2 ~]$ opatchauto apply /u01/patches/30501910 ... ---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /u01/app/oracle/product/19.3.0.0/dbhome_1, host: node2. Command failed: /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/opatchauto apply /u01/patches/30501910 -oh /u01/app/oracle/product/19.3.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.3.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.3.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_node2_rac_2.ser -analyzedresult /u01/app/oracle/product/19.3.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_node2_rac_2.ser Command failure output: ==Following patches FAILED in apply: Patch: /u01/patches/30501910/30489227 Log: Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject Possible causes are: ORACLE_HOME/inventory/oneoffs/30489227 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19.3.0.0/dbhome_1/inventory/oneoffs/30489227/etc/config/actions" or "/u01/app/oracle/product/19.3.0.0/dbhome_1/inventory/oneoffs/30489227/etc/config/inventory" does not exist. After fixing the cause of failure Run opatchauto resume ...
For this I found another note 2286025.1 in which it is saying to do any of the following :
- Opatch lsinventory
- opatch -analyze
- Restore Database home backup for Hostname2.
But after doing points 1 and 2, I kept getting the same error, so what I proceeded to do is patch the GI on node2
## Note.- I trimmed the log for readability [root@node2 ~]$ . oraenv ORACLE_SID = [root] ? +ASM2 The Oracle base has been set to /u01/app/grid [root@node2 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH [root@node2 ~]$ opatchauto apply /u01/patches/30501910 -oh $ORACLE_HOME OPatchauto session is initiated at Tue Oct 6 12:28:25 2020 ... --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:node2 CRS Home:/u01/app/19.3.0.0/grid Version:19.0.0.0.0 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /u01/patches/30501910/30489227 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log Patch: /u01/patches/30501910/30489632 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log Patch: /u01/patches/30501910/30557433 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log Patch: /u01/patches/30501910/30655595 Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-06_12-33-02PM_1.log OPatchauto session completed at Tue Oct 6 12:49:02 2020 Time taken to complete the session 20 minutes, 38 seconds
For the RDBMS I followed my own blog post (How to clone an oracle home in 19c) and cloned the Oracle Home from node1 , as this one was already patched correctly with 30501910
[oracle@node1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
[oracle@node1 ~]$ $ORACLE_HOME/runInstaller -silent -createGoldImage -destinationLocation /u01/patches
Launching Oracle Database Setup Wizard...
Successfully Setup Software.
Gold Image location: /u01/patches/db_home_2020-10-06_02-11-59PM.zip
[oracle@node1 ~]$ unset ORACLE_HOME
[oracle@node1 ~]$ unset ORACLE_SID
[oracle@node1 ~]$ unset ORACLE_RSID
[oracle@node1 ~]$ unset ORACLE_UNQNAME
[oracle@node1 ~]$ unset ORACLE_BASE
[oracle@node1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_2
[oracle@node1 ~]$ export ORACLE_HOSTNAME=node1
[oracle@node1 ~]$ export ORA_INVENTORY=/u01/app/oraInventory
[oracle@node1 ~]$ export NODE1_HOSTNAME=node1
[oracle@node1 ~]$ export NODE2_HOSTNAME=node2
[oracle@node1 ~]$ export ORACLE_BASE=/u01/app/oracle
[oracle@node1 ~]$ unzip -q /u01/patches/db_home_2020-10-06_02-11-59PM.zip -d $ORACLE_HOME
[oracle@node1 ~]$ env | egrep "ORA|NODE"
ORACLE_BASE=/u01/app/oracle
NODE2_HOSTNAME=node2
ORACLE_HOSTNAME=node1
NODE1_HOSTNAME=node1
ORA_INVENTORY=/u01/app/oraInventory
ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_2
[oracle@node1 ~]$ cd $ORACLE_HOME
[oracle@node1 dbhome_2]$ ${ORACLE_HOME}/runInstaller -ignorePrereq -waitforcompletion -silent \
> -responseFile ${ORACLE_HOME}/install/response/db_install.rsp \
> oracle.install.option=INSTALL_DB_SWONLY \
> ORACLE_HOSTNAME=${ORACLE_HOSTNAME} \
> UNIX_GROUP_NAME=oinstall \
> INVENTORY_LOCATION=${ORA_INVENTORY} \
> SELECTED_LANGUAGES=en \
> ORACLE_HOME=${ORACLE_HOME} \
> ORACLE_BASE=${ORACLE_BASE} \
> oracle.install.db.InstallEdition=EE \
> oracle.install.db.OSDBA_GROUP=dba \
> oracle.install.db.OSOPER_GROUP=dba \
> oracle.install.db.OSBACKUPDBA_GROUP=dba \
> oracle.install.db.OSDGDBA_GROUP=dba \
> oracle.install.db.OSKMDBA_GROUP=dba \
> oracle.install.db.OSRACDBA_GROUP=dba \
> oracle.install.db.CLUSTER_NODES=${NODE1_HOSTNAME},${NODE2_HOSTNAME} \
> oracle.install.db.isRACOneInstall=false \
> oracle.install.db.rac.serverpoolCardinality=0 \
> oracle.install.db.config.starterdb.type=GENERAL_PURPOSE \
> oracle.install.db.ConfigureAsContainerDB=false \
> SECURITY_UPDATES_VIA_MYORACLESUPPORT=false \
> DECLINE_SECURITY_UPDATES=true
Launching Oracle Database Setup Wizard...
...
The response file for this session can be found at:
/u01/app/oracle/product/19.3.0.0/dbhome_2/install/response/db_2020-10-06_03-06-10PM.rsp
You can find the log of this install session at:
/u01/app/oraInventory/logs/InstallActions2020-10-06_03-06-10PM/installActions2020-10-06_03-06-10PM.log
As a root user, execute the following script(s):
1. /u01/app/oracle/product/19.3.0.0/dbhome_2/root.sh
Execute /u01/app/oracle/product/19.3.0.0/dbhome_2/root.sh on the following nodes:
[node1, node2]
After doing this, the only thing I needed to do was verify that the Oracle Home is patched correctly and switch the database to the new Oracle Home
[oracle@node1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches 30557433;Database Release Update : 19.6.0.0.200114 (30557433) 30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227) OPatch succeeded. [oracle@node2 ~]$ $ORACLE_HOME/OPatch/opatch lspatches 30557433;Database Release Update : 19.6.0.0.200114 (30557433) 30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227) OPatch succeeded. [oracle@node1 ~]$ srvctl modify database -db test -oraclehome /u01/app/oracle/product/19.3.0.0/dbhome_2 [oracle@node1 ~]$ srvctl config database -db test -a Database unique name: test Database name: test Oracle home: /u01/app/oracle/product/19.3.0.0/dbhome_2 Oracle user: oracle Spfile: +DATA/test/PARAMETERFILE/spfile.270.1053084685 Password file: +DATA/test/PASSWORD/pwdtest.258.1053083711 Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: Disk Groups: RECO,DATA Mount point paths: Services: Type: RAC Start concurrency: Stop concurrency: OSDBA group: dba OSOPER group: oper Database instances: test1,test2 Configured nodes: node1,node2 CSS critical: no CPU count: 0 Memory target: 0 Maximum memory: 0 Default network number for database services: Database is administrator managed
I will try again this exercise by doing the copy of the oui-patch.xml before patching and see if that fixes this error, but hopefully this blog post helps you in case you face these bugs,
Sorry, the comment form is closed at this time.