Error ORA-01033 after doing a switchover in a 12.1 RAC environment

10 Apr Error ORA-01033 after doing a switchover in a 12.1 RAC environment

I am mostly writing this so I don’t forget , it will be a small post. The other day I did a switchover in a RAC environment , which went pretty smooth , but after doing the switchover in the primary I kept getting the following error :

select dest_name,status,error from gv$archive_dest_status where dest_id=2;

DEST_NAME
--------------------------------------------------------------------------------
STATUS ERROR
--------- -----------------------------------------------------------------
LOG_ARCHIVE_DEST_2
ERROR ORA-01033: ORACLE initialization or shutdown in progress

LOG_ARCHIVE_DEST_2
ERROR ORA-01033: ORACLE initialization or shutdown in progress

So I went and checked the standby, and saw the standby was in recover mode and waiting for the redo log

PROCESS STATUS CLIENT_P CLIENT_PID THREAD# SEQUENCE# BLOCK# ACTIVE_AGENTS KNOWN_AGENTS
------- ------------ -------- ---------- ---------------- ---------------- ---------------- ---------------- ----------------
ARCH CONNECTED ARCH 44474 0 0 0 0 0
RFS IDLE ARCH 133318 0 0 0 0 0
RFS IDLE ARCH 50602 0 0 0 0 0
ARCH CLOSING ARCH 44470 1 21623 14336 0 0
ARCH CLOSING ARCH 44476 1 21624 1 0 0
ARCH CLOSING ARCH 44472 2 19221 96256 0 0
RFS IDLE LGWR 133322 1 21625 17157 0 0
RFS IDLE LGWR 50620 2 19222 36611 0 0
MRP0 WAIT_FOR_LOG N/A N/A 2 19222 36617 0 0

So my first train of though was that the password file was incorrect , so I recreated them and copied them from the primary to the standby nodes , but I still kept getting the same error. I reviewed the environment with the scripts in DOC ID 1581388.1 and everything seemed ok. So it really kept bugging me that the logs were not being applied even though the logs were being shipped to the standby, so it did have to do with the password file, but what really bugged me, was that I had just recreated the password file in $ORACLE_HOME/dbs and I still kept getting the same error.

So after a while of troubleshooting , I found that in the new primary , the password file was residing in an ASM Diskgroup, and that was the main culprit, so what I had to do was copy the password file from the ASM diskgroup in the primary to the standby.
Primary

[oracle@localhost trace]$ srvctl config database -d renedb
Database unique name: renedb
Database name:
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATA1/renedb/spfilerenedb.ora
Password file: +DATA1/renedb/PASSWORD/pwrenedb
Domain:
Start options: open
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools:
Disk Groups: ARCH1,DATA1,REDO
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: renedb1,renedb2
Configured nodes: localhost,localhost
Database is administrator managed
[oracle@localhost trace]$ exit
-bash-4.1$ sudo su - grid
[sudo] password for pythian:
[grid@localhost ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base remains unchanged with value /u01/app/grid
[grid@localhost ~]$ asmcmd
ASMCMD> pwcopy +DATA1/renedb/PASSWORD/pwrenedb /tmp/pwrenedb
copying +DATA1/renedb/PASSWORD/pwrenedb > /tmp/pwrenedb
ASMCMD> exit

Standby

[oracle@localhost dbs]$ scp 10.10.0.1:/tmp/pwrenedb /tmp/pwrenedb_stby
pwrenedb_stby_phdb 100% 7680 7.5KB/s 00:00
[oracle@localhost dbs]$ exit
logout
[pythian@localhost ~]$ sudo su - grid
[sudo] password for pythian:
Last login: Fri Mar 31 21:55:53 MST 2017
[grid@localhost ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@localhost ~]$ asmcmd
ASMCMD> mkdir DATA/RENEDB/PASSWORD
ASMCMD> pwcopy /tmp/pwrenedb_stby_phdb +DATA/RENEDB/PASSWORD/pwrenedb_stby
copying /tmp/pwrenedb_stby_phdb > +DATA/RENEDB/PASSWORD/pwrenedb_stby
ASMCMD> exit
[grid@localhost ~]$ exit
logout
[pythian@localhost ~]$ sudo su - oracle
Last login: Sat Apr 1 01:35:46 MST 2017 on pts/4
The Oracle base has been set to /u01/app/oracle
[oracle@localhost dbs]$ srvctl modify database -d renedb_stby -pwfile +DATA/RENEDB/PASSWORD/pwrenedb_stby
[oracle@localhost dbs]$ srvctl config database -d renedb_stby
Database unique name: renedb_stby
Database name:
Oracle home: /u01/app/oracle/product/12.1.0/dbhome_1
Oracle user: oracle
Spfile: /u01/app/oracle/product/12.1.0/dbhome_1/dbs/spfilerenedb_stby.ora
Password file: +DATA/RENEDB/PASSWORD/pwrenedb_stby
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: ARCH,DATA,REDO
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: renedb_stby1,renedb_stby2
Configured nodes: *******,***********
Database is administrator managed

Once I did this, the standby started applying the redo logs and after the gap was closed the Primary switchover status was “TO STANDBY”
Primary

Primary Site last generated SCN

*******************************

DB_UNIQUE_NAME SWITCHOVER_STATUS CURRENT_SCN
--------------- -------------------- ----------------
renedb TO STANDBY 134480468945

Standby

Data Guard Apply Lag

********************

NAME LAG_TIME DATUM_TIME TIME_COMPUTED
------------ -------------------- -------------------- --------------------
apply lag +00 00:00:00 04/01/2017 04:05:51 04/01/2017 04:05:52

1 row selected.

Data Guard Gap Problems

***********************

no rows selected

PROCESS STATUS CLIENT_P CLIENT_PID THREAD# SEQUENCE# BLOCK# ACTIVE_AGENTS KNOWN_AGENTS
------- ------------ -------- ---------- ---------------- ---------------- ---------------- ---------------- ----------------
ARCH CONNECTED ARCH 44474 0 0 0 0 0
RFS IDLE ARCH 133318 0 0 0 0 0
RFS IDLE ARCH 50602 0 0 0 0 0
ARCH CLOSING ARCH 44470 1 21623 14336 0 0
ARCH CLOSING ARCH 44476 1 21624 1 0 0
ARCH CLOSING ARCH 44472 2 19221 96256 0 0
RFS IDLE LGWR 133322 1 21625 17157 0 0
RFS IDLE LGWR 50620 2 19222 36611 0 0
MRP0 APPLYING_LOG N/A N/A 2 19222 36617 33 33

9 rows selected.

Conclusion
In 12.1 it is recommended as per DOC ID 1984091.1, to have the password file in ASM diskgroups. So once I did this, I was able to workaround error ORA-01033 and able to sleep well :).