Error ORA-28353: failed to open wallet after applying October 2018 BP for 11.2.0.4 or upgrade to 12/18 or 19c

25 Sep Error ORA-28353: failed to open wallet after applying October 2018 BP for 11.2.0.4 or upgrade to 12/18 or 19c

So this is a post that was in draft mode for almost 1.5 years. But even though that I know most clients are no longer in 11.2.0.4 , this is valid for anyone that still are and are thinking of an upgrade from 11.2 to 12/18/19.

I had been doing several test on my Spanish RAC Attack for 12.2 and creating several RAC VMs so that I can do testing to help several clients to do their patching to October 2018 PSU so that it gives them enough leverage from a security perspective to not patch their database and do their DB upgrade to 18c. This client has TDE with FIPS 140(Which I will explain a bit more down below on what it is) enabled, so I setup my environment the same way.

So the patching went great , but the problem came after applying the patch, as I wasn’t able to open the database , even though I had the correct password for the encryption key.

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-28365: wallet is not open

SQL> alter system set encryption key identified by “xxx”;
alter system set encryption key identified by “xxxx”
*
ERROR at line 1:
ORA-28353: failed to open wallet

SQL> select * from v$encryption_wallet;

INST_ID WRL_TYPE
---------- --------------------
WRL_PARAMETER
------------------------
STATUS
------------------
1 file
/etc/oracle/wallets/$ORACLE_UNQNAME
CLOSED

SQL> ALTER SYSTEM SET ENCRYPTION WALLET OPEN IDENTIFIED BY *****;
ALTER SYSTEM SET ENCRYPTION WALLET OPEN IDENTIFIED BY *****
*
ERROR at line 1:
ORA-28353: failed to open wallet

What you started to see in the alert.log was the below when I tried to open the database :

Tue Feb 16 11:21:56 2018
QMNC started with pid=81, OS id=86184
kcbztek_get_tbskey: decrypting encrypted key for tablespace 16 without opening the wallet

So what I did was a rollback of the patch , and as soon as you rolled back the patch, the database opened

SQL> select name,open_mode ,database_role from v$database;

NAME OPEN_MODE DATABASE_ROLE
--------- -------------------- ----------------
TEST READ WRITE PRIMARY

After a lot of days of investigation as I couldn’t find information regarding the error, one thing I saw was that FIPS 140-2 was enabled. Federal Information Processing Standard (FIPS) standard, 140-2, which is a U.S. government standard that defines cryptographic module security requirements.

You can see that is enabled for SSL in the following file

[oracle@collabn1 /u01/app/oracle/product/11.2.0.4/dbhome_1/ldap/admin ]$ cat fips.ora
#
# file to be reside in $ORACLE_HOME/ldap/admin
#
# sec-220
#
sslfips_140=true

I was able to find a document called After Applying October 2018 CPU/PSU, Auto-Login Wallet Stops Working For TDE With FIPS Mode Enabled (Doc ID 2474806.1) .

So the solution is to patch the DB with October 2018 PSU, once the binaries are patched, recreate the auto login file cwallet.sso with a compatibility of version 12. Once you have done this, you will be able to open your DB normally.

[oracle@collabn1 ~]$ cd /etc/oracle/wallets/test
[oracle@collabn1 /etc/oracle/wallets/test]$ mv cwallet.sso cwallet.sso.original

[oracle@collabn1 /etc/oracle/wallets/test]$ orapki wallet create -wallet /etc/oracle/wallets/test -auto_login -compat_v12
Oracle PKI Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
Enter wallet password: ******

[oracle@collabn1 ~]$ srvctl status database -d test
Instance test1 is running on node collabn1
Instance test2 is running on node collabn2

SQL> select name,open_mode ,database_role from v$database;

NAME OPEN_MODE DATABASE_ROLE
--------- -------------------- ----------------
TEST READ WRITE PRIMARY

SQL> select * from v$encryption_wallet;

INST_ID WRL_TYPE
---------- --------------------
WRL_PARAMETER
------------------------
STATUS
------------------
1 file
/etc/oracle/wallets/$ORACLE_UNQNAME
OPEN

IMPORTANT: DO NOT recreate the ewallet.p12 file! This will likely cause data loss, as you will lose the master key required to decrypt your encrypted data. Below is an example of what you DO NOT WANT TO DO

[oracle@collabn1 ~]$ cd /etc/oracle/wallets/test
[oracle@collabn1 /etc/oracle/wallets/test]$ mv ewallet.p12 ewallet.p12.old

[oracle@collabn1 /etc/oracle/wallets/test]$ sqlplus / as sysdba
SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY Welcome1 WITH BACKUP;

keystore altered

I also want to make a note, that this applies to Jan 2019 Database BP as well or an upgrade from 11.2.0.4 to 12/18/19c. So anything after October 2018 if you are using TDE and SSL with FIPS , you will face this issue.