ORA-00600 [kcbtse_encdec_tbsblk_1] When Duplicating PDB IN 19C

29 Jul ORA-00600 [kcbtse_encdec_tbsblk_1] When Duplicating PDB IN 19C

Seems that lately I have been getting all of the ORA-00600 errors possible when duplicating databases, and this is just another one in the list. This will be a very short blog post on how I worked around ORA-00600 [KCBTSE_ENCDEC_TBSBLK_1]  when duplicating a PDB and TDE is enabled in the Source PDB in a 19.10 version.

Over the last months, the team had been seeing this ORA-00600 intermittently when issuing the duplicate command below:

RMAN> CONNECT AUXILIARY *
2> CONNECT CATALOG *
 
RMAN> run { 
 allocate auxiliary channel ch1 type 'sbt_tape' PARMS="SBT_LIBRARY=/u01/app/CV/commvault/Base/libobk.so, BLKSIZE=1048576 ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1)" TRACE 0; 
 allocate auxiliary channel ch2 type 'sbt_tape' PARMS="SBT_LIBRARY=/u01/app/CV/commvault/Base/libobk.so, BLKSIZE=1048576 ENV=(CV_mmsApiVsn=2,CV_channelPar=ch2)" TRACE 0; 
 DUPLICATE DATABASE CDBPROD TO CDBSTAGE PLUGGABLE DATABASE 'PDBRENE' ; 
} 

When I finally got involved and the problem was explained, I started to see a trend, whenever the duplicate command used an L0 + L1 backup for the restore, we were hitting the ORA-00600. Searching MOS, there are several bugs out there, but most are on the backup side of things, not on the restore side of it, so I couldn’t get an exact match.
Looking at the log and trace, you could see this happening on the incremental restore part of it.

## Database Log
PDBRENE(5):Incremental restore complete of datafile 60 +DATAC1/CDBSTAGE/B871E50DDA0DCFD5E0530AFE10ACC7B6/DATAFILE/bill_01.7819.1079101617
PDBRENE(5):  checkpoint is 426591948421
PDBRENE(5):  last deallocation scn is 271840228442
Errors in file /u02/app/oracle/diag/rdbms/cdbstage/CDBSTAGE1/trace/CDBSTAGE1_ora_246596.trc  (incident=115225) (PDBNAME=PDBRENE):
ORA-00600: internal error code, arguments: [kcbtse_encdec_tbsblk_1], [5], [10], [616562690], [86], [2], [20], [4294967295], [], [], [], []
PDBRENE(5):Incident details in: /u02/app/oracle/diag/rdbms/cdbstage/CDBSTAGE1/incident/incdir_115225/CDBSTAGE1_ora_246596_i115225.trc

## Trace File
[TOC00000]
Jump to table of contents
Dump continued from file: /u02/app/oracle/diag/rdbms/cdbstage/CDBSTAGE1/trace/CDBSTAGE1_ora_246596.trc
[TOC00001]
ORA-00600: internal error code, arguments: [kcbtse_encdec_tbsblk_1], [5], [10], [616562690], [86], [2], [20], [4294967295], [], [], [], []

[TOC00001-END]
[TOC00002]
========= Dump for incident 115225 (ORA 600 [kcbtse_encdec_tbsblk_1]) ========
[TOC00003]
----- Beginning of Customized Incident Dump(s) -----
----- Abridged Call Stack Trace -----
<Skipped 2 stack frames>
kcbtse_encdec_tbsblk_pdb1<-krbr3b2<-krbr1b1<-krbrValidateData<-krbrpr<-krbirbp<-pevm_icd_call_common<-pfrinstr_ICAL<-pfrrun_no_tool<-pfrrun<-plsql_run<-pricar<-pricbr<-prient2<-prient<-kkxrpc<-kporpc<-opiodr<-ttcpip<-opitsk<-opiino<-opiodr<-opidrv<-sou2o<-opimai_real
<-ssthrdmain<-main<-__libc_start_main 
----- End of Abridged Call Stack Trace -----
Partial short call stack signature: 0x90dfb4faae7825b1
.
Tablespace key not found as expected. 
Key versions: pkv 4294967295 pkv2 4294967295 gkv 4294967295 gkv2 4294967295 input 4294967295 fon 1 cfv 0 tekafnrl (nil)
...
Error Descriptor: ORA-600 [kcbtse_encdec_tbsblk_1] [5] [10] [616562690] [86] [2] [20] [4294967295] [] [] [] []
Error class: 0
Problem Key # of args: 1
Number of actions: 18
----- Incident Context Dump -----
Address: 0x7f5f9f9b4148
Incident ID: 115225
Problem Key: ORA 600 [kcbtse_encdec_tbsblk_1]
Error: ORA-600 [kcbtse_encdec_tbsblk_1] [5] [10] [616562690] [86] [2] [20] [4294967295] [] [] [] []
[00]: dbgexExplicitEndInc [diag_dde]
[01]: dbgeEndDDEInvocationImpl [diag_dde]
[02]: kcbz_encdec_tbsblk [CACHE_RCV]<-- Signaling
[03]: kcbtse_encdec_tbsblk_pdb1 [CACHE_RCV]
[04]: krbr3b2 [KRB]
[05]: krbr1b1 [KRB]
[06]: krbrValidateData [KRB]
[07]: krbrpr [KRB]
[08]: krbirbp [KRB]
[09]: pevm_icd_call_common [PLSQL_Code_Execution]
[10]: pfrinstr_ICAL [PLSQL_Code_Execution]
[11]: pfrrun_no_tool [PLSQL_Code_Execution]
[12]: pfrrun [PLSQL_Code_Execution]
[13]: plsql_run [PLSQL_PSD_Standalones]
[14]: pricar [PLSQL_Code_Execution]
[15]: pricbr [PLSQL_Code_Execution]
[16]: prient2 [PLSQL_Code_Execution]
[17]: prient [PLSQL_Code_Execution]
[18]: kkxrpc [PROGINT_PLSQL]
[19]: kporpc [PROGINT_MISC]
[20]: opiodr [OPI]
[21]: ttcpip []
[22]: opitsk [OPI]
[23]: opiino [OPI]
[24]: opiodr [OPI]
[25]: opidrv [OPI]
[26]: sou2o []
[27]: opimai_real [OPI]
[28]: ssthrdmain []
[29]: main []
[30]: __libc_start_main []
MD [00]: 'SID'='294.8383' (0x2)
MD [01]: 'ProcId'='67.1' (0x2)
MD [02]: 'Module'='restore incr datafile' (0x200)
MD [03]: 'Action'='0000819 STARTED40' (0x200)
MD [04]: 'PdbName'='PDBRENE' (0x0)
MD [05]: 'Client ProcId'='*************** (TNS V1-V3).246596_140048676942848' (0x0)
Impact 0: 
Impact 1: 
Impact 2: 
Impact 3: 
Derived Impact: 
----- END Incident Context Dump -----

The workaround I applied to this bug is to use only the L0 backup for the duplicate command. Once I used that, the command will succeed.

channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=16
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:02
Finished restore at 25-JUL-21
Performing import of metadata...
Finished Duplicate PDB at 25-JUL-21

We are currently working with Oracle on identifying the bug, but for now using only the L0 backup is how we are working around it.