ODA X5 – Database Home Creation Error (racclonepl.sh)

Introduction

ODA (Oracle Data Appliance) virtualized platform is orchestrated by OAKCLI tool, changes like Database Creation, Database Home Creation, deleting database homes, etc need to be performed by OAKCLI Tool. As this is fully scripted, we should have a clear understanding of Oracle internals to troubleshoot issues encountered via OAKCLI Tool.

Most of the issues encountered during OAKCLI command execution are complicated but solutions are simpler. Even it comes to 19.13.0.0 environment is tune and sound. But to troubleshoot the issue my advice would be to analyze the log to find the root cause of the problem.

In addition, Oracle announces the discontinuation of OVM (Oracle Virtual Machine), going forward Oracle OVM will be replaced by KVM (Kernal Virtual Machine)

ODA (Oracle Data Appliance) virtualized (OVM) based coming to the end of support from 19.13. After 19.13 oracle will move with KVM-based virtualization. In my view, KVM (Kernal Virtual Machine ) is much more stable than OVM.

In this article, I will cover to steps to create a new Oracle 19.13 home and the issues we faced during the new database home creation.

Gather Database Home Information

For the Oracle Database Appliance, we do not need to execute the runInstaller for the installation, the Image is available on Oracle Support. We need to download the below-mentioned 19.13 RDBMS clone and unpack it on Node01.

Patch 30403653: ORACLE DATABASE APPLIANCE 19.13.0.0.211019 END-USER (RDBMS CLONE) DOWNLOAD for VM Platform

Once we download and stage the binary we need to unpack the binary on node01. I have shared the sample wget command to download the 30403653 RDBMS clone.

wget --http-user=oracle_support_email_addres --http-password=xxxxx --no-check-certificate --output-document=p30403653_1913000_Linux-x86-64.zip \
 "https://updates.oracle.com/Orion/Services/download/p30403653_1913000_Linux-x86-64.zip?aru=24531546&patch_file=p30403653_1913000_Linux-x86-64.zip"

Unpacking the RDBMS Clone

Make sure enough space is there on /u01 mount point, we need close to 12GB. Unpacking needs to be done on only one node, during the creation node01 will transfer the binary to Node02.

Note : Unpack only on one node 
oakcli unpack -package /db_backup/19.13.DB_CLONE/p30403653_1913000_Linux-x86-64.zip

[root@ecl-odabase-0 19.13.DB_CLONE]# oakcli unpack -package /db_backup/19.13.DB_CLONE/p30403653_1913000_Linux-x86-64.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.
[root@ecl-odabase-0 19.13.DB_CLONE]#

First, gather the current Database Home information.

[root@ecl-odabase-0 19.13.DB_CLONE]# oakcli show dbhomes
Oracle Home Name      Oracle Home version Home Location                              Home Edition
----------------      ------------------- ------------                               ------------
OraDb12102_home1      12.1.0.2.191015     /u01/app/oracle/product/12.1.0.2/dbhome_1  Enterprise
OraDb12102_home2      12.1.0.2.191015     /u01/app/oracle/product/12.1.0.2/dbhome_2  Enterprise
[root@ecl-odabase-0 19.13.DB_CLONE]#

Create a 19.13 Database Home

Use Oracle OAKCLI to create dbhome -version 19.13.0.0.211019 to create the database home. But creation failed due to while executing racclonepl.sh script.

[root@ecl-odabase-0 oraInventory]# oakcli create dbhome -version 19.13.0.0.211019
INFO: 2022-07-18 11:54:44: Please check the logfile  '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/createdbhome_99502.log' for more details
Please enter the 'SYSASM'  password : (During deployment we set the SYSASM password to 'welcome1'):
Please re-enter the 'SYSASM' password:
INFO: 2022-07-18 11:56:00: Installing a new Home : OraDb19000_home2 at /u01/app/oracle/product/19.0.0.0/dbhome_2
Please select one of the following for Database edition  [1 .. 2] :
1    => Enterprise Edition
2    => Standard Edition
1
The selected value is : Enterprise Edition
INFO   : Logging all actions in the file /opt/oracle/oak/log/ecl-odabase-0/patch/19.13.0.0.0/ecl-odabase-0-20220718115655.log and traces in  the file /opt/oracle/oak/log/ecl-odabase-0/patch/19.13.0.0.0/ecl-odabase-0-20220718115655.trc
INFO   : Loading the configuration file /opt/oracle/oak/onecmd/create_database.params...
INFO   : Creating the node list files...
INFO   : Setting up ssh for root...
INFO   : Setting up SSH across the Private Network...
...INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.27 /root/DoAllcmds.sh
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.28 /root/DoAllcmds.sh
INFO   : Background process 13611 (node: 192.168.16.27) gets done with the exit code 0
INFO   : Background process 13660 (node: 192.168.16.28) gets done with the exit code 0
INFO   : Setting up SSH completed successfully
INFO   : Running the command /usr/bin/rsync -tarqvz /opt/oracle/oak/onecmd/ root@192.168.16.28:/opt/oracle/oak/onecmd --exclude=*zip --exclude=*gz --exclude=*log --exclude=*trc --exclude=*rpm to sync directory /opt/oracle/oak/onecmd  on node  192.168.16.28 
SUCCESS: Ran /usr/bin/rsync -tarqvz /opt/oracle/oak/onecmd/ root@192.168.16.28:/opt/oracle/oak/onecmd --exclude=*zip --exclude=*gz --exclude=*log --exclude=*trc --exclude=*rpm and it returned: RC=0
...INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /root/DoAllcmds.sh
INFO   : Background process 14506 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /root/DoAllcmds.sh
INFO   : Background process 14551 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115740.sh
INFO   : Background process 14692 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115740.sh
INFO   : Background process 14735 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Setting up SSH for user oracle...
...INFO   : checking nodes in /opt/oracle/oak/onecmd/tmp/db_nodes...
...
SUCCESS: All nodes in /opt/oracle/oak/onecmd/tmp/db_nodes are pingable and alive.
INFO   : Checking SSH setup for user (oracle) on nodes in /opt/oracle/oak/onecmd/tmp/db_nodes...
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115823.sh
INFO   : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115823.sh
INFO   : Background process 17042 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115823.sh
INFO   : Background process 17106 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Creating DB homes on the nodes ecl-odabase-0 ecl-odabase-1....
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115824.sh
INFO   : INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115824.sh
Background process 17259 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 17311 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115825.sh
INFO   : Background process 17532 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115825.sh
INFO   : Background process 17580 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Running /usr/bin/scp /opt/oracle/oak/pkgrepos/orapkgs/DB/19.13.0.0.211019/Base/db19.211019.tar.gz root@192.168.16.28:/opt/oracle/oak/pkgrepos/orapkgs/DB/19.13.0.0.211019/Base/db19.211019.tar.gz to copy DB clone to the other node...
SUCCESS: Ran /usr/bin/scp /opt/oracle/oak/pkgrepos/orapkgs/DB/19.13.0.0.211019/Base/db19.211019.tar.gz root@192.168.16.28:/opt/oracle/oak/pkgrepos/orapkgs/DB/19.13.0.0.211019/Base/db19.211019.tar.gz and it returned: RC=0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115904.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115904.sh
INFO   : Background process 22451 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 22496 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115905.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115905.sh
INFO   : Background process 22650 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 22694 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Restoring the copy of DB home on all nodes...
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115906.sh
INFO   : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115906.sh
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718115906.sh
INFO   : Background process 22873 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 22915 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718120235.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220718120235.sh
INFO   : Background process 36956 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Background process 36908 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Cloning and relinking RAC home
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/fuser_mv.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/fuser_mv.sh
INFO   : Background process 37316 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Background process 37272 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running clone.pl on RAC home. Look at /opt/oracle/oak/onecmd/tmp/racclonepl.sh-20220718115655-log for more details
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/racclonepl.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/racclonepl.sh
ERROR  : Ran '/usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/racclonepl.sh' and it returned code(1) and output is:
         Cloning failed .... exiting.
INFO   : Background process 37536 (node: ecl-odabase-1) gets done with the exit code 1
INFO   : Background process 37494 (node: ecl-odabase-0) gets done with the exit code 0
ERROR  : Failure in copying /opt/oracle/oak/onecmd/tmp/racclonepl.sh to DB nodes and executing it as root in parallel
Exiting...
[root@ecl-odabase-0 oraInventory]#

racclonepl.sh Log Output

This error was caused due to a permission issue on the /u01/app/oraInventory/ContentsXML/oui-patch.xml file. 

view /opt/oracle/oak/onecmd/tmp/racclonepl.sh-20220718111922-log

Setup Inventory in progress.
SEVERE:OUI-10094:Problem in saving inventory. Installation cannot continue. oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)
[FATAL] OUI-10094:Problem in saving inventory. Installation cannot continue. oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)

Solution

List the files and get the note down the current permission. To overcome this we need to give 775 permission on the ContentsXML folder.

############ Solution 
su - grid
cd /u01/app/oraInventory/
[root@ecl-odabase-0 ~]# ls -l /u01/app/oraInventory/
total 44
drwxrwxr-x  2 grid   oinstall  4096 Jul 20 13:02 ContentsXML
drwxrwx--- 24 grid   oinstall  4096 Jul 20 13:02 backup
drwxrwx---  2 oracle oinstall  4096 Jul 20 13:02 locks
drwxrwx---  7 grid   oinstall 20480 Jul 20 13:02 logs
-rw-rw----  1 grid   oinstall    56 Mar 26  2020 oraInst.loc
-rwxrwx---  1 grid   oinstall  1620 Mar 26  2020 orainstRoot.sh
drwxrwx---  2 oracle oinstall  4096 Sep  8  2021 oui
[root@ecl-odabase-0 ~]#
chmod -R 775 ContentsXML

Creation of the 19.13 2nd Attempt

Even though this was unsuccessful, I noticed that the Oracle Home was already created on both Nodes. Before starting the home creation, delete the old home via OAKCLI delete dbhome -oh command, and then remove the /u01/app/oracle/product/19.0.0.0/dbhome_1 folder.

Delete Failed Oracle Home

Please find the OAKCLI delete dbhome command Expected Output

############# Delete dbhome 

[root@ecl-odabase-0 oraInventory]# oakcli delete dbhome -oh OraDb19000_home1
INFO: 2022-07-18 11:50:13: Look at the log file '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/deletedbhome_69731.log' for more details

Please enter the 'SYS'  password :
Please re-enter the 'SYS' password:
...SUCCESS: Ran /usr/bin/rsync -tarqvz /opt/oracle/oak/onecmd/ root@192.168.16.28:/opt/oracle/oak/onecmd --exclude=*zip --exclude=*gz --exclude=*log --exclude=*trc --exclude=*rpm and it returned: RC=0

.........
SUCCESS: All nodes in /opt/oracle/oak/onecmd/tmp/db_nodes are pingable and alive.
SUCCESS: 2022-07-18 11:53:32: Successfully deinstalled the dbhome OraDb19000_home1
[root@ecl-odabase-0 oraInventory]#

New 19.13 Home

After changing the permission on the ContentsXML folder, execute the OAKCLI to create dbhome for the new home.

[root@ecl-odabase-0 ~]# oakcli create dbhome -version 19.13.0.0.211019
INFO: 2022-07-20 12:55:53: Please check the logfile  '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/createdbhome_41158.log' for more details
Please enter the 'SYSASM'  password : (During deployment we set the SYSASM password to 'welcome1'):
Please re-enter the 'SYSASM' password:
INFO: 2022-07-20 12:56:13: Installing a new Home : OraDb19000_home1 at /u01/app/oracle/product/19.0.0.0/dbhome_1
Please select one of the following for Database edition  [1 .. 2] :
1    => Enterprise Edition
2    => Standard Edition
1
The selected value is : Enterprise Edition
INFO   : Logging all actions in the file /opt/oracle/oak/log/ecl-odabase-0/patch/19.13.0.0.0/ecl-odabase-0-20220720125625.log and traces in  the file /opt/oracle/oak/log/ecl-odabase-0/patch/19.13.0.0.0/ecl-odabase-0-20220720125625.trc
INFO   : Loading the configuration file /opt/oracle/oak/onecmd/create_database.params...
INFO   : Creating the node list files...
INFO   : Setting up ssh for root...
INFO   : Setting up SSH across the Private Network...
...INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.27 /root/DoAllcmds.sh
INFO   : Background process 44881 (node: 192.168.16.27) gets done with the exit code 0
INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.28 /root/DoAllcmds.sh
INFO   : Background process 44926 (node: 192.168.16.28) gets done with the exit code 0
INFO   : Setting up SSH completed successfully
INFO   : Running the command /usr/bin/rsync -tarqvz /opt/oracle/oak/onecmd/ root@192.168.16.28:/opt/oracle/oak/onecmd --exclude=*zip --exclude=*gz --exclude=*log --exclude=*trc --exclude=*rpm to sync directory/opt/oracle/oak/onecmd on node 192.168.16.28
SUCCESS: Ran /usr/bin/rsync -tarqvz /opt/oracle/oak/onecmd/ root@192.168.16.28:/opt/oracle/oak/onecmd --exclude=*zip --exclude=*gz --exclude=*log --exclude=*trc --exclude=*rpm and it returned: RC=0
...INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /root/DoAllcmds.sh
INFO   : Background process 48165 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /root/DoAllcmds.sh
INFO   : Background process 48249 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125710.sh
INFO   : Background process 48454 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125710.sh
INFO   : Background process 48577 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Setting up SSH for user oracle...
...INFO   : checking nodes in /opt/oracle/oak/onecmd/tmp/db_nodes...
...
SUCCESS: All nodes in /opt/oracle/oak/onecmd/tmp/db_nodes are pingable and alive.
INFO   : Checking SSH setup for user (oracle) on nodes in /opt/oracle/oak/onecmd/tmp/db_nodes...
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125752.sh
INFO   : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125752.sh
INFO   : Background process 51158 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125752.sh
INFO   : Background process 51212 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Creating DB homes on the nodes ecl-odabase-0 ecl-odabase-1....
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125753.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125753.sh
INFO   : Background process 51328 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 51371 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125754.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125754.sh
INFO   : Background process 51507 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 51550 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Running /usr/bin/scp /opt/oracle/oak/pkgrepos/orapkgs/DB/19.13.0.0.211019/Base/db19.211019.tar.gz root@192.168.16.28:/opt/oracle/oak/pkgrepos/orapkgs/DB/19.13.0.0.211019/Base/db19.211019.tar.gz to copy DB clone to the other node...
SUCCESS: Ran /usr/bin/scp /opt/oracle/oak/pkgrepos/orapkgs/DB/19.13.0.0.211019/Base/db19.211019.tar.gz root@192.168.16.28:/opt/oracle/oak/pkgrepos/orapkgs/DB/19.13.0.0.211019/Base/db19.211019.tar.gz and it returned: RC=0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125855.sh
INFO   : Background process 57056 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125855.sh
INFO   : Background process 57102 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125856.sh
INFO   : INFO   : Background process 57228 (node: ecl-odabase-0) gets done with the exit code 0
Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125856.sh
INFO   : Background process 57271 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Restoring the copy of DB home on all nodes...
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125857.sh
INFO   : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125857.sh
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720125857.sh
INFO   : Background process 57442 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Background process 57401 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720130109.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720130109.sh
INFO   : Background process 65144 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 65150 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Cloning and relinking RAC home
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/fuser_mv.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/fuser_mv.sh
INFO   : Background process 65377 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Background process 65333 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running clone.pl on RAC home. Look at /opt/oracle/oak/onecmd/tmp/racclonepl.sh-20220720125625-log for more details
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/racclonepl.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/racclonepl.sh
INFO   : Background process 65527 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 65572 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Runing root.sh on all nodes...
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720130309.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/DoAllcmds-20220720130309.sh
INFO   : Background process 77864 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 77930 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Removing temporary templates
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/remove_temp.sh
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/remove_temp.sh
INFO   : Background process 78163 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Background process 78208 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Running asmgid
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/set_asm_gid.sh
INFO   : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/set_asm_gid.sh
INFO   : Background process 78309 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/set_asm_gid.sh
INFO   : Background process 78356 (node: ecl-odabase-1) gets done with the exit code 0
INFO   : Did not do scp for node : ecl-odabase-0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/fuser_mv_bk.sh
INFO   : Background process 78482 (node: ecl-odabase-0) gets done with the exit code 0
INFO   : Running as root: /usr/bin/ssh -l root ecl-odabase-1 /opt/oracle/oak/onecmd/tmp/fuser_mv_bk.sh
INFO   : Background process 78527 (node: ecl-odabase-1) gets done with the exit code 0
SUCCESS: 2022-07-20 13:03:13: Successfully created the Database Home : OraDb19000_home1
[root@ecl-odabase-0 ~]#

Verification

OAKCLI show dbhomes can be used to validate the created home. Please find the expected output below.

[root@ecl-odabase-0 ~]# oakcli show dbhomes
Oracle Home Name      Oracle Home version Home Location                              Home Edition
----------------      ------------------- ------------                               ------------
OraDb12102_home2      12.1.0.2.191015     /u01/app/oracle/product/12.1.0.2/dbhome_2  Enterprise
OraDb19000_home1      19.13.0.0.211019    /u01/app/oracle/product/19.0.0.0/dbhome_1  Enterprise
[root@ecl-odabase-0 ~]#

Conclusion

Database home creation should be clean before proceeding with the database upgrade. Before making any changes take time to analyze the log files to find the root cause of the issue. In my next article, I will cover the database upgrade steps.