Oracle Database Appliance (ODA) virtualized eases the database upgrade process using oakcli orchestration utility. This tool became more stable after 19.8. All the ODA components can be orchestrated via this oakcli utility. However, 19.13 will be the last upgrade for the virtualized platform, after that Oracle going to discontinue OVM and they are moving to KVM-based virtualization. This is going to be a revolutionary change for ODA.

In this article, I will cover the steps to perform database upgrades via oakcli tool.

There are pre-requisite steps that need to be performed before the upgrade

  • Gain Space in/u01 for an Upgrade
  • Create a New database home. I have covered this new database home creation in a previous blog post: https://chanaka-dbhelp.blogspot.com/2022/07/oda-x5-db-home-creation-error.html
  • Execute Pre-upgrade Check
  • Fixed the Pre-upgrade Issues
  • Backup Database Before Upgrade
  • Upgrade Database

 

1. We can gain space by removing the old repository files, this can be achieved by executing the manage clear repo command

oakcli manage cleanrepo --ver 19.8.0.0.0

Check the database homes that are configured, As per this example we have 3 database homes two 19c and one 12c

[root@ecl-odabase-0 app]# oakcli show dbhomes
Oracle Home Name Oracle Home version Home Location Home Edition
---------------- ------------------- ------------ ------------
OraDb12102_home2 12.1.0.2.191015 /u01/app/oracle/product/12.1.0.2/dbhome_2 Enterprise
OraDb19000_home1 19.13.0.0.211019 /u01/app/oracle/product/19.0.0.0/dbhome_1 Enterprise
OraDb19000_home2 19.13.0.0.211019 /u01/app/oracle/product/19.0.0.0/dbhome_2 Enterprise
[root@ecl-odabase-0 app]#

2.  Download the pre-upgrade utility from Oracle support

How to Download and Run Oracle’s Database Pre-Upgrade Utility (Doc ID 884522.1)

Note: We noticed that even pre-upgrade is handled by oakcli utility. But as a best practice, I would recommend to fixed those pre-upgrade suggestions before running the upgrade.

Please find the sample pre-upgrade execution output

[oracle@ecl-odabase-0 TESTDB]$ /u01/app/oracle/product/12.1.0.2/dbhome_2/jdk/bin/java -jar /home/oracle/PRE_UPGRADE/TESTDB/preupgrade.jar FILE DIR /home/oracle/PRE_UPGRADE/TESTDB/
==================
PREUPGRADE SUMMARY
==================
/home/oracle/PRE_UPGRADE/TESTDB/preupgrade.log
/home/oracle/PRE_UPGRADE/TESTDB/preupgrade_fixups.sql
/home/oracle/PRE_UPGRADE/TESTDB/postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade:

Log into the database and execute the preupgrade fixups
@/home/oracle/PRE_UPGRADE/TESTDB/preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@/home/oracle/PRE_UPGRADE/TESTDB/postupgrade_fixups.sql

Preupgrade complete: 2023-03-20T13:55:32
[oracle@ecl-odabase-0 TESTDB]$

 

Upgrade Database 

As per this example, we are upgrading testdb to 19c, the command required to point 19c home. The upgrade will take close to 1hr to complete all the tasks, Make sure to enable the screen before executing this command.

[root@ecl-odabase-0 incident]# oakcli upgrade database -db testdb -to OraDb19000_home1
INFO: 2023-03-20 14:31:04: Look at the log file '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/dbupgrade_96057.log' for more details

Please enter the 'SYS' password :
Please re-enter the 'SYS' password:
INFO: 2023-03-20 14:31:52: Upgrading the database testdb. It will take few minutes. Please wait...
...

 

After Upgrade

This is oakcli output after the upgrade

[root@ecl-odabase-0 incident]# screen -ls
There is a screen on:
96799.pts-1.ecl-odabase-0 (Attached)
1 Socket in /var/run/screen/S-root.

[root@ecl-odabase-0 incident]# oakcli upgrade database -db testdb -to OraDb19000_home1
INFO: 2023-03-20 14:31:04: Look at the log file '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/dbupgrade_96057.log' for more details

Please enter the 'SYS' password :
Please re-enter the 'SYS' password:
INFO: 2023-03-20 14:31:52: Upgrading the database testdb. It will take few minutes. Please wait...
...


...
SUCCESS: 2023-03-20 15:04:10: Successfully upgraded the database testdb

I have shared the complete log to get the proper understanding, oakcli utility is handling all the tasks pre-patching, patching, and post-patching.

 

Complete Log

[root@ecl-odabase-0 ~]# cat /opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/dbupgrade_96057.log
2023-03-20 14:31:04: INFO: Look at the log file '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/dbupgrade_
96057.log' for more details
2023-03-20 14:31:04: Command received is : oakcli upgrade database -db testdb -to OraDb19000_home1
2023-03-20 14:31:04: This is V4 machine
2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:05: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:05: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:05: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:05: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:05: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:05: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:06: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:06: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:06: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:06: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:06: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:06: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:18: Checking SSH equivalence for root.
2023-03-20 14:31:18: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster
2023-03-20 14:31:18: Command output:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online ,
>End Command output
2023-03-20 14:31:18: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:18: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:18: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-0 /bin/true" as user "root"
2023-03-20 14:31:18: Executing cmd: /bin/su root -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-0 /bin/true"
2023-03-20 14:31:18: SSH equivalence for root to ecl-odabase-0: success
2023-03-20 14:31:18: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-1 /bin/true" as user "root"
2023-03-20 14:31:18: Executing cmd: /bin/su root -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-1 /bin/true"
2023-03-20 14:31:18: SSH equivalence for root to ecl-odabase-1: success
2023-03-20 14:31:18: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.27 /bin/true" as user "root"
2023-03-20 14:31:18: Executing cmd: /bin/su root -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.27 /bin/true"
2023-03-20 14:31:19: SSH equivalence for root to 192.168.16.27: success
2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.28 /bin/true" as user "root"
2023-03-20 14:31:19: Executing cmd: /bin/su root -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.28 /bin/true"
2023-03-20 14:31:19: SSH equivalence for root to 192.168.16.28: success
2023-03-20 14:31:19: Checking SSH equivalence for oracle.
2023-03-20 14:31:19: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster
2023-03-20 14:31:19: Command output:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online ,
>End Command output
2023-03-20 14:31:19: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:19: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-0 /bin/true" as user "oracle"
2023-03-20 14:31:19: Executing cmd: /bin/su oracle -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-0 /bin/true"
2023-03-20 14:31:19: SSH equivalence for oracle to ecl-odabase-0: success
2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-1 /bin/true" as user "oracle"
2023-03-20 14:31:19: Executing cmd: /bin/su oracle -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-1 /bin/true"
2023-03-20 14:31:19: SSH equivalence for oracle to ecl-odabase-1: success
2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.27 /bin/true" as user "oracle"
2023-03-20 14:31:19: Executing cmd: /bin/su oracle -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.27 /bin/true"
2023-03-20 14:31:19: SSH equivalence for oracle to 192.168.16.27: success
2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.28 /bin/true" as user "oracle"
2023-03-20 14:31:19: Executing cmd: /bin/su oracle -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.28 /bin/true"
2023-03-20 14:31:20: SSH equivalence for oracle to 192.168.16.28: success
2023-03-20 14:31:32: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:32: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:32: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:32: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:32: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:32: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:33: Executing cmd: export ORACLE_
HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl status database -d testdb
2023-03-20 14:31:33: Command output:
> Instance testdb1 is running on node ecl-odabase-0
> Instance testdb2 is running on node ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:33: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:33: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:33: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:33: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 14:31:33: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:33: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:34: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:34: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:34: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:34: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 14:31:34: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:34: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:34: run_as_user2: Running /bin/su oracle -c ' export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2;/u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl config database -d testdb '
2023-03-20 14:31:35: Removing file /tmp/file2c3Y6L
2023-03-20 14:31:35:
2023-03-20 14:31:35: Successfully removed file: /tmp/file2c3Y6L
2023-03-20 14:31:35: /bin/su successfully executed

2023-03-20 14:31:35: Getting the JOB_
Queue_Processes for the database : testdb
2023-03-20 14:31:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:35: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:35: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:35: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:35: Executing cmd: export ORACLE_
HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl config database -db testdb
2023-03-20 14:31:36: Command output:
> Database unique name: testdb
> Database name: testdb
> Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_2
> Oracle user: oracle
> Spfile: +DATA/TESTDB/PARAMETERFILE/spfile.296.1131972439
> Password file: +DATA/TESTDB/PASSWORD/pwdtestdb.269.1131971689
> Domain:
> Start options: open
> Stop options: immediate
> Database role: PRIMARY
> Management policy: AUTOMATIC
> Server pools:
> Disk Groups: DATA,REDO
> Mount point paths:
> Services:
> Type: RAC
> Start concurrency:
> Stop concurrency:
> OSDBA group: dba
> OSOPER group: racoper
> Database instances: testdb1,testdb2
> Configured nodes: ecl-odabase-0,ecl-odabase-1
> Database is administrator managed ,
>End Command output
2023-03-20 14:31:36: db_domain for the database testdb is

2023-03-20 14:31:36: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan
2023-03-20 14:31:37: Command output:
> SCAN name: ecl-oda-scan, Network: 1
> Subnet IPv4: 10.11.30.0/255.255.255.0/eth0, static
> Subnet IPv6:
> SCAN 1 IPv4 VIP: 10.11.30.48
> SCAN VIP is enabled.
> SCAN 2 IPv4 VIP: 10.11.30.49
> SCAN VIP is enabled.
> SCAN 3 IPv4 VIP: 10.11.30.50
> SCAN VIP is enabled. ,
>End Command output
2023-03-20 14:31:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan_listener -i 1 -S 1
2023-03-20 14:31:37: Command output:
> #@=result[0]: res_
name={ora.LISTENER_SCAN1.lsnr} netnum={1} lsnr_name={LISTENER_SCAN1} port={1521,1531} ports={null} enabled={true} enabled_nodes={} disabled_nodes={} ,
>End Command output
2023-03-20 14:31:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:38: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:38: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:38: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:38: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:38: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:38: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:38: Will be running following sql statements as user: oracle:

export oracle_
home=/u01/app/oracle/product/12.1.0.2/dbhome_2,
export sid= testdb1,
/u01/app/oracle/product/12.1.0.2/dbhome_
2/bin/sqlplus -L sys/password@ecl-oda-scan/testdb as sysdba

set heading off
set echo off
set lines 200
show parameter job_queue_processes

2023-03-20 14:31:40: Removing file /tmp/fileKzXXW6
2023-03-20 14:31:40:
2023-03-20 14:31:40: Successfully removed file: /tmp/fileKzXXW6
2023-03-20 14:31:40: /bin/su successfully executed

2023-03-20 14:31:40: Output is :
job_queue_processes integer 1000

2023-03-20 14:31:40: JOB_Queue_Processes for the database testdb is 1000
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:40: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:40: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:40: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:40: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan
2023-03-20 14:31:41: Command output:
> SCAN name: ecl-oda-scan, Network: 1
> Subnet IPv4: 10.11.30.0/255.255.255.0/eth0, static
> Subnet IPv6:
> SCAN 1 IPv4 VIP: 10.11.30.48
> SCAN VIP is enabled.
> SCAN 2 IPv4 VIP: 10.11.30.49
> SCAN VIP is enabled.
> SCAN 3 IPv4 VIP: 10.11.30.50
> SCAN VIP is enabled. ,
>End Command output
2023-03-20 14:31:41: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan_listener -i 1
2023-03-20 14:31:41: Command output:
> SCAN Listeners for network 1:
> Registration invited nodes:
> Registration invited subnets:
> Endpoints: TCP:1521,1531
> SCAN Listener LISTENER_
SCAN1 exists
> SCAN Listener is enabled. ,
>End Command output
2023-03-20 14:31:41: Executing cmd: /u01/app/19.0.0.0/grid/bin/cemutlo -n
2023-03-20 14:31:41: Command output:
> ecl-oda-lab-c ,
>End Command output
2023-03-20 14:31:41: Executing cmd: hostname -d
2023-03-20 14:31:41: Command output:
> newco.local ,
>End Command output
2023-03-20 14:31:41: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config nodeapps -a
2023-03-20 14:31:42: Command output:
> Network 1 exists
> Subnet IPv4: 10.11.30.0/255.255.255.0/eth0, static
> Subnet IPv6:
> Ping Targets:
> Network is enabled
> Network is individually enabled on nodes:
> Network is individually disabled on nodes:
> VIP exists: network number 1, hosting node ecl-odabase-0
> VIP Name: ecl-oda-0-vip.newco.local
> VIP IPv4 Address: 10.11.30.157
> VIP IPv6 Address:
> VIP is enabled.
> VIP is individually enabled on nodes:
> VIP is individually disabled on nodes:
> VIP exists: network number 1, hosting node ecl-odabase-1
> VIP Name: ecl-oda-1-vip.newco.local
> VIP IPv4 Address: 10.11.30.158
> VIP IPv6 Address:
> VIP is enabled.
> VIP is individually enabled on nodes:
> VIP is individually disabled on nodes: ,
>End Command output
2023-03-20 14:31:43: INFO : Logging all actions in the file /opt/oracle/oak/onecmd/tmp/ecl-odabase-0-20230320143143.log and traces in the file /opt/oracle/oak/onecmd/tmp/ecl-odabase-0-20230320143143.trc
2023-03-20 14:31:43: INFO : Loading the configuration file /opt/oracle/oak/onecmd/upgrade_databse.params...
2023-03-20 14:31:45: INFO : Creating the node list files...
2023-03-20 14:31:45: Executing cmd: ssh -o StrictHostKeyChecking=no root@ecl-odabase-0 "/bin/df -k /u01 /tmp"
2023-03-20 14:31:45: Command output:
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/xvdb1 95988492 78908600 12197204 87% /u01
> /dev/xvda2 57060636 43633540 10521856 81% / ,
>End Command output
2023-03-20 14:31:45: Free space on /u01 on ecl-odabase-0 is 12197204 1K-blocks
2023-03-20 14:31:45: Free space on / on ecl-odabase-0 is 10521856 1K-blocks
2023-03-20 14:31:45: Executing cmd: ssh -o StrictHostKeyChecking=no root@ecl-odabase-1 "/bin/df -k /u01 /tmp"
2023-03-20 14:31:45: Command output:
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/xvdb1 95988492 77826264 13279540 86% /u01
> /dev/xvda2 57060636 41370112 12785284 77% / ,
>End Command output
2023-03-20 14:31:45: Free space on /u01 on ecl-odabase-1 is 13279540 1K-blocks
2023-03-20 14:31:45: Free space on / on ecl-odabase-1 is 12785284 1K-blocks
2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:45: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:45: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:45: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:45: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:46: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:46: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:46: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:46: Executing cmd: export ORACLE_
HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl config database -db testdb
2023-03-20 14:31:47: Command output:
> Database unique name: testdb
> Database name: testdb
> Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_2
> Oracle user: oracle
> Spfile: +DATA/TESTDB/PARAMETERFILE/spfile.296.1131972439
> Password file: +DATA/TESTDB/PASSWORD/pwdtestdb.269.1131971689
> Domain:
> Start options: open
> Stop options: immediate
> Database role: PRIMARY
> Management policy: AUTOMATIC
> Server pools:
> Disk Groups: DATA,REDO
> Mount point paths:
> Services:
> Type: RAC
> Start concurrency:
> Stop concurrency:
> OSDBA group: dba
> OSOPER group: racoper
> Database instances: testdb1,testdb2
> Configured nodes: ecl-odabase-0,ecl-odabase-1
> Database is administrator managed ,
>End Command output
2023-03-20 14:31:47: db_domain for the database testdb is

2023-03-20 14:31:47: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan
2023-03-20 14:31:47: Command output:
> SCAN name: ecl-oda-scan, Network: 1
> Subnet IPv4: 10.11.30.0/255.255.255.0/eth0, static
> Subnet IPv6:
> SCAN 1 IPv4 VIP: 10.11.30.48
> SCAN VIP is enabled.
> SCAN 2 IPv4 VIP: 10.11.30.49
> SCAN VIP is enabled.
> SCAN 3 IPv4 VIP: 10.11.30.50
> SCAN VIP is enabled. ,
>End Command output
2023-03-20 14:31:47: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan_listener -i 1 -S 1
2023-03-20 14:31:48: Command output:
> #@=result[0]: res_
name={ora.LISTENER_SCAN1.lsnr} netnum={1} lsnr_name={LISTENER_SCAN1} port={1521,1531} ports={null} enabled={true} enabled_nodes={} disabled_nodes={} ,
>End Command output
2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:48: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:48: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:48: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:48: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:49: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:49: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:49: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:49: Will be running following sql statements as user: oracle:

export oracle_
home=/u01/app/oracle/product/12.1.0.2/dbhome_2,
export sid= testdb1,
/u01/app/oracle/product/12.1.0.2/dbhome_
2/bin/sqlplus -L sys/password@ecl-oda-scan/testdb as sysdba

set echo off
set heading off
select value from v$parameter where name='pga_aggregate_target'

2023-03-20 14:31:50: Removing file /tmp/filezIGyIR
2023-03-20 14:31:50:
2023-03-20 14:31:50: Successfully removed file: /tmp/filezIGyIR
2023-03-20 14:31:50: /bin/su successfully executed

2023-03-20 14:31:50: Output is :
2147483648

1 row selected.


2023-03-20 14:31:50: Will be running following sql statements as user: oracle:

export oracle_home=/u01/app/oracle/product/12.1.0.2/dbhome_2,
export sid= testdb1,
/u01/app/oracle/product/12.1.0.2/dbhome_2/bin/sqlplus -L sys/password@ecl-oda-scan/testdb as sysdba

set echo off
set heading off
select value from v$parameter where name='pga_aggregate_limit'

2023-03-20 14:31:51: Removing file /tmp/fileBNgBgF
2023-03-20 14:31:51:
2023-03-20 14:31:51: Successfully removed file: /tmp/fileBNgBgF
2023-03-20 14:31:51: /bin/su successfully executed

2023-03-20 14:31:51: Output is :
4294967296

1 row selected.


2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:51: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:51: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:51: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:51: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:52: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 14:31:52: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:52: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:52: INFO: Upgrading the database testdb. It will take few minutes. Please wait...
2023-03-20 14:31:52: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:52: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:52: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:52: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 14:31:52: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:52: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:52: Executing cmd: export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl status database -d testdb
2023-03-20 14:31:53: Command output:
> Instance testdb1 is running on node ecl-odabase-0
> Instance testdb2 is running on node ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:53: INFO : Running the command /u01/app/19.0.0.0/grid/bin/crsctl stat resource ora.testdb.db -p
2023-03-20 14:31:58: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:58: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:58: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:58: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 14:31:58: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:58: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:58: INFO : Did not do scp for node : ecl-odabase-0
2023-03-20 14:31:59: INFO : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/dbstats_
testdb.sh
2023-03-20 14:31:59: INFO : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/dbstats_testdb.sh
2023-03-20 14:32:37: INFO : Running dbua to upgrade testdb
2023-03-20 14:32:37: INFO : Running dbua command
2023-03-20 14:32:38: INFO : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /u01/app/oracle/product/19.0.0.0/dbhome_
1/bin/dbua -silent -performFixUp true -dbName testdb
2023-03-20 14:32:38: INFO : Running on the local node: /bin/su oracle -c /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/dbua -silent -performFixUp true -dbName testdb
2023-03-20 15:03:35: INFO : upgrade testdb <>
2023-03-20 15:03:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:35: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:03:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:35: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:03:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:35: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:35: Executing cmd: export ORACLE_
HOME=/u01/app/oracle/product/19.0.0.0/dbhome_1; /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/srvctl status database -d testdb
2023-03-20 15:03:36: Command output:
> Instance testdb1 is running on node ecl-odabase-0
> Instance testdb2 is running on node ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:36: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:36: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:03:36: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:36: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 15:03:36: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:37: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:37: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:37: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:37: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:37: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:37: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:37: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:38: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:38: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:38: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:38: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************

> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:38: Command output:
> **************************************************************

> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,

>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:38: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:39: run_as_user2: Running /bin/su oracle -c ' export ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/dbhome_1;/u01/app/oracle/product/19.0.0.0/dbhome_1/bin/srvctl status instance -d testdb -n ecl-odabase-0 '
2023-03-20 15:03:39: Removing file /tmp/filetjwniy
2023-03-20 15:03:39:
2023-03-20 15:03:39: Successfully removed file: /tmp/filetjwniy
2023-03-20 15:03:39: /bin/su successfully executed

2023-03-20 15:03:39: INFO : Did not do scp for node : ecl-odabase-0
2023-03-20 15:03:40: INFO : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/runDatapatch.sh
2023-03-20 15:03:40: INFO : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/runDatapatch.sh
2023-03-20 15:04:09: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:04:10: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:04:10: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:04:10: Command output:
> **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** ,
>End Command output
2023-03-20 15:04:10: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:04:10: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:04:10: SUCCESS: Successfully upgraded the database testdb
[root@ecl-odabase-0 ~]#

 

Conclusion

ODA is the best option for small and medium enterprises. Oracle Database Appliance simplifies deployment, maintenance, and support for high-availability database solutions built from Oracle Database templates designed for optimal performance on this platform. For virtualized environments, oakcli simplifies upgrading the database to 19c.  19.13 is the most stable version of virtualized ODA and this is the last support version of virtualized ODA.
 

oda

Oracle Database Appliance (ODA) virtualized eases the database upgrade process using oakcli orchestration utility. This tool became more stable after 19.8. All the ODA components can be orchestrated via this oakcli utility. However, 19.13 will be the last upgrade for the virtualized platform, after that Oracle going to discontinue OVM and they are moving to KVM-based virtualization. This is going to be a revolutionary change for ODA.

In this article, I will cover the steps to perform database upgrades via oakcli tool.

There are pre-requisite steps that need to be performed before the upgrade

  • Gain Space in/u01 for an Upgrade
  • Create a New database home. I have covered this new database home creation in a previous blog post: https://chanaka-dbhelp.blogspot.com/2022/07/oda-x5-db-home-creation-error.html
  • Execute Pre-upgrade Check
  • Fixed the Pre-upgrade Issues
  • Backup Database Before Upgrade
  • Upgrade Database

 

1. We can gain space by removing the old repository files, this can be achieved by executing the manage clear repo command

oakcli manage cleanrepo --ver 19.8.0.0.0

Check the database homes that are configured, As per this example we have 3 database homes two 19c and one 12c

[root@ecl-odabase-0 app]# oakcli show dbhomes
Oracle Home Name      Oracle Home version Home Location                              Home Edition
----------------      ------------------- ------------                               ------------
OraDb12102_home2      12.1.0.2.191015     /u01/app/oracle/product/12.1.0.2/dbhome_2  Enterprise
OraDb19000_home1      19.13.0.0.211019    /u01/app/oracle/product/19.0.0.0/dbhome_1  Enterprise
OraDb19000_home2      19.13.0.0.211019    /u01/app/oracle/product/19.0.0.0/dbhome_2  Enterprise
[root@ecl-odabase-0 app]#

2.  Download the pre-upgrade utility from Oracle support

How to Download and Run Oracle’s Database Pre-Upgrade Utility (Doc ID 884522.1)

Note: We noticed that even pre-upgrade is handled by oakcli utility. But as a best practice, I would recommend to fixed those pre-upgrade suggestions before running the upgrade.

Please find the sample pre-upgrade execution output

[oracle@ecl-odabase-0 TESTDB]$ /u01/app/oracle/product/12.1.0.2/dbhome_2/jdk/bin/java -jar /home/oracle/PRE_UPGRADE/TESTDB/preupgrade.jar FILE DIR /home/oracle/PRE_UPGRADE/TESTDB/ ================== PREUPGRADE SUMMARY ================== /home/oracle/PRE_UPGRADE/TESTDB/preupgrade.log /home/oracle/PRE_UPGRADE/TESTDB/preupgrade_fixups.sql /home/oracle/PRE_UPGRADE/TESTDB/postupgrade_fixups.sql Execute fixup scripts as indicated below: Before upgrade: Log into the database and execute the preupgrade fixups @/home/oracle/PRE_UPGRADE/TESTDB/preupgrade_fixups.sql After the upgrade: Log into the database and execute the postupgrade fixups @/home/oracle/PRE_UPGRADE/TESTDB/postupgrade_fixups.sql Preupgrade complete: 2023-03-20T13:55:32 [oracle@ecl-odabase-0 TESTDB]$ 

 

Upgrade Database 

As per this example, we are upgrading testdb to 19c, the command required to point 19c home. The upgrade will take close to 1hr to complete all the tasks, Make sure to enable the screen before executing this command.

[root@ecl-odabase-0 incident]# oakcli upgrade database -db testdb -to OraDb19000_home1
INFO: 2023-03-20 14:31:04: Look at the log file '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/dbupgrade_96057.log' for more details

Please enter the 'SYS'  password :
Please re-enter the 'SYS' password:
INFO: 2023-03-20 14:31:52: Upgrading the database testdb. It will take few minutes. Please wait...
...

 

After Upgrade

This is oakcli output after the upgrade

[root@ecl-odabase-0 incident]# screen -ls
There is a screen on:
        96799.pts-1.ecl-odabase-0       (Attached)
1 Socket in /var/run/screen/S-root.

[root@ecl-odabase-0 incident]# oakcli upgrade database -db testdb -to OraDb19000_home1
INFO: 2023-03-20 14:31:04: Look at the log file '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/dbupgrade_96057.log' for more details

Please enter the 'SYS'  password :
Please re-enter the 'SYS' password:
INFO: 2023-03-20 14:31:52: Upgrading the database testdb. It will take few minutes. Please wait...
...


...
SUCCESS: 2023-03-20 15:04:10: Successfully upgraded the database testdb

I have shared the complete log to get the proper understanding, oakcli utility is handling all the tasks pre-patching, patching, and post-patching.

 

Complete Log

[root@ecl-odabase-0 ~]# cat /opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/dbupgrade_96057.log 2023-03-20 14:31:04: INFO: Look at the log file '/opt/oracle/oak/log/ecl-odabase-0/tools/19.13.0.0.0/dbupgrade_96057.log' for more details
2023-03-20 14:31:04: Command received is : oakcli upgrade database -db testdb -to OraDb19000_home1 2023-03-20 14:31:04: This is V4 machine 2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:05: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:05: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:05: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:05: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:05: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:05: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:05: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:06: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:06: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:06: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:06: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:06: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:06: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:18: Checking SSH equivalence for root. 2023-03-20 14:31:18: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster 2023-03-20 14:31:18: Command output: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online , >End Command output 2023-03-20 14:31:18: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:18: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:18: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-0 /bin/true" as user "root" 2023-03-20 14:31:18: Executing cmd: /bin/su root -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-0 /bin/true" 2023-03-20 14:31:18: SSH equivalence for root to ecl-odabase-0: success 2023-03-20 14:31:18: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-1 /bin/true" as user "root" 2023-03-20 14:31:18: Executing cmd: /bin/su root -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-1 /bin/true" 2023-03-20 14:31:18: SSH equivalence for root to ecl-odabase-1: success 2023-03-20 14:31:18: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.27 /bin/true" as user "root" 2023-03-20 14:31:18: Executing cmd: /bin/su root -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.27 /bin/true" 2023-03-20 14:31:19: SSH equivalence for root to 192.168.16.27: success 2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.28 /bin/true" as user "root" 2023-03-20 14:31:19: Executing cmd: /bin/su root -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.28 /bin/true" 2023-03-20 14:31:19: SSH equivalence for root to 192.168.16.28: success 2023-03-20 14:31:19: Checking SSH equivalence for oracle. 2023-03-20 14:31:19: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster 2023-03-20 14:31:19: Command output: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online , >End Command output 2023-03-20 14:31:19: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:19: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-0 /bin/true" as user "oracle" 2023-03-20 14:31:19: Executing cmd: /bin/su oracle -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-0 /bin/true" 2023-03-20 14:31:19: SSH equivalence for oracle to ecl-odabase-0: success 2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-1 /bin/true" as user "oracle" 2023-03-20 14:31:19: Executing cmd: /bin/su oracle -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 ecl-odabase-1 /bin/true" 2023-03-20 14:31:19: SSH equivalence for oracle to ecl-odabase-1: success 2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.27 /bin/true" as user "oracle" 2023-03-20 14:31:19: Executing cmd: /bin/su oracle -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.27 /bin/true" 2023-03-20 14:31:19: SSH equivalence for oracle to 192.168.16.27: success 2023-03-20 14:31:19: Invoking "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.28 /bin/true" as user "oracle" 2023-03-20 14:31:19: Executing cmd: /bin/su oracle -c "/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 192.168.16.28 /bin/true" 2023-03-20 14:31:20: SSH equivalence for oracle to 192.168.16.28: success 2023-03-20 14:31:32: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:32: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:32: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:32: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:32: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:32: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:33: Executing cmd: export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl status database -d testdb
2023-03-20 14:31:33: Command output:
> Instance testdb1 is running on node ecl-odabase-0
> Instance testdb2 is running on node ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:33: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:33: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 14:31:33: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:33: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 14:31:33: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:33: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:34: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:34: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 14:31:34: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:34: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 14:31:34: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:34: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:34: run_as_user2: Running /bin/su oracle -c ' export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2;/u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl config database -d testdb ' 2023-03-20 14:31:35: Removing file /tmp/file2c3Y6L 2023-03-20 14:31:35: 2023-03-20 14:31:35: Successfully removed file: /tmp/file2c3Y6L 2023-03-20 14:31:35: /bin/su successfully executed 2023-03-20 14:31:35: Getting the JOB_Queue_Processes for the database : testdb 2023-03-20 14:31:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:35: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:35: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:35: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:35: Executing cmd: export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl config database -db testdb
2023-03-20 14:31:36: Command output:
> Database unique name: testdb
> Database name: testdb
> Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_2 > Oracle user: oracle > Spfile: +DATA/TESTDB/PARAMETERFILE/spfile.296.1131972439 > Password file: +DATA/TESTDB/PASSWORD/pwdtestdb.269.1131971689 > Domain: > Start options: open > Stop options: immediate > Database role: PRIMARY > Management policy: AUTOMATIC > Server pools: > Disk Groups: DATA,REDO > Mount point paths: > Services: > Type: RAC > Start concurrency: > Stop concurrency: > OSDBA group: dba > OSOPER group: racoper > Database instances: testdb1,testdb2 > Configured nodes: ecl-odabase-0,ecl-odabase-1 > Database is administrator managed , >End Command output 2023-03-20 14:31:36: db_domain for the database testdb is
2023-03-20 14:31:36: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan
2023-03-20 14:31:37: Command output:
> SCAN name: ecl-oda-scan, Network: 1
> Subnet IPv4: 10.11.30.0/255.255.255.0/eth0, static
> Subnet IPv6:
> SCAN 1 IPv4 VIP: 10.11.30.48
> SCAN VIP is enabled.
> SCAN 2 IPv4 VIP: 10.11.30.49
> SCAN VIP is enabled.
> SCAN 3 IPv4 VIP: 10.11.30.50
> SCAN VIP is enabled. ,
>End Command output
2023-03-20 14:31:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan_listener -i 1 -S 1 2023-03-20 14:31:37: Command output: > #@=result[0]: res_name={ora.LISTENER_SCAN1.lsnr} netnum={1} lsnr_name={LISTENER_SCAN1} port={1521,1531} ports={null} enabled={true} enabled_nodes={} disabled_nodes={} , >End Command output 2023-03-20 14:31:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:38: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:38: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:38: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:38: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:38: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:38: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:38: Will be running following sql statements as user: oracle: export oracle_home=/u01/app/oracle/product/12.1.0.2/dbhome_2, export sid= testdb1, /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/sqlplus -L sys/password@ecl-oda-scan/testdb as sysdba

 set heading off set echo off set lines 200 show parameter job_queue_processes 
2023-03-20 14:31:40: Removing file /tmp/fileKzXXW6
2023-03-20 14:31:40:
2023-03-20 14:31:40: Successfully removed file: /tmp/fileKzXXW6
2023-03-20 14:31:40: /bin/su successfully executed

2023-03-20 14:31:40: Output is :
 job_queue_processes                  integer     1000

2023-03-20 14:31:40: JOB_Queue_Processes for the database testdb is 1000
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:40: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:40: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:40: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:40: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:40: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan
2023-03-20 14:31:41: Command output:
> SCAN name: ecl-oda-scan, Network: 1
> Subnet IPv4: 10.11.30.0/255.255.255.0/eth0, static
> Subnet IPv6:
> SCAN 1 IPv4 VIP: 10.11.30.48
> SCAN VIP is enabled.
> SCAN 2 IPv4 VIP: 10.11.30.49
> SCAN VIP is enabled.
> SCAN 3 IPv4 VIP: 10.11.30.50
> SCAN VIP is enabled. ,
>End Command output
2023-03-20 14:31:41: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan_listener -i 1 2023-03-20 14:31:41: Command output: > SCAN Listeners for network 1: > Registration invited nodes: > Registration invited subnets: > Endpoints: TCP:1521,1531 > SCAN Listener LISTENER_SCAN1 exists
> SCAN Listener is enabled. ,
>End Command output
2023-03-20 14:31:41: Executing cmd: /u01/app/19.0.0.0/grid/bin/cemutlo -n
2023-03-20 14:31:41: Command output:
> ecl-oda-lab-c ,
>End Command output
2023-03-20 14:31:41: Executing cmd: hostname -d
2023-03-20 14:31:41: Command output:
> newco.local ,
>End Command output
2023-03-20 14:31:41: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config nodeapps -a
2023-03-20 14:31:42: Command output:
> Network 1 exists
> Subnet IPv4: 10.11.30.0/255.255.255.0/eth0, static
> Subnet IPv6:
> Ping Targets:
> Network is enabled
> Network is individually enabled on nodes:
> Network is individually disabled on nodes:
> VIP exists: network number 1, hosting node ecl-odabase-0
> VIP Name: ecl-oda-0-vip.newco.local
> VIP IPv4 Address: 10.11.30.157
> VIP IPv6 Address:
> VIP is enabled.
> VIP is individually enabled on nodes:
> VIP is individually disabled on nodes:
> VIP exists: network number 1, hosting node ecl-odabase-1
> VIP Name: ecl-oda-1-vip.newco.local
> VIP IPv4 Address: 10.11.30.158
> VIP IPv6 Address:
> VIP is enabled.
> VIP is individually enabled on nodes:
> VIP is individually disabled on nodes: ,
>End Command output
2023-03-20 14:31:43: INFO : Logging all actions in the file /opt/oracle/oak/onecmd/tmp/ecl-odabase-0-20230320143143.log and traces in the file /opt/oracle/oak/onecmd/tmp/ecl-odabase-0-20230320143143.trc
2023-03-20 14:31:43: INFO : Loading the configuration file /opt/oracle/oak/onecmd/upgrade_databse.params... 2023-03-20 14:31:45: INFO : Creating the node list files... 2023-03-20 14:31:45: Executing cmd: ssh -o StrictHostKeyChecking=no root@ecl-odabase-0 "/bin/df -k /u01 /tmp" 2023-03-20 14:31:45: Command output: > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/xvdb1 95988492 78908600 12197204 87% /u01 > /dev/xvda2 57060636 43633540 10521856 81% / , >End Command output 2023-03-20 14:31:45: Free space on /u01 on ecl-odabase-0 is 12197204 1K-blocks 2023-03-20 14:31:45: Free space on / on ecl-odabase-0 is 10521856 1K-blocks 2023-03-20 14:31:45: Executing cmd: ssh -o StrictHostKeyChecking=no root@ecl-odabase-1 "/bin/df -k /u01 /tmp" 2023-03-20 14:31:45: Command output: > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/xvdb1 95988492 77826264 13279540 86% /u01 > /dev/xvda2 57060636 41370112 12785284 77% / , >End Command output 2023-03-20 14:31:45: Free space on /u01 on ecl-odabase-1 is 13279540 1K-blocks 2023-03-20 14:31:45: Free space on / on ecl-odabase-1 is 12785284 1K-blocks 2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:45: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:45: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:45: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:45: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:45: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:46: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:46: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:46: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:46: Executing cmd: export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl config database -db testdb
2023-03-20 14:31:47: Command output:
> Database unique name: testdb
> Database name: testdb
> Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_2 > Oracle user: oracle > Spfile: +DATA/TESTDB/PARAMETERFILE/spfile.296.1131972439 > Password file: +DATA/TESTDB/PASSWORD/pwdtestdb.269.1131971689 > Domain: > Start options: open > Stop options: immediate > Database role: PRIMARY > Management policy: AUTOMATIC > Server pools: > Disk Groups: DATA,REDO > Mount point paths: > Services: > Type: RAC > Start concurrency: > Stop concurrency: > OSDBA group: dba > OSOPER group: racoper > Database instances: testdb1,testdb2 > Configured nodes: ecl-odabase-0,ecl-odabase-1 > Database is administrator managed , >End Command output 2023-03-20 14:31:47: db_domain for the database testdb is
2023-03-20 14:31:47: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan
2023-03-20 14:31:47: Command output:
> SCAN name: ecl-oda-scan, Network: 1
> Subnet IPv4: 10.11.30.0/255.255.255.0/eth0, static
> Subnet IPv6:
> SCAN 1 IPv4 VIP: 10.11.30.48
> SCAN VIP is enabled.
> SCAN 2 IPv4 VIP: 10.11.30.49
> SCAN VIP is enabled.
> SCAN 3 IPv4 VIP: 10.11.30.50
> SCAN VIP is enabled. ,
>End Command output
2023-03-20 14:31:47: Executing cmd: /u01/app/19.0.0.0/grid/bin/srvctl config scan_listener -i 1 -S 1 2023-03-20 14:31:48: Command output: > #@=result[0]: res_name={ora.LISTENER_SCAN1.lsnr} netnum={1} lsnr_name={LISTENER_SCAN1} port={1521,1531} ports={null} enabled={true} enabled_nodes={} disabled_nodes={} , >End Command output 2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:48: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:48: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:48: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:48: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:48: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:49: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:49: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:49: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:49: Will be running following sql statements as user: oracle: export oracle_home=/u01/app/oracle/product/12.1.0.2/dbhome_2, export sid= testdb1, /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/sqlplus -L sys/password@ecl-oda-scan/testdb as sysdba

 set echo off set heading off select value from v$parameter where name='pga_aggregate_target' 
2023-03-20 14:31:50: Removing file /tmp/filezIGyIR
2023-03-20 14:31:50:
2023-03-20 14:31:50: Successfully removed file: /tmp/filezIGyIR
2023-03-20 14:31:50: /bin/su successfully executed

2023-03-20 14:31:50: Output is :
 2147483648

 1 row selected.


2023-03-20 14:31:50: Will be running following sql statements as user: oracle:

 export oracle_home=/u01/app/oracle/product/12.1.0.2/dbhome_2, export sid= testdb1, /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/sqlplus -L sys/password@ecl-oda-scan/testdb as sysdba 
 set echo off set heading off select value from v$parameter where name='pga_aggregate_limit' 
2023-03-20 14:31:51: Removing file /tmp/fileBNgBgF
2023-03-20 14:31:51:
2023-03-20 14:31:51: Successfully removed file: /tmp/fileBNgBgF
2023-03-20 14:31:51: /bin/su successfully executed

2023-03-20 14:31:51: Output is :
 4294967296

 1 row selected.


2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:51: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:51: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:51: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:51: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 14:31:51: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:52: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 14:31:52: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:52: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:52: INFO:  Upgrading the database testdb. It will take few minutes. Please wait...
2023-03-20 14:31:52: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 14:31:52: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 14:31:52: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:52: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 14:31:52: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 14:31:52: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 14:31:52: Executing cmd: export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/srvctl status database -d testdb 2023-03-20 14:31:53: Command output: > Instance testdb1 is running on node ecl-odabase-0 > Instance testdb2 is running on node ecl-odabase-1 , >End Command output 2023-03-20 14:31:53: INFO : Running the command /u01/app/19.0.0.0/grid/bin/crsctl stat resource ora.testdb.db -p 2023-03-20 14:31:58: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:58: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:58: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 14:31:58: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 14:31:58: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 14:31:58: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 14:31:58: INFO : Did not do scp for node : ecl-odabase-0 2023-03-20 14:31:59: INFO : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/dbstats_testdb.sh
2023-03-20 14:31:59: INFO : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/dbstats_testdb.sh 2023-03-20 14:32:37: INFO : Running dbua to upgrade testdb 2023-03-20 14:32:37: INFO : Running dbua command 2023-03-20 14:32:38: INFO : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/dbua -silent -performFixUp true -dbName testdb
2023-03-20 14:32:38: INFO : Running on the local node: /bin/su oracle -c /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/dbua -silent -performFixUp true -dbName testdb 2023-03-20 15:03:35: INFO : upgrade testdb <> 2023-03-20 15:03:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:03:35: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 15:03:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:03:35: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 15:03:35: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 15:03:35: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 15:03:35: Executing cmd: export ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/dbhome_1; /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/srvctl status database -d testdb
2023-03-20 15:03:36: Command output:
> Instance testdb1 is running on node ecl-odabase-0
> Instance testdb2 is running on node ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:36: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:36: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 15:03:36: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:03:36: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 15:03:36: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:37: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:37: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:03:37: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:37: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:37: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:03:37: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 15:03:37: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:37: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:38: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:03:38: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:38: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all
2023-03-20 15:03:38: Command output:
> ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > **************************************************************
> ecl-odabase-1:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** , >End Command output 2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:03:38: Command output: > **************************************************************
> ecl-odabase-0:
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
> ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** ,
>End Command output
2023-03-20 15:03:38: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes
2023-03-20 15:03:38: Command output:
> ecl-odabase-0
> ecl-odabase-1 ,
>End Command output
2023-03-20 15:03:39: run_as_user2: Running /bin/su oracle -c ' export ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/dbhome_1;/u01/app/oracle/product/19.0.0.0/dbhome_1/bin/srvctl status instance -d testdb -n ecl-odabase-0 ' 2023-03-20 15:03:39: Removing file /tmp/filetjwniy 2023-03-20 15:03:39: 2023-03-20 15:03:39: Successfully removed file: /tmp/filetjwniy 2023-03-20 15:03:39: /bin/su successfully executed 2023-03-20 15:03:39: INFO : Did not do scp for node : ecl-odabase-0 2023-03-20 15:03:40: INFO : This is root, will become oracle and run: /bin/su oracle -c /usr/bin/ssh -l oracle ecl-odabase-0 /opt/oracle/oak/onecmd/tmp/runDatapatch.sh 2023-03-20 15:03:40: INFO : Running on the local node: /bin/su oracle -c /opt/oracle/oak/onecmd/tmp/runDatapatch.sh 2023-03-20 15:04:09: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:04:10: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 15:04:10: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check cluster -all 2023-03-20 15:04:10: Command output: > ************************************************************** > ecl-odabase-0: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** > ecl-odabase-1: > CRS-4537: Cluster Ready Services is online > CRS-4529: Cluster Synchronization Services is online > CRS-4533: Event Manager is online > ************************************************************** , >End Command output 2023-03-20 15:04:10: Executing cmd: /u01/app/19.0.0.0/grid/bin/olsnodes 2023-03-20 15:04:10: Command output: > ecl-odabase-0 > ecl-odabase-1 , >End Command output 2023-03-20 15:04:10: SUCCESS: Successfully upgraded the database testdb [root@ecl-odabase-0 ~]# 

 

Conclusion

ODA is the best option for small and medium enterprises. Oracle Database Appliance simplifies deployment, maintenance, and support for high-availability database solutions built from Oracle Database templates designed for optimal performance on this platform. For virtualized environments, oakcli simplifies upgrading the database to 19c.  19.13 is the most stable version of virtualized ODA and this is the last support version of virtualized ODA.

More from this Author

OCI Tablespace Creation Error ORA 28361 Master Key not yet set for CREATE TABLESPACE in OCI DBCS

OCI Tablespace Creation Error – ORA-28361: Master Key not yet set for CREATE TABLESPACE in OCI DBCS

Introduction We are living in a data era. Every organization invests in a colossal sum of money to secure its IT infrastructure environment. ... Read More

OLVM Renew Engine and KVM Certificate

OLVM: Renew Engine and KVM Certificate

Introduction Virtualization made a significant change in the IT (Information Technology) industry. This technology helped many organizations to use ... Read More