OLVM Upgrade 4.4.8 – 4.4.10

OLVM (Oracle Linux Virtualization Manager) release 4.4 is based on the oVirt 4.4.0 through 4.4.10 release. Ovirt already released version 4.5, But Oracle is behind on releasing the 4.5. Currently, they are working on testing the 4.5 release for bugs.  If you are planning to upgrade OLVM and KVMs make sure to upgrade the OLVM engine first. Once the engine is up to date you can plan for KVM upgrades.

Note: OLVM-Engine upgrade can be performed without downtime. There is no impact on running VM and KVMs.

Please find the Oracle and Ovirt documentation to read the new feature upgrade from 4.4.8 to 4.4.10.

Oracle:

https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-manager/relnotes/relnotes-whatsnew.html#whatsnew

Ovirt: 

https://www.ovirt.org/release/4.4.10/

In this article, I will illustrate how can we upgrade the OLVM engine from 4.4.8 to 4.4.10. 

Please find the Oracle note for the upgrade :  OLVM: How to set OLVM Engine and KVM Hosts to Maintenance Mode? (Doc ID 2915795.1)

Pre-requisite Steps Before Upgrading the Engine

• If this is running in the virtualized platform, I would recommend taking a consistent snapshot. (To take a Consistent snapshot need to shut down the VM and take the snap).
• Back up the OLVM engine database using the below-mentioned command.
• Execute the engine-upgrade-check command which will check for the latest rpms.

OLVM Backup Command

engine-backup --scope=all --mode=backup --file=/root/backup_Before_Upgrade_05June2023.bck --log=/root/backuplog_before_upgrade.log

Sample Backup Output

[root@local-olvm-01 ~]# engine-backup --scope=all --mode=backup --file=/root/backup_Before_Upgrade_05June2023.bck --log=/root/backuplog_before_upgrade.log
Start of engine-backup with mode 'backup'
scope: all
archive file: /root/backup_Before_Upgrade_05June2023.bck
log file: /root/backuplog_before_upgrade.log
Backing up:
Notifying engine
- Files
- Engine database 'engine'
- DWH database 'ovirt_engine_history'
- Grafana database '/var/lib/grafana/grafana.db'
Packing into file '/root/backup_Before_Upgrade_05June2023.bck'
Notifying engine
Done.
[root@local-olvm-01 ~]#

Pre-Upgrade Check

• Execute engine-upgrade-check to get the latest rpm upgrade for the engine.

Sample pre-upgrade engine log

[root@local-olvm-01 ~]# engine-upgrade-check
VERB: Creating transaction
VERB: Queue package ovirt-engine-setup for update
VERB: Building transaction
VERB: Transaction built
VERB: Transaction Summary:
VERB:     install   : ovirt-engine-setup-plugin-websocket-proxy-4.4.10.7-1.0.22.el8.noarch
VERB:     install   : ovirt-engine-setup-plugin-cinderlib-4.4.10.7-1.0.22.el8.noarch
VERB:     install   : python3-ovirt-engine-lib-4.4.10.7-1.0.22.el8.noarch
VERB:     install   : ovirt-engine-setup-plugin-imageio-4.4.10.7-1.0.22.el8.noarch
VERB:     install   : ovirt-engine-setup-plugin-ovirt-engine-4.4.10.7-1.0.22.el8.noarch
VERB:     install   : ovirt-engine-setup-plugin-ovirt-engine-common-4.4.10.7-1.0.22.el8.noarch
VERB:     install   : ovirt-engine-setup-4.4.10.7-1.0.22.el8.noarch
VERB:     install   : ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.4.10.7-1.0.22.el8.noarch
VERB:     install   : ovirt-engine-setup-base-4.4.10.7-1.0.22.el8.noarch
VERB:     remove    : ovirt-engine-setup-4.4.8.6-1.0.11.el8.noarch
VERB:     remove    : ovirt-engine-setup-base-4.4.8.6-1.0.11.el8.noarch
VERB:     remove    : ovirt-engine-setup-plugin-cinderlib-4.4.8.6-1.0.11.el8.noarch
VERB:     remove    : ovirt-engine-setup-plugin-imageio-4.4.8.6-1.0.11.el8.noarch
VERB:     remove    : ovirt-engine-setup-plugin-ovirt-engine-4.4.8.6-1.0.11.el8.noarch
VERB:     remove    : ovirt-engine-setup-plugin-ovirt-engine-common-4.4.8.6-1.0.11.el8.noarch
VERB:     remove    : ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.4.8.6-1.0.11.el8.noarch
VERB:     remove    : ovirt-engine-setup-plugin-websocket-proxy-4.4.8.6-1.0.11.el8.noarch
VERB:     remove    : python3-ovirt-engine-lib-4.4.8.6-1.0.11.el8.noarch
VERB: Closing transaction with commit
Upgrade available.
[root@local-olvm-01 ~]#
[0] 0:platform-python*   

Upgrade Engine

I would recommend executing this in screen or tmux to avoid any terminal connection interruption. This process will be close to 30 to 45 min. you can validate the rpm update by looking at /var/log/dnf.rpm.log

Sample engine upgrade log

[root@local-olvm-01 ~]# dnf update ovirt\*setup\*
Last metadata expiration check: 4:30:51 ago on Mon 05 Jun 2023 04:00:02 PM EDT.
Dependencies resolved.
=============================================================================================================================================================================================================
 Package                                                                      Architecture                       Version                                         Repository                             Size
=============================================================================================================================================================================================================
Upgrading:
 ovirt-engine-dwh-grafana-integration-setup                                   noarch                             4.4.10-1.0.3.el8                                ovirt-4.4                              89 k
 ovirt-engine-dwh-setup                                                       noarch                             4.4.10-1.0.3.el8                                ovirt-4.4                              96 k
 ovirt-engine-setup                                                           noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                              22 k
 ovirt-engine-setup-base                                                      noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                             119 k
 ovirt-engine-setup-plugin-cinderlib                                          noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                              43 k
 ovirt-engine-setup-plugin-imageio                                            noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                              30 k
 ovirt-engine-setup-plugin-ovirt-engine                                       noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                             208 k
 ovirt-engine-setup-plugin-ovirt-engine-common                                noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                             126 k
 ovirt-engine-setup-plugin-vmconsole-proxy-helper                             noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                              42 k
 ovirt-engine-setup-plugin-websocket-proxy                                    noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                              43 k
 python3-ovirt-engine-lib                                                     noarch                             4.4.10.7-1.0.22.el8                             ovirt-4.4                              44 k

Transaction Summary
=============================================================================================================================================================================================================
Upgrade  11 Packages

Total download size: 862 k
Is this ok [y/N]: y

Engine-step

This will reconfigure the engine with the latest inputs.

Sample engine-setup log

[root@local-olvm-01 ~]# engine-setup
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20220725103208-vp6a4t.log
          Version: otopi-1.9.5 (otopi-1.9.5-1.el8)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup (late)
[ INFO  ] Stage: Environment customization

          --== PRODUCT OPTIONS ==--

          Configure Cinderlib integration (Currently in tech preview) (Yes, No) [No]:
          Configure Engine on this host (Yes, No) [Yes]:

          Configuring ovirt-provider-ovn also sets the Default cluster's default network provider to ovirt-provider-ovn.
          Non-Default clusters may be configured with an OVN after installation.
          Configure ovirt-provider-ovn (Yes, No) [Yes]:
          Configure WebSocket Proxy on this host (Yes, No) [Yes]:

          * Please note * : Data Warehouse is required for the engine.
          If you choose to not configure it on this host, you have to configure
          it on a remote host, and then configure the engine on this host so
          that it can access the database of the remote Data Warehouse host.
          Configure Data Warehouse on this host (Yes, No) [Yes]:
          Configure VM Console Proxy on this host (Yes, No) [Yes]:
          Configure Grafana on this host (Yes, No) [Yes]:

          --== PACKAGES ==--

[ INFO  ] Checking for product updates...
[ INFO  ] DNF Package grafana-postgres available, but not installed.
[ INFO  ] No product updates found

          --== NETWORK CONFIGURATION ==--

          Host fully qualified DNS name of this server [local-olvm-01.local.com]:
[WARNING] Failed to resolve kvm02.local.com using DNS, it can be resolved only locally

          Setup can automatically configure the firewall on this system.
          Note: automatic configuration of the firewall may overwrite current settings.
          Do you want Setup to configure the firewall? (Yes, No) [Yes]:
[ INFO  ] firewalld will be configured as firewall manager.

          --== DATABASE CONFIGURATION ==--

          Where is the DWH database located? (Local, Remote) [Local]:

          Setup can configure the local postgresql server automatically for the DWH to run. This may conflict with existing applications.
          Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:
          Where is the Engine database located? (Local, Remote) [Local]:

          Setup can configure the local postgresql server automatically for the engine to run. This may conflict with existing applications.
          Would you like Setup to automatically configure postgresql and create Engine database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:

          --== OVIRT ENGINE CONFIGURATION ==--

          Engine admin password:
          Confirm engine admin password:
[WARNING] Password is weak: The password is shorter than 8 characters
          Use weak password? (Yes, No) [No]: Yes
          Application mode (Virt, Gluster, Both) [Both]:
          Use default credentials (admin@internal) for ovirt-provider-ovn (Yes, No) [Yes]:

          --== STORAGE CONFIGURATION ==--

          Default SAN wipe after delete (Yes, No) [No]:

          --== PKI CONFIGURATION ==--

          Organization name for certificate [local.com]:

          --== APACHE CONFIGURATION ==--

          Setup can configure the default page of the web server to present the application home page. This may conflict with existing applications.
          Do you wish to set the application as the default page of the web server? (Yes, No) [Yes]:

          Setup can configure apache to use SSL using a certificate issued from the internal CA.
          Do you wish Setup to configure that, or prefer to perform that manually? (Automatic, Manual) [Automatic]:

          --== SYSTEM CONFIGURATION ==--


          --== MISC CONFIGURATION ==--

          Please choose Data Warehouse sampling scale:
          (1) Basic
          (2) Full
          (1, 2)[1]:
          Use Engine admin password as initial Grafana admin password (Yes, No) [Yes]:

          --== END OF CONFIGURATION ==--

[ INFO  ] Stage: Setup validation
[WARNING] Less than 16384MB of memory is available

          --== CONFIGURATION PREVIEW ==--

          Application mode                        : both
          Default SAN wipe after delete           : False
          Host FQDN                               : local-olvm-01.local.com
          Firewall manager                        : firewalld
          Update Firewall                         : True
          Set up Cinderlib integration            : False
          Configure local Engine database         : True
          Set application as default page         : True
          Configure Apache SSL                    : True
          Engine database host                    : localhost
          Engine database port                    : 5432
          Engine database secured connection      : False
          Engine database host name validation    : False
          Engine database name                    : engine
          Engine database user name               : engine
          Engine installation                     : True
          PKI organization                        : local.com
          Set up ovirt-provider-ovn               : True
          Grafana integration                     : True
          Grafana database user name              : ovirt_engine_history_grafana
          Configure WebSocket Proxy               : True
          DWH installation                        : True
          DWH database host                       : localhost
          DWH database port                       : 5432
          DWH database secured connection         : False
          DWH database host name validation       : False
          DWH database name                       : ovirt_engine_history
          Configure local DWH database            : True
          Configure VMConsole Proxy               : True

          Please confirm installation settings (OK, Cancel) [OK]:
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stopping engine service
[ INFO  ] Stopping ovirt-fence-kdump-listener service
[ INFO  ] Stopping dwh service
[ INFO  ] Stopping vmconsole-proxy service
[ INFO  ] Stopping websocket-proxy service
[ INFO  ] Stage: Misc configuration (early)
[ INFO  ] Stage: Package installation
[ INFO  ] DNF Downloading 4 files, 16083.99KB
[ INFO  ] DNF Downloaded ovirt-vmconsole-1.0.9-3.el8.noarch.rpm
[ INFO  ] DNF Downloaded selinux-policy-3.14.3-80.0.4.el8_5.2.noarch.rpm
[ INFO  ] DNF Downloaded selinux-policy-targeted-3.14.3-80.0.4.el8_5.2.noarch.rpm
[ INFO  ] DNF Downloaded ovirt-vmconsole-proxy-1.0.9-3.el8.noarch.rpm
[ INFO  ] DNF Upgraded: selinux-policy-3.14.3-80.0.4.el8_5.2.noarch
[ INFO  ] DNF Upgraded: selinux-policy-targeted-3.14.3-80.0.4.el8_5.2.noarch
[ INFO  ] DNF Upgraded: ovirt-vmconsole-1.0.9-3.el8.noarch
[ INFO  ] DNF Upgraded: ovirt-vmconsole-proxy-1.0.9-3.el8.noarch
[ INFO  ] DNF Unknown: ovirt-vmconsole-proxy-1.0.9-2.el8.noarch
[ INFO  ] DNF Unknown: ovirt-vmconsole-1.0.9-2.el8.noarch
[ INFO  ] DNF Unknown: selinux-policy-targeted-3.14.3-80.0.1.el8.noarch
[ INFO  ] DNF Unknown: selinux-policy-3.14.3-80.0.1.el8.noarch
[ INFO  ] DNF Verify: selinux-policy-3.14.3-80.0.4.el8_5.2.noarch 1/8
[ INFO  ] DNF Verify: selinux-policy-3.14.3-80.0.1.el8.noarch 2/8
[ INFO  ] DNF Verify: selinux-policy-targeted-3.14.3-80.0.4.el8_5.2.noarch 3/8
[ INFO  ] DNF Verify: selinux-policy-targeted-3.14.3-80.0.1.el8.noarch 4/8
[ INFO  ] DNF Verify: ovirt-vmconsole-1.0.9-3.el8.noarch 5/8
[ INFO  ] DNF Verify: ovirt-vmconsole-1.0.9-2.el8.noarch 6/8
[ INFO  ] DNF Verify: ovirt-vmconsole-proxy-1.0.9-3.el8.noarch 7/8
[ INFO  ] DNF Verify: ovirt-vmconsole-proxy-1.0.9-2.el8.noarch 8/8
[ INFO  ] Stage: Misc configuration
[ INFO  ] Upgrading CA
[ INFO  ] Initializing PostgreSQL
[ INFO  ] Creating PostgreSQL 'engine' database
[ INFO  ] Configuring PostgreSQL
[ INFO  ] Creating PostgreSQL 'ovirt_engine_history' database
[ INFO  ] Configuring PostgreSQL
[ INFO  ] Creating CA: /etc/pki/ovirt-engine/ca.pem
[ INFO  ] Creating CA: /etc/pki/ovirt-engine/qemu-ca.pem
[ INFO  ] Updating OVN SSL configuration
[ INFO  ] Updating OVN timeout configuration
[ INFO  ] Creating/refreshing DWH database schema
[ INFO  ] Setting up ovirt-vmconsole proxy helper PKI artifacts
[ INFO  ] Setting up ovirt-vmconsole SSH PKI artifacts
[ INFO  ] Configuring WebSocket Proxy
[ INFO  ] Creating/refreshing Engine database schema
[ INFO  ] Creating a user for Grafana
[ INFO  ] Creating/refreshing Engine 'internal' domain database schema
[ INFO  ] Creating default mac pool range
[ INFO  ] Adding default OVN provider to database
[ INFO  ] Adding OVN provider secret to database
[ INFO  ] Setting a password for internal user admin
[ INFO  ] Install selinux module /usr/share/ovirt-engine/selinux/ansible-runner-service.cil
[ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Starting engine service
[ INFO  ] Starting dwh service
[ INFO  ] Starting Grafana service
[ INFO  ] Restarting ovirt-vmconsole proxy service

          --== SUMMARY ==--

[ INFO  ] Restarting httpd
          Please use the user 'admin@internal' and password specified in order to login
          Web access is enabled at:
              http://local-olvm-01.local.com:80/ovirt-engine
              https://local-olvm-01.local.com:443/ovirt-engine
          Internal CA CF:C8:A2:E0:42:FE:5F:19:55:B3:E2:9F:A9:7F:4C:DC:49:8D:C7:CB
          SSH fingerprint: SHA256:aRJ1E8zUzNaYsXG2tCfvV4EGTMxw/mMzrKQQt2ZFZKE
[WARNING] Less than 16384MB of memory is available
          Web access for grafana is enabled at:
              https://local-olvm-01.local.com/ovirt-engine-grafana/
          Please run the following command on the engine machine kvm02.local.com, for SSO to work:
          systemctl restart ovirt-engine

          --== END OF SUMMARY ==--

[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220725103208-vp6a4t.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220725103526-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Execution of setup completed successfully
[root@local-olvm-01 ~]#

Engine Startup Issue

After the upgrade, we were not able to start the OLVM engine because of the JDBC driver issue. 

Error

Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [select * from  gettagsbyparent_id(?, ?, ?, ?, ?, ?, ?, ?, ?)]; nested exception is org.postgresql.util.PSQLException: ERROR: function gettagsbyparent_id(uuid, unknown, character varying, character varying, unknown, boolean, integer, unknown, unknown) does not exist
  Hint: No function matches the given name and argument types. You might need to add explicit type casts.
  Position: 16

Complete Error Log

2023-06-05 20:44:07,605-04 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 50) 
MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: 
WFLYEE0042: Failed to construct component instance
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.base/java.lang.Thread.run(Thread.java:829)
        at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:127)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:141)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
        ... 8 more
Caused by: javax.ejb.EJBException: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke protected void org.ovirt.engine.core.bll.TagsDirector.init() on org.ovirt.engine.core.bll.TagsDirector@2bd6ae3f
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:239)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:70)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.weld@23.0.2.Final//org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
        ... 13 more
                at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee@23.0.2.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
        at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3@23.0.2.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232)
        ... 28 more
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.weld.core@3.1.6.Final//org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:83)
        ... 59 more
Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar 
[select * from  gettagsbyparent_id(?, ?, ?, ?, ?, ?, ?, ?, ?)]; nested exception is org.postgresql.util.PSQLException: 
ERROR: function gettagsbyparent_id(uuid, unknown, character varying, character varying, unknown, boolean, integer, unknown, unknown) does not exist
  Hint: No function matches the given name and argument types. You might need to add explicit type casts.
  Position: 16

Solution

This error was caused due to old Postgres JDBC driver issue, Oracle has a meta link note to address this JDBC driver failure. OLVM: 500 – Internal Server Error after OLVM 4.4.10 dnf update (Doc ID 2909844.1)

Validate the Postgres JDBC Driver Version 

First, validate the current JDBC driver version as mentioned below. This will help to understand driver is up to date or not. The issue can be fixed by installing the postgresql-jdbc-42.2.14-1.el8.noarch. Also, the latest rpm can be validated from /vat/log/dnf.rpm.log .

[root@local-olvm-01 ovirt-engine]# rpm -qa | grep jdbc*
postgresql-jdbc-42.2.3-3.el8_2.noarch
ovirt-engine-extension-aaa-jdbc-1.2.0-1.el8.noarch

Upgrade Postgres JDBC driver.

[root@sofe-olvm-01 log]# dnf update postgresql-jdbc-42.2.14-1.el8.noarch
Last metadata expiration check: 0:19:54 ago on Mon 05 Jun 2023 08:43:55 PM EDT.
Dependencies resolved.
=============================================================================================================================================================================================================
 Package                                             Architecture                               Version                                              Repository                                         Size
=============================================================================================================================================================================================================
Upgrading:
 postgresql-jdbc                                     noarch                                     42.2.14-1.el8                                        ol8_appstream                                     753 k
Transaction Summary
=============================================================================================================================================================================================================
Upgrade  1 Package
Total download size: 753 k
Is this ok [y/N]: y
Downloading Packages:
postgresql-jdbc-42.2.14-1.el8.noarch.rpm                                                                                                                                     788 kB/s | 753 kB     00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                        786 kB/s | 753 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                     1/1
  Upgrading        : postgresql-jdbc-42.2.14-1.el8.noarch                                                                                                                                                1/2
  Cleanup          : postgresql-jdbc-42.2.3-3.el8_2.noarch                                                                                                                                               2/2
  Verifying        : postgresql-jdbc-42.2.14-1.el8.noarch                                                                                                                                                1/2
  Verifying        : postgresql-jdbc-42.2.3-3.el8_2.noarch                                                                                                                                               2/2
Upgraded:
  postgresql-jdbc-42.2.14-1.el8.noarch
Complete!
[root@sofe-olvm-01 log]#

Conclusion

OLVM 4.4.10 looks much more stable than 4.4.8 and also came up with bug fixes for VM high availability and snapshots.  Before upgrading OLVM, I would always recommend taking a backup of the OLVM engine. In case of failure, we can restore from the backup. 

Also, these upgrades are smooth and most of the upgrade issues are covered with Oracle meta link notes.