OLVM: Renew Engine and KVM Certificate

Introduction

Virtualization made a significant change in the IT (Information Technology) industry. This technology helped many organizations to use server resources efficiently. Even though cloud technology is emerging some companies are not ready to move their workloads to the cloud due to data sensitivity and business obstacles. So the only option to save the IT infrastructure cost using virtualization technology.

As organizations hosting their critical database workloads, we need to ensure virtualization platforms are fully secure. Oracle Linux Virtualization Manager has these security certificates that need to be renewed every year.  

In this article, I will elaborate on how we can renew these certificates from the OLVM engine.

Use below mentioned metalink note for the OLVM renewal process : 

OLVM: GUI shows “Engine’s certification is about to expire at yyyy-mm-dd. Please renew the engine’s certification.” (Doc ID 2956779.1)

OLVM: How to renew OLVM Hosts Certificate in OLVM Environment/Infrastructure (Doc ID 2885203.1)

Validate the current Certificate Expiration Time:

Execute the OpenSSL command to validate the certificate expiration time.

Validate Engine Certificate:

Note: First we need to renew the engine certificate and then we can renew the KVM certificates.

[root@sofe-olvm-01 ~]# openssl x509 -in /etc/pki/ovirt-engine/certs/engine.cer -noout -dates
notBefore=Jul 26 13:31:25 2022 GMT
notAfter=Aug 29 13:31:25 2023 GMT
[root@sofe-olvm-01 ~]#

Validate KVM Certificate:

Execute OpenSSL command to validate the KVM certificate

As per below mentioned outputs KVM certificates are expiring on Oct 11th, 2023. These certificates are valid only for 1 year.

Note: We need to renew these certificates before these expiry dates, otherwise it’s impacting managing these KVMs via the OLVM engine.

[root@kvm01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates
notBefore=Sep  7 20:05:13 2022 GMT
notAfter=Oct 11 20:05:13 2023 GMT
[root@kvm01 ~]#

[root@kvm02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates
notBefore=Sep  7 20:41:43 2022 GMT
notAfter=Oct 11 20:41:43 2023 GMT
[root@kvm02 ~]#

Review Engine Certificates

First, we need to renew the engine certificate using the below-mentioned steps. For this renewal, we do not need to take any downtime.

As a best practice let’s back the engine.

engine-backup --scope=all --mode=backup --file=/root/backup_renew_certificate_30Aug2023.bck --log=/root/backup_renew_certificate.log

[root@engine-olvm-01 ~]# engine-backup --scope=all --mode=backup --file=/root/backup_renew_certificate_30Aug2023.bck --log=/root/backup_renew_certificate.log
Start of engine-backup with mode 'backup'
scope: all
archive file: /root/backup_renew_certificate_30Aug2023.bck
log file: /root/backup_renew_certificate.log
Backing up:
Notifying engine
- Files
- Engine database 'engine'
- DWH database 'ovirt_engine_history'
- Grafana database '/var/lib/grafana/grafana.db'
Packing into file '/root/backup_renew_certificate_30Aug2023.bck'
Notifying engine
Done.
[root@engine-olvm-01 ~]#

Validate all the certificates in the engine

It’s better to validate all the certificates before the renewal, This gives you a clear idea about what we need to renew

Login to the engine server and navigate to “/etc/pki/ovirt-engine/certs/” folder and execute for cert in *.cer; do echo “Certificate: $cert”; openssl x509 -in “$cert” -noout -dates; echo; do a script to get all the certificates.

cd /etc/pki/ovirt-engine/certs/
for cert in *.cer; do echo "Certificate: $cert"; openssl x509 -in "$cert" -noout -dates; echo;  done

[root@engine-olvm-01 certs]# for cert in *.cer; do echo "Certificate: $cert"; openssl x509 -in "$cert" -noout -dates; echo;  done
Certificate: apache.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT

Certificate: engine.cer
notBefore=Jul 26 13:31:25 2022 GMT
notAfter=Aug 29 13:31:25 2023 GMT

Certificate: jboss.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT

Certificate: KVM120.sofe.ca.cer
notBefore=Sep  7 20:05:13 2022 GMT
notAfter=Oct 11 20:05:13 2023 GMT

Certificate: KVM120.sofe.ca-ssh.cer
notBefore=Sep  7 20:06:34 2022 GMT
notAfter=Oct 11 20:06:34 2023 GMT

Certificate: KVM121.sofe.ca.cer
notBefore=Sep  7 20:41:43 2022 GMT
notAfter=Oct 11 20:41:43 2023 GMT

Certificate: KVM121.sofe.ca-ssh.cer
notBefore=Sep  7 20:43:08 2022 GMT
notAfter=Oct 11 20:43:08 2023 GMT

Certificate: ovirt-provider-ovn.cer
notBefore=Jul 26 13:31:27 2022 GMT
notAfter=Aug 29 13:31:27 2023 GMT

Certificate: ovn-ndb.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT

Certificate: ovn-sdb.cer
notBefore=Jul 26 13:31:27 2022 GMT
notAfter=Aug 29 13:31:27 2023 GMT

Certificate: reports.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT

Certificate: vmconsole-proxy-helper.cer
notBefore=Jul 26 13:31:42 2022 GMT
notAfter=Aug 29 13:31:42 2023 GMT

Certificate: vmconsole-proxy-host.cer
notBefore=Jul 26 13:31:43 2022 GMT
notAfter=Aug 29 13:31:43 2023 GMT

Certificate: vmconsole-proxy-user.cer
notBefore=Jul 26 13:31:43 2022 GMT
notAfter=Aug 29 13:31:43 2023 GMT

Certificate: websocket-proxy.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT

[root@engine-olvm-01 certs]#

How to renew a Certificate

To renew a certificate we need to execute the “engine-setup –offline” command, This generates all the new certificates.

[root@sofe-olvm-01 ~]# engine-setup --offline
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20230830141846-ehsd5w.log
          Version: otopi-1.9.5 (otopi-1.9.5-1.el8)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup (late)
[ INFO  ] Stage: Environment customization

          --== PRODUCT OPTIONS ==--

[ INFO  ] ovirt-provider-ovn already installed, skipping.

          --== PACKAGES ==--


          --== NETWORK CONFIGURATION ==--


          Setup can automatically configure the firewall on this system.
          Note: automatic configuration of the firewall may overwrite current settings.
          Do you want Setup to configure the firewall? (Yes, No) [Yes]: No

          --== DATABASE CONFIGURATION ==--

          The detected DWH database size is 138.05131435394287 MB.
          Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.
          If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.
          Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: No
[WARNING] Are you sure you do not want to backup the DWH database?
          A positive reply makes sense only if you do not need the data in DWH, or have some other, external means to restore it to a working state.
          Are you sure you do not want to backup the DWH database?(Yes, No) [No]:
          The DWH Database will be backed up prior to upgrade.
          Perform full vacuum on the oVirt engine history
          database ovirt_engine_history@localhost?
          This operation may take a while depending on this setup health and the
          configuration of the db vacuum process.
          See https://www.postgresql.org/docs/12/sql-vacuum.html
          (Yes, No) [No]:

          --== OVIRT ENGINE CONFIGURATION ==--

          Perform full vacuum on the engine database engine@localhost?
          This operation may take a while depending on this setup health and the
          configuration of the db vacuum process.
          See https://www.postgresql.org/docs/12/sql-vacuum.html
          (Yes, No) [No]:

          --== STORAGE CONFIGURATION ==--


          --== PKI CONFIGURATION ==--

          One or more of the certificates should be renewed, because they expire soon, or include an invalid expiry date, or they were created with validity period longer than 398 days, or do not include the subjectAltName extension, which can cause them to be rejected by recent browsers and up to date hosts.
          See https://www.ovirt.org/develop/release-management/features/infra/pki-renew/ for more details.
          Renew certificates? (Yes, No) [No]: Yes

          --== APACHE CONFIGURATION ==--


          --== SYSTEM CONFIGURATION ==--


          --== MISC CONFIGURATION ==--


          --== END OF CONFIGURATION ==--

[ INFO  ] Stage: Setup validation
          During execution engine service will be stopped (OK, Cancel) [OK]:
[ INFO  ] Cleaning stale zombie tasks and commands

          --== CONFIGURATION PREVIEW ==--

          Default SAN wipe after delete           : False
          Host FQDN                               : oracle-olvm-01.sofe.ca
          Firewall manager                        : firewalld
          Update Firewall                         : False
          Set up Cinderlib integration            : False
          Engine database host                    : localhost
          Engine database port                    : 5432
          Engine database secured connection      : False
          Engine database host name validation    : False
          Engine database name                    : engine
          Engine database user name               : engine
          Engine installation                     : True
          PKI organization                        : sofe.ca
          Renew PKI                               : True
          Set up ovirt-provider-ovn               : True
          Grafana integration                     : True
          Grafana database user name              : ovirt_engine_history_grafana
          Configure WebSocket Proxy               : True
          DWH installation                        : True
          DWH database host                       : localhost
          DWH database port                       : 5432
          DWH database secured connection         : False
          DWH database host name validation       : False
          DWH database name                       : ovirt_engine_history
          DWH database user name                  : ovirt_engine_history
          Backup DWH database                     : True
          Configure VMConsole Proxy               : True

          Please confirm installation settings (OK, Cancel) [OK]:
[ INFO  ] Cleaning async tasks and compensations
[ INFO  ] Unlocking existing entities
[ INFO  ] Checking the Engine database consistency
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stopping engine service
[ INFO  ] Stopping ovirt-fence-kdump-listener service
[ INFO  ] Stopping dwh service
[ INFO  ] Stopping vmconsole-proxy service
[ INFO  ] Stopping websocket-proxy service
[ INFO  ] Stopping service: grafana-server
[ INFO  ] Stage: Misc configuration (early)
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Upgrading CA
[ INFO  ] Renewing engine certificate
[ INFO  ] Renewing jboss certificate
[ INFO  ] Renewing websocket-proxy certificate
[ INFO  ] Renewing apache certificate
[ INFO  ] Renewing reports certificate
[ INFO  ] Updating OVN SSL configuration
[ INFO  ] Updating OVN timeout configuration
[ INFO  ] Backing up database localhost:ovirt_engine_history to '/var/lib/ovirt-engine-dwh/backups/dwh-20230830142301.tv3pj2k_.dump'.
[ INFO  ] Creating/refreshing DWH database schema
[ INFO  ] Configuring WebSocket Proxy
[ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20230830142309.uai8smw2.dump'.
[ INFO  ] Creating/refreshing Engine database schema
[ INFO  ] Creating/refreshing Engine 'internal' domain database schema
          Unregistering existing client registration info.
[ INFO  ] Install selinux module /usr/share/ovirt-engine/selinux/ansible-runner-service.cil
[ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Starting service: grafana-server
[ INFO  ] Starting engine service
[ INFO  ] Starting dwh service
[ INFO  ] Starting Grafana service
[ INFO  ] Restarting ovirt-vmconsole proxy service

          --== SUMMARY ==--

[ INFO  ] Restarting httpd
          Web access is enabled at:
              http://oracle-olvm-01.sofe.ca:80/ovirt-engine
              https://oracle-olvm-01.sofe.ca:443/ovirt-engine
          Internal CA DD:55:2B:EF:9C:D7:93:F9:A1:AF:9B:64:54:02:0A:FF:88:6F:5B:BD
          SSH fingerprint: SHA256:XPSQG8LoQaL2r65lTUeNkzdChjFZUXaowO6kNUTKZaQ
          Web access for grafana is enabled at:
              https://oracle-olvm-01.sofe.ca/ovirt-engine-grafana/
          Please run the following command on the engine machine oracle-olvm-01.sofe.ca, for SSO to work:
          systemctl restart ovirt-engine

          --== END OF SUMMARY ==--

[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20230830141846-ehsd5w.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20230830142400-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Execution of setup completed successfully
[root@engine-olvm-01 ~]#

Post Certificate Renewal

After executing the engine renewal command, Make sure to restart the overt-engine so this will help to log to the OLVM Console page.

[root@engine-olvm-01 ~]# systemctl restart ovirt-engine.service
[root@engine-olvm-01 ~]# systemctl status ovirt-engine.service
● ovirt-engine.service - oVirt Engine
   Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-08-30 14:25:24 EDT; 6s ago
 Main PID: 3289277 (ovirt-engine.py)
    Tasks: 104 (limit: 100229)
   Memory: 685.7M
   CGroup: /system.slice/ovirt-engine.service
           ├─3289277 /usr/libexec/platform-python /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify start
           └─3289464 ovirt-engine --add-modules java.se -server -XX:+TieredCompilation -Xms3930M -Xmx3930M -Xss1M -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcIn>
Aug 30 14:25:24 engine-olvm-01.sofe.ca systemd[1]: Starting oVirt Engine...
Aug 30 14:25:24 engine-olvm-01.sofe.ca systemd[1]: Started oVirt Engine.
Aug 30 14:25:24 engine-olvm-01.sofe.ca ovirt-engine.py[3289277]: 2023-08-30 14:25:24,999-0400 ovirt-engine: INFO _detectJBossVersion:186 Detecting JBoss version. Running: /usr/lib/jvm/jre-11-openjdk-11.0.16>
Aug 30 14:25:25 engine-olvm-01.sofe.ca ovirt-engine.py[3289277]: 2023-08-30 14:25:25,838-0400 ovirt-engine: INFO _detectJBossVersion:206 Return code: 0,  | stdout: '['WildFly Full 23.0.2.Final (WildFly Core>
lines 1-14/14 (END)

Conclusion

Any platform security is a major concern, As system and database architects we need to make sure these major SSL certificate renewals are happening at the right time. Renewing the certificate every year gives you a fully secure compliant virtualized platform and reduces the changes to OLVM engine vulnerabilities.