Contents
An upgrade without issues is no upgrade at all! Let’s go through a few of those when upgrading to OHS 12.2.1.4.
1. Domain Reconfiguration
Improper file permissions cause standalone OHS domain reconfiguration to fail.
cd $ORACLE_HOME/oracle_common/common/bin
./reconfig.sh -log=ohs12214_reconfg.log -log_priority=ALL
2023-04-12 23:09:27,018 FINE [55] com.oracle.cie.domain.ConfigHelper – copyFile failed: from=/u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/ohs1/cgi-bin/printoam to=/u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/printoam
java.io.FileNotFoundException: /u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/printoam (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:162)
at com.oracle.cie.domain.ConfigHelper.copyFile(ConfigHelper.java:317)
at com.oracle.cie.domain.WLSGeneratorHelper.copyDirectory(WLSGeneratorHelper.java:2060)
at com.oracle.cie.domain.WLSGeneratorHelper.copyDirectory(WLSGeneratorHelper.java:2049)
at com.oracle.cie.domain.WLSGeneratorHelper.copyDirectory(WLSGeneratorHelper.java:2049)
at com.oracle.cie.domain.ohs.OHSHelper.createSystemComponentRuntime(OHSHelper.java:65)
at com.oracle.cie.domain.ohs.OHSSystemComponentHandler.save(OHSSystemComponentHandler.java:123)
at com.oracle.cie.domain.camext.impl.PrimaryCAMExtension.save(PrimaryCAMExtension.java:518)
at com.oracle.cie.domain.progress.template.importer.ImporterGenerationPhase.execute(ImporterGenerationPhase.java:356)
at com.oracle.cie.domain.progress.AbstractProgressGenerator.run(AbstractProgressGenerator.java:94)
at java.lang.Thread.run(Thread.java:750)
2023-04-12 23:09:27,018 SEVERE [55] com.oracle.cie.domain.progress.AbstractProgressGenerator – Error occurred in phase {Artifacts Generation} execution.
com.oracle.cie.domain.GenerationException: Processing CAM config:
Root Cause: java.io.FileNotFoundException: /u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/printoam (Permission denied)
at com.oracle.cie.domain.ohs.OHSSystemComponentHandler.save(OHSSystemComponentHandler.java:127)
at com.oracle.cie.domain.camext.impl.PrimaryCAMExtension.save(PrimaryCAMExtension.java:518)
at com.oracle.cie.domain.progress.template.importer.ImporterGenerationPhase.execute(ImporterGenerationPhase.java:356)
at com.oracle.cie.domain.progress.AbstractProgressGenerator.run(AbstractProgressGenerator.java:94)
at java.lang.Thread.run(Thread.java:750)
2023-04-12 23:09:27,020 SEVERE [55] com.oracle.cie.domain.progress.domain.reconfig.ReconfigurationGenerator – Failed to process macro rules error during reconfiguration!
com.oracle.cie.domain.GenerationException: Processing CAM config:
Root Cause: java.io.FileNotFoundException: /u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/printoam (Permission denied)
at com.oracle.cie.domain.ohs.OHSSystemComponentHandler.save(OHSSystemComponentHandler.java:127)
at com.oracle.cie.domain.camext.impl.PrimaryCAMExtension.save(PrimaryCAMExtension.java:518)
at com.oracle.cie.domain.progress.template.importer.ImporterGenerationPhase.execute(ImporterGenerationPhase.java:356)
at com.oracle.cie.domain.progress.AbstractProgressGenerator.run(AbstractProgressGenerator.java:94)
at java.lang.Thread.run(Thread.java:750)
2023-04-12 23:09:34,633 FINEST [16] com.oracle.cie.wizard.internal.engine.EmbeddedEngine – Waiting for new commands…
Since some of the domain configuration templates are already updated, restore the pre-upgrade OHS domain from the backup, update the permissions of the files in question, and rerun Reconfiguration Wizard.
cd $DOMAIN_HOME/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/
ls -lrt
total 16
-rw-r—– 1 oracle oinstall 144 Aug 17 2017 README.txt
-r-xr-x— 1 oracle oinstall 650 May 25 2022 authtest
-r-xr-x— 1 oracle oinstall 650 May 25 2022 printenv3
-r-xr-x— 1 oracle oinstall 296 May 25 2022 printoam
cp printoam printoam_bkp
cp printenv3 printenv3_bkp
cp authtest authtest_bkp
chmod 755 printoam authtest printenv3
A successful domain reconfiguration output is shown below
2. Missing Node Manager Server Certificate
When attempting to start the Oracle HTTP Server component ohs1 using startComponent.sh, WLST failed to connect to Node Manager.
cd $DOMAIN_HOME/bin
./startComponent.sh ohs1
Error occured while starting ohs1
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Apr 13 16:37:31 xxxx-yyyy.cs.uoguelph.ca startComponent.sh[26113]: Error: Error occurred while performing nmConnect : Cannot connect to Node Manager. : sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
In this case, Node Manager used a custom identity keystore(as can be confirmed from the nodemanager.properties file). By importing the server certificate to the trust stores, the issue can be resolved.
NOTE: Backup files before making any changes.
cd $DOMAIN_HOME/security/
ls -lrt
total 40
-rw-r—– 1 oracle oinstall 64 Apr 26 2022 SerializedSystemIni.dat
-rw——- 1 oracle oinstall 0 Apr 26 2022 cwallet.sso.lck
-rw——- 1 oracle oinstall 4093 Apr 26 2022 cwallet.sso
-rw-r—– 1 oracle oinstall 1011 Apr 26 2022 CustomTrust.jks
-rw-r—– 1 oracle oinstall 2303 Apr 26 2022 CustomIdentity.jks
-rw-r—– 1 oracle oinstall 2352 Apr 12 23:52 DemoIdentity.jks
— Export Node Manager’s certificate.
keytool -exportcert -alias server-cert -keystore CustomIdentity.jks -file /u01/oracle/stage/pre-upg_bkp/trust.crt
Enter keystore password:
Certificate stored in file </u01/oracle/stage/pre-upg_bkp/trust.crt>
— Import the certificate into the demo, custom trust store, and JDK cacerts.
cd $WLS_HOME/server/lib/
ls -lrt
-rw-r—– 1 oracle oinstall 945 Apr 17 14:17 trust.crt
-rw——- 1 oracle oinstall 1610 Feb 13 2017 DemoTrust.jks
keytool -import -trustcacerts -alias server-cert -file trust.crt -keystore DemoTrust.jks
cd $DOMAIN_HOME/security
keytool -import -trustcacerts -alias server-cert -file trust.crt -keystore CustomTrust.jks
Set CONFIG_JVM_ARGS env variable to include custom keystore details before starting OHS.
cd $JAVA_HOME/jre/lib/security
keytool -import -trustcacerts -alias server-cert -file trust.crt -v -keystore cacerts
3. OHS unable to start due to an expired self-signed certificate
Using orapki utility, recreate the default wallet and self-signed certificate to test SSL functionality.
— Make sure orapki is in $PATH env variable
export PATH=$ORACLE_HOME/oracle_common/bin:$PATH
— Backup existing wallet
$DOMAIN_HOME/config/fmwconfig/components/OHS/instances/ohs2/keystores/default/
mv cwallet.sso cwallet.sso.expired
— Create an auto-login wallet
orapki wallet create -wallet . -auto_login_only
— Recreate self-signed cert using the same DN as used in the original wallet
orapki wallet adds -wallet. -dn ‘CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY’ -keysize 2048 – self_signed -validity 3650 -auto_login_only
— Check the wallet contents
orapki wallet display -wallet .
OHS should start successfully unless there are other issues specific to your configuration.
References: WLST Fails to Connect to Node Manger when Starting OHS – Doc ID 2215312.1
12c: How to Recreate the Default Wallet that has Expired from Oracle HTTP Server – Doc ID 2729766.1