An upgrade without issues is no upgrade at all! Let’s go through a few of those when upgrading to OHS 12.2.1.4.
Improper file permissions cause standalone OHS domain reconfiguration to fail.
cd $ORACLE_HOME/oracle_common/common/bin
./reconfig.sh -log=ohs12214_reconfg.log -log_priority=ALL
2023-04-12 23:09:27,018 FINE [55] com.oracle.cie.domain.ConfigHelper – copyFile failed: from=/u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/ohs1/cgi-bin/printoam to=/u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/printoam
java.io.FileNotFoundException: /u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/printoam (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:162)
at com.oracle.cie.domain.ConfigHelper.copyFile(ConfigHelper.java:317)
at com.oracle.cie.domain.WLSGeneratorHelper.copyDirectory(WLSGeneratorHelper.java:2060)
at com.oracle.cie.domain.WLSGeneratorHelper.copyDirectory(WLSGeneratorHelper.java:2049)
at com.oracle.cie.domain.WLSGeneratorHelper.copyDirectory(WLSGeneratorHelper.java:2049)
at com.oracle.cie.domain.ohs.OHSHelper.createSystemComponentRuntime(OHSHelper.java:65)
at com.oracle.cie.domain.ohs.OHSSystemComponentHandler.save(OHSSystemComponentHandler.java:123)
at com.oracle.cie.domain.camext.impl.PrimaryCAMExtension.save(PrimaryCAMExtension.java:518)
at com.oracle.cie.domain.progress.template.importer.ImporterGenerationPhase.execute(ImporterGenerationPhase.java:356)
at com.oracle.cie.domain.progress.AbstractProgressGenerator.run(AbstractProgressGenerator.java:94)
at java.lang.Thread.run(Thread.java:750)
2023-04-12 23:09:27,018 SEVERE [55] com.oracle.cie.domain.progress.AbstractProgressGenerator – Error occurred in phase {Artifacts Generation} execution.
com.oracle.cie.domain.GenerationException: Processing CAM config:
Root Cause: java.io.FileNotFoundException: /u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/printoam (Permission denied)
at com.oracle.cie.domain.ohs.OHSSystemComponentHandler.save(OHSSystemComponentHandler.java:127)
at com.oracle.cie.domain.camext.impl.PrimaryCAMExtension.save(PrimaryCAMExtension.java:518)
at com.oracle.cie.domain.progress.template.importer.ImporterGenerationPhase.execute(ImporterGenerationPhase.java:356)
at com.oracle.cie.domain.progress.AbstractProgressGenerator.run(AbstractProgressGenerator.java:94)
at java.lang.Thread.run(Thread.java:750)
2023-04-12 23:09:27,020 SEVERE [55] com.oracle.cie.domain.progress.domain.reconfig.ReconfigurationGenerator – Failed to process macro rules error during reconfiguration!
com.oracle.cie.domain.GenerationException: Processing CAM config:
Root Cause: java.io.FileNotFoundException: /u02/private/oracle/config/domains/ohs/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/printoam (Permission denied)
at com.oracle.cie.domain.ohs.OHSSystemComponentHandler.save(OHSSystemComponentHandler.java:127)
at com.oracle.cie.domain.camext.impl.PrimaryCAMExtension.save(PrimaryCAMExtension.java:518)
at com.oracle.cie.domain.progress.template.importer.ImporterGenerationPhase.execute(ImporterGenerationPhase.java:356)
at com.oracle.cie.domain.progress.AbstractProgressGenerator.run(AbstractProgressGenerator.java:94)
at java.lang.Thread.run(Thread.java:750)
2023-04-12 23:09:34,633 FINEST [16] com.oracle.cie.wizard.internal.engine.EmbeddedEngine – Waiting for new commands…
Since some of the domain configuration templates are already updated, restore the pre-upgrade OHS domain from the backup, update the permissions of the files in question, and rerun Reconfiguration Wizard.
cd $DOMAIN_HOME/config/fmwconfig/components/OHS/instances/ohs1/cgi-bin/
ls -lrt
total 16
-rw-r—– 1 oracle oinstall 144 Aug 17 2017 README.txt
-r-xr-x— 1 oracle oinstall 650 May 25 2022 authtest
-r-xr-x— 1 oracle oinstall 650 May 25 2022 printenv3
-r-xr-x— 1 oracle oinstall 296 May 25 2022 printoam
cp printoam printoam_bkp
cp printenv3 printenv3_bkp
cp authtest authtest_bkp
chmod 755 printoam authtest printenv3
A successful domain reconfiguration output is shown below
When attempting to start the Oracle HTTP Server component ohs1 using startComponent.sh, WLST failed to connect to Node Manager.
cd $DOMAIN_HOME/bin
./startComponent.sh ohs1
Error occured while starting ohs1
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Apr 13 16:37:31 xxxx-yyyy.cs.uoguelph.ca startComponent.sh[26113]: Error: Error occurred while performing nmConnect : Cannot connect to Node Manager. : sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
In this case, Node Manager used a custom identity keystore(as can be confirmed from the nodemanager.properties file). By importing the server certificate to the trust stores, the issue can be resolved.
NOTE: Backup files before making any changes.
cd $DOMAIN_HOME/security/
ls -lrt
total 40
-rw-r—– 1 oracle oinstall 64 Apr 26 2022 SerializedSystemIni.dat
-rw——- 1 oracle oinstall 0 Apr 26 2022 cwallet.sso.lck
-rw——- 1 oracle oinstall 4093 Apr 26 2022 cwallet.sso
-rw-r—– 1 oracle oinstall 1011 Apr 26 2022 CustomTrust.jks
-rw-r—– 1 oracle oinstall 2303 Apr 26 2022 CustomIdentity.jks
-rw-r—– 1 oracle oinstall 2352 Apr 12 23:52 DemoIdentity.jks
— Export Node Manager’s certificate.
keytool -exportcert -alias server-cert -keystore CustomIdentity.jks -file /u01/oracle/stage/pre-upg_bkp/trust.crt
Enter keystore password:
Certificate stored in file </u01/oracle/stage/pre-upg_bkp/trust.crt>
— Import the certificate into the demo, custom trust store, and JDK cacerts.
cd $WLS_HOME/server/lib/
ls -lrt
-rw-r—– 1 oracle oinstall 945 Apr 17 14:17 trust.crt
-rw——- 1 oracle oinstall 1610 Feb 13 2017 DemoTrust.jks
keytool -import -trustcacerts -alias server-cert -file trust.crt -keystore DemoTrust.jks
cd $DOMAIN_HOME/security
keytool -import -trustcacerts -alias server-cert -file trust.crt -keystore CustomTrust.jks
Set CONFIG_JVM_ARGS env variable to include custom keystore details before starting OHS.
cd $JAVA_HOME/jre/lib/security
keytool -import -trustcacerts -alias server-cert -file trust.crt -v -keystore cacerts
Using orapki utility, recreate the default wallet and self-signed certificate to test SSL functionality.
— Make sure orapki is in $PATH env variable
export PATH=$ORACLE_HOME/oracle_common/bin:$PATH
— Backup existing wallet
$DOMAIN_HOME/config/fmwconfig/components/OHS/instances/ohs2/keystores/default/
mv cwallet.sso cwallet.sso.expired
— Create an auto-login wallet
orapki wallet create -wallet . -auto_login_only
— Recreate self-signed cert using the same DN as used in the original wallet
orapki wallet adds -wallet. -dn ‘CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY’ -keysize 2048 – self_signed -validity 3650 -auto_login_only
— Check the wallet contents
orapki wallet display -wallet .
OHS should start successfully unless there are other issues specific to your configuration.
References: WLST Fails to Connect to Node Manger when Starting OHS – Doc ID 2215312.1
12c: How to Recreate the Default Wallet that has Expired from Oracle HTTP Server – Doc ID 2729766.1