I was called back to a previous client site where I had previously helped with some Oracle Application Server (10.1.2.2) post-install configuration. In that previous visit, I got oriented to the environment they use and the packaged application they were deploying. The packaged application uses JSP, Oracle Forms, and Oracle Reports (possibly also Discoverer). The deployment environment is all Microsoft Windows servers with two Oracle Application Server homes per application server since the vendor’s deployment requires that JSPs be deployed in a separate O_H from the Oracle Forms and Oracle Reports environment (that’s the first eyebrow-raise I did, but whatever).

This customer had an environment that was configured by the vendor for testing purposes and it works fine. However, it uses HTTP and they want to use HTTPS for all client-server traffic. They also wanted to be able to manage the environment and be better equipped to support it, so they left the vendor-installed environment as is and built a new environment on new servers so they’d get first-hand views of the install and configuration procedures. Since all the application servers are virtual machines, they could easily create additional machines.

My first job back a few weeks ago was to configure SSL for all the client-server configuration. That took a couple of hours to finalize while explaining the steps I performed along the way. I also did some database configuration to configure basic backups of the databases (these are the first and only Oracle databases in the environment). The customer then fought through finding all the places where URLs were either in database tables or hard-coded in properties and configuration files to make the appropriate change for the new HTTPS URLs. We also configured the whole environment to use WebCache to load-balance the two application tiers (2 “Forms tier” and 2 “JSP tier” servers total). After much hunting and a little tracing of SQL to find the URLs in the DB, it all worked fine.

This week, the plan was to invite the vendor’s installation consultant along with me to a joint installfest. We would start on Monday morning with a database server that had databases installed, patched, and created along with two “brand new” application servers that had no Oracle software on them at all. The tasks were to install and configure the vendor’s application and go-live on it (cutting over the existing environments from the vendor-installed environment that was in use at the time) by Wednesday afternoon. That’s 3 days. Same week, same year…you read it right.

While this seems (and is) something of a horrible idea, it did have some logic that helped make it seem a little less bad:

• They only had one module (payroll–so an important module) for one small sub-company (a small number of employees) live on the current system. The rest of the modules and other entities will go live later, following data migration testing and training.
• They saw it as an opportunity to go live, maybe hit a few bumps, but have the two best resources to fix the bumps (the vendor installation consultant and me) onsite and ready to address whatever comes up. The alternative was to go live next week when they’d be there without additional support and have to address the issues “alone.”

The short version of the story is that they went live at around 5pm on Wednesday afternoon. True to their goal, it was three days from first Oracle Univesal Installer run to application go-live. Of course, they had done testing quite a lot on other systems/environments, so they already had a list of workarounds and fixes to implement as part of the three-day plan.

While I still don’t believe that such a short implementation cycle was the best plan, it did happen and they went in eyes wide open knowing the risks and potential issues. Regardless, this is probably the fastest implementation go-live I’ve ever witnessed and can’t imagine any other ERP implementation project doing the same thing no matter how small it is!

We increased many timeouts for OPMN as we believed that OPMN was just incorrectly “seeing” the containers as down and restarting them. OPMN restarts them by shutting them down first and then starting them.

We filed cases with Oracle support to no avail–they didn’t come up with any useful suggestions in a week or more. They were trying, but didn’t come up with the solution.

The system administrator developed a theory based on what he believed was a pattern. Every time he did a deployment, he would notice a crash of all the non-Oracle default containers. That is, the home and OC4J_WebCenter containers didn’t crash.

The deployment process that he followed resulted in him connecting to the server using remote desktop. His remote desktop client was configured with the /console option which was required by some other servers he managed, more about that later.

Once he was able to demonstrate that he could make the containers crash each time he logged off, we started testing variations using the system console, the remote desktop client with and without the /console option and found a pattern. The remote desktop client without the /console option did not cause a crash, but all other combinations did. Through all of this, the home and OC4J_WebCenter containers remained up and running.

Bottom line: Read Metalink Note 245609.1 which documents the apparently, well-known fact that logging out from the Windows console causes JVM termination. The very simple fix is to start the containers with the “-Xrs” option which tells the JVM to ignore certain signals from the OS.

The really terrible thing about all this is that Oracle puts the -Xrs option on the containers deployed during the installation, but the OEM tool doesn’t add them to the container startup parameters for the custom containers. Easy to fix, even easy to find once you know what to look for.

This begs two questions:

1. Why doesn’t Oracle add -Xrs to the startup options for the containers created after the initial installation? That would have avoided all the problems and there’s apparently no negative side effect–at least not that we’ve seen.
2. How could an SR analyst not find this Metalink note and refer us to the simple solution? Granted, we didn’t find it easily in our searches either, but eventually it was one of us that found the article and solution. Now that we know the fix, a simple search for -Xrs on Metalink gets plenty of hits. As they say, hindsight is 20/20.

Hopefully, this information will help some of you that are lucky enough to work on OC4J deployments on Windows.