App Server – dannorris.com blog

Install to go-live, 3 days

This has been an interesting week, but not really that surprising. I was called back to a previous client site where I had previously helped with some Oracle Application Server (10.1.2.2) post-install configuration. In that previous visit, I got oriented to the environment they use and the packaged application they were deploying. The packaged application … Continue reading “Install to go-live, 3 days”

This has been an interesting week, but not really that surprising.

I was called back to a previous client site where I had previously helped with some Oracle Application Server (10.1.2.2) post-install configuration. In that previous visit, I got oriented to the environment they use and the packaged application they were deploying. The packaged application uses JSP, Oracle Forms, and Oracle Reports (possibly also Discoverer). The deployment environment is all Microsoft Windows servers with two Oracle Application Server homes per application server since the vendor’s deployment requires that JSPs be deployed in a separate O_H from the Oracle Forms and Oracle Reports environment (that’s the first eyebrow-raise I did, but whatever). Continue reading “Install to go-live, 3 days”

Concatenating lines in ldapsearch results

Many of us have had reasons to migrate Oracle Application Server (specifically, Portal) environments from one server or group of servers to another. This is often the case when hardware upgrades are needed and the whole environment must be moved to another set of hosts. Recently, I was helping move an Oracle Portal (10.1.2.0.2) environment … Continue reading “Concatenating lines in ldapsearch results”

Recently, I was helping move an Oracle Portal (10.1.2.0.2) environment from one host to another. This was due to a company spin off, so the “sticky” part of this move was that the domain name and resulting realm changed (more on that in a minute).

First, if you’ve had to perform this task, you should have already identified Metalink Note 251776.1 which describes the process necessary for moving users and groups from one OracleAS Infrastructure to another. The note’s step 3 mentions that the LDIF file must be edited to replace all references to the old realm with the new realm in the target system. However, this can prove difficult if you do actually have to change the realm name because of the way that ldapsearch produces output. The LDIF standard specifies that lines can be continued on the following line if a space is the first character on the line. The corresponding ldapadd command can properly import lines that are broken into multiple lines, but the standard search and replace tools (in notepad, vi or any other standard text editor) can’t find the occurrences properly to replace them. So, some entries are able to be replaced easily like this one (assume we need to replace “dc=dannorris,dc=local” with “dc=newcorp,dc=com“): Continue reading “Concatenating lines in ldapsearch results”

Configuring Multiple Weblogic IIS Plug-Ins On Same IIS Server

As some of you may know, I’ve been working on a deployment of BEA Weblogic these past couple of weeks. We were doing some testing today and found an interesting side effect that was positively unexpected. Let me first say that the issues we encountered were with IIS configuration, not specifically with Weblogic. However, the … Continue reading “Configuring Multiple Weblogic IIS Plug-Ins On Same IIS Server”

As some of you may know, I’ve been working on a deployment of BEA Weblogic these past couple of weeks. We were doing some testing today and found an interesting side effect that was positively unexpected. Let me first say that the issues we encountered were with IIS configuration, not specifically with Weblogic. However, the issue wouldn’t have come up if we weren’t working on configuring the BEA-provided iisproxy.dll IIS plug-in.

Here’s the issue: We want to configure our production server to run two sites. The primary site is the production site and the secondary site is a staging site which we’re going to try to configure to behave exactly like production and have a configuration that matches production as well. So, we want to have two separate Weblogic Domains (that listen on different ports) and two separate IIS servers (that listen on separate ports). The desired configuration looks something like this:

Continue reading “Configuring Multiple Weblogic IIS Plug-Ins On Same IIS Server”

Configuring Weblogic For Restart After Reboot on Windows

I have been working on a customer project involving new installations of Oracle Database (a two-node failover cluster using Oracle Clusterware–good stuff) and two load-balanced BEA Oracle Weblogic servers for the middle tier. The middle tier environment runs on Windows Server 2003 Enterprise x64 Edition and is managed by an outsourced hosting facility. I had … Continue reading “Configuring Weblogic For Restart After Reboot on Windows”

I had worked with Weblogic before this project, but not on Windows and I was a little surprised by the difficulty finding relatively simple information on BEA’s support site as well as in their documentation.

<rant>I have to say that I’m pleased that Oracle closed this deal so that (I never thought I’d say this) they can make BEA’s support as good as Oracle Support and Metalink. I know it will take quite a while–probably 6-12 months–but it absolutely needs to happen. It’s little reminders like these that make me glad I work with Oracle (or the companies they ultimately buy) and not other software. I’m as guilty as anyone of not knowing how good we Oracle professionals have it when it comes to good, well-organized documentation, a solid support site with a killer knowledge repository (with an awesome search), and a warm, active community of users (too many “thanks to” to mention at this point).</rant>

Back to the reason you’re reading: this post will summarize what turns out to be the relatively easy changes required to ensure that your Windows-based Weblogic managed servers will start up at boot time.

When creating a new Weblogic Domain on Windows, you’ll sometimes have the opportunity to choose the Sun JDK or BEA JRockit JDK to use for your domain’s Java engine. Be conscious of which one you choose (for WL 10.0, it’s on the screen where you choose a Development or Production configuration). I chose BEA JRockit for my environment’s servers because on 64-bit Windows, it’s the only one supported. For 32-bit Windows, you have the option to use Sun’s JDK, but then we’d have a mix and more potential for new bugs to pop up due to the JDK differences. There’s a small change in this process when using Sun’s JDK versus BEA JRockit and I’ll note that as well.

To ensure that your Windows-based BEA Weblogic Managed Server(s) start at boot time, follow these steps:

After creating the new domain and the new Managed Server (Managed Server is a Weblogic term to identify the differences between an Admin Server and the application server where applications should be deployed), modify the Managed Server settings.
Lock & Edit the configuration and proceed to Environment > Servers > (name of your Managed Server) > Configuration tab > Server Start subtab. Then scroll down to the Arguments box and enter -Xnohup in the box. If you’re using Sun’s JDK instead of BEA JRockit, enter -Xrs in that box instead. This is documented at http://edocs.bea.com/wls/docs100/server_start/nodemgr.html#wp1101004.
Once that’s in place, you can Activate Changes.
The admin server says that no restarts are needed, but I would restart it just to make sure. I’m not sure how the admin server can change a JDK flag without restarting the application server. Maybe I’m just not knowledgeable enough to know how it works, but I think it’s just not smart enough to know that you *do* have to restart.
The last change was the one that I missed initially. Briefly, the reason I missed it was that it has to do with crash recovery and in my opinion, a server reboot shouldn’t cause an application server to crash, so I ignored this part of the documentation. One has to ask why you wouldn’t want crash recovery enabled by default anyway, but that’s probably for another rant some other day. Anyway, the final change is to modify a property for the node manager process. Edit BEA_HOME\wlserver_10.0\common\nodemanager\nodemanager.properties and set CrashRecoveryEnabled=true (it is in the file set to false by default). Save and exit the editor.
This is Windows, so go to the Services control panel and restart the BEA Products NodeManager service to put the change into effect.
Ensure that your managed server is up and running. Then, test the changes you’ve made by rebooting the node and see that your managed server restarts after the reboot is complete (and you’ve given time for node manager to start the managed server).

If you have problems, check the node manager logfile (BEA_HOME\wlserver_10.0\common\nodemanager\nodemanager.log) as it will be most useful in determining what happened. If you don’t see any hint that it even tried to restart the server after the reboot, then it’s probably because the crash recovery setting is not enabled–make sure you changed the right thing in the right file.

I didn’t test to see if this process will restart the admin server as well, but I think it probably will or at least should. With a production configuration, you have to enter the username/password for the admin server when starting it, so you may have to store that in the admin server configuration, but that should be a relatively easy fix. In my case, we didn’t want our admin server running all the time and only start it when needed, so having it start after a reboot wasn’t necessary or desired.

2:30am will never happen on Sunday morning

For those that didn’t bother to install the OS, Oracle or other DST patches (to handle the adjusted DST change schedule) last year and elected to just change the time on your servers by hand twice a year, get ready to do that again this weekend. Personally, I’d rather burn two whole regular work days … Continue reading “2:30am will never happen on Sunday morning”

For those that didn’t bother to install the OS, Oracle or other DST patches (to handle the adjusted DST change schedule) last year and elected to just change the time on your servers by hand twice a year, get ready to do that again this weekend. Personally, I’d rather burn two whole regular work days figuring out and installing the patches than get up to modify the time on a server at 2am on Sunday morning, but that’s just me I guess.

For those that forgot the crontab rule about not scheduling things to occur between 1am and 3am (because they either get skipped or run twice once a year), prepare to be confused as 2:30am doesn’t ever happen on clocks in most US states/regions (here are the exceptions).

I’ll look forward to a little more daylight during the hours that count and one hour less sleep on Saturday night.

On a side note, let’s hope that this little posting will get me started on regular posting here…at least for a while!

Cool thing happened on Twitter today…

A neat thing happened today on Twitter. While I admit that I don’t necessarily “get it” as fast as some of my “web 2.0” friends do, I haven’t seen this happen too much on Twitter since I’ve been following it in the past several months. I’m sure it probably happens all the time to cool … Continue reading “Cool thing happened on Twitter today…”

A neat thing happened today on Twitter. While I admit that I don’t necessarily “get it” as fast as some of my “web 2.0” friends do, I haven’t seen this happen too much on Twitter since I’ve been following it in the past several months. I’m sure it probably happens all the time to cool people, but I was lucky enough to cross over for a few minutes and that’s notable.

Basically, the “thing” was that someone needed help understanding how to get started with an OID installation for managing TNS connect descriptors. He wanted (and needed) to use an existing database since he was resource-constrained and wasn’t sure what the installation process looks like for such an installation.

Here’s the combined thread between @fuadar, @topperge and me (@dannorris) just a few minutes ago:

fuadar: looking for someone or some document to install oid in an existing 10.2 database need only names service resolution
dannorris: @fudar It’s much easier to just have it install its own DB. If you use existing DB, you must run metadata repos creation asst first.
fuadar: @dnanorris out of space already have a database out there for other functions. trying to setup oid to solve our tnsnames issues
dannorris: @fudar Issues? Honestly, OID usually introduces more issues than it solves when it comes to TNS. It’s a lot more complex than a text file.
fuadar: @dannorris true but i’m trying to come up with some way to manage acouple of hundred servers and a couple of thousand clients
dannorris: @fudar It’s definitely the right direction to head–just need realistic expectations about complexity and manageability–not easier!
fuadar: @dannorris agree just looking for better documentation
topperge: @fuadar fudar, all you need is RepCA and install the identity repos, http://tinyurl.com/yweyr8
dannorris: @fuadar Better free up some space first–you’ll need a gig or two I’d expect. (ps sorry for misspelling your handle)
fuadar: @topperge so what you are saying is just go thru the oid software install process and then so the repca manually
fuadar: @topperge i am using the Oracle Identity management dvd’s 10.1.4.0.1
dannorris: @fuadar Install RepCA first, run it, then install OID from IdM and tell it to use the repos you created.
dannorris: @fuadar be sure to check DB prereqs (version, pkgs, options, etc.). Follow section here http://snurl.com/1xzda
fuadar: @dannorris thanks reinstalling the software now
topperge: @fuadar There is a 10.1.4 MRCA with the DVDs, install from that first , then install from the OIM Infrastructure CD second
topperge: @fuadar then make sure you patch to 10.1.4.2 which is patch 5983637 on metalink (doing the same install right now)

Even patch numbers! Posting that same question to a forum would likely have taken several hours to get responses–and precise responses as well. Now, I don’t want everyone to believe that @topperge (Matt Topper) and I sit around all day looking for questions we can answer on Twitter. However, I am on Twitter most of the time (even though I don’t tweet that often) and occasionally will throw a response or post in when I think of it. Matt is usually there and seems to behave similarly most of the time.

The bottom line: today, Twitter helped someone solve a real technical problem much faster than they were likely to solve it via other means (web 2 dot oh or otherwise). I don’t know that it happens every day, but we can only save one life at a time :).

You can follow me (@dannorris) on twitter, but as I don’t say much, you won’t likely be impressed. After all, I’m no Jake Kuramoto.

Using mod_rewrite to rewrite OC4J-served URLs – a complete review

We recently ran into an issue in a customer configuration where rewriting URLs using pass-through didn’t function as expected with OC4J-deployed applications. As it turned out, there’s a bug in the OC4J container and a relatively easy workaround for some. The situation was this (names changed to protect the innocent): An existing Java application deployment … Continue reading “Using mod_rewrite to rewrite OC4J-served URLs – a complete review”

The situation was this (names changed to protect the innocent):

An existing Java application deployment existed using JRun on Solaris. In that deployment, an application called “abc” would be called as “http://abcapp.corp.com/servlet/login”
Applications were to be migrated to Oracle Application Server 10.1.3.1.0.
Deployments on OAS were required to prefix the application with something and they used the application name. So, on the new site, the application would need to be called as “http://abcapp.corp.com/abc/servlet/login”. This was undesirable since bookmarks would have to be updated. While it could easily be handled with redirections, the desired behavior was to have all URLs match what they were on the old deployment.

On the surface, this seems like a relatively simple problem to solve using a RewriteRule with the [PT] option and few RewriteConds in the Apache configuration. That is, until you find the bug in OC4J that makes it impossible. First, let’s review the configuration parameters. Continue reading “Using mod_rewrite to rewrite OC4J-served URLs – a complete review”

Another “special” circumstance when running OAS on Windows

We encountered an “interesting” challenge recently where some, not all, OC4J containers in an Oracle Application Server 10.1.3.1.0 installation would “crash” (they would stop running). There was no apparent pattern to the “crazy” crashing containers. The system administrator was actively doing application (re)deployments at the rate of 3-4 per week. The containers seemed to be … Continue reading “Another “special” circumstance when running OAS on Windows”

We encountered an “interesting” challenge recently where some, not all, OC4J containers in an Oracle Application Server 10.1.3.1.0 installation would “crash” (they would stop running). There was no apparent pattern to the “crazy” crashing containers. The system administrator was actively doing application (re)deployments at the rate of 3-4 per week. The containers seemed to be “crashing” randomly, sometimes throughout the day, sometimes just after a deployment.

We increased many timeouts for OPMN as we believed that OPMN was just incorrectly “seeing” the containers as down and restarting them. OPMN restarts them by shutting them down first and then starting them.

We filed cases with Oracle support to no avail–they didn’t come up with any useful suggestions in a week or more. They were trying, but didn’t come up with the solution.

The system administrator developed a theory based on what he believed was a pattern. Every time he did a deployment, he would notice a crash of all the non-Oracle default containers. That is, the home and OC4J_WebCenter containers didn’t crash.

The deployment process that he followed resulted in him connecting to the server using remote desktop. His remote desktop client was configured with the /console option which was required by some other servers he managed, more about that later.

Once he was able to demonstrate that he could make the containers crash each time he logged off, we started testing variations using the system console, the remote desktop client with and without the /console option and found a pattern. The remote desktop client without the /console option did not cause a crash, but all other combinations did. Through all of this, the home and OC4J_WebCenter containers remained up and running.

Bottom line: Read Metalink Note 245609.1 which documents the apparently, well-known fact that logging out from the Windows console causes JVM termination. The very simple fix is to start the containers with the “-Xrs” option which tells the JVM to ignore certain signals from the OS.

The really terrible thing about all this is that Oracle puts the -Xrs option on the containers deployed during the installation, but the OEM tool doesn’t add them to the container startup parameters for the custom containers. Easy to fix, even easy to find once you know what to look for.

This begs two questions:

Why doesn’t Oracle add -Xrs to the startup options for the containers created after the initial installation? That would have avoided all the problems and there’s apparently no negative side effect–at least not that we’ve seen.
How could an SR analyst not find this Metalink note and refer us to the simple solution? Granted, we didn’t find it easily in our searches either, but eventually it was one of us that found the article and solution. Now that we know the fix, a simple search for -Xrs on Metalink gets plenty of hits. As they say, hindsight is 20/20.

Hopefully, this information will help some of you that are lucky enough to work on OC4J deployments on Windows.

Another Oracle Certification Exam

Tonight I took another Oracle certification beta exam, Oracle Application Server 10g: Administration II (1Z1-312). Since it was a beta, the fee was only $50 and I knew some of the topics to be covered, so I figured I’d wing it and see how I did. I doubt I passed as I wasn’t well-prepared–especially for the questions related to Application Server Guard and some of the questions on Cold Failover Clusters. We’ll see in a few months if I managed to squeak by it or not (they don’t announce scores for the beta exams for about 10 weeks after the beta period ends). The good part about beta exams is the price, but the bad part is that they have you answer all the questions in the test pool. For this exam, there were over 215 questions in 180 minutes (3 hours). I should know better than to schedule such a span through dinner time, but that was all I could fit in to my schedule this time!

Another reason it was challenging for me was due to a thought that occurred to me as I got about half way through the exam. That is, why are Oracle ACE Directors (for Middleware, Database, or otherwise) not required to have completed some certification. I’ll be the first to agree that having a certification doesn’t necessarily mean you know what you’re talking about. I also know from friends that have already been given the ACE Director honor, the process can be a long one and, at least for them, there were several technical interviews that were required as well. I guess if I were in Oracle’s Certification Program Office, I’d sure like the ACE Directors to take and pass my exams as a sign that the exams were worth taking and that they actually stood for something meaningful. After all, if the ACE Directors are required to take them, it would add at least a little legitimacy to the certification program, wouldn’t it?

I’m not looking to start a flame war or drag the ACE Director program over the coals. I am wondering what others may think of certifications. Note that I’ve already posted my thoughts on certifications, so you’ll see I’m not proposing that certifications be the sole measure of anything. However, they are an interesting tool and provide at least one relatively objective metric as a starting point for evaluating a candidate (for a job or for an elite honorary title like ACE Director).

Let’s see if anyone’s reading…comment away! 🙂

Oracle buys Bharosa

I’m completely impressed just by the descriptions of the new tools that Oracle is getting with its Bharosa acquisition. If this stuff does even half of what it is advertising it can do, I’ll be very impressed. If you’re wondering what this is all about, please check out the post on the Talking Identity blog … Continue reading “Oracle buys Bharosa”

As I learn more, I’ll be continuing to post more information here. For now, it’s on to a vacation day tomorrow!