My new Oracle-provided laptop arrived around mid-week and I realized that, at least for now, I’ll have to revert back to using the Windows-based laptop and (hopefully temporarily) put my MacBook Pro on the shelf. Actually, my wife is very excited since she’ll get the MBP to use now and we’ll do the usual “trickle down” to the kids so that the oldest computer in the “fleet” will get ditched.

I tried “upgrading” to a new DSL line (from cable modem) last week too, but that appears to have failed as the DSL modem drops my connection on a regular basis for a few seconds. That’s just long enough to break the VPN connection and make it appear that I’m bouncing off and on instant messaging every hour or so. Annoying.

I was excited to finally get on to some real technical work late last week and got to login to my first Exadata storage server. The chores were to re-image 4 servers and apply the latest patches to them. The real fun is starting now!

Of course, the most important part of any job (in my opinion) is people. I have had several virtual meetings with my peers and manager so far and, as expected, they’re all superb. I’m looking forward to working more with the team and others in our companion teams as well as the days roll ahead.

For those certified scuba divers that will (or can) be in the Monterey Bay area on 21-June, I invite you to come diving with me. I’ve arranged some reserved spots on the Beachhopper II dive charter that I’ve dove with before. Brian and Mary Jo (the captain, crew, and bottle washers) are top notch and we had a great time last fall at the first annual pre-OpenWorld scuba event (look for more details on the 2nd annual event later this summer). The boat isn’t huge, but 10 divers is enough for a lot of fun.

The pre-Kaleidoscope dive day is Sunday, 21-June (Father’s Day). The boat will depart the K dock at Monterey Bay harbor at 8am, so load-up is 7:30am. We’ll have a nice morning, drain 2 tanks at some of the best sites you’ll see in northern California (specific sites will be determined that morning by the captain and diver requests), and then motor back to the harbor probably shortly after noon or 1pm. Mary Jo said that she’d also entertain the option of an afternoon 2-tank trip as well, if there is interest (I know I’m interested). Oh, I almost forgot to mention that snacks are provided and they are amazing–made by Mary Jo herself!

The boat costs break down like this:

• $70 for the boat trip (weights are not included)
• plus $20 for two tanks of air ($90 total)
• or $30 for 2 tanks of Nitrox ($100 total)

The charter doesn’t offer gear rental, so we’ll have to pick that up separately. I previously rented from Glenn’s Aquarius 2 which is located pretty close to the harbor and opens at 7am for morning pickup. Their pricing for rental are:

• Weights only: $8
• Wetsuit, hood, gloves: $21
• Full gear (BCD, reg, exposure suit, etc.): $65

We’re less than 1 month away (I just found out I was going to be able to attend last week), so let me know ASAP if you’re interested in diving with us. Once you contact me, I’ll send you the signup instructions. I’m releasing the remaining open seats on 29-May, but there may still be open spots after that, so contact me (comment below, or email) if you’re interested.

As a special treat, Stanley will be joining us for his first scuba dive as well!

So, this new opportunity came along a little while ago and I thought it sounded just perfect for me. Many of you that know me will recall that I’ve had trouble finding the right company that fits with all aspects of my personality, goals, and philosophy which has led me to “try” a few of them in the past several years. I don’t regret the choices I’ve made and I’ve learned an awful lot from each of my employers. Most importantly, I’ve created new relationships at each of my past companies that I still maintain today.

In looking at all the past experiences, I’ve concluded that I am ready for a change in direction. Things I enjoy:
Oracle products
People (customers and Oracle employees) that work with Oracle software
RAC
High Availability
Large, complex environments
Servers, OS, and storage infrastructures
Working with lots of new people, especially creating, managing, and growing new relationships
Presenting my knowledge to others, both one-on-one and to groups (and conference events that surround such gatherings)
A little bit of travel (which helps support my love of…)
Scuba diving

So, when an opportunity came along to get paid to do something that combines almost all of the things I enjoy (except scuba), I couldn’t pass it up. Starting on May 18, 2009, I will be the newest member of the new group at Oracle that’s known as the “X-Team”. This team is responsible for working with customers and prospective customers of the Oracle Exadata and HP Oracle Database Machine products to help them adopt these new technologies. The group is a part of the Maximum Availability Architecture group at Oracle that authors many of the best practices whitepapers and presentations you have likely seen online. For those at Oracle that know what this means, the group is a part of the Server Technology development organization under Juan Loaiza.

For those of you that have been my past consulting customers, first of all, Thank You. I’m no longer consulting and while I won’t be able to provide an “Oracle-sanctioned” recommendation to other consultants that may be able to help, I do have a large network of friends and one of them can likely help you. Please never hesitate to keep in touch!

On a logistical note, I’m not moving and will hopefully continue to be involved with local events in Chicagoland. However, I will be traveling part of the time to visit customers and other Oracle facilities, so keep an eye on my twitter feed, Britekite location, and Tripit plans and let me know if there’s a chance to have a meeting IRL.

Finally, this decision to join Oracle means that I’ll be sacrificing several things. First and probably most near and dear to me is the RAC SIG. In September 2008, I took over as the RAC SIG President. The RAC SIG is as strong as ever and there are a good group of volunteers involved in leading the group as it continues to grow and evolve. I’ll always be a member of the RAC SIG and will continue to watch it closely and volunteer when and where I can. The RAC SIG is associated with the IOUG, the Independent Oracle User Group, and Oracle employees shouldn’t be too involved in “independent” groups. So, this year, the RAC SIG will once again elect a new president. I will remain president until Oracle Open World in October 2009 in order to provide continuity to the group’s leadership and ensure a smooth transition. You can nominate yourself for a RAC SIG office soon via our website nomination form (nominations will be open soon and stay open until July 31, 2009).

I’m also going to relinquish my appointment as an Oracle ACE Director. While I think I’ll still be considered an Oracle Employee ACE, I’ll remember fondly the fame that Oracle Technology Network affords the Oracle ACE program and the individuals that are given the honor. Thanks to Justin, Vikki, Lillian, Todd, and the others at Oracle for allowing me to be a part of that program. I’ll certainly miss the perks!

That’s about it for now, I’m off to the new job and will once again begin learning. Luckily, I’m apparently the only person named Dan Norris at Oracle (last someone checked for me), so you can contact me at dan.norris@oracle.com in a couple weeks.

First, where I’ll be presenting:

• Monday, 10:45-11:45am, #301, “Avoiding Common RAC Problems”
• Tuesday, 9:45am-12pm, #332, “Installing RAC From The Ground Up”
• Wednesday, 9:45-10:45am, #121, “Troubleshooting Oracle Clusterware”

I’m also currently the President of the Oracle RAC Special Interest Group (RAC SIG). The RAC SIG is hosting several great sessions (I’m moderating a couple of these panels) at Collaborate 09 as well:

• Sunday, 6-7:30pm, IOUG/SIG Welcome Reception (each SIG will have representatives there–this is open to all IOUG attendees)
• Monday, 8-9am, RAC SIG Orientation
• Tuesday, 12:15-1:15pm, RAC SIG Birds of a Feather
• Tuesday, 4:30-5:30pm, RAC SIG Expert Panel
• Wednesday, 4:30-5:30pm, RAC SIG Customer Panel (not in online scheduler at the moment, check again later)
• Thursday, 8:30am-12pm, RAC Attack (University Session – Additional fee required)

The RAC SIG has also assembled this list of RAC-related sessions at Collaborate 09 to help you plan your conference agenda.

Be sure to set up your personal agenda using the agenda builder and add these sessions to your agenda. I think that if you have these in your agenda and details (like date or room assignments) change, you’ll be notified via email (not sure, but I think that’s how it works).

Also, you can follow @IOUG on Twitter (follow me too if you’d like) and that will help you find where the action is during the event next week. It’s going to be a great event and I look forward to seeing you there!

Each participant will have their own private RAC cluster to use. You’ll be able to install a new cluster, test session failover, perform backup and recovery and just about anything else you’d like to try (time permitting). The session will have lab outlines with very specific instructions that cater to beginners. Advanced users are welcome to test anything they like. If you try something that doesn’t work, we have mechanisms in place to help “reset” your cluster in 15 minutes and let you continue working and testing.

Here’s the official conference abstract for the session:

“Whether you’re new to or familiar with Real Application Clusters (RAC), you do not want to miss the IOUG RAC Attack! hands-on lab. The hands-on lab will cover: cluster installation prerequisites on Linux, installing Oracle clusterware, installing Oracle RDBMS, creating RAC database, failure testing and backup/recovery testing.

Also, you’ll have a chance to interact with some RAC experts from IOUG, Oracle and the RAC SIG, as they’ll be available to help you as you navigate the hands-on exercises. These volunteers are sharing their knowledge to help you be successful with your learning experience. This opportunity doesn’t come along often, so don’t miss a chance to pick the brains of our experts!”

There are still seats available for this event, but there is a limit, so don’t delay. It is an additional cost above the Collaborate09 conference registration, but I think you’ll find it to be packed with knowledge and experts to help you get the hands-on experience you need to grow and succeed with Oracle RAC. The session will be staffed with Oracle, IOUG and RAC SIG experts ready to help you and answer questions about RAC and clustering. Don’t miss out!

If you’ve already registered, you can add on the University Session by contacting IOUG. If you haven’t registered yet, there is still time left–sign up now!

Here are some choice excerpts from the transcript that I find telling about the future of Exadata:

Larry Ellison:

“So, that’s looking back. Now looking forward, I think the most exciting product we’ve had in many, many years is our Exadata Database Server.”

“Exadata is 100% innovation on top of our very large and very strong database business. And the early results have been remarkable. Charles Phillips will go into a lot of detail but I’ll just throw a couple of numbers out there.

One of our customers, and Charles will describe this customer, one of our customers saw a 28x performance improvement over an existing Oracle database. Another customer saw a monthly aggregation drop from 4.5 hours just to 3 minutes.

When compared to Teradata, a competitive database machine that’s been in the market for a very, very long time, another customer saw that we were 6x faster than their existing Teradata application, when using Exadata versus Teradata.

Another customer saw a batch process fall from 8 hours to 30 minutes. Charles will go into more detail on all this, he will repeat those numbers, because I think they’re worth mentioning twice.”

Charles Phillips:

“So now just a few comments by area. On databases, Larry mentioned, we’re very excited about how the HP Oracle database machine is performing. The increases have just been stunning and so we are getting great feedback from our customers and the pipeline is the largest build I’ve ever seen in terms of a new product.

And as he mentioned, the numbers are just stunning. The major European retailer who reduced the batch processing time from 8 hours to 30 minutes did not believe the process had completed. We had to convince him that’s actually how it’s done.

And so, as Larry mentioned, this is the reminder that this is an internally developed technology in the midst of all the discussion of acquisitions. People forget that we’re actually spending $3.0 billion a year on research and development and this is why we do it.”

From these snippets, you can see that the top executives at Oracle are excited about Exadata. If you’re a techie (if you’re not, how’d you get to this blog?), you’ll probably already know about Kevin Closson’s popular blog on all things related to Oracle and storage. Kevin is giving a webcast next week on Exadata where we expect he’ll discuss some of the technical workings of the product–deeper than the overview information many of us have heard before. If you’re interested, I strongly encourage you to sign up for the event and attend. There is no better authority on Exadata than Kevin and this is a great opportunity!

Jeremy is the main creator of the IOUG RAC Attack! event that was held for the first time back in August 2008. He (with help from others) will also be putting it on as a half-day session at Collaborate 09. It’s a University Seminar on Thursday morning. All hands-on, all RAC, all the time. I’m looking forward to the event (I’m volunteering as a staffer). You should sign up now before it’s full! I can almost guarantee you’ll learn something.

Besides his work on this hands on lab/class for RAC, Jeremy has many other community contributions. His blog is full of excellent technical bits that always seem to come from a significant amount of research. He contributes occasionally to the Oracle-L mailing list. He also contributed some code to OCFS (v1) several years ago, so you can guess he understands a thing or two about programming and Linux, too.

His ACE Profile isn’t posted yet, but look for it to arrive soon. In the meantime, read some of the good stuff he wrote on his blog and look for him (and me too) at Collaborate in early May!

Congratulations, Jeremy!

In an Oracle RAC cluster, services can be started, stopped, and relocated from one instance to another. However, if you have multiple services for your database, then it becomes difficult to start them at a cold start. Due to dependencies in Oracle Clusterware, services are stopped when all instances of a database are stopped. However, when you start instances, services are left OFFLINE and you’re forced to start them manually. If you have just one additional service (first of all, you should probably use more than one unless you have a very small user community), you can just start the service and that will start the instances necessary to bring the service up via dependencies in Oracle Clusterware. If you have multiple services, though, you still have to start the rest of them manually.

There is a solution to this problem that uses the normal Oracle Clusterware operations to automatically start any services for the instance that was started on the local node automatically. This is done with a small perl script that is run as an Oracle Clusterware cluster callout. Cluster callouts are scripts or programs that Oracle Clusterware will run whenever a cluster “event” (like starting or stopping an instance) occurs. It calls these programs with many command-line arguments that can be parsed to determine what the event is about and why it was fired. The program can use these arguments to filter out the events that aren’t interesting to it and act on the events that it wishes to act upon. You create a new “callout” by placing an executable (script or binary) in the $ORA_CRS_HOME/racg/usrco directory. Each executable file found in that directory will be executed for each event that happens locally.

In our case, we want to look for instance startup events and then query Oracle Clusterware to determine which services should be started and start them. A script to do exactly that is provided by the Oracle RAC and Clusterware team on their sample code page titled “Start Services on Instance Up“. There you will find a small zip archive that has a couple of perl scripts and a short readme file explaining how to install and use them. In my case, I had to modify the script to handle some hostname parsing. In particular, I had to modify line 54 in startMatchedSRVonUp.pl to read like this “$myHost = (split(/\./,$_))[0];“. That is because my “/bin/hostname” command returns the fully-qualified domain name (node1.dannorris.net) instead of just the short hostname (node1) as the script expects by default. So, a small change there may help you too. It logs by default in the /tmp directory and doesn’t delete its log, so you should be able to easily debug it using that information.

Coincidentally, I had occasion to install this script at a new customer implementation of Oracle RAC recently and it also came up in my session at Hotsos Symposium 09 earlier this week as well. If you’ve found a better way to handle this issue, please fire off a comment below for the good of all mankind.

I was called back to a previous client site where I had previously helped with some Oracle Application Server (10.1.2.2) post-install configuration. In that previous visit, I got oriented to the environment they use and the packaged application they were deploying. The packaged application uses JSP, Oracle Forms, and Oracle Reports (possibly also Discoverer). The deployment environment is all Microsoft Windows servers with two Oracle Application Server homes per application server since the vendor’s deployment requires that JSPs be deployed in a separate O_H from the Oracle Forms and Oracle Reports environment (that’s the first eyebrow-raise I did, but whatever).

This customer had an environment that was configured by the vendor for testing purposes and it works fine. However, it uses HTTP and they want to use HTTPS for all client-server traffic. They also wanted to be able to manage the environment and be better equipped to support it, so they left the vendor-installed environment as is and built a new environment on new servers so they’d get first-hand views of the install and configuration procedures. Since all the application servers are virtual machines, they could easily create additional machines.

My first job back a few weeks ago was to configure SSL for all the client-server configuration. That took a couple of hours to finalize while explaining the steps I performed along the way. I also did some database configuration to configure basic backups of the databases (these are the first and only Oracle databases in the environment). The customer then fought through finding all the places where URLs were either in database tables or hard-coded in properties and configuration files to make the appropriate change for the new HTTPS URLs. We also configured the whole environment to use WebCache to load-balance the two application tiers (2 “Forms tier” and 2 “JSP tier” servers total). After much hunting and a little tracing of SQL to find the URLs in the DB, it all worked fine.

This week, the plan was to invite the vendor’s installation consultant along with me to a joint installfest. We would start on Monday morning with a database server that had databases installed, patched, and created along with two “brand new” application servers that had no Oracle software on them at all. The tasks were to install and configure the vendor’s application and go-live on it (cutting over the existing environments from the vendor-installed environment that was in use at the time) by Wednesday afternoon. That’s 3 days. Same week, same year…you read it right.

While this seems (and is) something of a horrible idea, it did have some logic that helped make it seem a little less bad:

• They only had one module (payroll–so an important module) for one small sub-company (a small number of employees) live on the current system. The rest of the modules and other entities will go live later, following data migration testing and training.
• They saw it as an opportunity to go live, maybe hit a few bumps, but have the two best resources to fix the bumps (the vendor installation consultant and me) onsite and ready to address whatever comes up. The alternative was to go live next week when they’d be there without additional support and have to address the issues “alone.”

The short version of the story is that they went live at around 5pm on Wednesday afternoon. True to their goal, it was three days from first Oracle Univesal Installer run to application go-live. Of course, they had done testing quite a lot on other systems/environments, so they already had a list of workarounds and fixes to implement as part of the three-day plan.

While I still don’t believe that such a short implementation cycle was the best plan, it did happen and they went in eyes wide open knowing the risks and potential issues. Regardless, this is probably the fastest implementation go-live I’ve ever witnessed and can’t imagine any other ERP implementation project doing the same thing no matter how small it is!

That’s right, in June 2009, I will visit Iceland and willfully plunge into the +2 C water that is the clearest body of water in the world. The reasons it is so clear have something to do with the fact that the water is the runoff from melting glaciers, filtered by volcanic rocks, and is very, very cold. It supports no wildlife (another reason it’s so clear/clean). Rumor has it that visibility is over 300 feet–that is something I really do have to see to believe.

The trip is being arranged by my friend Mogens Nørgaard who may very well be completely crazy. If you ever get a chance to meet and engage in conversation with him (a.k.a. “Moans Nogood”), do it. You won’t regret it, guaranteed.

The trip is highlighted on DIVE.is, Iceland’s (probably only) dive shop website. Oh, I forgot to mention that the lake bottom is where two tectonic plates (the North American and Eurasian plates, to be precise) meet up (!), so you’re essentially diving on or in one of the continental divides.

Of course, I’m very excited about this trip and hope that Ice, land can continue to function as their economic issues seem to be a little worse than everyone else’s. In the small world department, I have made contact with an Iceland native that I worked with back at Tandem (acquired by Compaq -> HP) in the late 90s. Hopefully, I can meet up with Leifur while I’m in the country. There are only about 300,000 people in the whole country, so he shouldn’t be *that* hard to find. On the other hand, it is possible that Leifur is like “John” is in the US. We’ll see.