Session ID: S316824
Title: Top 10 Lessons Learned in Deploying the Oracle Exadata
Tuesday, September 21, 12:30PM
Location: Moscone South, Rm 307

Check the OOW 2010 content catalog for updated room assignments and times.

Even better than a technical session is the interview and Q&A session I’m doing on Oracle Technology Network Live which is 30 minutes of pure technical talk about Exadata. The session is properly titled “Exadata for Geeks” and I’ll be joining Justin Kestelyn, editor of Oracle Technology Network at the OTN Lounge which is located in the Mason Street tent this year (*not* the previous location in Moscone West).

Significantly, this year I elected not to organize what would have been the 3rd annual pre-OOW scuba dive in Monterey Bay. Time and my work requirements are the primary reasons for this, but it also is a result of the fact that not a single person asked me about it, so apparently it was just for me after all . Instead, I’m hoping that I might get to visit Alcatraz this year. I’ve been to SF so very many times in the past 12 years, but have yet to take that tour, so I think it’s time (I’ve heard it is a really interesting tour).

See you in SF!

I recently had a need to “unplumb” (from Solaris fame) or make interfaces on Linux “disappear” from the ifconfig list. It could be that I don’t know how to completely deconfigure an interface, but I didn’t find any methods to unassign an IP address from a Linux Ethernet interface after it was assigned. You can take interfaces down (ifconfig eth3 down) and reconfigure them to assign different addresses, but not remove the address completely.

After many searches and finding nothing that matched my need, I turned to my fellow Oakies (thanks, Mark!) who turned up this post from 2 years ago that hinted at a solution. It is driver-specific which is not ideal, but that makes sense given what I’m trying to do.

Here’s the generic version of the solution:

echo "<interface_name>" > /sys/bus/pci/drivers/<driver_name>/unbind

Determining the driver_name is pretty simple: check the /etc/modprobe.conf file (on OEL/RHEL 5.x). In that file, you’ll find things like this:

...
alias eth0 igb
alias eth1 igb
alias eth2 igb
alias eth3 igb
...

These lines indicate that the Ethernet driver used on this system by eth[0-3] is the igb driver. Now that you know the driver name, the tricky part is figuring out what the driver wants you to use as the interface name. I’ll give a few examples (and I haven’t figured out the scientific way to determine what the driver expects short of reading source code).

For the bnx2 driver, you can use the relatively simple ethernet interface name, like this:

echo "eth2" > /sys/bus/pci/drivers/bnx2/unbind

For my test system, the igb driver doesn’t use the “simple” Ethernet interface name like the bnx2 driver does. Instead, when trying that, it gives an error that the interface doesn’t exist. Time to dig in a little deeper.

On this system, the igb directory looks like this:

# ls -l /sys/bus/pci/drivers/igb/
total 0
lrwxrwxrwx 1 root root    0 Jul 16 13:12 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root    0 Jul 16 13:12 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
lrwxrwxrwx 1 root root    0 Jul 16 13:12 0000:07:00.0 -> ../../../../devices/pci0000:00/0000:00:02.0/0000:07:00.0
lrwxrwxrwx 1 root root    0 Jul 16 13:12 0000:07:00.1 -> ../../../../devices/pci0000:00/0000:00:02.0/0000:07:00.1
--w------- 1 root root 4096 Jul 16 13:12 bind
lrwxrwxrwx 1 root root    0 Jul 16 13:12 module -> ../../../../module/igb
--w------- 1 root root 4096 Jul 16 13:12 new_id
--w------- 1 root root 4096 Jul 16 13:12 remove_id
--w------- 1 root root 4096 Jul 16 13:12 unbind
#

So, knowing that I have 4 interfaces on the system, I made the correlation to the 4 addresses that appear as symlinks in the driver’s directory and expect that they indicate the interface name. Checking a couple of those (each symlink references a directory), I see this:

# ls -Ll /sys/bus/pci/drivers/igb/0000:01:00.0
total 0
-rw-r--r-- 1 root root   4096 Jul 16 13:15 broken_parity_status
drwxr-xr-x 5 root root      0 Jul 16 13:12 bus
-r--r--r-- 1 root root   4096 Jul 16 12:57 class
-rw-r--r-- 1 root root   4096 Jul 16 12:57 config
-r--r--r-- 1 root root   4096 Jul 16 12:57 device
drwxr-xr-x 2 root root      0 Jul 16 13:12 driver
-rw------- 1 root root   4096 Jul 16 13:15 enable
-r--r--r-- 1 root root   4096 Jul 16 12:57 irq
-r--r--r-- 1 root root   4096 Jul 16 13:00 local_cpus
-r--r--r-- 1 root root   4096 Jul 16 13:15 modalias
drwxr-xr-x 3 root root      0 Jul 16 12:57 net:eth0
drwxr-xr-x 2 root root      0 Jul 16 12:55 power
-r--r--r-- 1 root root   4096 Jul 16 12:57 resource
-rw------- 1 root root 131072 Jul 16 13:15 resource0
-rw------- 1 root root 131072 Jul 16 13:15 resource1
-rw------- 1 root root     32 Jul 16 13:15 resource2
-rw------- 1 root root  16384 Jul 16 13:15 resource3
-r-------- 1 root root 131072 Jul 16 13:15 rom
drwxr-xr-x 5 root root      0 Jul 16 13:12 subsystem
-r--r--r-- 1 root root   4096 Jul 16 13:15 subsystem_device
-r--r--r-- 1 root root   4096 Jul 16 13:15 subsystem_vendor
--w------- 1 root root   4096 Jul 16 13:15 uevent
-r--r--r-- 1 root root   4096 Jul 16 12:57 vendor
# ls -Ll /sys/bus/pci/drivers/igb/0000:07:00.0
total 0
-rw-r--r-- 1 root root   4096 Jul 16 13:13 broken_parity_status
drwxr-xr-x 5 root root      0 Jul 16 13:12 bus
-r--r--r-- 1 root root   4096 Jul 16 12:57 class
-rw-r--r-- 1 root root   4096 Jul 16 12:57 config
-r--r--r-- 1 root root   4096 Jul 16 12:57 device
drwxr-xr-x 2 root root      0 Jul 16 13:12 driver
-rw------- 1 root root   4096 Jul 16 13:13 enable
-r--r--r-- 1 root root   4096 Jul 16 12:57 irq
-r--r--r-- 1 root root   4096 Jul 16 13:00 local_cpus
-r--r--r-- 1 root root   4096 Jul 16 13:13 modalias
drwxr-xr-x 3 root root      0 Jul 16 12:57 net:eth2
drwxr-xr-x 2 root root      0 Jul 16 13:13 power
-r--r--r-- 1 root root   4096 Jul 16 12:57 resource
-rw------- 1 root root 131072 Jul 16 13:13 resource0
-rw------- 1 root root 131072 Jul 16 13:13 resource1
-rw------- 1 root root     32 Jul 16 13:13 resource2
-rw------- 1 root root  16384 Jul 16 13:13 resource3
-r-------- 1 root root 131072 Jul 16 13:13 rom
drwxr-xr-x 5 root root      0 Jul 16 13:12 subsystem
-r--r--r-- 1 root root   4096 Jul 16 13:13 subsystem_device
-r--r--r-- 1 root root   4096 Jul 16 13:13 subsystem_vendor
--w------- 1 root root   4096 Jul 16 13:15 uevent
-r--r--r-- 1 root root   4096 Jul 16 12:57 vendor

You can see the directory with name “net:<interface_name>” as a subdirectory in each listing above. This tells us which interface from /sys/bus/pci/drivers/igb/0000* corresponds with which of the Linux Ethernet interface names. From this, we can see that eth2 is really 0000:07:00.0. So, in order to unbind this interface such that it no longer appears in the “ifconfig -a” output, we run this command:

echo "0000:07:00.0" > /sys/bus/pci/drivers/igb/unbind

and then it no longer appears in the “ifconfig -a” output. If you wanted to make this permanent, you should comment out the corresponding line from /etc/modprobe.conf so that it won’t be configured at boot time. Using the echo command above takes effect immediately, but won’t persist through a reboot (after reboot, the interface will return) unless the /etc/modprobe.conf changes are made.

Now, hopefully the next blog post after this one won’t require 14 more months of preparation!

My new Oracle-provided laptop arrived around mid-week and I realized that, at least for now, I’ll have to revert back to using the Windows-based laptop and (hopefully temporarily) put my MacBook Pro on the shelf. Actually, my wife is very excited since she’ll get the MBP to use now and we’ll do the usual “trickle down” to the kids so that the oldest computer in the “fleet” will get ditched.

I tried “upgrading” to a new DSL line (from cable modem) last week too, but that appears to have failed as the DSL modem drops my connection on a regular basis for a few seconds. That’s just long enough to break the VPN connection and make it appear that I’m bouncing off and on instant messaging every hour or so. Annoying.

I was excited to finally get on to some real technical work late last week and got to login to my first Exadata storage server. The chores were to re-image 4 servers and apply the latest patches to them. The real fun is starting now!

Of course, the most important part of any job (in my opinion) is people. I have had several virtual meetings with my peers and manager so far and, as expected, they’re all superb. I’m looking forward to working more with the team and others in our companion teams as well as the days roll ahead.

For those certified scuba divers that will (or can) be in the Monterey Bay area on 21-June, I invite you to come diving with me. I’ve arranged some reserved spots on the Beachhopper II dive charter that I’ve dove with before. Brian and Mary Jo (the captain, crew, and bottle washers) are top notch and we had a great time last fall at the first annual pre-OpenWorld scuba event (look for more details on the 2nd annual event later this summer). The boat isn’t huge, but 10 divers is enough for a lot of fun.

The pre-Kaleidoscope dive day is Sunday, 21-June (Father’s Day). The boat will depart the K dock at Monterey Bay harbor at 8am, so load-up is 7:30am. We’ll have a nice morning, drain 2 tanks at some of the best sites you’ll see in northern California (specific sites will be determined that morning by the captain and diver requests), and then motor back to the harbor probably shortly after noon or 1pm. Mary Jo said that she’d also entertain the option of an afternoon 2-tank trip as well, if there is interest (I know I’m interested). Oh, I almost forgot to mention that snacks are provided and they are amazing–made by Mary Jo herself!

The boat costs break down like this:

• $70 for the boat trip (weights are not included)
• plus $20 for two tanks of air ($90 total)
• or $30 for 2 tanks of Nitrox ($100 total)

The charter doesn’t offer gear rental, so we’ll have to pick that up separately. I previously rented from Glenn’s Aquarius 2 which is located pretty close to the harbor and opens at 7am for morning pickup. Their pricing for rental are:

• Weights only: $8
• Wetsuit, hood, gloves: $21
• Full gear (BCD, reg, exposure suit, etc.): $65

We’re less than 1 month away (I just found out I was going to be able to attend last week), so let me know ASAP if you’re interested in diving with us. Once you contact me, I’ll send you the signup instructions. I’m releasing the remaining open seats on 29-May, but there may still be open spots after that, so contact me (comment below, or email) if you’re interested.

As a special treat, Stanley will be joining us for his first scuba dive as well!

So, this new opportunity came along a little while ago and I thought it sounded just perfect for me. Many of you that know me will recall that I’ve had trouble finding the right company that fits with all aspects of my personality, goals, and philosophy which has led me to “try” a few of them in the past several years. I don’t regret the choices I’ve made and I’ve learned an awful lot from each of my employers. Most importantly, I’ve created new relationships at each of my past companies that I still maintain today.

In looking at all the past experiences, I’ve concluded that I am ready for a change in direction. Things I enjoy:
Oracle products
People (customers and Oracle employees) that work with Oracle software
RAC
High Availability
Large, complex environments
Servers, OS, and storage infrastructures
Working with lots of new people, especially creating, managing, and growing new relationships
Presenting my knowledge to others, both one-on-one and to groups (and conference events that surround such gatherings)
A little bit of travel (which helps support my love of…)
Scuba diving

So, when an opportunity came along to get paid to do something that combines almost all of the things I enjoy (except scuba), I couldn’t pass it up. Starting on May 18, 2009, I will be the newest member of the new group at Oracle that’s known as the “X-Team”. This team is responsible for working with customers and prospective customers of the Oracle Exadata and HP Oracle Database Machine products to help them adopt these new technologies. The group is a part of the Maximum Availability Architecture group at Oracle that authors many of the best practices whitepapers and presentations you have likely seen online. For those at Oracle that know what this means, the group is a part of the Server Technology development organization under Juan Loaiza.

For those of you that have been my past consulting customers, first of all, Thank You. I’m no longer consulting and while I won’t be able to provide an “Oracle-sanctioned” recommendation to other consultants that may be able to help, I do have a large network of friends and one of them can likely help you. Please never hesitate to keep in touch!

On a logistical note, I’m not moving and will hopefully continue to be involved with local events in Chicagoland. However, I will be traveling part of the time to visit customers and other Oracle facilities, so keep an eye on my twitter feed, Britekite location, and Tripit plans and let me know if there’s a chance to have a meeting IRL.

Finally, this decision to join Oracle means that I’ll be sacrificing several things. First and probably most near and dear to me is the RAC SIG. In September 2008, I took over as the RAC SIG President. The RAC SIG is as strong as ever and there are a good group of volunteers involved in leading the group as it continues to grow and evolve. I’ll always be a member of the RAC SIG and will continue to watch it closely and volunteer when and where I can. The RAC SIG is associated with the IOUG, the Independent Oracle User Group, and Oracle employees shouldn’t be too involved in “independent” groups. So, this year, the RAC SIG will once again elect a new president. I will remain president until Oracle Open World in October 2009 in order to provide continuity to the group’s leadership and ensure a smooth transition. You can nominate yourself for a RAC SIG office soon via our website nomination form (nominations will be open soon and stay open until July 31, 2009).

I’m also going to relinquish my appointment as an Oracle ACE Director. While I think I’ll still be considered an Oracle Employee ACE, I’ll remember fondly the fame that Oracle Technology Network affords the Oracle ACE program and the individuals that are given the honor. Thanks to Justin, Vikki, Lillian, Todd, and the others at Oracle for allowing me to be a part of that program. I’ll certainly miss the perks!

That’s about it for now, I’m off to the new job and will once again begin learning. Luckily, I’m apparently the only person named Dan Norris at Oracle (last someone checked for me), so you can contact me at dan.norris@oracle.com in a couple weeks.

First, where I’ll be presenting:

• Monday, 10:45-11:45am, #301, “Avoiding Common RAC Problems”
• Tuesday, 9:45am-12pm, #332, “Installing RAC From The Ground Up”
• Wednesday, 9:45-10:45am, #121, “Troubleshooting Oracle Clusterware”

I’m also currently the President of the Oracle RAC Special Interest Group (RAC SIG). The RAC SIG is hosting several great sessions (I’m moderating a couple of these panels) at Collaborate 09 as well:

• Sunday, 6-7:30pm, IOUG/SIG Welcome Reception (each SIG will have representatives there–this is open to all IOUG attendees)
• Monday, 8-9am, RAC SIG Orientation
• Tuesday, 12:15-1:15pm, RAC SIG Birds of a Feather
• Tuesday, 4:30-5:30pm, RAC SIG Expert Panel
• Wednesday, 4:30-5:30pm, RAC SIG Customer Panel (not in online scheduler at the moment, check again later)
• Thursday, 8:30am-12pm, RAC Attack (University Session – Additional fee required)

The RAC SIG has also assembled this list of RAC-related sessions at Collaborate 09 to help you plan your conference agenda.

Be sure to set up your personal agenda using the agenda builder and add these sessions to your agenda. I think that if you have these in your agenda and details (like date or room assignments) change, you’ll be notified via email (not sure, but I think that’s how it works).

Also, you can follow @IOUG on Twitter (follow me too if you’d like) and that will help you find where the action is during the event next week. It’s going to be a great event and I look forward to seeing you there!

Each participant will have their own private RAC cluster to use. You’ll be able to install a new cluster, test session failover, perform backup and recovery and just about anything else you’d like to try (time permitting). The session will have lab outlines with very specific instructions that cater to beginners. Advanced users are welcome to test anything they like. If you try something that doesn’t work, we have mechanisms in place to help “reset” your cluster in 15 minutes and let you continue working and testing.

Here’s the official conference abstract for the session:

“Whether you’re new to or familiar with Real Application Clusters (RAC), you do not want to miss the IOUG RAC Attack! hands-on lab. The hands-on lab will cover: cluster installation prerequisites on Linux, installing Oracle clusterware, installing Oracle RDBMS, creating RAC database, failure testing and backup/recovery testing.

Also, you’ll have a chance to interact with some RAC experts from IOUG, Oracle and the RAC SIG, as they’ll be available to help you as you navigate the hands-on exercises. These volunteers are sharing their knowledge to help you be successful with your learning experience. This opportunity doesn’t come along often, so don’t miss a chance to pick the brains of our experts!”

There are still seats available for this event, but there is a limit, so don’t delay. It is an additional cost above the Collaborate09 conference registration, but I think you’ll find it to be packed with knowledge and experts to help you get the hands-on experience you need to grow and succeed with Oracle RAC. The session will be staffed with Oracle, IOUG and RAC SIG experts ready to help you and answer questions about RAC and clustering. Don’t miss out!

If you’ve already registered, you can add on the University Session by contacting IOUG. If you haven’t registered yet, there is still time left–sign up now!

Here are some choice excerpts from the transcript that I find telling about the future of Exadata:

Larry Ellison:

“So, that’s looking back. Now looking forward, I think the most exciting product we’ve had in many, many years is our Exadata Database Server.”

“Exadata is 100% innovation on top of our very large and very strong database business. And the early results have been remarkable. Charles Phillips will go into a lot of detail but I’ll just throw a couple of numbers out there.

One of our customers, and Charles will describe this customer, one of our customers saw a 28x performance improvement over an existing Oracle database. Another customer saw a monthly aggregation drop from 4.5 hours just to 3 minutes.

When compared to Teradata, a competitive database machine that’s been in the market for a very, very long time, another customer saw that we were 6x faster than their existing Teradata application, when using Exadata versus Teradata.

Another customer saw a batch process fall from 8 hours to 30 minutes. Charles will go into more detail on all this, he will repeat those numbers, because I think they’re worth mentioning twice.”

Charles Phillips:

“So now just a few comments by area. On databases, Larry mentioned, we’re very excited about how the HP Oracle database machine is performing. The increases have just been stunning and so we are getting great feedback from our customers and the pipeline is the largest build I’ve ever seen in terms of a new product.

And as he mentioned, the numbers are just stunning. The major European retailer who reduced the batch processing time from 8 hours to 30 minutes did not believe the process had completed. We had to convince him that’s actually how it’s done.

And so, as Larry mentioned, this is the reminder that this is an internally developed technology in the midst of all the discussion of acquisitions. People forget that we’re actually spending $3.0 billion a year on research and development and this is why we do it.”

From these snippets, you can see that the top executives at Oracle are excited about Exadata. If you’re a techie (if you’re not, how’d you get to this blog?), you’ll probably already know about Kevin Closson’s popular blog on all things related to Oracle and storage. Kevin is giving a webcast next week on Exadata where we expect he’ll discuss some of the technical workings of the product–deeper than the overview information many of us have heard before. If you’re interested, I strongly encourage you to sign up for the event and attend. There is no better authority on Exadata than Kevin and this is a great opportunity!

Jeremy is the main creator of the IOUG RAC Attack! event that was held for the first time back in August 2008. He (with help from others) will also be putting it on as a half-day session at Collaborate 09. It’s a University Seminar on Thursday morning. All hands-on, all RAC, all the time. I’m looking forward to the event (I’m volunteering as a staffer). You should sign up now before it’s full! I can almost guarantee you’ll learn something.

Besides his work on this hands on lab/class for RAC, Jeremy has many other community contributions. His blog is full of excellent technical bits that always seem to come from a significant amount of research. He contributes occasionally to the Oracle-L mailing list. He also contributed some code to OCFS (v1) several years ago, so you can guess he understands a thing or two about programming and Linux, too.

His ACE Profile isn’t posted yet, but look for it to arrive soon. In the meantime, read some of the good stuff he wrote on his blog and look for him (and me too) at Collaborate in early May!

Congratulations, Jeremy!

In an Oracle RAC cluster, services can be started, stopped, and relocated from one instance to another. However, if you have multiple services for your database, then it becomes difficult to start them at a cold start. Due to dependencies in Oracle Clusterware, services are stopped when all instances of a database are stopped. However, when you start instances, services are left OFFLINE and you’re forced to start them manually. If you have just one additional service (first of all, you should probably use more than one unless you have a very small user community), you can just start the service and that will start the instances necessary to bring the service up via dependencies in Oracle Clusterware. If you have multiple services, though, you still have to start the rest of them manually.

There is a solution to this problem that uses the normal Oracle Clusterware operations to automatically start any services for the instance that was started on the local node automatically. This is done with a small perl script that is run as an Oracle Clusterware cluster callout. Cluster callouts are scripts or programs that Oracle Clusterware will run whenever a cluster “event” (like starting or stopping an instance) occurs. It calls these programs with many command-line arguments that can be parsed to determine what the event is about and why it was fired. The program can use these arguments to filter out the events that aren’t interesting to it and act on the events that it wishes to act upon. You create a new “callout” by placing an executable (script or binary) in the $ORA_CRS_HOME/racg/usrco directory. Each executable file found in that directory will be executed for each event that happens locally.

In our case, we want to look for instance startup events and then query Oracle Clusterware to determine which services should be started and start them. A script to do exactly that is provided by the Oracle RAC and Clusterware team on their sample code page titled “Start Services on Instance Up“. There you will find a small zip archive that has a couple of perl scripts and a short readme file explaining how to install and use them. In my case, I had to modify the script to handle some hostname parsing. In particular, I had to modify line 54 in startMatchedSRVonUp.pl to read like this “$myHost = (split(/\./,$_))[0];“. That is because my “/bin/hostname” command returns the fully-qualified domain name (node1.dannorris.net) instead of just the short hostname (node1) as the script expects by default. So, a small change there may help you too. It logs by default in the /tmp directory and doesn’t delete its log, so you should be able to easily debug it using that information.

Coincidentally, I had occasion to install this script at a new customer implementation of Oracle RAC recently and it also came up in my session at Hotsos Symposium 09 earlier this week as well. If you’ve found a better way to handle this issue, please fire off a comment below for the good of all mankind.