Each participant will have their own private RAC cluster to use. You’ll be able to install a new cluster, test session failover, perform backup and recovery and just about anything else you’d like to try (time permitting). The session will have lab outlines with very specific instructions that cater to beginners. Advanced users are welcome to test anything they like. If you try something that doesn’t work, we have mechanisms in place to help “reset” your cluster in 15 minutes and let you continue working and testing.

Here’s the official conference abstract for the session:

“Whether you’re new to or familiar with Real Application Clusters (RAC), you do not want to miss the IOUG RAC Attack! hands-on lab. The hands-on lab will cover: cluster installation prerequisites on Linux, installing Oracle clusterware, installing Oracle RDBMS, creating RAC database, failure testing and backup/recovery testing.

Also, you’ll have a chance to interact with some RAC experts from IOUG, Oracle and the RAC SIG, as they’ll be available to help you as you navigate the hands-on exercises. These volunteers are sharing their knowledge to help you be successful with your learning experience. This opportunity doesn’t come along often, so don’t miss a chance to pick the brains of our experts!”

There are still seats available for this event, but there is a limit, so don’t delay. It is an additional cost above the Collaborate09 conference registration, but I think you’ll find it to be packed with knowledge and experts to help you get the hands-on experience you need to grow and succeed with Oracle RAC. The session will be staffed with Oracle, IOUG and RAC SIG experts ready to help you and answer questions about RAC and clustering. Don’t miss out!

If you’ve already registered, you can add on the University Session by contacting IOUG. If you haven’t registered yet, there is still time left–sign up now!

In an Oracle RAC cluster, services can be started, stopped, and relocated from one instance to another. However, if you have multiple services for your database, then it becomes difficult to start them at a cold start. Due to dependencies in Oracle Clusterware, services are stopped when all instances of a database are stopped. However, when you start instances, services are left OFFLINE and you’re forced to start them manually. If you have just one additional service (first of all, you should probably use more than one unless you have a very small user community), you can just start the service and that will start the instances necessary to bring the service up via dependencies in Oracle Clusterware. If you have multiple services, though, you still have to start the rest of them manually.

There is a solution to this problem that uses the normal Oracle Clusterware operations to automatically start any services for the instance that was started on the local node automatically. This is done with a small perl script that is run as an Oracle Clusterware cluster callout. Cluster callouts are scripts or programs that Oracle Clusterware will run whenever a cluster “event” (like starting or stopping an instance) occurs. It calls these programs with many command-line arguments that can be parsed to determine what the event is about and why it was fired. The program can use these arguments to filter out the events that aren’t interesting to it and act on the events that it wishes to act upon. You create a new “callout” by placing an executable (script or binary) in the $ORA_CRS_HOME/racg/usrco directory. Each executable file found in that directory will be executed for each event that happens locally.

In our case, we want to look for instance startup events and then query Oracle Clusterware to determine which services should be started and start them. A script to do exactly that is provided by the Oracle RAC and Clusterware team on their sample code page titled “Start Services on Instance Up“. There you will find a small zip archive that has a couple of perl scripts and a short readme file explaining how to install and use them. In my case, I had to modify the script to handle some hostname parsing. In particular, I had to modify line 54 in startMatchedSRVonUp.pl to read like this “$myHost = (split(/\./,$_))[0];“. That is because my “/bin/hostname” command returns the fully-qualified domain name (node1.dannorris.net) instead of just the short hostname (node1) as the script expects by default. So, a small change there may help you too. It logs by default in the /tmp directory and doesn’t delete its log, so you should be able to easily debug it using that information.

Coincidentally, I had occasion to install this script at a new customer implementation of Oracle RAC recently and it also came up in my session at Hotsos Symposium 09 earlier this week as well. If you’ve found a better way to handle this issue, please fire off a comment below for the good of all mankind.

• Everything is about people, who you know, and knowing what those people do. With people and connections, you can learn a lot more than doing it on your own.
• Identity Management is sort of important, but not important enough to get a session room in Moscone–you have to go to Marriott’s basement to catch an IdM session.
• It’s pretty cool to be mentioned in the opening keynote address, but it’d be cooler if you knew you were going to be mentioned ahead of time .
• Lots of people are taking pictures of slides in sessions even if they know that slides will be posted later. Not sure why.
• I see about 75% of people taking notes on paper with legacy devices like pens. Why not type them in to a text file instead?
• It seems that news about “X” is leaking out and if the rumors I’ve heard are true, it’ll be a very big deal. All eyes will be on Larry’s keynote today–hopefully he’ll provide enough technical information to make everyone understand what’s up.
• Oracle Beehive looks really cool, but personally, I don’t think anyone is taking it seriously since Oracle Collab Suite just died on the vine after a similar launch a few years ago.
• The OTN Lounge is a great thing and keeps getting better every year. Justin and the OTN gang do a great job making a place for people to do Good Things for the community and magic does happen occasionally.
• Unconference sessions are absolutely excellent ways to get information and engage in good discussions instead of death by powerpoint.
• Staying hydrated is, as mentioned by Judy Sim, a great idea.
• Time zones in SF suck for many people–especially those from Oz.
• There is a growing community around Oracle and it’s full of great individuals.

Tonight is the “appreciation event” and while the acts aren’t as interesting to me as past years, it still sounds like a fun night and a chance to see more people and have a little fun, too. See you there!

Following the session, I met with Ignacio Ruiz of databases-la.com for an interview that he’s going to post sometime later. He’s got a great amount of enthusiasm for his venture and I hope he’s able to create a large following for his publications in his part of the world. I’ll post here when he gets the interview (including video) up on his site.

We also had a great turnout at the RAC SIG Birds of a Feather session where we introduced the new officers and facilitated some great discussions. The new officers are excited and motivated to make some great things happen in the next couple of years in addition to maintaining the great offerings that the SIG has been providing since its beginning.

After the day’s sessions, I headed over to the Blogger Meetup that Eddie coordinated. It was a great gathering and well-attended. Thanks Justin for making it possible! I got to meet lots of new people and see all my good friends that I don’t get to see IRL very often.

I contributed one article titled “RAC For Beginners: The Basics” and also helped identify the other contributors to help give the issue a significant focus on RAC. It was a great coincidence that the issue came out just before the RAC Attack! event we held in early August. Each attendee at that event received a copy of the issue as part of their event registration.

You’ll need your IOUG login in order to read the articles, but you can at least browse the article titles on the SELECT Journal website without logging in. If you aren’t an IOUG member, you should consider joining–SELECT Journal is just one of many member benefits.

For potential authors reading this entry, SELECT Journal contributing authors do receive some nice gifts like an exclusive embroidered shirt and other goodies in addition to the 15 minutes of fame (actually 3 months since it’s a quarterly publication!). Hope you enjoy the article and if you have suggestions for improving it, please let me know directly via email.

Also, don’t forget to check out the Oracle RAC SIG website and sign up for membership there to access our document library of more great articles like the ones featured in SELECT.

As one of the RAC SIG board members, I’ve been involved with the planning and development of this event over the last 9 months and I think it’s going to be one of the best opportunities to get focused, high-quality education on RAC available today. Plus, you’ll get the chance to network with a group of people that are focused on RAC and it’s uses.

Of course, I’ll be there, so if you do make it to the event, please be sure to say hello. I’ll likely spend much of my time in the hands-on lab (when I’m not presenting my technical session) helping those that need it to build their cluster, test backup and recovery or exercise some new features.

Hope to see you there!

First, some background on how these items work is necessary. The TNS listener (tnslsnr process on *nix) process listens on a specific network address for connection requests to one of the services from one of the database instances that it services. When requested, it either spawns a server process (dedicated server environment) and connects the user to that process or forwards the connection request to a dispatcher (shared server environment) for service to the database service requested. Alternatively, if the listener knows of more than one instance providing the requested service, it may direct the client to an alternate listener (usually on a different node) that will service the request.

In any Oracle database configuration, listeners define the instances as local or remote (in single-instance environments, normally everything is local). You can see this behavior when examining the “lsnrctl services <listener_name>” output (lsnrctl syntax reference here). A listener’s services are those services that have been registered with it by instances. A listener will accept registration from any instance (this may be a weak point of security, but that’s another topic) and listeners have no outbound communication with any other entity in the Oracle environment (or beyond).

The remote_listener parameter specifies a list of listening endpoints that the local instance should contact to register its services. This list is usually defined in a TNS entry in the tnsnames.ora file and then the TNS alias set as the value of the remote_listener parameter. Here’s a sample of what that entry might look like:

LISTENERS_CLUSTERNAME=
  (ADDRESS_LIST=
    (ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip)(PORT=1521))
    (ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip)(PORT=1521))
    (ADDRESS=(PROTOCOL=TCP)(HOST=node3-vip)(PORT=1521))
  )

The local_listener parameter is sometimes confusing. It defines where to connect to the local instance, but its most important function is related to remote listeners. The contents of the local_listener parameter are passed along to the remote listeners during remote registration so that when those remote listeners wish to refer a connection request to the local instance, they refer the client (requestor) to the proper listening endpoint so it can get connected. The local_listener should contain the ADDRESS section of the TNS entry and the HOST portion should reference the VIP address, like this: (ADDRESS=(PROTOCOL=TCP)(HOST=node3-vip)(PORT=1521))

To properly (manually) configure listeners in a RAC environment, I follow the steps like the ones below. Note that in most cases, the Oracle Net Configuration Assistant (netca) will do this for you as part of the database creation process.

1. Create individual listener.ora files for each listener. Make sure that the HOST= lines in the listener.ora definition reference the VIP addresses (and only the VIP address). I prefer to specify IP addresses instead of hostnames or DNS names here to avoid possible lookup issues and/or confusion.
2. Create a TNS entry (on each node) that looks like the one below to specify a single TNS entry that references all the listeners in the cluster. Note that the HOST= parts reference the VIP addresses of each node (I used names instead of IP addresses here to avoid reader confusion–I’d put in IP addresses in the HOST= attributes when using this for a real configuration.).

   LISTENERS_CLUSTERNAME =
     (ADDRESS_LIST =
       (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
       (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
     )

3. Set the remote_listener parameter in the instances (a global parameter, not an instance-specific parameter) to be the name of the TNS entry you created in the previous step. This is done with “alter system set remote_listener = 'LISTENERS_CLUSTERNAME'; “
4. Set the local_listener parameter to be the ADDRESS string for the local instance. This parameter must be an instance-specific parameter with each instance having a similar, but unique value since each instance runs on a different HOST. If the local instance (called inst1 in the example here) runs on a node with the node VIP of 10.3.121.54, then set the local_listener parameter accordingly for each instance (it is instance-specific, so use the sid= syntax): alter system set local_listener = '(ADDRESS=(PROTOCOL=TCP)(HOST=10.3.121.54)(PORT=1521))' sid='inst1';
5. On each instance, you can run “alter system register;” to force immediate registration with the listeners. If you don’t do this, the listener registration will usually be updated within a minute or two anyway (automatically), but this command can help shorten debugging cycles when necessary.

One final tip is to use the lsnrctl program to get information about the current state of the listener. I find it helpful sometimes to get verbose information about the services and here’s how I do that. In this example, the listener name is LISTENER_NODE1 (listeners usually get named as LISTENER_<hostname>).

LSNRCTL> set curr listener_node1
Current Listener is listener_node1
LSNRCTL> set display verbose
Service display mode is VERBOSE
LSNRCTL> servic
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip)(PORT=1521)(IP=FIRST)))
Services Summary...
Service "myservice.world" has 2 instance(s).
  Instance "rac1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:blocked
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip.dannorris.net)(PORT=1521))
  Instance "rac2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip.dannorris.net)(PORT=1521))
Service "rac.world" has 2 instance(s).
  Instance "rac1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:blocked
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip.dannorris.net)(PORT=1521))
  Instance "rac2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip.dannorris.net)(PORT=1521))
Service "racXDB.world" has 2 instance(s).
  Instance "rac1", status READY, has 0 handler(s) for this service...
  Instance "rac2", status READY, has 0 handler(s) for this service...
Service "rac_XPT.world" has 2 instance(s).
  Instance "rac1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:blocked
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip.dannorris.net)(PORT=1521))
  Instance "rac2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip.dannorris.net)(PORT=1521))
The command completed successfully

If you have questions or if you’ve found alternate ways to configure listeners for RAC that works differently, please comment below so we can all learn some new things.

• Sunday, 4pm, Moscone West 2002-L2: RAC SIG Birds of a Feather Discussion Session, S292952
• Monday, 12:30pm, Moscone South 310: RAC SIG Customer Panel, S291621
• Tuesday, 10:45am, Moscone South 310: Panel Discussion: Deploying Siebel, SAP, Oracle E-Business Suite, PeopleSoft, and Hyperion on a Grid, S291659
• Tuesday, 12:15pm, Moscone South 310: RAC SIG Expert Panel, S291665

Besides great content at these events, you’ll find many of the RAC SIG board members attending, hosting, moderating, and participating in these events. If you’re interested in becoming more involved with the group, please seek out one of us to discuss how you can help. Also, we’re always looking for potential presenters for our live webcast series, so we may be seeking you out to solicit your participation in that as well!

Either way, attending these sessions or not, I encourage you to sign up for an account on the RAC SIG website. This will enable you to access our recorded past webcasts (on demand playback at any time) and receive email reminders about upcoming events.

See you in SF! Be sure to confirm the room assignments and session times as they sometimes change in last-minute planning by OOW staff.