First session of the day was Tim Gorman‘s “Getting the Most Out of AWR” session. Tim did a great review of the standard interfaces for creating, managing, and purging AWR data as well as the standard reporting available for AWR information. He even reviewed a sample AWR report and AWR SQL report to highlight the information that he typically uses to diagnose issues. After the review of standard components, he offered up several of his custom scripts/reports against AWR data. Those scripts are available on his website too. He did importantly highlight that if you use AWR, even just his custom scripts against some standard views, you must license the Oracle Enterprise Manager Diagnostics Pack which is an add-on license for the RDBMS. Tim is a great speaker and someone I’ll look for at future events.

The second session of the day for me was my friend Cary Millsap‘s new session titled “For Developers: Making Friends with the Oracle Database”. I had already read his whitepaper by the same title, so the information wasn’t new to me, but I wanted to see what he emphasized and how he presented it. As expected, Cary did a masterful job of presenting this relatively complex and sometimes “scary” information in a way that made attendees comfortable, educated, and motivated by the extreme results that such a small effort can yield. I think there must have been many believers in the audience and some of them are (hopefully) performing their first extended SQL traces in the next few days. Cary’s presentation style is always very comfortable and his ability to drive home a key point while still interjecting plenty of dry wit humor makes his sessions a must-see for me. He’s also a Mac convert…another credit to his ability to evolve!

After Cary’s session, I had a very difficult decision in the next time slot. I met Jeff Needham at RMOUG on Wednesday night and instantly knew that I wanted to hear more of what he had to say. Jeff was presenting opposite a good topic from my friend Gaja Krishna Vaidyanatha on SAN performance evaluation. In the end, I decided to attend Gaja’s presentation as I haven’t seen him present before. His session was a two-part session that spanned two one-hour slots before and after lunch. While the material wasn’t unfamiliar to me, I was comforted by the fact that Gaja’s techniques are similar to my own and it was nice getting to hear his speaking style. The biggest take-away from his session was that you need to be absolutely, positively sure that you’re using Direct I/O if your database lives on a filesystem. It will (almost positively) make a large impact on the overall system performance, not just the database.

I had exchanged emails and seen several messages from Daniel Fink on the Oracle-L list that I monitor and occasionally participate in. So, I didn’t miss the opportunity to attend his session about a disciplined approach to problem diagnosis. The session wasn’t quite what I expected, but I did enjoy it. I realize that many of the principles and processes he outlined are not familiar to many technicians. I think that after you hear a good process outlined by a smart person that lays it out step-by-step, it just makes sense. Selfishly, as a consultant, I sort of hope that people don’t follow Daniel’s advice because I think that many consulting opportunities wouldn’t be available if everyone just followed his approach.

Finally, the excellent end to the conference was a dubious pair of presenters, my friends Dan Morgan and Mogens Nørgaard. The session that these two assembled wasn’t so much a session as it was an extended, “open” format Q & A on topics ranging from BAARF to beer (of course) to AWR to data centers at the South Pole. As usual, Dan and Mogens artfully and authoritatively crafted perfect responses to each question in their own special ways.

I’ve gotten to know Mogens especially well over the past few times we’ve been in the same places and this week at RMOUG we shared a hotel room as well. That led to some late-night conversations about many important topics over some fine 18-year-old scotch–I would highly recommend this method for getting to know someone (especially Mogens!) well.

Following the conference, I enjoyed happy hour with a great crowd including Mike Swing, Carol Dacko, John King, Peggy King, Robyn Sands, Debra Lilley, Mogens, Sue Harper, Daniel Fink, Dan Morgan (together, we are the three Dans) and some other people that came and went too fast for me to remember. Once that concluded, I enjoyed dinner and conversation with Kevin Closson, Jeff Needham, and Christo Kutrovsky before officially calling it quits for the conference. Tomorrow it will be off to the airport and heading home!

Only a few weeks until the Hotsos Symposium where I’ll present a new session. Then, look ahead, I hope to attend the ODTUG (Monterey), MOOW (Amsterdam), and OOW (San Francisco) events in June, September, and October, respectively.

My first session of the day was Graham Wood’s session on adaptive thresholds for monitoring in 11g. I didn’t know very much about these new methods for setting alerts, but they are certain to be useful. Some of the methods for adaptive thresholds were available in 10g, but many enhancements were made in 11g. Most importantly, the cyclic nature of workload has periods that can be automatically determined in 11g whereas in 10g, they had to be specified manually. Graham talked briefly about using DB Time and Average Active Sessions as important metrics for tuning, but for alerting, adaptive thresholds make the most sense. Setting a hard limit means that you will likely miss many issues. If your system is normally 10% utilized overnight, but spends all night at 60% utilization, you would like to know about it. However, during the daytime, if the system normally operates at 75% utilization, a hard alert limit at 80% would have miss the 6x utilization increase from the overnight hours. Adaptive thresholds wouldn’t miss the aberration and would alert you of the 60% utilization in the overnight period. That would allow you time to attempt to resolve the problem before daytime hours start.

At 10:30, I listened to Kevin Closson talk about Exadata. As you might expect if you read Kevin’s blog, the session was quite technical and he detailed much of the architecture of the Exadata Storage Cell and how it interoperates with ASM and the RDBMS. Over the course of the 2 days I’ve been at RMOUG, I’ve been lucky enough to enjoy several conversations with Kevin, so I had sort of a preview of his session already, but I still managed to learn a few things. Specifically, one of the most interesting things I learned was about the I/O Resource Manager (IORM). IORM is the storage counterpart to the Database Resource Manager and allows you to meter or limit I/O in an Exadata cell based on an I/O Resource plan that is managed in the database. The Exadata cell is told about the IORM plan at the time the request is made since the RDBMS piggybacks the IORM plan metadata with the query request. IORM plans can have declarations for transactions, databases, or usage categories.

After lunch, I attended Tom Kyte‘s presentation about reorganizing objects. Tom proceeded to directly address some myths in his usual way with lots of examples. My takeaways were that indexes with monotonically increasing keys (sequences or current date) can benefit from periodic rebuilds. Tables rarely benefit from rebuilding, so business needs are often required to determine if the downtime required to rebuild the table outweigh the benefits of the rebuild. In many cases, a relatively simple test can show how much benefit may be obtained, if any.

In the vendor presentation session, I listened to Tom Kyte present Exadata. While much of the information is the same as I’ve heard in other presentations, it’s always interesting to see someone new present the same information. They often think of different examples and offer their own experiences. As expected, Tom incorporated some of his own examples.

The last session slot of the day was easy for me to choose–Riyaj Shamsudeen presented a session about new features in Oracle Database 11g that are specifically related to performance. He reviewed a large number of topics in a relatively short period. I have many notes, but the main topics were: onilne index rebuild enhancements, invisible indexes, virtual columns, LOB performance improvements, CBO extended statistics (dbms_stats.create_extended_statistics), fine-grained dependency checking, SQL result cache, PL/SQL function cache, PL/SQL INLINE pragma, compound triggers, and adaptive cursor sharing. There was a lot to absorb and Riyaj offered many examples of these features in action.

The end of the day was the reception in the exhibit hall which was nice for visiting and meeting new people. After the reception, the Oracle ACEs gathered for dinner at a local restaurant. Not wanting to miss opportunities to network with the great minds at RMOUG, Robyn Sands, Kevin Closson, Mogens Nørgaard, and I adjourned to the Hyatt lobby for a nitecap and some great conversation to finish the evening.

The day’s events left me exhausted and energized to wake up again for more great information. I’m not sure if Thursday can top Wednesday’s events, but I’m trying! I sure hope I can make it back to RMOUG again next year and even if I left last night, I would have gotten my money’s worth and then some!

After all, I know of no other conference where you can have meaningful, interesting conversations with all of these fine individuals in a single evening: Mogens Nørgaard, Debra Lilley, Graham Wood, Kevin Closson, Daniel Liu, Gaja Krishna Vaidyanatha, Jeff Needham, Christo Kutrovsky, Mike Ault, John King, Joze Senegacnik, Tim Gorman, Duncan Mills, Lynn Munsinger, and Peggy King. And those are just the people I got to talk to (there were many other well-respected technicians and sharers of knowledge around that I didn’t have time to speak with. I haven’t even gotten to see Cary Millsap, Robert Freeman, Craig Shallahamer, Riyaj Shamsudeen, Tanel Poder, Jeremiah Wilton, Tom Kyte, Iggy Fernandez, or Daniel Fink yet.

This is my first time to RMOUG and you may be thinking that this is some sort of fluke to have all these great researchers and presenters at one event, especially a relatively small event where there are ample opportunities to network with them directly. I’ve reviewed the RMOUG agenda for the last 3 years and this agenda is representative of the quality that they’ve managed to schedule for the event every year. Plus, unlike other conferences, the RMOUG attendee tuition is priced to help them break even, not make a large bankroll. My biggest regret is that I couldn’t manage to get here before this year!

At Hotsos Symposium 2009 (March 8-12, 2009 in Dallas), I’ll be presenting “Avoiding Common RAC Problems”. The abstract is available online.

In the IOUG DBA/Clustering track at Collaborate 2009 (May 3-7, 2009 in Orlando), I’ll be presenting 3 sessions. They are listed below:

1. “Avoiding Common RAC Problems” – RAC doesn’t work for everyone in every situation, that’s a fact. However, in many cases, RAC can work and work well if configured and used properly. Some issues are caused by simple misconfiguration, others are more complex like application-related logic issues that only surface in RAC environments. There are some issues that appear with a relatively high frequency and many of them can be fixed without knowledge of internals. Specifically, this session will review common issues in the areas of networking, storage, people, applications, and testing, examining the issues and their solutions.
2. “Troubleshooting Oracle Clusterware” – Installing RAC is something that typically happens very few times in the life of a database or project. However, dissecting the installation process can be very instructive when learning how RAC works and how the multiple components work together to support a complex, multi-instance database.This session will review a RAC installation from start to finish. We’ll examine the OS prerequisites, network setup, kernel parameters, Clusterware prerequisites, Clusterware installation, ASM considerations, ASM installation and configuration, and finally RDBMS installation and configuration. We’ll also discuss options for patching these installations and provide recommendations on ways to keep this environment stable and manageable as it grows.
3. “Installing RAC From The Ground Up” – Oracle Clusterware is present in all Oracle RAC configurations, but it is often overlooked as a managed piece of software. Since it isn’t a database, many DBAs aren’t familiar with its function or techniques for troubleshooting its behaviors.To better understand and be ready for troubleshooting this relatively new software, we start by examining its components and their purposes. An understanding of how each piece fits together will better enable DBAs to quickly identify the most likely source of problems in the stack. Familiarity with the proper operating system and Oracle tools enables DBAs to reveal the right information at the right times to debug the Clusterware components and processes.Finally, adding a troubleshooting process to the sometimes overwhelming stack of software that makes up a RAC cluster can be the key to combining knowledge of Clusterware architecture and processes with the proper tools to quickly identify the right resolution to most issues. Clusterware has ability to reboot servers without warning and sometimes the reasons for its actions are not easily identified. With so much power to affect overall availability, the Clusterware is a critical component and it is important to understand its troubleshooting just as thoroughly as the database layer troubleshooting.

We also recently received word that we’ll be holding a 1/2-day hands on session for RAC at Collaborate on Thursday morning. Stay tuned for more details on the RAC SIG website!

Finally, I have also had one presentation accepted for the ODTUG Kaleidoscope conference (June 21-25, 2009 in Monterey, CA). That session is titled “DBA-speak for Developers”.

1. “DBA-speak for Developers”: The most common problem between DBA and Development teams is a lack of communication. When communication breaks down, it is common for a “Us vs. Them” mentality to develop. DBAs are sometimes guilty of using “DBA-speak” terms that are intended for other DBAs to understand, but Developers aren’t always DBA-speak literate. Developers, on the other hand, have many languages as well: Java, PHP, Ruby, and Perl, to name a few. In this technical session, we will focus on helping Developers by translating some of the most common things that their DBAs may say into something that the common Developer can better understand. We’ll no doubt have some fun trying to make sense of the madness that is the Oracle DBA, but we’ll also dig in to the real phrases Developers are likely to hear. Attendees are also invited to bring their own DBA-speak phrases for translation as well.

Looking ahead, I’m highly anticipating Oracle Open World (October 11-15, 2009 – San Francisco) and Miracle Oracle Open World (October?, 2009, near Copenhagen, Denmark). These are both great events (one very large, one small) with tons of great information and great people to meet and talk with at every turn.

So, if you’re at any of these events, please do look me up. I’m on the schedule for many events and enjoy meeting new faces, fellow RAC SIGers, and anyone interested in a chat about RAC, clustering, storage, or scuba diving!

On a more immediate note, I’ll be attending the RMOUG conference next week in Denver. This has got to be one of the best buys around for two days of great sessions by some wonderful experts. The networking opportunities are top-notch as well with the event being a relatively small event and the people (like Debra Lilley, Cary Millsap, Kevin Closson, Mogens Nørgaard, and Tim Gorman, to name a few) very personable. If you’re going, I’ll enjoy seeing you there! If you’re not…put it on your calendar for next year. Even with budget cuts, this is still an affordable event!

• June 15-19, ODTUG Kaleidoscope 2008, New Orleans, LA
• September 21-25, Oracle OpenWorld, San Francisco, CA
• December 1-5, UKOUG Conference, Birmingham, UK
• February, 2009 (TBD), RMOUG Training Days, Denver, CO
• March 8-12, 2009, Hotsos Symposium, Dallas, TX
• April 20-24, 2009, RSA Conference, San Francisco, CA
• May 3-7, 2009, Collaborate 09, Orlando, FL

In between these events (hopefully, I’ll get to attend and/or participate in all of them), I’ll also be planning to present at some regional user group events in Charlotte (CLTOUG), Chicago (COUG), Northern California (NoCOUG) and others as time allows.

If you’re going to be at any of these events, please get in touch so we can meet up there. I’m hoping to make more time this year for attending these events and be a little less involved in them so that I can make more time for talking with individuals and/or customers. Hope to see you in the next year at one or more of these venues!