The Oracle ACE program is one way that Oracle recognizes community members that make significant contributions to the Oracle community through blogging, forum participation, user group presentations, and other similar volunteering activities. As one of the Oracle ACE Directors, I have tried to promote the program by raising awareness of its existence and the importance of spreading your knowledge for the good of all Oracle technologists. With help from Google, almost everyone consumes the knowledge posted by the good deeds of others, but a relative few (but growing) contribute to the body of knowledge available online.

There are a lot of smart people working in technology communities these days. Oracle’s community has been growing steadily and I think relatively rapidly in the last few years. Other non-Oracle communities have deep roots and dedicated individuals volunteering lots of their time to help build and maintain networks of technologists too. This afternoon, I read a blog post written by Sheeri Cabral who is a bona fide MySQL community leader and has the awards to prove it. Her post offers a bullet-list of tasks that, if followed, will put you on the road to being a community leader as well.

I think it’s a good time to note that community involvement is becoming a bigger factor in the job market. As a consulting practice manager that regularly interviews and occasionally hires talented individuals, I look at community involvement as a significant factor in my evaluation process. Those that are engaged in the community are more likely to get my attention and those that lead parts of the community receive and deserve a special place near the front of the line in my book. Right or wrong, those involved with the community have typically been more resourceful, harder working, and easier to work with in my experiences. Of course, you also have to “know your stuff”, but that’s becoming the easy part with such an active community producing tons of valuable technical content daily.

So, consider the blueprint Sheeri offers as the motivation to get you more involved. I know I will be working to check off the items on that list for my own community involvement in the coming months! For example, the ODTUG Kaleidoscope and Oracle OpenWorld events are going to be here before you know it and presenting at these events is a great way to give back some knowledge to the rest of the community.

If you want to present at Oracle OpenWorld, you’ve got an opportunity to do so (yes, YOU!). Oracle has made a few session slots available to those that have good ideas. To get started, see the blog postings about the submission process and then go to Oracle Mix and submit your idea! Once submitted, start blogging about it yourself and get others to vote for your idea so you can present at OOW08. If your idea doesn’t get picked, you can always choose to present at one of the OTN Unconference slots at OOW08 too. If timing doesn’t work out for you to attend OOW this year, we’re only a few months away from the start of the call for speakers for the Collaborate 09 conference (in Orlando, May, 2009). IOUG starts their call for papers in the fall, probably sometime in August or September. Watch the IOUG home page for your chance to submit a session proposal there too.

Finally, congratulations, Sheeri, on your well-deserved award and thanks for offering sage advice on community involvement!

I’m thinking of organizing a 2-tank (or more) scuba dive down in Monterrey, California, a little ways south of SF for the weekend prior to OOW 08. You can’t really scuba in San Francisco Bay–not much there, murky, ships, etc. In Monterrey, it’s a whole different story and there are many dive operators and charters servicing the area. I’m proposing that we dive on Saturday, September 20 (my birthday!)–2 or maybe 3 tanks. For those like me that live in landlocked areas, getting to dive with the sea lions and wildlife of northern California is a rare opportunity.

If there’s enough interest, I’ll coordinate a group and find a charter that will take us out and a shop that will rent all the gear (if you don’t want to lug your own). If there are enough of us, I’ll even figure out a way to transport us from the SF area down to Monterrey and back. At this point, I haven’t made plans, just seeing how much interest there might be. I’ll figure out costs once I figure out how big the group may be.

If you’re interested, comment on this posting and/or email me directly (dannorris@dannorris.com) and I’ll keep a little list of who’s interested. If you’re not attending OOW, but can make it to Monterrey for a dive, you’re welcome too–I just need a dive buddy and I don’t particularly care if they know anything about Oracle :). See you underwater (hopefully)!

I have been working on a customer project involving new installations of Oracle Database (a two-node failover cluster using Oracle Clusterware–good stuff) and two load-balanced BEA Oracle Weblogic servers for the middle tier. The middle tier environment runs on Windows Server 2003 Enterprise x64 Edition and is managed by an outsourced hosting facility.

I had worked with Weblogic before this project, but not on Windows and I was a little surprised by the difficulty finding relatively simple information on BEA’s support site as well as in their documentation.

<rant>I have to say that I’m pleased that Oracle closed this deal so that (I never thought I’d say this) they can make BEA’s support as good as Oracle Support and Metalink. I know it will take quite a while–probably 6-12 months–but it absolutely needs to happen. It’s little reminders like these that make me glad I work with Oracle (or the companies they ultimately buy) and not other software. I’m as guilty as anyone of not knowing how good we Oracle professionals have it when it comes to good, well-organized documentation, a solid support site with a killer knowledge repository (with an awesome search), and a warm, active community of users (too many “thanks to” to mention at this point).</rant>

Back to the reason you’re reading: this post will summarize what turns out to be the relatively easy changes required to ensure that your Windows-based Weblogic managed servers will start up at boot time.

When creating a new Weblogic Domain on Windows, you’ll sometimes have the opportunity to choose the Sun JDK or BEA JRockit JDK to use for your domain’s Java engine. Be conscious of which one you choose (for WL 10.0, it’s on the screen where you choose a Development or Production configuration). I chose BEA JRockit for my environment’s servers because on 64-bit Windows, it’s the only one supported. For 32-bit Windows, you have the option to use Sun’s JDK, but then we’d have a mix and more potential for new bugs to pop up due to the JDK differences. There’s a small change in this process when using Sun’s JDK versus BEA JRockit and I’ll note that as well.

To ensure that your Windows-based BEA Weblogic Managed Server(s) start at boot time, follow these steps:

1. After creating the new domain and the new Managed Server (Managed Server is a Weblogic term to identify the differences between an Admin Server and the application server where applications should be deployed), modify the Managed Server settings.
2. Lock & Edit the configuration and proceed to Environment > Servers > (name of your Managed Server) > Configuration tab > Server Start subtab. Then scroll down to the Arguments box and enter -Xnohup in the box. If you’re using Sun’s JDK instead of BEA JRockit, enter -Xrs in that box instead. This is documented at http://edocs.bea.com/wls/docs100/server_start/nodemgr.html#wp1101004.
3. Once that’s in place, you can Activate Changes.
4. The admin server says that no restarts are needed, but I would restart it just to make sure. I’m not sure how the admin server can change a JDK flag without restarting the application server. Maybe I’m just not knowledgeable enough to know how it works, but I think it’s just not smart enough to know that you *do* have to restart.
5. The last change was the one that I missed initially. Briefly, the reason I missed it was that it has to do with crash recovery and in my opinion, a server reboot shouldn’t cause an application server to crash, so I ignored this part of the documentation. One has to ask why you wouldn’t want crash recovery enabled by default anyway, but that’s probably for another rant some other day. Anyway, the final change is to modify a property for the node manager process. Edit BEA_HOME\wlserver_10.0\common\nodemanager\nodemanager.properties and set CrashRecoveryEnabled=true (it is in the file set to false by default). Save and exit the editor.
6. This is Windows, so go to the Services control panel and restart the BEA Products NodeManager service to put the change into effect.
7. Ensure that your managed server is up and running. Then, test the changes you’ve made by rebooting the node and see that your managed server restarts after the reboot is complete (and you’ve given time for node manager to start the managed server).

If you have problems, check the node manager logfile (BEA_HOME\wlserver_10.0\common\nodemanager\nodemanager.log) as it will be most useful in determining what happened. If you don’t see any hint that it even tried to restart the server after the reboot, then it’s probably because the crash recovery setting is not enabled–make sure you changed the right thing in the right file.

I didn’t test to see if this process will restart the admin server as well, but I think it probably will or at least should. With a production configuration, you have to enter the username/password for the admin server when starting it, so you may have to store that in the admin server configuration, but that should be a relatively easy fix. In my case, we didn’t want our admin server running all the time and only start it when needed, so having it start after a reboot wasn’t necessary or desired.

I upgraded to Wordpress 2.5 tonight and also found a new theme that I like better than my dull old one (hat tip to BAAG and Alex for the theme). I’ll be working out the kinks with the word wrapping and updating some tags (I was foolish and used spaces instead of commas, so need to update those posts to fix the tag cloud). Hopefully, in the next couple of days, I’ll get things updated and hopefully the site will look and work better. Then I’ll hopefully continue working through my blog article backlog. Thanks for your patience while I remodel.

This is a quick note about upcoming major events on my Oracle event calendar.

• June 15-19, ODTUG Kaleidoscope 2008, New Orleans, LA
• September 21-25, Oracle OpenWorld, San Francisco, CA
• December 1-5, UKOUG Conference, Birmingham, UK
• February, 2009 (TBD), RMOUG Training Days, Denver, CO
• March 8-12, 2009, Hotsos Symposium, Dallas, TX
• April 20-24, 2009, RSA Conference, San Francisco, CA
• May 3-7, 2009, Collaborate 09, Orlando, FL

In between these events (hopefully, I’ll get to attend and/or participate in all of them), I’ll also be planning to present at some regional user group events in Charlotte (CLTOUG), Chicago (COUG), Northern California (NoCOUG) and others as time allows.

If you’re going to be at any of these events, please get in touch so we can meet up there. I’m hoping to make more time this year for attending these events and be a little less involved in them so that I can make more time for talking with individuals and/or customers. Hope to see you in the next year at one or more of these venues!

I took a few extra days to decompress from last week’s events at Collaborate 08 before writing up a summary of my thoughts. I didn’t think it was really possible to be more busy than I have been in previous years, but I think I matched my previous year’s schedule pretty easily. This year, instead of spending a lot of time in the Piocon exhibit hall booth (we had an excellent location this time), I replaced much of the time I usually spend carrying out the corporate duties with volunteer activities for the IOUG Collaborate 08 Conference Committee.

I was one of the three IOUG DBA Track Managers (along with Bonnie Bizzaro and Nick Marcovecchio) whose responsibilities were to select the DBA sessions for the conference, review the whitepapers and presentations for each speaker (with the help of our excellent reviewers), ensure that the speakers made it to their sessions, and provide feedback to speakers after their sessions (especially first-time speakers). These duties, along with coordinating 3 RAC SIG events, giving 2 expert technical sessions, one technical session, and assembling and moderating a panel discussion made for a busy week. I can say, though, that being on the conference committee is a rewarding experience and something that volunteers should aspire to as they become more involved with the IOUG. Anyone can apply for the Collaborate 09 conference committee when the application becomes available later this summer.

As mentioned, I had some extra duties this year that made it almost impossible for me to sit still through an entire session. However, I did get to sit in on parts of many sessions. We had about 400 total sessions in the IOUG Forum this year, up from about 250 last year. So, as expected, we had over 80 first-time IOUG speakers. I was delighted to see parts of many new speaker sessions and found most of them to be very informative, clear, and concise. Some of the speakers were a little nervous and a few finished a little early (due to speeding through their material a little too fast), but that’s about the worst I saw (which isn’t bad at all!).

My own sessions went well and other than the 8:30am session (on Oracle Adaptive Access Manager), they were attended pretty well too. I think my favorite session was the panel discussion I moderated on Thursday morning titled “To RAC or Not To RAC: What’s Best For HA?” The panelists were great and we had a lively discussion and dispelled several myths about RAC. I only wish we had more time to take questions from the audience, but several attendees still commented that they enjoyed the session and the panelists. Everyone seemed to enjoy the Saturday Night Live skits I played prior to the start of my presentations. My whitepapers and presentations are now available here.

There was a great RAC hands-on session put together at the last minute by Jeremy Schneider. Jeremy heard about an opening in the hands-on lab during our new speaker briefing on Sunday and by Wednesday, he had built several complete recipe procedures for installation and configuration as well as 3 sets of RAC virtual machines. Attendees could choose from different options for starting points to make it easier for each skill level to do something productive in the relatively short 90-minute hands-on session. The only unfortunate part of the RAC hands-on session was that it was scheduled concurrently with the RAC Birds of a Feather session and another RAC technical session as well. With 400+ sessions, scheduling two sessions from the same topic area such that they don’t overlap is next to impossible. Even with the scheduling conflict, there were about 15 RAC hands-on attendees and about 15 people attending the RAC Birds of a Feather session.

The RAC Birds of a Feather session was a lively discussion where I was able to gather some good ideas for future RAC SIG events and topics for possible hands-on sessions as well as webcasts. If you have more ideas, please send them to me.
My non-conference highlight of the week was when a group of 16 Pioteam members and friends of Piocon attended a playoff game in the quest for the Stanley Cup (that’s NHL for those that don’t know hockey). The Colorado Avalanche defeated the Minnesota Wild 5-1 with lots of action to make the match entertaining as well as victorious for the home team. It was a good time for all and allowed for some good discussions with our partners and customers that attended. Other than that, we also had a good time at the customer appreciation event on Wednesday night. They had some rock climbing, DJ music, pool tables, and (fake) gambling tables. I played fake Texas Hold ‘Em Poker for about an hour or so.

Of course, probably the most important and rewarding part of the Collaborate conferences for me is the mini-reunion of all the individuals that I generally only see once a year at these events (aside from “seeing” them online). From what I remember, I saw and/or hung out with Jake K, Matt, Jeremy, Bex, Michelle, Vinod, Venkat, Jake V, Paul, Tony, Vikki, George, Alex, Logan, and that doesn’t include all the IOUG volunteers that I worked so closely with since October. To all those that I forgot to mention, I enjoyed seeing you too! And especially to those I met for the first time, I hope to see you again online as well as at future conference events. Thanks to everyone that took the time to say hello and/or introduce yourself.

See you next year in Orlando, May 3-7, 2009 for Collaborate 09!

Collaborate 08 is here and I’m in Denver already to prepare for the start of the conference. This year, I won’t be blogging a day-by-day accounting of my whereabouts and happenings. Instead, I’ll save up for a summary of what I saw, heard, and vibes I felt during the week.

If you’re still one of the many that hasn’t quite finalized your itinerary for the week, check out these events from the RAC SIG:

• Sun, 13-Apr, 7:30pm, Hyatt Granite ABC: IOUG SIG Reception (this is all SIGs, not just RAC SIG)
• Mon, 14-Apr, 10:30am, Korbel 1F: RAC SIG Customer Panel
• Tue, 15-Apr, 12:15pm, Korbel 4C: RAC SIG Experts Panel
• Wed, 16-Apr, 1:30pm, Korbel 4F: RAC SIG Birds of a Feather

Also check out all the great RAC-related sessions happening this week. For convenience, the RAC SIG has assembled a summary of all the Collaborate 08 sessions related to RAC on their website for download.

Of course, I also would love to see you in my own sessions this week. Here is my presentation list this week:

• Mon, 14-Apr, 9:15am, Korbel 3B: SSL, Load Balancers, Rewrite, Redirect, and More Advanced Configuration (2-hour expert session co-presentation with Matt Topper)
• Tue, 15-Apr, 3:30pm, Four Seasons 2/3: Building a RAC Test Environment On VMWare For Free (2-hour expert session)
• Wed, 16-Apr, 8:30am, Korbel 3B: Oracle Adaptive Access Manager: What, Why, How
• Thu, 17-Apr, 9:45am, Korbel 2A: To RAC or Not To RAC: What’s Best For HA? Panel debate session (I’m the moderator)

When I’m not at one of those sessions or a RAC SIG session, I may be attending other sessions and will also be spending some time at the Piocon Piobooth #1619 in the exhibit hall (I actually have a real job!). My Piocon team members are presenting quite a few interesting sessions as well, so if RAC and IdM aren’t your things, check out these sessions as well.

If you’re at the conference and want to meet up, tweet me using twitter.com (I’m dannorris).

I recently had an opportunity to do a deeper dive into a performance issue related to the performance of a PIPELINED function. The results weren’t quite what I had expected, so I thought they may be of general interest. As with many performance issues, the issues encountered here are likely data- and database-specific. This was testing performed on 10.2.0.3 64-bit non-RAC on Solaris 9.

The PIPELINED function was called like this:

select * from TABLE(pkg1.f1(arg1,arg2,arg3));

And pkg1 is defined like this:

CREATE OR REPLACE PACKAGE pkg1 AS
TYPE coll_table IS TABLE OF custom_persistent_obj_type;
FUNCTION f1 (
p1        IN   NUMBER,
p2             IN   NUMBER,
p3        IN   NUMBER
) RETURN coll_table PIPELINED;
END;
/

The function pkg1.f1 returns a collection of object types, is a little more than 1400 lines long, and contains more than 10 distinct, separate queries, each of them significantly complex involving at least 2 tables, many of them with multiple levels of subqueries. Anyhoo, my chore was to open the idea bin and see if there were some possible enhancements that hadn’t been considered yet.

While reviewing overall performance metrics for the instance, I noted that the DBMS_PICKLER package appeared in the top 10 resource-intensive SQL statements on a regular basis. More digging identified this package as being the primary workhorse behind the TABLE function. So, then the test was clear. Let’s see what happens if we remove PIPELINED, remove TABLE and call the function in a more conventional way. So, my version of the pkg.f would create a collection, add records to it throughout the function’s execution, then return the whole collection at the end with a RETURN statement. The calling function would then need to process this collection into some form to make it more usable by the calling application (similar to the functionality that TABLE provides). For this task, I wrote a stand alone function called DAN.

The new package, pkg2, is defined like this:

CREATE OR REPLACE PACKAGE pkg2 AS
  TYPE coll_table IS TABLE OF custom_persistent_obj_type;
  FUNCTION f2 (
    p1        IN   NUMBER,
    p2             IN   NUMBER,
    p3        IN   NUMBER
  )  RETURN coll_table;
END;
/

DAN is defined as:

CREATE OR REPLACE FUNCTION dan (p_b_tab  IN  pkg2.coll_table,
    p_delimiter     IN  VARCHAR2 DEFAULT ',')
  RETURN VARCHAR2 IS
    l_string     VARCHAR2(32767);
  BEGIN
    for i in p_b_tab.FIRST .. p_b_tab.LAST LOOP
      l_string := l_string || to_char(p_b_tab(i).id1)  ||p_delimiter
                           ||         p_b_tab(i).id2   ||p_delimiter
                           || to_char(p_b_tab(i).array_id)   ||p_delimiter
                           ||         p_b_tab(i).value
      ;
    END LOOP;
  RETURN l_string;
END dan;
/

So, my new units would be called like this:

DECLARE
  l_string      VARCHAR2(32767);
BEGIN
  l_string := dan(pkg2.f2(arg1,arg2,arg3));
  DBMS_OUTPUT.PUT_LINE(l_string);
END;
/

Of course, performance is our primary objective in this case, so timing these two examples side-by-side is the primary chore. I was satisfied that they performed equivalent work and provided equivalent results using different methods. The calling program (custom java, in this case) would have to be modified to handle the changes required, but that may be worth it depending on the time difference.

So, I devised a script to do some very rough testing. The script looked like this:

set timing on echo on termout on pages 0 lines 450 feedb on serveroutput on size 100000
select * from table(pkg1.f1(1095482041,60983,1));
DECLARE
  l_string      VARCHAR2(32767);
BEGIN
  l_string := dan(pkg2.f2(1095482041,60983,1));
  DBMS_OUTPUT.PUT_LINE(l_string);
END;
/

The results were somewhat surprising. Keep in mind that all I’m changing is the way in which data is returned and processed from the function. The data returned, though in slightly different formats, is exactly the same and is not filtered further after it is returned (so there’s no benefit in having it available to run SQL against it).

As you can see, the results are pretty clear that there is a performance improvement in this case when we avoid using PIPELINED functions. I will also say that the result set from the function body is only about 30 rows. From my review of common uses for PIPELINED functions, it appears that larger collections (result sets) may benefit more from PIPELINED functions as they can likely take more advantage of the parallelism allowed with PIPELINED functions.

For those that didn’t bother to install the OS, Oracle or other DST patches (to handle the adjusted DST change schedule) last year and elected to just change the time on your servers by hand twice a year, get ready to do that again this weekend. Personally, I’d rather burn two whole regular work days figuring out and installing the patches than get up to modify the time on a server at 2am on Sunday morning, but that’s just me I guess.

For those that forgot the crontab rule about not scheduling things to occur between 1am and 3am (because they either get skipped or run twice once a year), prepare to be confused as 2:30am doesn’t ever happen on clocks in most US states/regions (here are the exceptions).

I’ll look forward to a little more daylight during the hours that count and one hour less sleep on Saturday night.

On a side note, let’s hope that this little posting will get me started on regular posting here…at least for a while!

I don’t drink coffee. Never have. However, I do love and need caffeine–in large and frequent doses, but carbonated and refrigerated (refrigeration optional when desperate). That’s why I’m always a tiny bit upset when I arrive at a new customer site and as part of the usual tour I find office supplies, bathrooms, printer, and all the free coffee you can drink. Some of them have a water cooler too (which I try to substitute…my physician tells me this is better than my caffeine addiction).

Why doesn’t Equal Opportunity apply? I’m discriminated against because of my beverage choice. Shouldn’t I expect equal access to all the things I need in order to perform my job? I submit that if others I work with need coffee to perform their daily tasks (I presume that’s why it’s part of everyone’s salary/benefits package–including mine), I should be afforded the same. I shouldn’t have to pay part of the coffee tax if I don’t take part in consuming the coffee.

In the interest of full disclosure, I do have to say that my current employer provides a magically perpetually stocked fridge of carbonated delights that I frequent when I’m in the office (they also provide bottomless pots of coffee). Unfortunately, I haven’t been in the office for a month, so that doesn’t help me right now!

Gotta go get something caffeinated to drink before I fall asleep…