This blog has moved here.

Wednesday, December 08, 2010

Upgrade GI to Simply Surprising...

I never thought I'd write a post about such a trivial task... Well, if you are going to upgrade from to be prepared for surprises.

The first surprise is given by the download page from the oracle support site (formally know as Oracle Metalink). The patch set has 4.8G! WTF?! Furthermore, it is split in 7 pieces... Despite of this huge size, the good thing is that, unlike the previous releases, this patch-set may be used as a self-contained Oracle installer, which means you don't have to install a base release and, after that, to apply the patch-set on top of it, but you may simply install the release directly. There's one more catch: if you want to upgrade just the Grid Infrastructure you don't need all 7 pieces from the patch-set. On the download page is not very clear mentioned but if you have the curiosity to open the README (and you should!) then you'll find the following:

Great! So, for the beginning we'd need the 3rd piece in order to upgrade our Grid Infrastructure.

The second surprise is the fact that the GI cannot be in-place upgraded. In previous releases we used to patch providing an existing home location. Starting with in-place upgrades for GI are not supported. According to the "Upgrade" guide:

As of Oracle Database 11g release 2 (11.2), the Oracle Clusterware software must be upgraded to a new home location in the Oracle grid infrastructure home. Additionally, Oracle ASM and Oracle Clusterware (and Oracle Restart for single-instance databases) must run in the same Oracle grid infrastructure home. When upgrading Oracle Clusterware to release 11.2, OUI automatically calls Oracle ASM Cluster Assistant (ASMCA) to perform the upgrade into the grid infrastructure home.

Okey, good to know! Let's start the upgrade process of GI. The wizard provided by the OUI is quite intuitive therefore I will not bother you with screenshots and other obvious things. However, the next surprise comes when you are running the script. The error is:
Failed to add (property/value):('OLD_OCR_ID/'-1') for checkpoint:ROOTCRS_OLDHOMEINFO.Error code is 256
 The fixes for bug 9413827 are not present in the crs home
 Apply the patches for these bugs in the crs home and then run /oragi/perl/bin/perl -I/oragi/perl/lib -I/oragi/crs/install /oragi/crs/install/ execution failed
WTF? You cannot patch if you don't have another stupid patch already there. Okey, as an Oracle DBA you have to be a patient guy... take a deep breath and start looking for bug 9413827. First of all there is the 10036834.8 note, which basically says that you might still get this error even if you apply the patch for the 9413827 bug. As an workaround they suggest to also apply the patch for 9655006 bug. That's madness! In the end it turns out that 9655006 patch is actually the July PSU. Okey, just download the appropriate version for your platform. Now, another surprise... you need an updated version of OPatch utility. Damn it! Back to metalink, search for patch 6880880 and download the version for your platform (Take care not to download the wrong version. By the way, did you noticed that you may download a wget script which can be used to download the patch without using a browser? Yea, finally something good on that shitty flash GUI.) According to the README they suggest to unzip the upgraded OPatch utility directly into your CRS home, using something like:
unzip [] -d [your GI home]
... which I did!
Now, you have to unzip the PSU patch into an empty folder, let's say /u01/stage, and run the following command as root:
/OPatch/opatch auto /u01/stage/ -och [your GI home]
In my case, the output was:
Executing /usr/bin/perl /u01/app/ -patchdir /u01 -patchn stage -och /u01/app/ -paramfile /u01/app/
2010-12-08 12:32:19: Parsing the host name
2010-12-08 12:32:19: Checking for super user privileges
2010-12-08 12:32:19: User has super user privileges
Using configuration parameter file: /u01/app/
The opatch Component check failed. This patch is not applicable for /u01/app/
The opatch Component check failed. This patch is not applicable for /u01/app/
Patch Component/Conflict  check failed for /u01/app/
Upssy! Another surprise! This patch is not applicable for bla bla bla? Are you serious? Let's check the logs. They should be in your $CRS_HOME/cfgtoollogs. Search for a log file named as opatchauto[timestamp].log. The important part for the log:
2010-12-08 12:32:19: The component check failed with following error
2010-12-08 12:32:19: bash: /u01/app/ Permission denied
Ha? I'm root! Aaaa... okey! Apparently it tries to run the OPatch tool under the grid user. Okey, let's fix the permissions.
chown root:oinstall /u01/app/ -R
chmod g+r /u01/app/
Now, try again! Yeap... now it's working.
After applying the patch we are ready for our It's interesting that the output still contains the Failed to add (property/value):('OLD_OCR_ID/'-1') message but the upgrade continues without any other complaints. Okey, let's perform a quick check:
srvctl config asm
ASM home: /u01/app/
ASM listener: LISTENER

srvctl config listener -a
Network: 1, Owner: grid
  /u01/app/ on node(s) owl
End points: TCP:1521
Great, the ASM and listeners are relocated to the new GI home. The next logical thing to do is to uninstall the old GI home, right? It's as simple as:
Oookey, meet SURPRISE Number 6:
ERROR: You must delete or downgrade the Oracle RAC databases and de-install the Oracle RAC homes before attempting to remove the Oracle Clusterware homes.
Isn't it great? On metalink I found Bug 10332736 and, on the WORKAROUND section, it says something about writing a note with a manual uninstall procedure. However, at the time of writing this, the note wasn't published yet. Yea... all I can say is that I'm tired of these stupid issues. What happend with the Oracle testing department? They encourage to patch frequently but, as far as I'm concerned, I always have this creepy feeling before doing it.


LeoJames said...

It is fun isn't it? Patch for patch?

I broke my head for a while to when I saw those errors during last week upgrade.

Good post... Thanks for sharing...

Garrett said...

Wish I would have found your blog post sooner. I just went through all this upgrading hell on my own, it would have gone MUCH faster if I had read all this first. Oh well, the silver lining to it all is I stumbled on your awesome blog and bookmarked it :)

Maria said...

Good post !
Me too have the same issue with deinstal grid home. The same error and the workaround note in metalink is still not available.


Anonymous said...

I also ran into the same issue with unzipping the archive as root, but then it tries to run the upgrade as the grid home owner.

Usually I don't log something with Oracle support, because they are no help, but in this case I did. They were no help. Eventually I did remember to read through the logs, and found the issue, what a headache.