Patching PSU patches GI with GIMR and RDBMS in different Patchset

Introduction

This article explains how to perform PSU patch on GI home with different version/release 12.1.0.2 and the RDBMS is in different version 12.1.0.1 and Grid infrastructure management repository is the part of GI home which is introduced in 12c. In this process we also see how to resolve the conflicts and understand which overlay patch to apply in order to fix the specific bug patch which was already installed.

Patching Plan and the Environment

This is very much important to reader to understand the environment to get idea whether this articles useful to refer or not. This is two nodes RAC cluster (ORA-R2N1, ORA-R2N2) and the database name HADB. The GI home is on 12.1.0.2 and the Database Home is 12.1.0.1. By using opatchauto we can run from root user and hence the patch will be applied on GI + RDBMS homes but here the GI home and RDBMS versions/releases are different, hence we have to deal the GI patch separately and RDBMS separately. On top of that Global Infrastructure Management Repository instance is must start from 12.1.0.2 and it is mandatory for the future upgrades of GI to 12.2 for example. Below is the list of the patches I have applied, however am not including the conflict patches and overlay patches information because it depends purely based on the environment and the patches which are already installed on the homes. Basically this article gives idea to you how to apply PSU on GI and DB homes are different and the GIMR is part of the configuration.

OPatch patch of version 12.1.0.1.10 for Oracle software releases 12.1.0.x (NOV 2015)
Patch 21951844 - Database Patch Set Update 12.1.0.1.160119 (Jan2016)
Patch 22191349 - Oracle Grid Infrastructure Patch Set Update 12.1.0.2.160119 (Jan2016)

What is GIMR?

GIMR container stores the diagnostic information collected by the cluster health monitor which helps to identify cluster ware related and operating system related problems. This is also a database but runs with the GI home but not from the RDBMS home and there is more flexibility to start the database from any node in case of failure at one node. Of course this option is available from 12.1.0.1 and it is upto you whether you prefer to install or not, if we preferred to install then cluster health monitor enabled to gather the diagnostic data. When we select the GIMR database option then you can see the instance will be created with the name “MGMTDB” and the instance name as “-MGMTDB”, There may be many confusions where the database will be created and so on? Yes that is expected. When we initiate OUI by default it picks the location of OCR/vote disk and still OUI allows flexibility to create database in any other disk groups as well.

[root@ORA-R2N1 patches]# ps -ef|grep pmon
oragrid 7388 1 0 08:02 ? 00:00:00 asm_pmon_+ASM1
oracle 9315 1 0 08:03 ? 00:00:00 ora_pmon_hadb1
oragrid 12516 1 0 08:03 ? 00:00:00 mdb_pmon_-MGMTDB
root 14623 40282 0 08:04 pts/1 00:00:00 grep --color=auto pmon
[root@ORA-R2N1 patches]#
SQL> select name,db_unique_name,open_mode from v$database;

NAME DB_UNIQUE_NAME OPEN_MODE
--------- ------------------------------ --------------------
_MGMTDB _mgmtdb READ WRITE

Patching the environment

We will start first patching the GI home, To use opatchauto to apply patch on oracle homes, then we must have root access, in case root access is not allowed then the patch should apply manually and these steps are available in readme file of the patch. Prior to patching there are few prerequisites which needs to complete them and they are below.

If the GI and RDBMS are in same version then one single opatchauto command from GI home will take care everything, but we are into two different patchset hence we have to handle patching individually.

Opatch Upgrade

Opatch is an utility to apply the patches and this opatch version should meet the requirement as per the patch readme file, the opatch will be available to download from MOS 6880880 and available with various versions and releases, we need to select the appropriate opatch. For example to apply the Jan2016 PSU the opatch version should be 12.1.0.1.7 but the version can be same or higher than that. After downloading we can unzip and then copy under GI_HOME, so that Oracle accesses the latest patch and the version should meets the requirement.

[oracle@ORA-R2N1 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH
[oracle@ORA-R2N1 ~]$ opatch version

OPatch Version: 12.1.0.1.3

OPatch succeeded.

[oracle@ORA-R2N1 ~]$ cd /home/oragrid/patches
[root@ORA-R2N1 patches]# ls -ltr
total 1214032
drwxrwxr-x 4 oragrid oinstall 45 Jul 7 2015 16870214
-rw-r----- 1 oragrid oinstall 481946 Jul 7 2015 p16870214_121020_Linux-x86-64.zip
drwxr-x--- 10 oragrid oinstall 4096 Nov 13 15:16 OPatch
-rw-r--r-- 1 oragrid oinstall 52774823 Nov 17 15:18 p6880880_121010_Linux-x86-64.zip
[oracle@ORA-R2N1 ~]$ unzip p6880880_121010_Linux-x86-64.zip
..
.
[oracle@ORA-R2N1 ~]$ opatch version
OPatch Version: 12.1.0.1.10
OPatch succeeded.

Backup of Oracle Home

Always it is best practice to take the backup of the Oracle Homes and it can be useful in case any unresolvable issues during the patching in order to avoid the downtime. This is applicable for both GI home and also RDBMS home. The backup can take using the Tar or zip binary. As we have two nodes there is no necessity to take backup from both nodes. One node backup of binaries can be cloned anytime.

[root@ORA-R2N1 patches]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'ORA-R2N1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'ORA-R2N1'
CRS-2673: Attempting to stop 'ora.REDODG4.dg' on 'ORA-R2N1'
CRS-2673: Attempting to stop 'ora.DATADG1.dg' on 'ORA-R2N1'
CRS-2673: Attempting to stop 'ora.FRADG2.dg' on 'ORA-R2N1'
. . .
. .
CRS-2673: Attempting to stop 'ora.cssd' on 'ORA-R2N1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'ORA-R2N2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ORA-R2N2'
CRS-2677: Stop of 'ora.cssd' on 'ORA-R2N1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'ORA-R2N2' succeeded
[root@ORA-R2N1 patches]#

[root@ORA-R2N1 root]# cd /u01/app/oracle/product/11.2.0/dbhome_1
[root@ORA-R2N1 dbhome_1]# tar -zcvf /backup/GI_DB_BKP_03_APR_CR1046883/GI12102.tgz .
./.patch_storage/20831110_Jul_11_2015_00_45_40/original_patch/
./.patch_storage/21359755_Oct_21_2015_02_52_58/
./.patch_storage/21359755_Oct_21_2015_02_52_58/backup/
./.patch_storage/21359755_Oct_21_2015_02_52_58/original_patch/
./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/
./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/patched/
./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/patched/assistants/
./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/patched/assistants/dbca/
./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/patched/assistants/dbca/jlib/
./.patch_storage/21948354_Dec_20_2015_23_39_33/
./.patch_storage/21948354_Dec_20_2015_23_39_33/backup/
./.patch_storage/21948354_Dec_20_2015_23_39_33/original_patch/
./.patch_storage/interim_inventory.txt
./.patch_storage/record_inventory.txt
./.patch_storage/patch_free
[root@ORA-R2N2 grid]#
$ pwd
/backup/GI_DB_BKP_03_APR_CR1046883
oracle@ORA-R2N2:ORA-R2N2:/backup/GI_DB_BKP_03_APR_CR1046883 >
$ ls -ltr
total 7604336
-rw-r--r-- 1 root root 5016080737 Apr 3 03:21 db12101.tgz
-rw-r--r-- 1 root root 2770746668 Apr 3 03:47 GI12102.tgz
oracle@ORA-R2N2:/backup/GI_DB_BKP_03_APR_CR1046883 >
$
Now we can start the cluster service again for the patching.
[root@ORA-R2N1 patches]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ORA-R2N1'
CRS-2672: Attempting to start 'ora.evmd' on 'ORA-R2N2'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ORA-R2N2'
CRS-2672: Attempting to start 'ora.evmd' on 'ORA-R2N1'
CRS-2676: Start of 'ora.cssdmonitor' on 'ORA-R2N1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ORA-R2N1'
. . .
. .
.
CRS-2672: Attempting to start 'ora.storage' on 'ORA-R2N2'
CRS-2676: Start of 'ora.storage' on 'ORA-R2N2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ORA-R2N2'
CRS-2676: Start of 'ora.crsd' on 'ORA-R2N2' succeeded
[root@ORA-R2N1 patches]#

Patch conflict detection

During the plan phase we can run the opatchauto using the analyze option, so that opatchauto will not update system, but it just checks conflicts with the patch and the oracle home. If there are any conflicts then these should be resolved by requesting merge patches. If the conflict patch contains in the superset patch which we are applying then it can be ignored.

[root@ORA-R2N1 patches]# opatchauto apply /home/oragrid/patches/22191349 -analyze
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
OPatchauto Version : 12.1.0.1.10
OUI Version : 12.1.0.2.0
Running from : /u01/app/oragrid/product/12.1.0/grid

opatchauto log file: /u01/app/oragrid/product/12.1.0/grid/cfgtoollogs/opatchauto/22191349/opatch_gi_2016-04-03_06-32-01_analyze.log

NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system.

OCM RSP file has been ignored in analyze mode.

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /home/oragrid/patches/22191349
Grid Infrastructure Patch(es): 21436941 21948341 21948344 21948354
DB Patch(es): 21948344 21948354

Patch Validation: Successful
Grid Infrastructure home:
/u01/app/oragrid/product/12.1.0/grid

Analyzing patch(es) on "/u01/app/oragrid/product/12.1.0/grid" ...
Patch "/home/oragrid/patches/22191349/21436941" analyzed on "/u01/app/oragrid/product/12.1.0/grid" with warning for apply.
Patch "/home/oragrid/patches/22191349/21948341" analyzed on "/u01/app/oragrid/product/12.1.0/grid" with warning for apply.
Patch "/home/oragrid/patches/22191349/21948344" analyzed on "/u01/app/oragrid/product/12.1.0/grid" with warning for apply.
Patch "/home/oragrid/patches/22191349/21948354" analyzed on "/u01/app/oragrid/product/12.1.0/grid" with warning for apply.

Apply Summary:

opatchauto ran into some warnings during analyze (Please see log file for details):
GI Home: /u01/app/oragrid/product/12.1.0/grid: 21436941,21948341,21948344,21948354

opatchauto completed with warnings.
[root@ORA-R2N1 patches]#

As we see above status of the opatchauto analyze is completed with warnings where we expected to be succeeded. We can refer the log files for more information whether the conflicts need to be resolved or the patch which will be rollback is part of the patch we are applying.

[Apr 3, 2016 1:58:39 AM] Whole composite patch Conflicts/Supersets are:

Composite Patch : 21948354

Bug Superset of 16870214

Detail Conflicts/Supersets for each patch are:

Sub-Patch : 19769480

Bug Superset of 16870214
Super set bugs are:
16870214
[Apr 3, 2016 1:58:39 AM] OUI-67303:
Patches [ 16870214 ] will be rolled back.

So as we see the patch 16870214 was already applied on GI Home and again the patch is part of the jan2016 because of cumulative behavior, hence the patch will be rollback and again applied. So we no need to worry about it anymore.

Patch installation

In order to apply patch the OCM response file is mandatory, the file will be available under $ORACLE_HOME/Opatch/ocm/bin. If the file does not exist then we can again create using the file emocmrsp as below. The patch we must apply on both bodes and the procedure is same so below output belongs to only one node.

$ cd /u01/app/oragrid/product/12.1.0/grid/OPatch/ocm/bin
oragrid@ORA-R2N1:+ASM1:/u01/app/oragrid/product/12.1.0/grid/OPatch/ocm/bin >
$ ./emocmrsp
OCM Installation Response Generator 10.3.7.0.0 - Production
Copyright (c) 2005, 2012, Oracle and/or its affiliates. All rights reserved.

Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([ Y ]es, [ N ]o) [ N ]: Y
The OCM configuration response file (ocm.rsp) was successfully created.
oragrid@ORA-R2N1:+ASM1:/u01/app/oragrid/product/12.1.0/grid/OPatch/ocm/bin >
$ ls -ltr
total 20
-rw-r--r-- 1 oragrid oinstall 18 Jul 8 2008 dummy
-rwxr-x--- 1 oragrid oinstall 9063 Apr 2 05:47 emocmrsp
-rw-r--r-- 1 oragrid oinstall 624 Apr 3 04:24 ocm.rsp

After the creation of ocm response file and we can start the patching, the important note is patching should not be run in parallel, of course there is no order on which node first we have to patch. Now the confusion is what about the MGMT database? Do we need to run the datapatch to load modified SQL after patching the GI home? The answer for the question is we no need to run the datapatch on MGMT database because GI(ASM) + MGMT all will be taken care. Again question is how to confirm whether the SQL patch was applied on the MGMT database? Yes we can refer the log files in order to gather the information.

[root@ORA-R2N1 ~]# opatchauto apply /home/oragrid/patches/22191349 -oh /u01/app/oragrid/product/12.1.0/grid -ocmrf /u01/app/oragrid/product/12.1.0/grid/OPatch/ocm/bin/ocm.rsp
OPatch Automation Tools
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version : 12.1.0.2.0
Running from : /u01/app/oragrid/product/12.1.0/grid

opatchauto log file: /u01/app/oragrid/product/12.1.0/grid/cfgtoollogs/opatchauto/22191349/opatch_gi_2016-04-03_06-57-55_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /home/oragrid/patches/22191349
Grid Infrastructure Patch(es): 21436941 21948341 21948344 21948354
DB Patch(es): 21948344 21948354

Patch Validation: Successful
User specified following Grid Infrastructure home:
/u01/app/oragrid/product/12.1.0/grid

Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/u01/app/oragrid/product/12.1.0/grid" ...
Patch "/home/oragrid/patches/22191349/21436941" applied to "/u01/app/oragrid/product/12.1.0/grid" with warning.
Patch "/home/oragrid/patches/22191349/21948341" applied to "/u01/app/oragrid/product/12.1.0/grid" with warning.
Patch "/home/oragrid/patches/22191349/21948344" applied to "/u01/app/oragrid/product/12.1.0/grid" with warning.
Patch "/home/oragrid/patches/22191349/21948354" applied to "/u01/app/oragrid/product/12.1.0/grid" with warning.

Performing postpatch operations on CRS Home... Successful

Apply Summary:

opatchauto ran into some warnings during patch installation (Please see log file for details):
GI Home: /u01/app/oragrid/product/12.1.0/grid: 21436941,21948341,21948344,21948354

opatchauto completed with warnings.
[root@ORA-R2N1 ~]#

From the above output we can see the patching was successful, but how to confirm whether the patching is also applicable on MGT database? Yes this information we can find from the log files of the location “$GRID_HOME/cfgtoollogs/crsconfig”. From the log file it is clear that sql patch also applied.

2016-04-03 06:54:37: Executing cmd: /u01/app/oragrid/product/12.1.0/grid/bin/srvctl status mgmtdb -S 1
2016-04-03 06:54:37: Command output:
> #@=result[0]: dbunique_name={_mgmtdb} inst_name={-MGMTDB} node_name={ORA-R2N1} up={true} state_details={Open} internal_state={STABLE}
>End Command output
2016-04-03 06:54:37: Mgmtdb is running on node: ORA-R2N1; local node: ORA-R2N1
2016-04-03 06:54:37: Mgmtdb is running on the local node
2016-04-03 06:54:37: Starting to patch Mgmt DB ...
2016-04-03 06:54:37: Invoking "/u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-04-03 06:54:37: Running as user oragrid: /u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB
2016-04-03 06:54:37: Invoking "/u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB" as user "oragrid"
2016-04-03 06:54:37: Executing /bin/su oragrid -c "/u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-04-03 06:54:37: Executing cmd: /bin/su oragrid -c "/u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB"
2016-04-03 06:55:48: Command output:
> SQL Patching tool version 12.1.0.2.0 on Sun Apr 3 06:54:37 2016
> Copyright (c) 2015, Oracle. All rights reserved.
>
> Connecting to database...OK
> Note: Datapatch will only apply or rollback SQL fixes for PDBs
> that are in an open state, no patches will be applied to closed PDBs.
> Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
> (Doc ID 1585822.1)
> Determining current state...done
> Adding patches to installation queue and performing prereq checks...done
> Installation queue:
> For the following PDBs: CDB$ROOT PDB$SEED ORA-R2N_CLUSTER
> Nothing to roll back
> The following patches will be applied:
> 21948354 (Database Patch Set Update : 12.1.0.2.160119 (21948354))
>
> Installing patches...
> Patch installation complete. Total patches installed: 3
>
> Validating logfiles...done
> SQL Patching tool complete on Sun Apr 3 06:55:48 2016
>End Command output
2016-04-03 06:55:48: Successfully patched Mgmt D

The patch is already applied, in order to know whether patch is applied on GI home, and we can run opatch lsinventory from the GI home to confirm.

$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oragrid/product/12.1.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/oragrid/product/12.1.0/grid/oraInst.loc
OPatch version : 12.1.0.1.10
OUI version : 12.1.0.2.0
Log file location : /u01/app/oragrid/product/12.1.0/grid/cfgtoollogs/opatch/opatch2016-04-03_07-38-59AM_1.log

Lsinventory Output file location : /u01/app/oragrid/product/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-04-03_07-38-59AM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: 10.128.206.11
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
Interim patches (4) :
Patch 21948354 : applied on Sun Apr 03 06:49:12 EDT 2016
Unique Patch ID: 19553095
Patch description: "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
Created on 20 Dec 2015, 23:39:33 hrs PST8PDT
Sub-patch 21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)"
Sub-patch 20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)"
Sub-patch 20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)"
Sub-patch 19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)"
Bugs fixed:
19189525, 19075256, 19141838, 19865345, 19791273, 19280225, 18845653
. . .
. .
.
Patch 21436941 : applied on Sun Apr 03 06:47:51 EDT 2016
Unique Patch ID: 19245012
Patch description: "WLM Patch Set Update: 12.1.0.2.5 (21436941)"
Created on 13 Aug 2015, 04:00:40 hrs PST8PDT
Bugs fixed:
20075154, 20641027, 20548410, 19016964, 19582630

Patch level status of Cluster nodes :

Patching Level Nodes
-------------- -----
2278979115 ORA-R2N2,ORA-R2N1

-----------------------------------------------------------
OPatch succeeded.
oragrid@ORA-R2N1:+ASM1:/home/oragrid >
$

By thus we have completed all the steps related to GI home and we can start patching the database home.

Patching Database Home

As said earlier, in fact patching on database home will take care by opatchauto from root user but that is applicable when the GI and database homes are in same version, release, patchset. Where there is difference in any of them we must consider to apply patch on GI home alone and later RDBMS home later. The prerequisites are same as GI home like Opatch version should met the requirements as per the readme file or it can be higher version. If opatch version is low then it can be downloaded from the patch number 6880880.

$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ opatch version
OPatch Version: 12.1.0.1.10
OPatch succeeded.
$

Patch Conflict Detection

Unzip the downloaded patch and prior to applying the patch, we must check whether the new patch we are going to apply can be able to patch or not.

$ cd /home/oracle/patches/db_patch/21951844
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation. All rights reserved.

PREREQ session

Oracle Home : /u01/app/oracle/product/12.1.0/db
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/db/oraInst.loc
OPatch version : 12.1.0.1.10
OUI version : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/opatch2016-04-02_08-17-27AM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

ZOP-40: The patch(es) has conflicts with other patches installed in the Oracle Home (or) among themselves.

Prereq "checkConflictAgainstOHWithDetail" failed.

Summary of Conflict Analysis:

There are no patches that can be applied now.

Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
16870214, 21951844

Whole composite patch Conflicts/Supersets are:

Composite Patch : 21951844

Conflict with 16870214

Detail Conflicts/Supersets for each patch are:

Sub-Patch : 19769486

Conflict with 16870214
Conflict details:
/u01/app/oracle/product/12.1.0/db/lib/libserver12.a:ksp.o
/u01/app/oracle/product/12.1.0/db/lib/libserver12.a:ksfd.o

OPatch succeeded.
$

We can see the conflict with the patch on the current RDBMS home, In order to apply patch we have to few things 1) rollback the patch 16870214 2) apply the PSU patch 3) apply overlay patch of 16870214. The patching we have to start once the overlay patch is available to download. Rollback of the patch we must do from both the nodes. Initially we will stop the oracle database prior to patching.

$ srvctl stop database -d hadb
oracle@ORA-R2N1:hadb1:/home/oracle >
$ srvctl status database -d hadb
Instance hadb1 is not running on node ORA-R2N1
Instance hadb2 is not running on node ORA-R2N2

$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ opatch rollback -id 16870214 -local
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/12.1.0/db
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/db/oraInst.loc
OPatch version : 12.1.0.1.10
OUI version : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/16870214_Apr_03_2016_05_18_50/rollback2016-04-03_05-18-50AM_1.log

RollbackSession rolling back interim patch '16870214' from OH '/u01/app/oracle/product/12.1.0/db'

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/12.1.0/db')

Is the local system ready for patching? [y|n]
y
User Responded with: Y

Patching component oracle.rdbms, 12.1.0.1.0...
RollbackSession removing interim patch '16870214' from inventory
Log file location: /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/16870214_Apr_03_2016_05_18_50/rollback2016-04-03_05-18-50AM_1.log

OPatch succeeded.
$

The above rollback procedure we have to run from 2nd node as well, After that we can start apply January 2016 PSU patch after performing conflict detection again whether the patch will succeed or not

$ cd /home/oracle/patches/db_patch/21951844
oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/21951844 >
$ opatch apply -local
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/12.1.0/db
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/db/oraInst.loc
OPatch version : 12.1.0.1.10
OUI version : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/opatch2016-04-03_07-52-30AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 19769486 20299016 20831107 21352619 21951844

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([ Y ]es, [ N ]o) [ N ]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/12.1.0/db')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '19769486' to OH '/u01/app/oracle/product/12.1.0/db'
ApplySession: Optional component(s) [ oracle.network.cman, 12.1.0.1.0 ] , [ oracle.oraolap, 12.1.0.1.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.network.rsf, 12.1.0.1.0...

Patching component oracle.ldap.rsf, 12.1.0.1.0...
. . .
. .
.
Patching component oracle.xdk, 12.1.0.1.0...

Patching component oracle.rdbms, 12.1.0.1.0...
Composite patch 21951844 successfully applied.
Log file location: /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/opatch2016-04-03_07-52-30AM_1.log

OPatch succeeded.
oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/21951844 >

Overlay for Rollback Patch

Before applying the Jan2016 PSU we have performed roll back of conflict patch and now we can apply the overlay patch based on the database version and release we are using. The overlay patch we can download from MOS and based on that we have to search with the same patch number. In my case the overlay patch can be downloaded fromhttp://updates.oracle.com/download/16870214.html

$ cd /home/oracle/patches/db_patch/overlay/16870214
oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/overlay/16870214 >
$ opatch apply -local
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/12.1.0/db
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/db/oraInst.loc
OPatch version : 12.1.0.1.10
OUI version : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/16870214_Apr_03_2016_07_55_37/apply2016-04-03_07-55-37AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 16870214

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([ Y ]es, [ N ]o) [ N ]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/12.1.0/db')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '16870214' to OH '/u01/app/oracle/product/12.1.0/db'
Patching component oracle.rdbms, 12.1.0.1.0...
Patch 16870214 successfully applied.
Log file location: /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/16870214_Apr_03_2016_07_55_37/apply2016-04-03_07-55-37AM_1.log

OPatch succeeded.
oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/overlay/16870214 >

Check for the applied patches using lsinventory

$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/12.1.0/db
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/db/oraInst.loc
OPatch version : 12.1.0.1.10
OUI version : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/opatch2016-04-03_05-30-55AM_1.log
Lsinventory Output file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/lsinv/lsinventory2016-04-03_05-30-55AM.txt

--------------------------------------------------------------------

Local Machine Information::
Hostname: ORA-R2N1.oracle-ckpt.com

ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Database 12c 12.1.0.1.0
There are 1 products installed in this Oracle Home.
Interim patches (2) :
Patch 16870214 : applied on Sun Apr 03 05:29:16 EDT 2016
Unique Patch ID: 18465450
Created on 19 Jan 2015, 22:43:42 hrs PST8PDT
Bugs fixed:
16870214
This patch overlays patches:
19769486
This patch needs patches:
19769486
as prerequisites

Patch 21951844 : applied on Sun Apr 03 05:22:43 EDT 2016
Unique Patch ID: 19567210
Patch description: "Database Patch Set Update : 12.1.0.1.160119 (21951844)"
Created on 17 Nov 2015, 01:48:00 hrs PST8PDT
Sub-patch 21352619; "Database Patch Set Update : 12.1.0.1.9 (21352619)"
Sub-patch 20831107; "Database Patch Set Update : 12.1.0.1.8 (20831107)"
Sub-patch 20299016; "Database Patch Set Update : 12.1.0.1.7 (20299016)"
Sub-patch 19769486; "Database Patch Set Update : 12.1.0.1.6 (19769486)"
. . .
. .
.
Rac system comprising of multiple nodes
Local node = ORA-R2N1
Remote node = ORA-R2N2

--------------------------------------------------------------------------------

OPatch succeeded.
$

Loading Modified SQL

After the all the patches applied on RDBMS home, we can run the datapatch from the database level as below after starting any one node.

SQL> startup
ORACLE instance started.

Total System Global Area 2.6724E+10 bytes
Fixed Size 7584272 bytes
Variable Size 1.4764E+10 bytes
Database Buffers 1.1878E+10 bytes
Redo Buffers 73891840 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 12c Release 12.1.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/overlay/16870214 >
$ cd $ORACLE_HOME/OPatch
oracle@ORA-R2N1:hadb1:/u01/app/oracle/product/12.1.0/db/OPatch >
$ ./datapatch -verbose
SQL Patching tool version 12.1.0.1.0 on Sun Apr 3 07:57:35 2016
Copyright (c) 2014, Oracle. All rights reserved.

Connecting to database...OK
Determining current state...
Currently installed SQL Patches:
Currently installed C Patches: 19769486,20299016,20831107,21352619,21951844
Adding patches to installation queue and performing prereq checks...
Installation queue:
Nothing to roll back
The following patches will be applied: 19769486,20299016,20831107,21352619,21951844
Installing patches...
Patch installation complete. Total patches installed: 5
Validating logfiles...
Patch 19769486 apply: SUCCESS
logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/19769486/19769486_apply_hadb_2016Apr03_07_58_41.log (no errors)
catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors)
catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors)
Patch 20299016 apply: SUCCESS
logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/20299016/20299016_apply_hadb_2016Apr03_08_00_15.log (no errors)
catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors)
catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors)
Patch 20831107 apply: SUCCESS
logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/20831107/20831107_apply_hadb_2016Apr03_08_00_15.log (no errors)
catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors)
catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors)
Patch 21352619 apply: SUCCESS
logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/21352619/21352619_apply_hadb_2016Apr03_08_00_15.log (no errors)
catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors)
catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors)
Patch 21951844 apply: SUCCESS
logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/21951844/21951844_apply_hadb_2016Apr03_08_00_16.log (no errors)
catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors)
catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors)
SQL Patching tool complete on Sun Apr 3 08:00:16 2016
oracle@ORA-R2N1:hadb1:/u01/app/oracle/product/12.1.0/db/OPatch >
$

From the log files, we can see there are no errors seen while datapatch is being applied, So we are done with all the steps related to Database home.

Ensure CRS and Database all services are up

Now start all the services and enable the crontab jobs and backup the database if necessary as required.

Summary

We have seen how to perform PSU patch when GI 12.1.0.2 when the MGMT database is running from GI home with step by step procedure along with the brief introduction of GIMR and also we have seen how to manage when the version/release/patchset is different to database home so on, We also have seen how to patching the database home and how to deal in case of conflicts.

=============================================================================

Oracle DBA Information at one place

Search This Blog