Best Practices for Upgrading Oracle RAC - WordPress.com

Best Practices for Upgrading Oracle RAC: 10g to 11g

Pasha, Mohammed Asmath Capital Market Authority 1

About Me !! Mohammed Asmath Pasha Senior Oracle Consultant / Team Lead Capital Market Authority Over 10 Years of Experience in Banking/Financial Sector

2

Objectives 

Why upgrade Oracle RAC



The benefits of doing an upgrade



Oracle RAC 11gR2 New Features at Glance



Oracle Grid Infrastructure 11gR2



Oracle 11gR2 GI - Clusterware New Features



Oracle 11gR2 GI – ASM New Features



Best Practices – Planning and Upgrading Oracle RAC to 11gR2 

How to plan the upgrade



How to minimize downtime during the upgrade



Possible challenges, with workarounds and solutions



Brief details of known issues and bugs 3

Why upgrade ? 

Oracle Support



New Features



Vendor Support



Bugs/Issues



Organization Policy

4

Why upgrade ?



Oracle Support o



Running out of premier support

New Features o

To leverage new features 5

Why upgrade ?



Vendor Applications o



New releases of an existing application or limitations in acquiring new applications Bugs/Issues

o

What are the issues are you facing today in your current environment.

o

What are the bugs you have encountered in your current environment.



Organization Policy o

Organization policy for software usage

Continued..6

Benefits of doing an upgrade



Reliability got increased further by introducing server pooling.



FAN and FCF or TAF makes it easy for application to mask component failures from user.



Error Detection and Automatic Recovery



Increased Availability by eliminating single point of failures.

7

11gR2 RAC New Features at Glance 11gR1/R2 ASM Fast Mirror Resync

Grid Infrastructure

ASM Preferred Read Disk Groups

Oracle ACFS

ASM Rolling Upgrade

Oracle ADVM RAC One Node Oracle Restart Grid Plug and Play

Improved I/O Fencing Processes Policy-Based Cluster and Capacity Management GI Configuration Wizard Role-Separated Management Enhanced Network Configuration (SCAN using DNS or GNS) Cluster Health Monitor (CHM) OCR and Voting disk support in ASM Redundant Interconnect Usage Cluster time synchronization service IPMI (Intelligent Management Platform) for Node Fencing Enhanced CVU Enhanced SRVCTL Minimized downtime patching

8

11gR2 Grid Infrastructure 

Starting 11gR2 ASM and Clusterware/Oracle Restart combined into single distribution called Grid Infrastructure.

Courtesy – Carol Palmer, Oracle. 

Unlike previous releases ASM cannot be installed as separate home 9

11gR2 Grid Infrastructure 

When Grid Infrastructure is installed on a standalone machine, Oracle automatically installs Oracle Restart component unlike Clusterware environment



Oracle Clusterware and Oracle ASM upgrades are always out-ofplace upgrades. With 11g release 2 (11.2), you cannot perform an inplace upgrade of Oracle Clusterware and Oracle ASM to existing homes. 10

Grid Infrastructure – New Features 

RAC One Node o Oracle Real Application Clusters One Node (Oracle RAC One Node) provides enhanced high availability for single-instance databases, protecting them from both planned and unplanned downtime. Oracle RAC One Node provides the following: o Always-on single-instance database services o Better consolidation for database servers o Enhanced server virtualization o Lower cost development and test platform for full Oracle RAC

11

GI – New Features 

Contd...

Oracle Restart o Oracle Restart enables the startup and automatic restart of singleinstance Oracle Database through the Cluster Ready Services (CRS) component of Oracle Clusterware on a single server. This provides higher availability and automated management of single-instance Oracle Database and Oracle ASM instances. o Oracle Restart is integrated with Oracle Data Guard srvctl config database -d orcl

Database unique name: orcl Database name: orcl Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1 Oracle user: oracle Spfile: +DATA/orcl/spfileorcl.ora Domain: ora.com Start options: open Stop options: immediate Database role: Management policy: automatic Disk Groups: DATA

12

GI – New Features

Contd...

 Grid Plug and Play o In the past, adding or removing servers in a cluster required extensive manual preparation. o Grid Plug and Play reduces the costs of installing, configuring, and managing server nodes by starting a grid naming service within the cluster to allow each node to perform the following tasks dynamically:

o Oracle Restart is integrated with Oracle Data Guard o Negotiating appropriate network identities for itself o Acquiring additional information it needs to operate from a configuration profile o Configuring or reconfiguring itself using profile data, making host names and addresses resolvable on the network 

Improved I/O Fencing Processes o Oracle Clusterware 11g release 2 (11.2) replaces the oprocd and Hangcheck processes with the cluster synchronization service daemon Agent and Monitor to provide more accurate recognition of hangs and to avoid false termination.

13


Contd...

Enhanced Network Configuration (SCAN using DNS or GNS) o SCAN (Single Client Access Name) has been introduced in 11gR2 to simplify the connectivity from the clients to a cluster database. In earlier releases when removal or addition of rac node takes place, the tnsnames.ora needs be updated accordingly. o Using the SCAN this problem is resolved by using single virtual name for the clients to connect, irrespective of infrastructural changes at the cluster level. o It can be implemented either using GNS (Grid Naming Service), a new service added in 11gR2 as part of CRS or by DNS. o In order to implement GNS a subdomain needs to be created in your company with DHCP enabled

Courtesy – Oracle

14


Contd...

Policy-Based Cluster and Capacity Management o With Oracle Clusterware 11g release 2 (11.2) and later, resources managed by Oracle Clusterware are contained in logical groups of servers called server pools. o Enables dynamic capacity assignment when needed to provide server capacity in accordance with the priorities you set with policies o Enables allocation of resources by importance, so that applications obtain the required minimum resources, whenever possible, and so that lower priority applications do not take resources from more important applications o Ensures isolation where necessary, so that you can provide dedicated servers in a cluster for applications and databases

15


Contd...

Role-Separated Management o Role-separated management for Oracle Clusterware allows certain administrative tasks to be delegated to different people, representing different roles in the company. It is based on the idea of an administrator role, called CRS Administrator that can create server pools and grant administrative tasks on a per resource basis.



Cluster Health Monitor (CHM) o The Cluster Health Monitor (CHM) stores real-time operating system metrics in the CHM repository that you can use for later triage with the help of Oracle Support should you have cluster issues. o System Monitor Service (osysmond) runs on each node of the cluster and Cluster logger Service (ologgerd) runs on one node of the cluster. Monitor service from each node send the data to logger service which stores the data in CHM repository. o Diag Collection automatically retrieves the information from CHM repository to be uploaded to Support. More over diag collection got enhanced to collect the data needed only. For example the incident time and incident duration parameters can be passed while collecting the data. Grid_home/bin/diagcollection.pl -collect –crshome Grid_home -chmoshome Grid_home -chmos -incidenttime 07/14/201201:00:00 -incidentduration 00:30 16




Contd...

OCR and Voting Disks (ASM) o OCR and Voting disk now can be stored in ASM, eliminating the need for third-party cluster volume manager. o Using RAW devices for OCR and Voting is not allowed anymore for fresh install. o In previous releases dd command was used to backup the voting disks, which is also desupported. o RAW devices are support only if you are upgrading!! Cluster time synchronization service (CTSS) o The Cluster Time Synchronization Service (CTSS) is installed as part of Oracle Clusterware and runs in observer mode if it detects a time synchronization service or a time synchronization service configuration, valid or broken, on the system. If CTSS detects that there is no time synchronization service or time synchronization service configuration on any node in the cluster, then CTSS goes into active mode and takes over time management for the cluster.



Redundant Interconnect Usage

o In previous releases, in order to have redundant connectivity for private interface bonding, teaming or similar technology was used. Grid Infrastructure and Oracle RAC now can make use of redundant network connectivity for private interface without using other network technology to enhance optimal communication in the cluster. 17

GI – New Features

Contd...

 Out-of-place Upgrade o In place upgrades are not available any more, starting 11gr2 patch sets are not released separately in fact the complete bundle released. o This is mainly to reduce the upgrade time!!!  IPMI (Intelligent Management Platform) for Node Fencing o Oracle Clusterware 11gR2 now supports IPMI, an industry standard management protocol to provide failure isolation support and to ensure cluster integrity. o You can either configure IPMI by selecting a node-termination protocol during the installation, or configure after installation using crsctl command.  Enhanced CVU o CVU can generate shell scripts (Fixup scripts) that perform the system configuration that is required for a successful installation, in addition to identifying system issues that can cause installation failures. o CVU provides additional checks to address install, configuration and operational issues.

18

GI – New Features

Contd...

 Enhanced SRVCTL o SRVCTL commands have been enhanced to manage the configuration of standalone server using Oracle Restart. o The new SRVCTL functionality enables you to register a non-cluster database that can be managed by Oracle Clusterware. Once registered, Oracle Clusterware can start, stop, monitor, and restart the database instance.  Minimized downtime patching for Oracle RAC o Opatch now can apply patches in multi-node, in multi-patch fashion. o Opatch now has ability to detect in one of the nodes is not patched.

19

ASM – New Features  Automatic Storage Management Cluster File System (Oracle ACFS) o Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is a new multi-platform, scalable file system, and storage management design that extends Oracle ASM technology to support all of your application data in both single host and cluster configurations. Additionally, Oracle ACFS provides snapshot functionality for a point-in-time copy of an Oracle ACFS system.

20

ASM – New Features

Contd...

 ASM Dynamic Volume Manager (Oracle ADVM) o Oracle ASM Dynamic Volume Manager (Oracle ADVM) provides volume management services and a standard disk device driver interface to clients. File systems and other disk-based applications send I/O requests to Oracle ADVM volume devices as they would to other storage devices on a vendor operating system. o Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is layered on Oracle ASM through the Oracle ADVM interface. With the addition of Oracle ADVM, Oracle ASM becomes a complete storage solution for both database and nondatabase file needs.

21

ASM – New Features

Contd...

o ASM Rolling Upgrades & ASM Configuration Assistant (ASMCA)  Starting 11.1.0.6 ASMCA performs a rolling upgrade and for earlier releases it performs normal upgrade during which ASMCA will bring down all ASM instances across the cluster. o Intelligent Data Placement  Intelligent Data Placement enables you to specify disk regions on Oracle ASM disks for best performance. Using the disk region settings, you can ensure that frequently accessed data is placed on the outermost (hot) tracks which have greater speed and higher bandwidth.

o Define Custom Allocation Unit o New ASMCMD Commands  Access Control Lists  ASM File Operations

22

ASM – New Features [Extended/Stretched Clusters] o ASM Preferred Read  ASM preferred mirror read is designed to indicate to Oracle which disk failover group is the preferred read disk group.

 If your ASM disk group is mirrored with local and remote disks, this attribute can be set for RAC instances to read from local disks to ensure better performance.

23

ASM – New Features [Extended/Stretched Clusters] Contd... o ASM Fast Mirror Resync/Rebalance  In 10g when redundancy is used and if transient failure occurs then the disk is immediately taken offline and dropped.  Whereas in 11g if transient failure occurs then the disk is taken offline but not dropped until it reaches the attribute value DISK_REPAIR_TIME. ASM keeps track of all modified extents during the failure. After the disk is fixed, only modified extents are resynchronized. (Redundancy failure recovers faster).  Failures are limited to cable, controller etc., and in the case of disk damage or data corruption the complete resync is needed after fixing the disk.  This feature also can be used in preventive maintenance of storage units.

24

Best Practices – Planning and Upgrading Oracle RAC to 11gR2 o Planning  Kick of meeting (Infrastructure Team, Application Team)  It always better to present what you are going to do… o Review your design document  This is needed to understand what other infrastructural changes that might collide or impact your upgrade plan. Example: Sys admins are upgrading all the system to RHEL 6.0 To simplify the stack and simplify vendor interactions, Oracle recommends avoiding 3rd party Clusterware, unless absolutely necessary.

25

Best Practices – Planning and Upgrading Oracle RAC to 11gR2 o Select the upgrade path that is needed for your environment  Supported Upgrade path for Grid Infrastructure 11gR2 Current Release

Upgrade Path

10.1.0.x 10.2.0.x 11.1.0.x 11.2.0.x

   

10.1.0.x 10.2.0.x 11.1.0.x 11.2.0.2

-> 10.1.0.5 -> 11.2.0.3 -> 10.2.0.3 -> 11.2.0.3 -> 11.2.0.3 -> 11.2.0.3

 Supported Upgrade patch for Oracle Database 11gR2 Current Release

Upgrade Path

9.0.1.x (or earlier) 9.2.0.x 10.1.0.x 10.2.0.x 11.1.0.x 11.2.0.x

     

9.0.1.x (or earlier) -> 9.0.1.4 -> 10.2.0.3 -> 11.2.0.3 9.2.0.x -> 9.2.0.8 -> 11.2.0.3 10.1.0.x -> 10.1.0.5 -> 11.2.0.3 10.2.0.x -> 10.2.0.3 -> 11.2.0.3 11.1.0.x -> 11.2.0.3 11.2.0.x -> 11.2.0.3

 Grid Infrastructure 11gR2 supports Oracle Database 11g release 2 (11.2), release 1 (11.1), Oracle Database 10g release 2 (10.2), and release 1 (10.1) Contd...26

Best Practices – Planning and Upgrading Oracle RAC to 11gR2 o Verify  Verify your Server hardware configuration

 Check the compatibility matrix for your O/S  List the additional patches needed for your O/S  Check additional network requirements  Check Storage configurations  Check compatibility with other tools like backup, mail etc.

 Create an Issue Log and Change Log (VVV Imp) #

Issue

Description

Status

Comments

1

Kernel issue

Kernel does not take affect after reboot

In-Progress

Will reviewd within RAC installtion

2

VIOs with Oracle RAC

Is VIOs supported on Oracle RAC ?

Completed

Confirmed from SYS Admin

3

Error related to Oracle user account

When run verification utility failed to PRVF-4007

Completed

Fixed by allow access ssh to same node

Contd...27

Best Practices – Planning and Upgrading Oracle RAC to 11gR2 o Prepare for Grid Infrastructure 11gR2 Upgrade    

Install additional OS patches/packages needed for 11gR2. Setup new BASE location for 11gR2 Setup new HOME for GI installation Setup SCAN IP’s (DNS or GNS) • 3 IP’s are recommended irrespective of number of nodes • Should be on the subnet of Public/VIP’s • Not recommended to use HOSTS file configuration

o Create additional OS groups needed /usr/sbin/groupadd -g 502 asmdba /usr/sbin/groupadd -g 503 oper /usr/sbin/groupadd -g 504 asmadmin /usr/sbin/groupadd -g 505 asmoper usermod -g oinstall -G dba,asmdba,oper,asmadmin,asmoper oracle

o Add additional interface on each node of RAC if redundant interconnect is needed

Contd...28

Best Practices – Planning and Upgrading Oracle RAC to 11gR2 o Apply additional patches needed for Grid Infrastructure 11gR2  If you are upgrading from 11.1 then you need to apply the patch for bug 7308467 on 11.1 clusterware home.  If you are upgrading from 11.2.0.2 then you must apply patch 11.2.0.2.1 (11.2.0.2 PSU 1) or later. For more patches that are needed for your environment, please refer to Things to Consider Before Upgrading to 11.2.0.3 Grid Infrastructure/ASM [ID 1363369.1] RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) [ID 810394.1] o Additional Storage Requirements, Check /tmp space!!

o Check CRS Version, the output should be same for the following: crsctl query crs activeversion crsctl query crs softwareversion

Contd...29

Best Practices – Planning and Upgrading Oracle RAC to 11gR2 o Run CLUVFY Utility  If upgrading < 11gR2 runcluvfy.sh stage –pre crsinst –n racn1,racn2 –fixup –fixupdir /tmp –verbose  If upgrading > 11gR2(11.2.0) to higher runcluvfy.sh stage –pre crsinst –upgrade –n racn1,racn2 –rolling –src_crshome -dest_crshome -dest_version 11.2.0.3 – fixup –fixupdir -verbose o Backup  Backup Clusterware, ASM, Oracle Homes (Full System backup)  Backup OCR  Backup Voting Most Importantly Plan your Fallback Strategy

Contd...30

Best Practices – Planning and Upgrading Oracle RAC to 11gR2

Upgrading Grid Infrastructure to 11gR2 o Login as “oracle”/same user of current Clusterware o UNSET Environment Variables o $ unset ORACLE_BASE o $ unset ORACLE_HOME o $ unset ORACLE_SID o $ unset ORA_CRS_HOME o $ unset TNS_ADMIN

o Launch Installer ./runInstaller.sh

31


Upgrading Grid Infrastructure to 11gR2 Choose the desired option according to your upgrade plan.

Contd...

To minimize the downtime, choose to install the software only. Later use: /u01/app/11.2.0/grid/crs/config/config.sh

32


Upgrading Grid Infrastructure to 11gR2

Contd...

o By default English will be selected, Add additional languages as needed.

33



Contd...

o Node Selection o This step also validates the SSH connectivity between all RAC nodes o On NEXT, a Pop-up message will appear if your environment is < 11.2.0.1

34



Contd...

Enter the virtual name of SCAN

Enter the SYS Password of your current ASM Instance

35



Contd...

Installer automatically picks up the groups as per the role Select the ORACLE_BASE, ORACLE_HOME, that was setup during the preparation

36



Contd...

o OCR Integrity check shows failed, You can ignore after verifying as above

o Kernel Parameter value of shmmni is set as 128 as needed, If your fixup script show the same value then you can safely ignore this. o Since NTP is not configured CTSS will be in Active mode o You will encounter this when both “domain” and “search” entries exists in resolv.conf Warning in "Task resolv.conf Integrity" [ID 1271996.1]

37



Contd...

o Once the software installation is done, execute rootupgrade.sh on local node first once completed, proceed with rest of the nodes. 38



Contd...

o Upgrade of Grid Infrastructure (Clusterware & ASM) completed successfully. Now proceed with Post-Upgrade steps. 39



Contd...

o Post Upgrade Checks  Verify CRS version on all Cluster nodes [oracle@racn1 bin]$ crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.2.0.3.0] [oracle@racn1 bin]$ crsctl query crs releaseversion Oracle High Availability Services release version on the local node is [11.2.0.3.0] [oracle@racn1 bin]$

[oracle@racn2 ~]$ crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.2.0.3.0] [oracle@racn2 ~]$ crsctl query crs releaseversion Oracle High Availability Services release version on the local node is [11.2.0.3.0] [oracle@racn2 ~]$

 Check Cluster State across all nodes [oracle@racn1 ~]$ crsctl check cluster -all ************************************************************** racn1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** racn2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online **************************************************************

40



Contd...

 Check the Voting & OCR disk information [oracle@racn2 crsd]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ------------------------- --------1. ONLINE 7200a9a1a244df0abffe040a4f6f8798 (/dev/raw/raw3) [] 2. ONLINE 0d690593a467df03ff181a184e84ffd3 (/dev/raw/raw4) [] 3. ONLINE 7cfd563c5110cf4cbfbda9fa5af30c38 (/dev/raw/raw5) [] Located 3 voting disk(s). [root@racn2 bin]# ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 520016 Used space (kbytes) : 6444 Available space (kbytes) : 513572 ID : 870042676 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/raw/raw2 Device/File integrity check succeeded Cluster registry integrity check succeeded

 Check if all nodes are Pinned [oracle@racn2 ~]$ olsnodes -t racn1 Pinned racn2 Pinned

You need to do this manually if you are performing new 11gR2 GI installation and want to create Pre 11gR2 database 41



Contd...

o Post Upgrade Activities o Migrate OCR to ASM ocrconfig -add +OCR ocrconfig -delete /dev/raw/raw1 ocrconfig -delete /dev/raw/raw2 o Migrate Voting to ASM crsctl replace votedisk +VOTE o Adding Redundancy for Interconnect oifcfg getif eth1 10.10.10.0 global cluster_interconnect eth0 192.168.117.0 global public

A New Interface added on both nodes with subnet 10.10.1.0

oifcfg setif -global eth2/10.10.1.0:cluster_interconnect oifcfg getif eth1 10.10.10.0 global cluster_interconnect eth0 192.168.117.0 global public eth2 10.10.1.0 global cluster_interconnect

How to Modify Private Network Information in Oracle Clusterware [ID 283684.1]

42



Contd...

o References for the issues during upgrade o How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on Linux/Unix [ID 969254.1] o Top 11gR2 Grid Infrastructure Upgrade Issues [ID 1366558.1] o ASMCA Fails When Upgrading to 11.2 due to Different ASM/CRS User or Non-default ASM Name [ID 1113073.1]

43


Upgrading Database to 11gR2 o Preparing for 11gR2 Database Installation and Upgrade  Check Space Requirements for new home of database

 Verify the pre-installation • cluvfy stage –pre dbinst –n racn1,racn2 -verbose  Time important Queries/Reports  Backup the database  Review the 11gR2 Upgrade Companion • Oracle 11gR2 Upgrade Companion(785351.1)  Run Oracle's Database Pre-Upgrade Utility [ID 884522.1]  Irrespective of the option you want to choose for the database (DBUA/Manual catupgrade) it is recommended to run Pre-Upgrade Utility (utlu112i.sql) and verify the requirements.

44


Upgrading Database to 11gR2

Cont..

o Installing 11gR2 Database  Login as “oracle” ./runInstaller.sh

45



Cont..

Choose the option according to your upgrade plan. To minimize the downtime install the software and later use DBUA/Manually to upgrade the database.

46



Cont..

o Select if additional language support needed o Select the Edition

47



Cont..

o Provide the locations for BASE, HOME o Groups will be picked up if created according to recommendations

48



Cont..

Execute root.sh on local node first and then proceed with second node. 49



Cont..

Once root.sh completed, DBUA will be launched to upgrade the database.

50



Cont..

51



Cont..

52



Cont..

53



Cont..

Installation, Configuration and Upgrade of database to 11gR2 completed successfully.

54



Cont..

o Post-Upgrade Steps  Check for invalid objects  Check the status of all database components SQL> @utlu112s.sql Oracle Database 11.2 Post-Upgrade Status Tool

09-24-2012 14:50:58

Component Current Version Elapsed Time Name Status Number HH:MM:SS Oracle Server VALID 11.2.0.3.0 00:36:02 Oracle Real Application Clusters VALID 11.2.0.3.0 00:00:02 Oracle Workspace Manager VALID 11.2.0.3.0 00:01:30 Gathering Statistics 00:04:11 Total Upgrade Time: 00:41:48

    

Gather Statistics Upgrade RMAN Catalog (If you are using) Backup the database Modify the COMPATIBLE parameter to 11gR2 Verify the timing of Important Queries /Reports

55



Cont..

o References      

DB Upgrade/Migration-Troubleshooting Tool [MOS] Information Center: Upgrading and Migration Oracle Database [ID 1351112.2] Master Note For Oracle Database Upgrades and Migrations [ID 1152016.1] Oracle Database Upgrade Path Reference List [ID 730365.1] Complete Checklist for Manual Upgrades to 11gR2 [ID 837570.1] Complete Checklist to Upgrade the Database to 11gR2 using DBUA [ID 870814.1]

 Oracle Database 11.2 Upgrade Methods http://www.oracle.com/technetwork/products/upgrade/upgrade-11-2-methods173002.pdf

56

Summary !!!  Planning and Preparation are the keys to a successful upgrade

 The DBUA is the recommended method for simplicity and ease-of-use  Take advantage of the Upgrade Planner from MYSUPPORT  Have a fallback strategy and test it  Oracle Grid Infrastructure/Database 11gR2Database 11g Release 2 is a stable release with lots of great features, so go for it!

57

Thank You !!! Email:

[email protected]

Blog:

http://masmathpasha.wordpress.com/

58