Oracle Database 19c: Clusterware Administration Workshop

Introduction to Clusterware

• Objectives
• Cluster
• Clusterware
• Oracle Clusterware
• Clusterware Architecture and Cluster Services
• Goals for Oracle Clusterware
• Oracle Clusterware Fencing
• Cluster Time Synchronization
• Network Resource Management
• Oracle Clusterware Operating System Requirements
• Oracle Clusterware Networking
• IP Addresses for Public Networks
• Private Network IPv6 Support
• Grid Naming Service (GNS)
• Grid Naming Service Configuration Options
• Shared GNS Across Multiple Clusters
• Highly Available Grid Naming Service
• Configuring Highly Available GNS
• Single-Client Access Name
• Quiz
• Summary

Oracle Clusterware Architecture

• Objectives
• Oracle Clusterware Technology Stack
• Cluster Ready Services Technology Stack
• OHAS Technology Stack
• Clusterware Component Processes and Services
• Oracle Clusterware Repository (OCR)
• CSS Voting Disk Function
• Voting Disk Considerations
• Oracle Local Registry and High Availability
• Oracle Clusterware Initialization
• Clusterware Startup Details
• Clusterware Startup: OHASD orarootagent
• Clusterware Startup Details: CRSD orarootagent
• Clusterware Startup Details: OHASD oraagent
• Controlling Oracle Clusterware
• Verifying the Status of Oracle Clusterware
• Viewing the High Availability Services Stack
• GPnP Architecture: Overview
• How GPnP Works: Cluster Node Startup
• Client Database Connections
• Quiz
• Summary
• Practice 2: Overview

Cluster Configuration Options

• Cluster Configuration Options
• Oracle Standalone Clusters
• Oracle Cluster Domain
• Oracle Cluster Domain: Overview
• Oracle Member Clusters
• Oracle Member Cluster for Oracle Databases
• Oracle Member Cluster for Applications
• Member Cluster Manifest File for Member Clusters
• Oracle Extended Clusters
• Option 1: Configure an Oracle Extended Cluster
• Assign Failure Groups to Sites
• Option 2: Configure Oracle Extended Clusters
• Assign Failure Groups to Sites Using ASMCA
• Quiz
• Summary

Grid Infrastructure: Preinstallation Tasks

• Objectives
• Shared Storage Planning for Grid Infrastructure and RAC
• Using a Shared File System with Grid Infrastructure
• Logical Volume Managers and Grid Infrastructure
• Managing Voting Disks in ASM
• Sizing Storage for Oracle Standalone Cluster
• GIMR Configuration Details
• Quiz
• Oracle Grid Infrastructure Installation
• General Server Minimum Requirements
• Checking System Requirements
• Enabling the Name Service Cache Daemon (nscd)
• Setting the Disk I/O Scheduler on Linux
• Cluster Name and SCAN Requirements
• Checking Network Requirements
• IP Address Requirements with GNS
• IP Address Requirements for Static Configuration
• Broadcast and Multicast Requirements
• Private Interconnect Network Requirements
• Interconnect NIC Guidelines
• Private Interconnect Redundant Network Requirements
• Interconnect Link Aggregation: Single Switch
• Interconnect Link Aggregation: Multiswitch
• Additional Interconnect Guidelines
• Cluster Time Synchronization
• Software Requirements (Kernel)
• Software Requirements: Packages
• Oracle Linux with the Unbreakable Enterprise Kernel
• Zero-Downtime Kernel Updates with Ksplice
• Oracle Preinstallation RPM
• Installing the cvuqdisk RPM for Linux
• Creating Groups and Users
• Creating Groups, Users, and Paths
• Shell Settings for the Grid Infrastructure User
• Determining Root Script Execution Plan
• Quiz
• Summary
• Practice 4: Overview

Grid Infrastructure Installation

• Objectives
• Installing Grid Infrastructure
• Choosing a Cluster Configuration
• Grid Plug and Play Support
• Configuring Shared GNS
• Cluster Node Information
• Specify Network Interface Usage
• Storage Option Information
• Create ASM Disk Group
• Create ASM Disk Group: Specify Failure Groups
• Specify ASM Password
• Failure Isolation Support with IPMI
• Specify Management Options
• Privileged Operating System Groups
• Specify Installation Location
• Create Inventory
• Root Script Execution Configuration
• Perform Prerequisite Checks
• Install Product
• Verifying the Grid Infrastructure Installation
• Understanding Offline Processes
• Check ASM Function for Oracle Clusterware Files
• Create a Fast Recovery Area Disk Group
• Modifying Oracle Clusterware Binaries After Installation
• Unconfiguring Oracle Clusterware Without Removing Binaries
• Quiz
• Summary
• Practice 5: Overview

Managing Cluster Nodes

• Objectives
• Adding a Cluster Node
• Prerequisite Steps for Adding a Node
• Adding a Node Using gridSetup.sh
• Using gridSetup.sh to Add a Node
• Adding a Node to a Cluster on Windows Systems
• Using Fleet Patching and Provisioning to Add a Node
• Deleting a Node from the Cluster
• Deleting a Node from a Windows-Based Cluster
• Additional Methods to Delete a Node from a Ccluster
• Summary
• Practice 6: Overview

Traditional Clusterware Management

• Objectives
• Managing Oracle Clusterware
• Role-Separated Management
• Configuring Horizontal Role Separation
• Controlling Oracle Clusterware
• Verifying the Status of Oracle Clusterware
• Determining the Location of Oracle Clusterware Configuration Files
• Checking the Integrity of Oracle Clusterware Configuration Files
• Locating the OCR Automatic Backups
• Changing the Automatic OCR Backup Location
• Adding, Replacing, and Repairing OCR Locations
• Removing an Oracle Cluster Registry Location
• Migrating OCR Locations to ASM
• Migrating OCR from ASM to Other Shared Storage
• Performing Manual OCR Backups
• Restoring the OCR on Linux or UNIX Systems
• Backing Up and Recovering the Voting Disk
• Adding, Deleting, or Migrating Voting Disks
• Restoring Voting Disks
• Oracle Local Registry
• Oracle Interface Configuration Tool: oifcfg
• Determining the Current Network Settings
• Configuring Redundant Interconnect Usage Using OIFCFG
• Changing the Virtual IP Addresses Using SRVCTL
• Changing the Interconnect Adapter Using OIFCFG
• Managing SCAN VIP and SCAN Listener Resources
• SCAN Listeners and Valid Node Checking
• What-If Command Evaluation
• Performing What-If Command Evaluation on Application Resources
• with CRSCTL
• Performing What-If Command Evaluation on Oracle Clusterware Resources
• with CRSCTL
• Formatting the Output for What-If Command Evaluation on Oracle
• Clusterware Resources
• Performing What-If Command Evaluation with SRVCTL
• Evaluating Failure Consequences with SRVCTL
• Reasoned Command Evaluation (Why-If)
• Why-If: Managing Servers, Server Pools, and Policies
• Quiz
• Summary
• Practice 7: Overview

Policy-Based Cluster and Capacity Management

• Objectives
• Policy-Based Cluster Management Enhancements: Overview
• Server Pools
• Server Pools and Policy-Based Management
• Server Pool Attributes
• Server Pool Attribute Considerations
• GENERIC and FREE Server Pools
• Assignment of Servers to Server Pools
• Creating Server Pools with crsctl and srvctl
• Managing Server Pools with srvctl and crsctl
• Moving Servers Between Server Pools
• Managing Server Pools Using Default Attributes
• Server State Attributes
• Server Categorization: Overview
• Server Categorization
• Administering Server Categorization: Server Attributes
• Administering Server Categorization: Server Categories
• Administering Server Categorization: Server Pools
• Policy Set: Overview
• Policy-Based Cluster Management and QoS Management
• Viewing the Policy Set
• Configuring a User-Defined Policy Set: Method 1
• Configuring a User-Defined Policy Set: Method 2
• Modifying a User-Defined Policy Set
• Activating a User-Defined Policy
• Load-Aware Resource Placement
• Server Weight-Based Node Eviction
• Assigning Weight to Servers and Resources
• Quiz
• Summary
• Practice 8 Overview: Using Policy-Based Cluster Management

Upgrading and Patching Grid Infrastructure

• Objectives
• Clusterware Upgrading and Patching: Overview
• Oracle Grid Infrastructure Upgrade
• Options for Oracle Grid Infrastructure Upgrades
• Pre-Upgrade Tasks
• Moving Oracle Clusterware Files to Oracle ASM
• Using CVU to Validate Readiness for Clusterware Upgrades
• Understanding Rolling Upgrades Using Batches
• Dry-run Upgrade
• Performing a Rolling Upgrade from an Earlier Release
• Completing a Clusterware Upgrade When Nodes Become Unreachable
• Deinstalling the Old Oracle Clusterware Installation
• Patching: Overview
• Types of Patches
• RU and RUR Download Assistant
• Apply Patches during an Oracle Grid Infrastructure Installation or Upgrade
• Grid Infrastructure Patching with OPatch
• Rolling Patches
• Checking Software Versions
• OPatch: Overview
• OPatch: General Usage
• Before Patching with OPatch
• Installing a Patch Manually Using OPatch
• OPatch Automation
• Installing a Patch Automatically Using OPatchAuto
• OPatch Log and Trace Files
• Queryable Patch Inventory
• Quiz
• Summary

Monitoring and Troubleshooting Oracle Clusterware

• Objectives
• “Golden Rule” in Debugging Oracle Clusterware
• Oracle Autonomous Health Framework
• Cluster Verify Utility (CVU)
• Clusterware resource (ora.cvu)
• CVU Heath Check Report: Example
• Cluster Verify Components
• Cluster Verify Output: Example
• Cluster Health Monitor (CHM)
• oclumon Utility
• clumon dumpnodeview Command
• oclumon dumpnodeview Command
• oclumon manage Command
• Oclumon dumpnodeview
• Oclumon Version / debug
• Cluster Health Advisor (CHA)
• Cluster Health Advisor: Overview
• Oracle Cluster Health Advisor Architecture
• Using the CHA Command Line Interface chactl
• Managing the CHA Models: Defining “normal”
• CHA Key Performance and Workload Indicators
• Using chactl query to View Problems and Diagnosis
• Managing the CHA Repository
• Trace File Analyzer (TFA) Collector
• TFA Collector Utility
• TFA Collector Analysis
• TFA Collector Repository
• Managing ADR Logs by Using tfactl managelogs
• Oracle Autonomous Health Framework Components
• Lesson Agenda
• Cluster Resource Activity Log (CALOG)
• Querying and Managing the CALOG
• Lesson Agenda
• ADR Directory Structure
• Files in the Trace Directory
• Clusterware Trace Files
• The Oracle Clusterware Alert Log
• Incident Trace Files
• Other Diagnostic Data
• Lesson Agenda
• Node Eviction: Overview
• Rebootless Node Eviction: Example
• Processes Roles For Node Reboots
• Reboot Advisory in clusterware alert.log
• Other Log & Trace Files to Review
• Possible Troubleshooting Scenario: Example
• Quiz
• Summary
• Practice 10: Overview

Making Applications Highly Available with Oracle Clusterware

• Objectives
• Oracle Clusterware High Availability (HA)
• Oracle Clusterware HA Components
• Clusterware Resource Modeling
• Agents
• Action Scripts
• Resource Types
• Adding Resource Types
• Adding a Resource Type with EM
• Using Clusterware to Enable High Availability
• Resource Attributes
• Resource States
• Resource Dependencies
• Start Dependencies
• Stop Dependencies
• Creating a Clusterware Managed Application VIP
• Creating an Application VIP Using EM
• Deciding on a Deployment Scheme
• Registering a Resource
• Registering a Resource: Example
• Adding Resources with EM
• Managing Resources with crsctl
• Managing Clusterware Resources with EM
• Clusterware Resource Groups
• Resource Group: Overview
• Automatic Resource Groups
• Resource Group Privileges
• Resource Group Dependencies
• Resource Group Dependency Types and Modifiers
• Failure and Recovery of Critical Resources
• Failure and Recovery of Non-Critical Resources
• Resource Group Types
• Using Resource Groups
• HA Events: ONS and FAN
• Managing Oracle Notification Server with srvctl
• Quiz
• Summary
• Practice 11: Overview