Oracle GoldenGate 19c: Troubleshooting and Tuning LVC

Introduction

• Objectives
• Course Goals
• Agenda: Day 1
• Agenda: Day 2
• Agenda: Day 3
• Agenda: Day 4
• Summary

Gathering Evidence

• Objectives
• GGSCI and adminclient Commands
• MA Administration Server: Status and Details
• Viewing Checkpoint Details
• CA Viewing Checkpoint Details: Integrated Extract
• CA Viewing Checkpoint Details: Integrated Extract (continued)
• MA Viewing Checkpoint Details: Integrated Extract
• MA Viewing Checkpoint Details: Integrated Extract (continued)
• MA Administration Server: Integrated Extract Checkpoints
• MA Viewing Checkpoint Details: Distribution Server Path
• MA Viewing Checkpoint Details: Distribution Server Path (continued)
• Viewing Checkpoint Details
• CA Viewing Checkpoint Details: Integrated Replicat
• MA Viewing Checkpoint Details: Integrated Replicat
• MA Administration Server: Integrated Replicat Checkpoints
• Viewing Status and Events: Oracle Management Pack for Oracle GoldenGate
• Process Report
• CA Process Report: Initial Load Example
• MA Process Report: Initial Load Example
• Process Report Not Created
• Event (Error) Log
• Managing Event Log: ogg-ggserr.xml
• Event Log: Sample
• Discard File
• Discard File: Sample
• MA: Service Manager Log Information
• MA: Administration Server Log Information
• MA: Distribution Server Log Information
• MA: Receiver Server Log Information
• MA: extract.log
• MA: replicat.log
• MA: ER-events.log and restapi.log
• MA: health deployment command
• System Logs
• Linux System Logs
• Windows Event Viewer: Sample
• Practice 2 Overview: Gathering Evidence
• Quiz
• Summary

Tools

• Objectives
• Tools: Overview
• Knowledge Documents
• Oracle GoldenGate Documentation
• oggerr Utility
• chkprm Utility
• ShowSyntax Parameter: Overview
• ShowSyntax Parameter: Usage
• ShowSyntax Parameter: Classic Replicat Output for CDB Target Database
• ShowSyntax Parameter: Integrated Replicat Output for CDB Target Database
• ShowSyntax Parameter: Classic Replicat Output for Non-CDB Target
• Database
• logdump Utility: Overview
• Viewing Canonical Trail Records
• logdump Utility: Header Fields
• Using logdump: Open
• Using logdump
• Using logdump: ScanForHeader
• Using logdump: ScanForEndTrans
• Using logdump: Filter
• Using logdump: Hex
• Using logdump: Position
• Using logdump: Count
• Using logdump: Save
• Using logdump: Env
• Using logdump: Logging
• Getting logdump Help and Exiting
• Other Useful logdump Commands
• logdump Filtering
• Metadata Encapsulation: Overview
• Metadata in Trail Files
• Quiz
• Practice 3 Overview: Examining the Evidence
• Summary

Basic Problems and Process Startup Problems

• Objectives
• Basic Troubleshooting: Example 1 (Slide 1 of 2)
• Basic Troubleshooting: Example 1 (Slide 2 of 2)
• Basic Troubleshooting: Example 2 (Slide 1 of 2)
• Basic Troubleshooting: Example 2 (Slide 2 of 2)
• Startup Problems: Overview
• Startup Problems: Oracle GoldenGate Configuration (Slide 1 of 5)
• Startup Problems: Oracle GoldenGate Configuration (Slide 2 of 5)
• Startup Problems: Oracle GoldenGate Configuration (Slide 3 of 5)
• Startup Problems: Oracle GoldenGate Configuration (Slide 4 of 5)
• Startup Problems: Oracle GoldenGate Configuration (Slide 5 of 5)
• Startup Problems: Operating System
• Startup Problems: Permissions
• Startup Problems: Database Login
• Startup Problems: Slow Startup
• Startup Speed: Building the Object Record
• Startup Speed: Resolving Wildcards
• Quiz
• Practice 4 Overview: The Stubborn Startup
• Summary

Extraction Problems

• Objectives
• Extract Is Slow (Slide 1 of 4)
• Extract Is Slow (Slide 2 of 4)
• Extract Is Slow (Slide 3 of 4)
• Extract Is Slow (Slide 4 of 4)
• Extract Abends (Slide 1 of 3)
• Extract Abends (Slide 2 of 3)
• Extract Abends (Slide 3 of 3)
• Data Pump Abends (Slide 1 of 4)
• Data Pump Abends (Slide 2 of 4)
• Data Pump Abends (Slide 3 of 4)
• Data Pump Abends (Slide 4 of 4)
• Quiz
• Practice 5 Overview: The Troublesome Extract
• Summary

Replication Problems

• Objectives
• Replicat Is Not Moving
• Replicat Abends: Example 1
• Replicat Abends: Example 2
• Replicat Abends with OGG-02191 Incompatible Record (Slide 1 of 5)
• Replicat Abends with OGG-02191 Incompatible Record (Slide 2 of 5)
• Replicat Abends with OGG-02191 Incompatible Record (Slide 3 of 5)
• Replicat Abends with OGG-02191 Incompatible Record (Slide 4 of 5)
• Replicat Abends with OGG-02191 Incompatible Record (Slide 5 of 5)
• Replicat Abends: Diagnosis
• Replicat Abends: Example 3
• Preventing Replicat from Exiting
• Quiz
• Practice 6 Overview: The Rebellious Replicat
• Summary

Missed Transactions

• Objectives
• What can cause missed transactions and/or divergent data? (Slide 1 of 4)
• What can cause missed transactions and/or divergent data? (Slide 2 of 4)
• What can cause missed transactions and/or divergent data? (Slide 3 of 4)
• What can cause missed transactions and/or divergent data? (Slide 4 of 4)
• How can missed transactions and/or divergent tables be discovered?
• Transactions Not Being Processed (Slide 1 of 3)
• Transactions Not Being Processed (Slide 2 of 3)
• Transactions Not Being Processed (Slide 3 of 3)
• CA Including Missed Transactions: GGSCI Example
• MA Including Missed Transactions: adminclient Example
• MA Including Missed Transactions: Administration Server Example
• (Slide 1 of 2)
• MA Including Missed Transactions: Administration Server Example
• (Slide 2 of 2)
• Quiz
• Practice Overview 7: The Dissimilar Data
• Summary

Mapping and Synchronization Problems

• Objectives
• Mapping Problems: Overview
• Mapping Problems: Source Fetch Fails
• Mapping Problems: Table and Column Names
• Mapping Problems: Definitions
• Mapping Problems: Column-Missing Error
• Synchronization Problems: Testing for Out-of-Sync Rows
• Synchronization Problems: Oracle GoldenGate Veridata Web GUI
• Synchronization Problems: Causes of Out-of-Sync Data
• Synchronization Problems: DML on Target Tables
• Synchronization Problems: Tables Without Keys
• Synchronization Problems: Character Sets in Oracle Databases
• Synchronization Problems: Different Structures or Keys
• Synchronization Problems: Options
• Synchronization Problems: Resynchronizing (Slide 1 of 4)
• Synchronization Problems: Resynchronizing (Slide 2 of 4)
• Synchronization Problems: Resynchronizing (Slide 3 of 4)
• Synchronization Problems: Resynchronizing (Slide 4 of 4)
• Quiz
• Practice 8 Overview: The Mangled Mapping
• Summary

SQLEXEC, File-Maintenance, and Other Problems

• Objectives
• SQLEXEC Problems: SQL Does Not Execute
• SQLEXEC Problems: Tracing SQLEXEC Problems
• File-Maintenance Problems: Discard File Is Not Created
• File-Maintenance Problems: Discard File Is Too Big
• File-Maintenance Problems: Trail File Delays
• File-Maintenance Problems: Trail Files Are Not Being Purged
• File-Maintenance Problems: Trail Files Are Purged Too Soon
• File-Maintenance Problems: Report File Is Too Big
• Other Problems: Time Differences
• Other Problems: Too Many Open Cursors
• Other Problems: INFO Command Shows Incorrect Status
• MA Patching Oracle Platform with OPatch
• CA Patching: Preparing to Patch Oracle GoldenGate for Oracle Database
• CA Patching: Patching for Oracle Platform
• CA Patching: Patching Oracle GoldenGate for Heterogeneous Databases
• Quiz
• Practice 9 Overview: The Filled Files
• Summary

Classic Performance Tuning

• Objectives
• Monitoring: Overview
• Monitoring: What is lag?
• Monitoring: Lag Time Differences
• Monitoring: Checking Lag Statistics
• MA Monitoring: Checking Lag Statistics
• CA Monitoring: Controlling How Lag Is Reported
• Monitoring: Checking Volume Statistics
• Monitoring: Using the Report Parameter
• Monitoring: Using ReportCount
• Monitoring: Using the Stats Command
• Identifying Bottlenecks
• Identifying Bottlenecks: Extract Lag
• Identifying Bottlenecks: Pump and/or Path Lag
• Identifying Bottlenecks: Replicat Lag
• Bottlenecks: Determining Lag (Latency)
• MA Heartbeat Table Example: GG_LAG_HISTORY View
• CA Heartbeat Table Example: GG_LAG_HISTORY View
• Bottlenecks: Determining Lag by Looking at Checkpoints
• Bottlenecks: MA Lag Report Tasks
• Bottlenecks: Monitoring Lag (Latency)
• Bottlenecks: Measuring Throughput
• Bottlenecks: Finding the Cause
• Bottlenecks: Uncovering a Network Bottleneck
• Bottlenecks: Network Bottleneck Solution
• Bottlenecks: MA Network Bottleneck Solution
• Bottlenecks: Network Bottleneck Solution
• Bottlenecks: Disk Reads
• Bottlenecks: Disk Reads Solutions
• Bottlenecks: Disk Writes
• Automatically Restarting Processes that Fail
• Tuning Extract: Filtering and Conversion
• Tuning Extract: Database Fetching
• Tuning Extract: Log Reads
• Tuning Classic Extract: Dividing the Load
• Tuning Extract: Databases
• Review: Comparison of Replicats
• Tuning Classic Replicat: Dividing the Load
• Tuning Coordinated Replicat: Dividing the Load
• Tuning Classic Replicat: Full Table Scans
• Tuning Classic Replicat: Transaction Type
• Tuning Classic Replicat: BatchSQL Overview
• Tuning Classic Replicat: BatchSQL Syntax
• Tuning Classic Replicat: Transaction Type
• Tuning Classic Replicat: Checkpoint Frequency
• Tuning Classic Replicat: Database
• Parallel Replicat: Highly Scalable Apply Engine
• Tuning Parallel Replicat: Control Parameters
• Quiz
• Practice 10 Overview: Performance Tuning
• Summary

Integrated Performance Tuning

• Objectives
• Integrated Capture: Architecture
• Logmining Server Architecture
• Integrated Capture: Architecture
• Integrated Capture: Deployment Options
• Integrated Capture: Syntax
• Oracle GoldenGate Database Views for Integrated Extract
• Integrated Replicat: Architecture
• Integrated Delivery
• Integrated Delivery in Detail
• Integrated Replicat Performance
• Integrated Replicat Performance Comparison
• Parallel Apply: Reasons to Use Dependency Computation and Scheduling
• Parallel Apply Processing
• Dependency-Aware Apply: Source Database Logging
• Dependency-Aware Apply: Trail File Logging
• Workload-Based Autotuning
• Autotune Parallelism
• Workload-Based Autotuning BATCHSQL_MODE
• Workload-Based Autotuning Options
• Direct Replicat (OCI) Processing
• Direct Replicat Processing
• BATCHSQL Support
• Bidirectional Replication
• Integrated Replicat: Syntax
• Changes to Replicat Parameters
• Quiz
• Oracle GoldenGate Database Views for Integrated Replicat
• Practice 11 Overview: Integrated Performance Tuning
• Summary

Integrated Capture and Delivery Troubleshooting

• Objectives
• Troubleshooting Aids
• Troubleshooting Aids: Where to Find Them
• Replication Performance Advisor: UTL_RPADV Package
• UTL_RPADV Report
• UTL_RPADV Performance Metrics
• UTL_RPADV Replication Paths
• UTL_RPADV Replication Component Analysis for Integrated Extract
• UTL_RPADV Rate Level Statistics for Integrated Extract
• Oracle GoldenGate Healthcheck Script: GoldenGate 12.3 and Higher
• Oracle GoldenGate Healthcheck Script: GoldenGate Prior to 12.3
• Oracle GoldenGate Healthcheck Script
• Healthcheck Script: Overview Example
• Healthcheck Script: Database Example
• Healthcheck Script: Extract Example
• Healthcheck Script: Replicat Example
• Healthcheck Script: Table Statistics and Errors Example
• Healthcheck Script: Tools Example
• Healthcheck Script: Report Map Example
• Oracle Database Enhanced AWR Report
• Oracle GoldenGate Replication Statistics
• Replication Performance at a Glance
• Analyzing Data Capture
• Oracle GoldenGate Apply Reader Statistics
• Apply Server and Apply Coordinator Statistics
• DML Operations
• Quiz
• Practice 12 Overview: Integrated Capture and Delivery Troubleshooting
• Summary