Oracle GoldenGate 12 - DOAG Deutsche ORACLE … · Oracle GoldenGate Administration Client...
Transcript of Oracle GoldenGate 12 - DOAG Deutsche ORACLE … · Oracle GoldenGate Administration Client...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate 12.3 Was ist neu an der Microservices Architecture
Thomas NiewelSystemberaterOracle Deutschland B.V. & Co K.G.November 2017
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
Oracle GoldenGate
Oracle GoldenGate Classic Architecture vs. Microservices Architecture
WebPage Interfaces
Directory Structure
Rest Api
1
2
3
4
5
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Cap
ture
Trai
l
Ro
ute
De
live
r
Pu
mp
Oracle GoldenGate Use cases
New DB/HW/OS/APP
Zero Downtime Upgrades & Data Migration
Fully Active Distributed
DB
High Availability& Disaster Recovery
ApplicationOffloading
Query & Report Offloading
Big Data, DW & Marts
Real-time BI, Hadoop Data Staging, Data Ingestion
Event Driven Architecture, SOA/JMS, Coherence
Message Bus& Data Grid
Data SynchronizationAcross the Enterprise
Global Data Centers
Real-time Analytics& Massive Parallelization
DataStreaming
GoldenGate
Real-timeData Delivery
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Flexible Deployment Models
UnidirectionalQuery OffloadingZero-Downtime Migration
Bi-DirectionalHot Standby orActive-Active for HA
Peer-to-PeerLoad BalancingMulti-Master
BroadcastData Distribution
Integration/ConsolidationData Warehouse JMS, Flat File
Oracle GoldenGateApplication Adapters
HDFS,HIVE,FLUME,Kafka,
HBASE. JDBC
Oracle GoldenGate for Big
Data
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
GoldenGate: Heterogeneous Replication
7
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
Oracle GoldenGate
Oracle GoldenGate Classic Architecture vs. Microservices Architecture
WebPage Interfaces
Directory Structure
Rest Api
1
2
3
4
5
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 9
OGG 12.3 Overview• 2 Installation Packages
– Classic Architecture (CA)
–Microservices Architecture (MA)
• Upgrading to 12.3
– Classic Architecture – as usual
–Microservices Architecture – must be fresh install
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Classic Architecture
DYNAMICPORTLIST
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Administration, Distribution, Receiver, Metrics Services with RESTful Service Interfaces
GoldenGate 12.3Microservices Architecture
Source Target
Trail Files
Extract
Admin Server
Trail Files
Replicat
Distribution Server
Receiver Server
SourceDatabase
TargetDatabase
HTTPS RESTFulService Interfaces
Admin Server
Metrics Server
Service Manager
Service Manager
Metrics Server
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 14
• Admin Server
– Replacement of the mgr process
• Distribution Server
– Replacement of the pump process
• Receiver Server
– Replacement of the collector process
• Metrics Server
– Real-time performance monitoring
Classic Architecture vs. Microservices Architecture
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 15
• Enables remote configuration, monitoring, and administration
• Provides formal RESTful service interfaces
– Industry-standard communications protocol: HTTPS, JSON
– Enables Applications to embed GoldenGate
• Secure – Authorize identity of clients with username / password or SSL client certificates
• No web server needed
Common Key Attributes and Benefits of Each Service
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Setup with oggca.sh
GoldenGate 12.3Microservices Architecture
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Setup with oggca.sh
GoldenGate 12.3Microservices Architecture
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Solutions For EveryoneAdministrators Traditionalists DevOps
• Simple to use Web interfaces
• Get up and running quickly
• Lower learning curve
• Command Line access via adminclient – run anywhere
• Similar GoldenGatecommands as in GGSCI
• Interaction the same as previous releases, just thin client
• Uses RESTful APIs
• Run anywhere
• Roll your own monitoring
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Standard TLS/SSL based authentication and Client Authorization
Security Model
Client Authorization
• Identity via SSL user certificates or via username/password
• Roles
– Security User
– Administrator
–Operator
– User
GoldenGateServices
SSL Server Certificate
Authorized Clients
Clients
SSL User Certificate
HTTPS
User/ Passwd
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Remote and secure administration and data communication in cloud environments
Cloud Ready
GoldenGateServices
DeploymentReverse
Proxy wss
HTTPS
• Remote Administration and Configuration
• No OS access required after installation
• Secure administration and data transport
• Works with all standard HTTP proxy and reverse proxy servers
GoldenGateServices
Deployment
Internal Network DMZ
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
Oracle GoldenGate
Oracle GoldenGate Classic Architecture vs. Microservices Architecture
WebPage Interfaces
Directory Structure
Rest Api
1
2
3
4
5
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Access via URL http(s)://<hostname>:<port>
WebPage Interfaces
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Access via URL (http(s)://<hostname>:<port>i.e. http://localhost:8209
Use Administration User for access:i.e. oggadmin/********
• Maintains inventory and configuration information about all deployments
• Watchdog process for Other OGG Services
• Runs as a daemon• Auto-Restart
Oracle GoldenGate Service Manager
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Diagnostic Page• Provides access to responses
that the ServiceManagerreceives
• $OGG_DEPLOYMENT/var/log/ServiceManager.log
Oracle GoldenGate ServiceManager
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Example: Security Roles
Oracle GoldenGate Administration Server - Housekeeping
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Administration Server - Housekeeping
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Access via URL (http(s)://<hostname>:<port>)i.e. http://localhost:8509
• Manage Extracts/Replicats• Create, delete, start/stop, reports,
statistics
• View critical events/messages
• View notifications per page
Oracle GoldenGate Administration Server - Overview
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Access via URL (http(s)://<hostname>:<port>)i.e. http://localhost:8509
• Manage Extracts/Replicats• Create, delete, start/stop, reports,
statistics
• View critical events/messages
• View notifications per page
Oracle GoldenGate Administration Server - Overview
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Example: Create Extract process
Oracle GoldenGate Administration Server - Configuration
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Example: Create Extract process
Oracle GoldenGate Administration Server - Configuration
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Example: Create Extract process (cont’ed)
Oracle GoldenGate Administration Server - Configuration
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Example: Create Extract process
Oracle GoldenGate Administration Server - Configuration
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Administration Server - Details
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Administration Server - Details
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Administration Server - Details
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Access via URL (http(s)://<hostname>:<port>)i.e. http://localhost:8510
• Replacement for the Extract Pump process
• Quickly identify source, trail file, and destination
• Distributes one or more trails to one or more destination
• Does not support transformations
• Supports multiple transport protocols• Tunable per network path
Oracle GoldenGate Distribution Server
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Industry standard HTTP(S) initiated full-duplex streaming protocol
Communication Protocol
Trail Files
Extract Trail Files
Replicat
Distribution Service
Receiver Service
• SSL based security
• Can seamlessly traverse through HTTP forward/reverse proxy servers
wssws
udt ogg
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Supports 4 transfer protocols – OGG, WS, WSS, UDT
Note: For WS/WSS, target credential needs to be setup that connects to the receiver service
• UDT • UDP transport for wide-area networks• Faster than TCP over wide-area with
high latency
• OGG• inter-operability with traditional
deployments
Oracle GoldenGate Distribution Server - Setup
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Quickly see path information• Status• Source/Target trails• DB Name• Trail name
• Configure Application Options
• Configure TCP Options
Oracle GoldenGate Distribution Server - Details
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Quickly see path information• Status• Source/Target trails• DB Name• Trail name
• Configure Application Options
• Configure TCP Options
Oracle GoldenGate Distribution Server - Details
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Access via URL (http(s)://<hostname>:<port>)i.e. http://localhost:8611
• Replacement for Collector
• Single source that handles multiple incoming trail files
• View statistics of path
• Tune network parameters
Oracle GoldenGate Receiver Server
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• See Network Statistics and File IO Statistics
• Statistics include:• Trail file name• Port• Bytes received• Bytes written out• Wait times
Oracle GoldenGate Receiver Server - Details
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Metrics Service• Separate the metrics collection and repository
from the administrative layer
• All GoldenGate processes send metrics via UDP to the metrics server
• Metrics server provide RESTful API to query metrics. Responses in either
– XML format (12.2)
– JSON format
• Supports integration with third party tools
• MUST have Oracle Management Pack for Oracle GoldenGate
Performance Metrics Server
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Metrics captured in real-time• No Repository
Oracle GoldenGate Metric Server
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• New thin client command line tool
• Offers similar functionality as GGSCI, now includes rlwrap
• Runs from $OGG_HOME/bin
– $OGG_HOME can be on any machine
• Supported on Windows and Linux
Oracle GoldenGate Administration Client (adminclient)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
[oracle@ogg123 sf_VM]$ ./adminclientOracle GoldenGate Administration Client for OracleVersion 12.3.0.1.0 OGGCORE_MAIN_PLATFORMS_170318.0742
Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.
Linux, x64, 64bit (optimized) on Mar 18 2017 13:30:38Operating system character set identified as UTF-8.
GG (not connected) 1> connect http://localhost:16000 deployment Atlanta as oggadmin password ********
OGG (http://localhost:16000 deployment_1) 2>
Address and Port for Service Manager
Deployment to connect to
Administrator name and password
AdminClient (connecting)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Interact with multiple deployments from a single command line tool
• Runs from any machine using RESTful calls
Administration Client (adminclient)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
Oracle GoldenGate
Oracle GoldenGate Classic Architecture vs. Microservices Architecture
WebPage Interfaces
Directory Structure
Rest Api
1
2
3
4
5
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Old File System layout – Classic Architecture
Directory Overview
• $OGG_HOME is read-write
• Can support multiple process groups but only one installation
• Everything in one directory – CREATE SUBDIRS
• Upgrade path - overwrite binaries, manual moving file
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
New File System layout – Microservices Architecture
Directory Overview
• $OGG_HOME should be read-only
• One to Multiple deployments can share same $OGG_HOME
• Deployment Homes can be customized
• One step upgrade
• New terms/variables to be used
• DEPLOYMENT_HOME
• $OGG_ETC_HOME / $OGG_VAR_HOME
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
New Directory Structure
• Number of Deployments: (ServiceManager + N)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Deployment Home – etc example
Deployment Homes
$OGG_ETC_HOME$OGG_VAR_HOME
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Deployment Home – var example
Deployment Homes
$OGG_ETC_HOME$OGG_VAR_HOME
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
Oracle GoldenGate
Oracle GoldenGate Classic Architecture vs. Microservices Architecture
WebPage Interfaces
Directory Structure
Rest Api
1
2
3
4
5
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate RESTful APIs• Representational State Transfer (RESTful) Application Program Interface
(API)
• Uses HTTP requests• GET, PUT, POST, and DELETE
• Can be used with many different programming languages• PL/SQL• Perl• Shell• JSON• Etc…
• Simple interaction from command line using CURL
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate RESTful APIs – JSON Documents • JavaScript Object Notation (JSON)
• An open-standard file format used to transmit data objects consisting of attribute-value pairs and array data types
• Media Type: application/json
• Example – create credential alias:
{"$schema":"ogg:credentials","userid":"ggate","password":"ggate"
}
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate RESTful APIs – GET • GET option allows you to retrieve items back from a service
• Example – GET associated services
curl -u gg123:gg123 -H "Content-
Type:application/json" -H
"Accept:application/json" -X GET
http://localhost:8209/services/v2/deplo
yments/ggsrc/services | python -
mjson.tool
.......
"$schema": "ogg:collection",
"items": [
{
"$schema": "ogg:collectionItem",
"links": [
{
"href":
"http://localhost:8209/services/v2/deployments/ggsrc/services",
"mediaType": "application/json",
"rel": "parent"
},
{
"href":
"http://localhost:8209/services/v2/deployments/ggsrc/services/adminsr
vr",
"mediaType": "application/json",
"rel": "canonical"
}
],
"name": "adminsrvr"
},
{
......
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate RESTful APIs – POST • GET option allows you to retrieve items back from a service
• Example – POST stats command
curl -u gg123:gg123 -H "Content-
Type:application/json" -H
"Accept:application/json" -X POST
http://localhost:8509/services/v2/extra
cts/GGSRC/command -d
'{"command":“STATS"}' | python -
mjson.tool
"$schema": "ogg:stats",
"stats": [
{
"operations": [
{
"count": 0,
"operation": "insert"
},
{
"count": 3,
"operation": "update"
},
{
"count": 0,
"operation": "delete"
},
{
"count": 0,
"operation": "discard"
},
{
"count": 3,
"operation": "operations"
}
],
"since": "2017-11-13T15:29:09.298Z",
"sourceTable": "GG123.GG_HEARTBEAT_SEED",
"targetTable": "GG123.GG_HEARTBEAT_SEED"
}
.....
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate RESTful APIs – POST • GET option allows you to retrieve items back from a service
• Example – POST stop extract
curl -u gg123:gg123 -H "Content-
Type:application/json" -H
"Accept:application/json" -X POST
http://localhost:8509/services/v2/extra
cts/GGSRC/command -d
'{"command":"STOP"}' | python -
mjson.tool
"messages": [],
"response": {
"$schema": "er:commandResult",
"reply": "PENDING",
"replyData": {}
}
}
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Additional Information
• Documentationhttps://docs.oracle.com/goldengate/c1230/gg-winux/index.html
• OTNhttp://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Cloud Workshops auf der DOAG Konferenz 2017
• Dienstag 21.11.2017
– 10.00 – 13.00 Uhr – Oracle Datenbank und GoldenGate in der Oracle Cloud
– 13.00 – 16.00 Uhr – IT Systems Management mit der Oracle Management Cloud
• Mittwoch 22.11.2017– 10.00 – 13.00 Uhr – Agile Entwicklung von Microservices in der Oracle Cloud
– 13.00 – 16.00 Uhr – Oracle Datenbank und GoldenGate in der Oracle Cloud
• Donnerstag 23.11.2017
– 10.00 – 13.00 Uhr – IT Systems Management mit der Oracle Management Cloud
– 13.00 – 16.00 Uhr – Agile Entwicklung von Microservices in der Oracle Cloud
65
Im Foyer Shanghai-Seoul – Einstieg in die jeweilige Session jederzeit möglich!