Oracle Data Integration - Overview
-
Upload
jeffrey-t-pollock -
Category
Technology
-
view
419 -
download
4
Transcript of Oracle Data Integration - Overview
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle Data IntegrationINTRODUCTION, CORE CAPABILITIES AND INVESTMENT ROADMAP
Jeff PollockVice President of ProductsOracle Data Integration September, 2016
#OpenWorld 2016
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
#OpenWorld 2016 2
Five CoreCapabilities
1. Business ContinuityDATA ALWAYS AVAILABLE
2. Data MovementDATA ANYWHERE IT’S NEEDED
3. Data TransformationDATA ACCESSIBLE IN ANY FORMAT
4. Data GovernanceDATA THAT CAN BE TRUSTED
5. Streaming DataDATA IN MOTION OR AT RES
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 4
Eight Core Products
Cloud or On-Premise
MostInnovativeTechnology
#1#1
Realtime / StreamingData Integration Tool
Pushdown / E-LTData Integration Tool
1st to certify replication withStreaming Big Data
1st to certify E-LT tool withApache Spark/Python
1st to power Data Preparationw/ML + NLP + Graph Data
1st to offer Self-Service &
Hybrid Cloud solution
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016
Data Integration ProductsGOLDENGATE – DATA INTEGRATOR – DATA PREPARATION –STREAM ANALYTICS – DATAFLOW ML – METADATA
MANAGEMENT – DATA QUALITY
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate
Realtime Performance
Extensible & Flexible
Proven & Reliable
Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of database transactions across homogeneous and heterogeneous environments in real-time with no distance limitations.
MostDatabases
DataEvents
Transaction Streams
Cloud
DBs
Big Data
Supports Databases, Big Data and NoSQL:
* The most popular enterprise integration tool in history
7
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integrator
Bulk Data Performance
Non Invasive Footprint
Future Proof IT Skills
Oracle Data Integrator provides high performance bulk data movement, massively parallel data transformation using database or big data technologies, and block-level data loading that leverages native data utilities
Bulk DataTransformation
Most Apps,Databases
& Cloud Bulk Data Movement
Cloud
DBs
Big Data
1000’s of customers –
more than other ETL tools
Flexible ELT workloads run
anywhere: DBs, Big Data, Cloud
Up to 2x faster batch processes
and 3x more efficient tooling
8
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Self-Service
Better Recommendations
Built-in Data Graph Zero software to install, easy to use browser based interface
Better automation and less grunt work
for humans
Graph database of real-world facts used for enrichment
Oracle Data Preparation
ReportingApps
FilesETL
Oracle Data Preparation is a self-service tool that makes it simple to transform, prepare, enrich and standardize business data – it can help IT accelerate solutions for the Business by giving control of data formatting directly to data analysts.
9
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Business Friendly
Extreme Performance
Spatial Awareness
Oracle Stream Analytics
DB
Web / Devices
DataEvent
Data & Transaction Streams
Downstream(eg; Hadoop)
DataEvent
Oracle Stream Analytics is a powerful analytic toolkit designed to work directly on data in motion – simple data correlations, complex event processing, geo-fencing, and advanced dashboards run on millions of events per second.
Innovative dual model for
Apache Spark or Coherence grid
Simple to use spatial and geo-fencing features an industry first
Includes Oracle GoldenGate for
streaming transactions
10
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Stream or Batch Data
Spark based Pipelines
ML-powered Profiling
Oracle Dataflow MLOracle Dataflow ML is big data solution for stream and batch processing in a single environment – Lambda based applications that can run streaming ETL for cloud based analytic solutions.
Batch and stream
processing at the same time
Machine learning guides users for data
profiling
Data movement across Oracle PaaS services
Most Apps,Databases
& Cloud
Bulk Data Movement
Streaming Data Cloud
DBs
Big Data
Big DataPipeline
11
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Business Glossary
End-to-End Lineage
100+ Supported Systems
Oracle Metadata ManagementOracle Metadata Management provides an integrated toolkit that combines business glossary, workflow, metadata harvesting and rich data steward collaboration features.
Supports Databases, Big Data, ETL Tools, BI Tools etc:
BI Report Lineage
Taxonomy Lineage
Data Model Lineage
12
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Simple to Use
High Performance Matching
Powerful DQ Rules
Oracle Enterprise Data QualityOracle Enterprise Data Quality establishes trusted business data by providing a foundation for data profiling, data standardization, match and merge capabilities and data cleansing.
Profile, Standardize, Match, Merge and Cleanse your data
DWMDM
Apps ETL
Health check for your data; quick & easy profiling and cleansing
Intuitive business user
friendly toolkit for quality rules
Suitable for high performance
Applications or Master Data
13
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016
Powerful Cloud SolutionsFor Data Integration
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 15
Self-Service, Streaming or Batch Data Integration
Data Integration Cloud
Easy Self Service Data Preparation
Machine Learning and Semantic Technology
Streaming or Batch Integrations
• Modern Lambda and Kappa style streaming applications for real-time data pipelines
• ETL integrations for classical Data Warehousing, Data Marts and Operational Databases
• Real-time Replication with Changed Data Capture
• Point and click deployment of Replication services, ETL, and Dataflow pipeline services
• Self-Service Data Preparation designed for non-programmers to prepare rich data transformations
• Spark-based ML platform for Dataflow deployments
• High value Natural Language Processing (NLP) and built-in data graph for Recommendation Service
DataPreparation
StreamingApplications
ETL DataProcessing
Ingest from Sources Automate Publish
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 16
Comprehensive and Differentiated Data Integration in the Public CloudData Integration Cloud – Core Capabilities
Ground to Cloud Data Streams Pushdown ELT/ETL
• GoldenGate realtime Replication from on-premise to Cloud
• Data Integrator bulk data movement into Cloud
• From on-premise to pipelines to analytics in sub-seconds
• Ingest, move, transform and analyze data 100% in memory
• Ultra-high performance ELT processing in Hadoop or DBaaS
• Simple to choose which PaaS you want the workload to run
streaming
batch
Pipelines Analytics
batchdata transformation
Five CloudSolutions
1. Data MigrationsREPEATABLE PROVEN DATA MIGRATION TOOLS
2. DW IntegrationENABLE CLOUD DATA WAREHOUSE STRATEGIES
3. Dev & Test CloudOPERATE DEV-TEST DB’S IN CLOUD
4. Data High AvailabilityDATA RELIABILITY AT 99.999% SERVICE LEVELS
5. Heterogeneous CloudBEST OF BREED FOR AMAZON AWS ETC.
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Key Benefits – Online/Offline Migrations
3rd Party Cloud Support
Database Migrations
• Seamless data migration from older databases or poorly optimized Cloud services
• Take advantage of newest Oracle DB versions in a simple subscription model
• No data loss for phased / online migrations
• Strong automation around provisioning and billing
• Reliable and proven core GoldenGate technology
• Migrate older Amazon RDS instances to Oracle
• Data delivery from on-premise to 3rd party Clouds
#OpenWorld 2016 18
Solution 1: Database Migrations in to Cloud
On Premise
RDS
Oracle DBaaS
I need to migrate my DBs to newer and less
expensive systems
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Key Capabilities: Comprehensive Solution
3rd Party Cloud Support
Data Warehouse Integration
• Operational or Advanced Analytics in the Cloud requires reliable data feeds
• Work with data coming from on-premise data sources or other cloud sources
• Prepare and Ingest data in realtime; built in support for replication and streaming ingestion to DW/Marts
• Transform and Cleanse data; innovative approach allows workloads to run on different PaaS resources
• Supports Amazon RDS databases and Redshift
• Big data support with Kinesis and EMR
#OpenWorld 2016 19
Solution 2: Data Warehouse Integration in the Cloud
DW on ExadataCloud Service
UN
STR
UC
TUR
EDST
RU
CTU
RED
BI & DataVisualization
RDS & REDSHIFT
Help me run my analytics
from the Cloud
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Key Benefits: Automate Data Movement
Oracle Database as a Service
Database Dev-Test Environments
• Keep development and test environments continuously in sync with minimum infrastructure or process overhead
• Leverage automation rather than manual syncs
• Low Overhead reduces manual and complex scripting
• Repeatable approach standardizes DBA processes
• Simple framework can be used across a variety of DBs
• All Supported Oracle Versions regardless of what you have running on premise
• Mix and match versions or patch levels
#OpenWorld 2016 20
Solution 3: Dev & Test Systems in the Cloud
On Premise
Oracle DBaaS12.1 & 12.2
10g & 11g
Make my DBA job easier and more
productive
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Key Benefits for DBaaS
Powerful Cloud Solution
Regional High Availability
• When you can’t afford to have your mission critical databases take downtime
• Particularly necessary when disaster recover is necessary across regional datacenters
• Standby Databases can be always on with Active Data Guard in the Cloud
• Active-Active can enable you to take read/write transactions on all databases.
• Strong Automation of service provisioning and billing
• Oracle or Amazon Cloud operate active-active Oracle Database in any Cloud across regional data centers
#OpenWorld 2016 21
Solution 4: High Availability for DBaaS in the Cloud
Oracle DBaaS
Datacenter Region 1
Active / ActiveOracle DBaaSor Amazon RDS
Datacenter Region 2
On Premise
My business applications can not go
down!
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Heterogeneous and Complete Solution
Bring Enterprise-strength to AWS
• Advanced data integration and availability capabilities for Kinesis, EMR, Redshift and RDS
• Proven data transfer rates of 10’s of thousands of DB transactions per second
• Security controls work correctly across Amazon secure data centers
• Flexible solution can leverage DB or Big Data from within the AWS services
• Most Trusted data replication engine over all others
• Top most popular ETL/ELT tool in worldwide market
• Fastest growing data integration for big data
• Oracle Data Integration is a heterogeneous capability that works across different data platforms and different cloud platforms.
#OpenWorld 2016 22
Solution 5: Make Amazon AWS Enterprise Grade
Oracle DatabaseActive-Active
Realtime Big Data
GoldenGate
Datacenter A Datacenter B
GoldenGate
Data Integrator
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016
Big Data SolutionsFor Data Integration & Governance
PragmaticBig DataSolutions
1. Data IngestionINDUSTRIAL STRENGTH DATA INGESTION LAYER
2. TransformationsFUTURE-PROOF TOOLING FOR CODE GENERATION
3. GovernanceMETADATA LINEAGE AND BUSINESS GLOSSARY
4. ConnectorsHIGH SPEED PARALLEL LOADERS FOR BULK DATA
5. Streaming Big DataINGEST & ANALYTICS 100% IN-MEMORY
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 25
BusinessData
ServingLayer
Apps
Analytics
EDWsRawData
Staging
Data Discovery
ETL Offload
Batch Layer
Data Streams
Social and Logs
Enterprise Data
Highly AvailableDatabases
Databus(Pub/Sub)
Streaming Analytics
Streaming Data Pipelines
...Speed Layer data broadcasting & streaming data processing
...Batch Layer data processing for huge data volumes
...Serving Layer for consumption
Pub / Sub
REST APIs
NoSQL
Bulk Data
Vision = Lambda/Kappa
Databus(Pub/Sub)
Speed Layer
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 26
Hybrid Open-Source...Open Source at the core of speed & batch processing engines
...Enterprise Vendor tools for connecting to existing IT system and
...Cloud Platforms for data fabric
BusinessData
ServingLayer
Apps
Analytics
Batch Layer
Data Streams
Social and Logs
Enterprise Data
Highly AvailableDatabases
Pub / Sub
REST APIs
NoSQL
Bulk Data
Speed LayerRaw Data Stream Processing
Batch Processing
Prepared Data
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Examples
#OpenWorld 2016 27
Reference Architecture
BusinessData
ServingLayer
Apps
Analytics
Batch Layer
Data Streams
Social and Logs
Enterprise Data
Highly AvailableDatabases
Pub / Sub
REST APIs
NoSQL
Bulk Data
Speed Layer
GoldenGate
Data Preparation
Data Quality, Metadata Management & Business Glossary
Oracle Data Integrator
Active DataGuard
Comprehensive architecture covers key areas – #1. Data Ingestion, #2. Data Preparation & Transformation, #3. Streaming Big Data, #4. Parallel Connectivity, and #5. Data Governance –and Oracle Data Integration has it covered.
Dataflow ML
Stream Analytics
Connectors
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
ApplicationsApplications DatabusApplications Speed Layer
Batch Layer
Cap
ture
Trai
l
Ro
ute
Del
iver
Pu
mp
#OpenWorld 2016 28
Streaming Analytics
Application
ServingLayer
RESTServices
VisualizationTools
ReportingTools
Data Marts
UserUpdates
DBMSUpdates
GoldenGate for Ingest
GG GG
Applications ServingLayer
Speed Layer
Batch Layer
Platforms
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 29
ODI for Transformations
ETL Engines
Big Data Frameworks
Speed Layer
Batch Layer
ServingLayer
ApplicationsApplications DatabusApplications
Application
RESTServices
VisualizationTools
ReportingTools
Data Marts
UserUpdates
DBMSUpdates
Applications ServingLayer
Speed Layer
Batch Layer
Oracle Data Integrator
Spark Streaming
Spark SQLSqoop
ERP
Oozie
Pig
Hive
Loaders
Kafka
NoSQL
OGG
SQL
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Data CatalogSpeed Layer
Batch Layer
ServingLayer
#OpenWorld 2016 30
OEMM for Data GovernanceApplicationsApplications DatabusApplications
Application
RESTServices
VisualizationTools
ReportingTools
Data Marts
UserUpdates
DBMSUpdates
Applications ServingLayer
Speed Layer
Batch Layer
KafkaGenerated Streaming
Generated ETL CodeSqoopOLTP Databases
HDFS Files
HCatalog
Hive
NoSQL
ETLTools
Data Warehouses
BI Models
ER Models
Oracle Enterprise Metadata Management
140+ Supported Tools
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
from Devices
Batch Layer
#OpenWorld 2016 31
Streaming Data
ApplicationsApplications
Databus
Applications
Speed Layer ServingLayer
RESTServices
VisualizationTools
ReportingTools
Data Marts
Applications
ServingLayer
Speed Layer
Batch Layer
Oracle Stream Analytics
Oracle Dataflow ML
Oracle GoldenGate
Application
ApplicationsApplicationsDevices
from Databases
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016
World Class SolutionsFor Data Integration & Governance
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 33
140+ Open Source, Standards and Third Party Integrations
Adaptive Altova Apache HCatalog Apache Hive/HQL Borland CA ERwin Cloudera Impala COBOL Copybook DataStax Embarcadero EMC ProActivity GentleWare Google BigQuery Grandite Hadapt Hive Hortonworks Hive IBM Cognos IBM DB2 IBM DataStage IBM Discovery IBM Federation Server IBM Lotus Notes IBM Netezza IBM Rational Rose IBM Rational Architect Informatica Metadata Manager
Informatica PowerCenter CoSORT ISO SQL Standard (DDL) MapR Hadoop Hive MicroFocus Microsoft Access Microsoft Office Excel Microsoft Visio Microsoft SQL Server Microsoft SSIS Microsoft Visual Studio Microstrategy Magic Draw OMG CWM Standard OMG UML Standard Oracle BI Answers Oracle BI Enterprise Edition Oracle BI Server Oracle DAC Oracle Data Integrator Oracle Data Modeler Oracle Database Oracle Designer Oracle Hyperion Applications Oracle Hyperion Essbase Oracle Warehouse Builder
Pivotal Greenplum PostgreSQL QlikView SAP BO Crystal Reports SAP BO Designer SAP BO Desktop Intelligence SAP BO Repository SAP BO Data Integrator SAP BO Data Steward SAP Master Data Management SAP Sybase PowerDesigner SAP Sybase ASE Database SAS Data Integration Studio SAS BI Server SAS Information Map SAS Metadata Management SAS OLAP Server Select Sparx Architect Syncsort Tableau Talend Teradata Tigris Visible W3C DTD & XSD Schema
Metadata Harvesting (Glossary, Lineage & Impact Analysis)Key Standards
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 34
Data Integration Cloud
Data Integration
SaaS Data Quality
DaaS Data Quality
ODI on Big DataCloud Service
Data Preparation
ODI CloudService
Dataflow MLCloud Service
EDQ CloudService
GoldenGateCloud Service
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 35
• Use Starbucks loyalty card to buy a Latte
• Pay for a purchase with PayPal
• Withdraw cash at an ATM from Bank of America, or Citibank, or Wells Fargo, or CapitalOne, or American Express, or JP Morgan Chase, or… most other banks in the world
• Click on an Google AdWords link
• Select a recommended item from EBay.com
• Buy something from Amazon
• Use a store coupon from Safeway
• Browse the catalog from Overstock.com
• Check flight availability on American Airlines
• Use the Cartwheel App at Target stores
• Take your General Motors car in for service
• Go shopping at Macy’s online store on Black Friday
• Deposit your payroll check issued by Paychex
• Take care of your small business accounting on Quickbooks.com
• Buy a song on Apple iTunes
• Change your profile or add a skill on LinkedIn.com
Everyday Moments
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Presen-tationson:
#OpenWorld 2016 36
Data Integration Solutions Program - tinyurl.com/DISOOW16
DemoStations:
Hands-on labs:
OracleEnterprise Metadata
Management
OracleEnterprise
Data Quality
Oracle GoldenGate
OracleData
Integrator
OracleBig Data
PreparationCloud Service
OracleEnterprise
Data QualityHOL7466
Oracle GoldenGateDeep DiveHOL7528
ODI and OGGfor Big Data
HOL7434
Oracle Big DataPreparation
Cloud ServiceHOL7432
MiddlewareDemoground
- Moscone South
Big Data Showcase
- Moscone South
DatabaseDemoground
- Moscone South
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | #OpenWorld 2016 37
Data Integration Solutions Program - tinyurl.com/DISOOW16
Monday, Sept 19• Oracle Data Integration Solutions – Platform Overview and Roadmap
[CON6619 ]• Oracle Data Integration: the Foundation for Cloud Integration [CON6620 ]• A Practical Path to Enterprise Data Governance with Cummins [CON6621]• Oracle Data Integrator Product Update and Strategy [CON6622]• Deep Dive into Oracle GoldenGate 12.3 New Features for the Oracle 12.2
Database [CON6555]
Tuesday, Sept 20• Oracle Big Data Integration in the Cloud [CON7472] • Oracle Data Integration Platform: a Cornerstone for Big Data [CON6624]• Oracle Data Integrator and Oracle GoldenGate for Big Data [HOL7434]• Oracle Enterprise Data Quality – Product Overview and Roadmap
[CON6627] • Self Service Data Preparation for Domain Experts – No Programming
Required [CON6630] • Oracle Big Data Preparation Cloud Service: Self-Service Data Prep for
Business Users [HOL7432] • Oracle GoldenGate 12.3 Product Update and Strategy [CON6631] • New GoldenGate 12.3 Services Architecture [CON6551] • Meet the Experts: Oracle GoldenGate Cloud Service [MTE7119]
Wednesday, Sept 21• Data Quality for the Cloud: Enabling Cloud Applications with Trusted Data
[CON6629] • Transforming Streaming Analytical Business Intelligence to Business
Advantage [CON7352]• Oracle Enterprise Data Quality for All Types of Data [HOL7466] • Oracle GoldenGate for Big Data [CON6632] • Accelerate Cloud On-Boarding using Oracle GoldenGate Cloud Service
[CON6633] • Oracle GoldenGate Deep Dive and Oracle GoldenGate Cloud Service for Cloud
Onboarding [HOL7528]
Thursday, Sept 22• Best Practices for Migrating to Oracle Data Integrator [CON6623] • Best Practices for Oracle Data Integrator: Hear from the Experts [CON6625]• Dataflow, Machine Learning and Streaming Big Data Preparation [CON6626] • Data Governance with Oracle Enterprise Data Quality and Metadata
Management [CON6628] • Faster Design, Development and Deployment with Oracle GoldenGate Studio
[CON6634] • Getting started with Oracle GoldenGate [CON7318] • Best Practice for High Availability and Performance Tuning for Oracle
GoldenGate [CON6558]
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Connect with Oracle Data Integration
@OracleDI
Blogs.oracle.com/DataIntegration/
Oracle Data Integration
Oracle Data Integration
#OpenWorld 2016 38
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
#OpenWorld 2016 40