OWB - Making the Move to Oracle Data Integrator

T : +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com

OWB - Making the Move to Oracle Data IntegratorMichael Rainey, Principal Consultant, Rittman MeadUKOUG, December 2012

mailto:[email protected]


http://www.rittmanmead.com



About me...

• Michael Rainey, Principal Consultant, Rittman Mead• Oracle Data Integration Specialist‣Oracle Data Integrator 11g, Oracle GoldenGate 11g

• Writer for Rittman Mead Blog: http://www.rittmanmead.com/blog• Email : [email protected]• Twitter : @mRainey





http://www.rittmanmead.com/blog

http://www.rittmanmead.com/blog




• Oracle BI&DW Experts• Consulting, Training and Support• Global Company, Offices in the US, UK, India and Australia• 1 Oracle ACE Director, 2 Oracle ACE’s• Longest-running Oracle (BI) Blog• Voted UKOUG BI Partner of the Year 2008, 2010, 2011• Oracle Community Advocates‣ Frequent speakers at user groups and conferences‣ 36 Sessions at Oracle Open World since 2010

• Twitter : @rittmanmead

About Rittman Mead...






• ODI is Oracle’s strategic product for data integration‣Describe ODI from an OWB developer’s perspective ‣Discuss key features of ODI 11g

• Plan the migration‣Analysis and preparation

• Perform the migration‣Manual‣Automated

- Partial conversion - Inline Views- Metadata migration - Utilize OWB metadata

• Test the migration‣Determine the definition of success

About this presentation...






Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and

may not be incorporated into any contract.It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making

purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.






“Oracle Data Integrator is Oracle’s Strategic Product for Data Integration”

Gray, Denis. “Oracle Data Integrator Product Update and Future Strategy.” Oracle OpenWorld 2012 San Francisco, CA.






ODI for OWB Developers

• Rittman Mead blog post series by Peter Scott (@dw_pete)http://www.rittmanmead.com/2011/10/oracle-warehouse-builder-and-data-integrator/‣ “Oracle Warehouse Builder and Data Integrator”‣ “Oracle Data Integrator Topologies for OWB

Developers”‣ “Oracle Data Integrator Designer for OWB

Developers”





http://www.rittmanmead.com/2011/10/oracle-warehouse-builder-and-data-integrator/





Oracle Data Integrator 11g Key Features

• ODI Declarative Design‣Separates business rules from data integration type

• Mapping tab allows creation of business logic‣ETL development‣ “The what”

• Flow tab implements data integration type‣Knowledge Modules (built-in templates)

- Incremental Update- Slowly Changing Dimension

‣ “The how”• Generates the “data flow” on execution






ODI Interfaces






ODI Topology, Projects, Metadata, Packages, Load Plans






Plan the Migration






Perform analysis to determine approach

• Analyze current ETL and data warehouse‣Highly customized or “out of the box” ETL?‣Highly integrated or distinct subject areas?‣Perform data profiling to understand quality of data‣How much downtime is allowed for migration?

• Gain understanding of Oracle Data Integrator• Perform Gap analysis‣What gaps do OWB customizations fill...and vice versa?

• Prepare and decide upon resources‣Database servers‣Specialists (testing, SMEs)‣Additional software (data comparison tools, etc)






Determine migration methodology

• Phased approach‣ Lower risk‣Allows lessons learned to be used in next phase‣Requires both ETL tools to be in production

• Parallel adoption‣Develop entire data warehouse in ODI and run in parallel with OWB through user acceptance

testing‣Can be difficult to keep in-synch with ongoing development

• Big bang‣Convert entire data warehouse and “flip the switch”‣Downtime required






Perform the Migration

• Manually rebuild all Mappings in ODI• Automatically - let the code do the work‣Partial conversion - Inline Views‣Metadata migration - Utilize OWB metadata

• A mix of approaches will most likely be necessary






Manual conversion

• Manually build each Interface based on the OWB Mapping logic• Pros‣Complex Mappings may not be easily migrated automatically‣Evaluate and update transformations

• Cons‣ Time consuming‣ Introduces the “human factor”

- Failing to perform or omitting a task- Performing the task incorrectly- Performing an extra or non-required task- Performing tasks out of sequence






Manual migration - OWB to ODI






Inline Views - Partial Migration

• Extract the SQL from the OWB Mapping to use as the source of the ODI Interface ‣Generate Scripts‣PL/SQL Package

• David Allan (@i_m_dave) of the Oracle Data Integration team developed the Knowledge Module and process‣Blog post:

https://blogs.oracle.com/warehousebuilder/entry/odi_11g_simple_flexible_powerful‣Creates an inline view based on a SQL statement that

can be used in another Interface





https://blogs.oracle.com/warehousebuilder/entry/odi_11g_simple_flexible_powerful





Inline Views - Partial Migration - Why?

• Pros‣Move out of OWB and into ODI‣Begin all new development in ODI‣Can gradually convert SQL as a Source Interfaces

• Cons‣Manually setup Topology, Project, Models, etc‣Complex Mappings will be difficult to migrate‣No metadata lineage






DemonstrationExtract OWB mapping SQL and load as inline view in ODI Interface






Metadata migration

• Utilize the OWB metadata‣Export MDL (Metadata Loader) file

• Import metadata into Oracle database schema‣Use ODI’s JDBC driver for XML

• Groovy script + ODI SDK‣Groovy scripts read the OWB metadata and create ODI

objects using the ODI SDK‣Build Topology, Models, Datastores, Interfaces, etc.

• Most objects will migrate, but not always 100%‣OWB and ODI are loaded with features, not all can be

migrated automatically






Metadata migration - Why?

• Automated migration will...‣Allow for full metadata lineage to remain intact‣Reduce errors

- Human factor errors go down to a minimum- Less rework & chasing errors caused by mistakes

‣Save time- Which in turn, saves cost- Allows focus on testing, where it is most needed






Export the OWB Metadata Loader (MDL) file






Load database schema using the JDBC driver for XML






DemonstrationMetadata migration using MDL, Groovy, and the ODI SDK






Test the migration






Test methodology

• Plan the test approach‣Compare with production (OWB) or validate independently‣Determine sample size‣Define exit criteria

• Always include Subject Matter Experts‣Understand the data and business rules‣Know common discrepancies & how to handle them

• Prepare for discrepancies‣Understand the differences between OWB and ODI

- Ensure all business rules were migrated‣ Timing of data gather






Testing...1, 2, 3

• Unit testing‣ Interface executes with no errors, data flows as expected‣Errors captured

• Integration testing‣Data not lost or corrupted

• Validation / User acceptance testing‣Data compares and is validated properly‣Performance acceptable

• Develop a process to automate testing‣Example: ODI Procedure to run compare after each load‣Store test results in a database schema for analysis‣Review results and mark items complete






In summary...

• Perform analysis & determine migration strategy• Determine the approach to use in making the move from OWB to ODI ‣Manually‣ Inline Views‣Metadata migration using MDL & Groovy

• Data validation and testing is the key to success - plan ahead

• More information can be found at http://www.rittmanmead.com• Contact us at [email protected] or [email protected]• Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/

rittmanmead)

Rittman Mead can help!












OWB - Making the Move to Oracle Data IntegratorMichael Rainey, Principal Consultant, Rittman MeadUKOUG, December 2012





OWB - Making the Move to Oracle Data Integrator

Technology

Transcript of OWB - Making the Move to Oracle Data Integrator