Data Migrations: How to avoid the #1 project pitfall
-
Upload
rtts -
Category
Technology
-
view
739 -
download
3
description
Transcript of Data Migrations: How to avoid the #1 project pitfall
built by
Bill HaydukCEO/President
RTTS
Data Migrations: How to avoid the #1 project pitfall
Christopher ThompsonSenior Domain Expert
RTTS
Laura PoggiMarketing Manager
RTTS
Webinar
Today’s Agenda
• Data Migration o Business Reasons o Riskso Costso Scary Metricso Common Defects
• #1 Project Pitfall
• QuerySurge Solution
• Demo
built by
AGENDA
Data Migration
o Business Reasons o Riskso Costso Scary Metricso Common Defects
#1 Project Pitfall
QuerySurge Solution
Demo
About
RTTS is the leading provider of software quality for critical business systems
FACTSFounded: 1996
Primary Focus:
consulting services, software
Locations: New York, Atlanta, Philly, Phoenix
Geographic region:North America
Customer profile:Fortune 1000, > 600 clients
Software:
QuerySurge
built by
80 percent of data migration projects fail to meet expectations, running over time and budget.
- Gartner
83 percent of migrations experience problems- IBM survey
80% of organizations… will underestimate the costs related to the data acquisition tasks by an average of 50 percent
- Gartner
Data Migrations
Data Migration – Top Reasons
built by
Top reasons for Data Migration• Decommissioning legacy systems
• Adding new line of business
• New deployment (i.e. ERP or CRM system)
• Consolidation of systems
• Relocation
• the average cost for a Data Migration = $875,000
• 34% of migrations have missing or lost data
• 38% have some form of data corruption
• 64% of migration projects have unexpected outage/downtime
• typical cost of downtime:
o $6.5 million/hour - Brokerage industry
o $2.8 million/hour - Energy Industry
• 72% of organizations deferred because data migration is too risky
Data Migration – Scary Metrics
built by
built by
• Unexpected downtime
• Budget overruns
• Customer or Brand Impact
• Data corruption
• Application performance issues
• Data loss
Data Migration – Key Risks
Key Risks
• Remediation activities
• Server outage
• Application outage
• Labor costs
• Software Tools
• System Maintenance
Data Migration – Cost Factors
built by
Key Cost Factors
Data Migration – Key Pitfalls
• Failing to engage the lines of business at the outset
• Absence of data governance policies and organizational structure
• Poor data quality in a legacy system
• Neglecting to validate and redefine business rules
• Failure to validate and test the data-migration process
Which is the #1 Pitfall?built by
Key Pitfalls of Migration Projects
• Failure to validate and test the data-migration process
"You really need to make sure that you're validating and testing throughout the process”.
“Unfortunately companies often don't spend enough time aligning the data testing, validation, and migration cycles to the project timeline”. - InformationWeek
Question:
How are you going to test the data?
Failure to validate and test the process
Data Migration – Popular Test Strategy
Post-Migration Testing• Row Counts• Comparing migrated records to
source using sampling
Question:
Is there a better way?
built by
12
Our Solution: Automated Testing of the
Migration Process
built by
QuerySurge is the
premier test tool built
to automate Data Testing
and the ETL Testing Process
What is QuerySurge?
built by
What does
QuerySurge ™do?
built by
QuerySurge finds bad data
• Most firms test < 1% of their data
• Databases have at best, untested data & at worst, bad data
• Bad data will cost companies between $8 - $100 million annually (Gartner)
• QuerySurge tests up to 100% of your data quickly & finds bad data
QuerySurge™ Architecture
built by
Target
Sources
Target Database
built by
Data Migration: Data Flow
Source Data Migration Process
ETL & EL
SQL
SQL
SQL
SQL
SQL
SQL
built by
Design Library Create Query Pairs (source & target queries)
17
QuerySurge™ Modules
Scheduling Build groups of Query Pairs Schedule Test Runs
Deep-Dive Reporting Examine and automatically
email test results
Run Dashboard View real-time execution Analyze real-time results
QuerySurge™ Modules
built by
automates the testing effort the kickoff, the tests, the comparison, emailing the results
speeds up testing up to 1,000 times faster than manual testing
tests across different platformsany JDBC-compliant db, DWH, DMart, flat file, XML, Hadoop
The QuerySurge solution…
built by
verifies more data verifies upwards of 100% of all data quickly
Automated Data Migration Strategy
built by
• Automated Row Counts• Automated Table Compares (no transformations)
#1: Automated testing with Data Wizards (no programming)
#2: Automated testing with custom QueryPairs (custom SQL)
• All tables with transformations are compared • Upwards of 100% coverage of all data quickly
SQL
SQLSQL
SQL
SQL
SQL SQL
SQL
2 Steps:
21
Fast and Easy. No programming needed.
built by
• Perfect for data migrations & upgrades
• Great for verifying that all rows have come over without data issues
• Superb at verifying table-to-table transfers
• Ability to graphically create SQL stubs for advanced scripting
Step #1:
Uses: Verify that the amount of rows match
the amount that were expected Great for smoke testing
built by
Result:
Quickly compares rows of all tables
SQL Skill set / Value added:
beginner: no coding needed
experienced user: saves time
Uses: - Verifies tables from source
and target match
built by
Result: Can perform 100s of tables in
a few minutes.
SQL Skill set / Value added: beginner: no coding neededexperienced user: saves
time
built by
Custom QueryPairs • Write Query against the source and against
the target• QuerySurge quickly compares tables after
transformation
Step #2:
QuerySurge Value-Add
QuerySurge provides value by either:
in testing data coverage from < 1% to upwards of 100%
in testing time by as much as 1,000 x
combination of in test coverage while in testing time
25built by
Return on Investment (ROI)
redeployment of head count because of an increase in coverage and decrease in need for testers
an increase in better data due to shorter / more thorough testing cycle, possibly saving $ millions by preventing bad data.
26built by
Ensuring Data Warehouse Quality
Demonstration
Christopher ThompsonSenior Domain Expert
RTTS
28
• SAP Data Migration: Turning Risk into Opportunity, Jerome Delune, EIM Global Practice Director, SAP Consulting
• Data Migration Best Practices and Nondisruptive Migration Service Capability for Enterprise Storage, Hitachi Data Systems, February 2012
• Risks and Challenges in Data Migrations and Conversions, Gartner, February 2009
• Data Migration – Key Considerations, Glen Bell, SearchDatamanagement.com, May, 2011
• Data Migration in the Global 2000, Bloor Research, September 2007
• Zero-Downtime Upgrades and Migrations with Oracle GoldenGate 11g, Oracle, 2009
• 10 Big Data Migration Mistakes, InformationWeek, August 9, 2012
• How to Implement an Effective Data Migration Testing Strategy, DataMigration Pro magazine, David Katzoff, Managing Director of Product Development, Valiance Partners
• The Hidden Costs of Data Migration, IBM, 2007
References
built by