Best Practices for Migrating to MySQL -...

Best Practices for Migratingto MySQL

Best Practices for Migrating to MySQL

For existing Oracle/SQL Server environments

Ronald BradfordPrincipal - 42SQLhttp://42sql.com

Washington DC 18th Feb 2009

1. Reasons to Migrate to MySQL

Workshop Agenda

1. Reasons to migrate to MySQL

2. Ideal application candidates

3. Migration process overview

4. Migration assistance tools

5. Specific migration issues

6. Ideals for minimizing future migrations

7. Additional MySQL information

Session 1 Agenda

• Is migration possible?

• MySQL in the workplace

• MySQL client examples

• Where do I start with migration?

• Reasons to migrate

• Roadblocks to migration

Is Migration Possible?

The short answer is“Yes!”

Is Migration Possible?

• Short answer is Yes?

• MySQL is enterprise ready

• 14 year old product

• Provides commercial support and indemnity

• Well established to deliver success on web

• Not just a web product

• You interact with MySQL supported systems every day

MySQL in the workplace

• Web

• Web 2.0

• Telecom

• On Demand/SaaS

• Hosting

• Enterprise

• OEM/Embedded

• Gaming

Usual Suspects

• Facebook

• Wikipedia

• Craigslist

• Yahoo

• YouTube

• Omniture

• Travelocity

• Twitter

• GoDaddy

• Flickr

• Fotolog

• TicketMaster

Marketing Examples

NASA• Application

• - Informing the general public on contract opportunities, managing large acquisitions on line

• Key Business Benefit• - Saves NASA and its partners $4 million annually

• Catalyst to Migration Decision• - Previous database vendor restructured its license program, causing NASA

to be faced with fees that were “more than twice the total annual budget”

• Post Migration Results• - License fees were significantly lower

• - MySQL averaged 28% faster, compared to other databases

• - Running 3 years without any loss of data or down time

• - “The cost of the optional technical support for MySQL was a fraction of that for the product previously used”

http://mysql.com/news-and-events/generate-article.php?type=ss&id=nasa

The Weather Channel

• Software & Hardware overhaul

• Reported

• 30% increased capacity

• 50% decreased cost

• Sun Solaris to x86 commodity H/W & Linux

• Websphere to Tomcat

• Oracle to MySQL

http://mysql.com/customers/customer.php?id=22

Google

• "Google runs critical business systems with InnoDB and MySQL. The systems require 24×7 operation with minimal downtime. The systems support large OLTP and reporting workloads. We are very happy with the scalability, reliability and manageability of this software."

http://www.innodb.com/innodb/users/

Embedded Deployment

• Easy not to realize where MySQL is in use

• e.g. Support systems

• Documentation

• Issue tracking

• Publication

• Ad tracking

MySQL and Wordpress

• New York Times

• CNN

• Fox News

• Wall Street Journal

• Reuters

• Time Inc

• Financial Times

• Rolling Stone Magazine

http://wordpress.com/notable-users/

• Harvard

• NASA

• Xerox

• Tech Crunch

• Nancy Pelosi

• Stephen Colbert

MySQL in the Enterprise

• Dow Jones

• NASDAQ

• Reuters

• ESPN

• NBC/GE

• New York Times

• Comcast

Embedded

•Cisco

•McAfee

•Adobe

MySQL Marketplace

•

Where do I start with Migration?

MySQL Migration References

• http://mysql.com/why-mysql/migration/

• White Papers

• http://mysql.com/why-mysql/white-papers/mysql_wp_oracle2mysql.php

• http://dev.mysql.com/tech-resources/articles/migrating-from-microsoft.html

• Customer Migrations

• http://www.mysql.com/customers/?migration

MySQL Migration References

• This workshop

• Migration overview

• Migration products

• Migration partners

• Ideal candidates

• Roadblocks

Reasons

Reasons to Migrate

• Cost / TCO

• e.g. Subscription v. License

• EOL H/W & S/W

• Integration into newer systems

• Asked to

• Forced to

• Physical resources

• Trend

Cost / TCO

• Truly difficult to value

• Software

• Hardware

• Support

• Skills

• Staff

• Can you state your $ per customer investment

Cost / TCO

• Easy when starting a new project now

• Open Source

• Zero entry point - Licenses

• Zero entry point - Subscription (if desired)

• VC preferred

• But

Cost / TCO

•Less measurable incidentals

•Staff skills / proficiency

•Infrastructure

• e.g. H/W

• Integration points

• Reference material / procedures

EOL

• Newer H/W is significantly cheaper

• More verses Larger

• Growth, Redundancy, Co Location

• Existing licensing may be prohibitive

• Potential environmental impact

Going Green

• "We do worry about how to maximize page views for kilowatt hours"

• Jim Buckmaster, CEO Craigslist

• Page views per kilowatt hour: The socially responsible benchmark?

http://blogs.zdnet.com/BTL/?p=3434

Integration

• Newer deployed systems for additional purposes

• More efficient to retool old systems to integrate with new

Acquisition

• Your company is acquired by another with different technology infrastructure

• Not likely in government situation

Forced to

• Management Decision

• e.g. "No new net Oracle licenses"

• New government initiative ?

Physical Resources

• Newer languages being taught

• Desire to update skills

• New staff acquisition

• Off shoring / outsourcing options

Open Source - Gardner

• By 2012, 80% of all commercial software will include elements of open-source technology.

• By 2012, at least one-third of business application software spending will be as service subscription instead of as product license.

• By 2011, early technology adopters will forgo capital expenditures and instead purchase 40 per cent of their IT infrastructure as a service.

http://gartner.com/it/page.jsp?id=593207

Open Source

• Sun releases Java

• Microsoft releases .Net components

• Adobe release Flex/Flash infrastructure

• MySQL, the 'M' of LAMP

• Be wary of different licensing agreements for software reuse

Road blocks

Barriers to success

• Resource commitment

• Set expectations

• e.g. it works in Oracle, it should work in MySQL

• It may be cheaper to migrate but it takes a long time

• Fear of change

• Licensing/IP/Patents/Copyright

Actual Client Example

• Management directs use of MySQL

• Technical resources

• Refuse to use optimal numeric datatypes (could be 25% disk saving)

• Refuse to use covering indexes (improved performance in queries)

• Fudge production simulation (3 full table scans in 15 sample SELECT's)

• Place junior inexperienced staff (in company and technology) on evaluation process

Summary

• MySQL can operate in the enterprise

• There are justifiable reasons to consider MySQL

• Migration to MySQL is possible

• Opportunities, Technology and Time to Market becoming more viable


• For existing Oracle/SQL Server environments




Workshop Agenda


2. Ideal Application Candidates





7. General MySQL Information

Session 2 Agenda

• Good Candidates

• Possible Candidates

• Easy Applications

• Not Ideal Candidates

• Are you serious?

• Challenge

MySQL in the workplace

• Web

• Web 2.0

• Telecom

• On Demand/SaaS

• Hosting

• Enterprise

• OEM/Embedded

• Gaming

Good Candidates

• Not heavy DB specific Logic

• e.g. PL/SQL, MS T-SQL

• Defined API Structure (e.g. MVC design)

• Have testing infrastructure

• Newer development languages

• Using ORM

• Native MySQL connector

Good Candidates

• No new development happening

• e.g. Focus is system redesign

• Data is partitioned / sharded

• e.g. No single large data storage

• Have capacity for downtime

• e.g. Support migration/upgrade process

Good Candidates

• No end user differences detected

• e.g. Client presentation unaffected

• Web based?

Possible Candidates

• Migration to existing MySQL based software product

• Issues

• Application feature match

• Data migration

• Staff retraining

• Documentation

Easy Applications

• Data Warehousing

• e.g. Driven by batch process

• Extract, Transform, Load (ETL)

• Business Intelligence (BI)

• Read only scalability

• Published information

• Self service content

Easy Applications

• Less critical systems

• Minimize resistance

• Build creditability in management and business

• Build staff skills, knowledge, confidence and expertise in new discipline

Not Ideal Candidates

• Heavy DB Specific Logic

• e.g. Stored Procedures, Triggers, Functions

• Lack of logic/presentation separation

• Complex authentication models

• No testing infrastructure

• Heavy investment in staff operations, documentation, infrastructure

Not Ideal Candidates

• Specific features in use

• millisecond/microsecond (no native support)

• Multiple procedural languages (patch)

• Tight integration

• into authentication models

• associated stack tools

Are you serious?

• Play to the strengths of MySQL

• Client example

• Refused to optimize numeric datatypes

• Refused to implement covering indexes

• Wrote false production simulations

• Chose obscure differences (e.g. 15/16)

• Assigned junior & inexperienced staff

Challenge

• Commit to deploying MySQL public facing

• Try

• Self service content

• Access to online publications

• Customer/client information gathering

• Customer/client feedback

Summary

• MySQL can run important systems

• There is a point to cost & efficiency

• Evaluation is a complex process

• The greatest barrier is perception

Workshop Agenda



3.Migration Process Overview





Session 3 Overview

• What to migrate

• Migration approaches

• Critical components

What to migrate?• Database Schema/Objects

• Data

• Application

• Administration

• e.g. Monitoring

• Development Process

• e.g. Developer tools

• Skill set

• Philosophies

• Terminology

Database Schema

• Tables

• Columns

• Indexes

• Constraints

• Views

• Relatively straight forward

• Real

Database Schema

• Synonyms

• Materialized Views

• Snapshots

• Database Links

•No native support, rework necessary

Database Objects

• Stored Procedures

• Functions

• Triggers

• Possibly Complex, relative to current complexity

Data

• Data Extraction

• Data Injection

• Data Cleansing

• Refactoring back into existing source

• Data Validation

• Relatively straight forward

Application

• Can existing technology be adapted?

• e.g. No UI Changes

• Migration becomes syntax specific

• Any feature reduction?

• Moderate complexity depending on development approach

Application

• Change in client software technology

• Change in User Interface

• Most complex, effectively an entirely new system deployment

Administration

• System Monitoring

• Database Monitoring

• Backups

• Disaster Recovery

• Relatively complex

Development Process

• IDE tools

• SQL tools

• SQL Syntax

• Moderate new skills necessary

Skill set

• The style of things

• In addition to new development tools

• Different support systems

• Different support resources

Hardware

• Regardless of approach

• Additional (double?) hardware required

• Impacts

• Costs

• Infrastructure (location, power, backups, monitoring)

Migration Approaches

• Direct changeover

• Parallel changeover

• Hybrid changeover

Direct Changeover

• Overview

• Stop using existing system

• Start using new system

Direct Changeover

•Issues•Data migration time (downtime)

•Staff training must be done upfront (for UI changes)

•Testing must be throughout

• No going back

Parallel Changeover

• Overview

• Run both systems in parallel

• Functionality is duplicated across systems

Parallel Changeover

•Issues•Data consistency

• e.g. Double Entry

• Rework application to double write

• 3rd party replay tool

•Two way data sync is not feasible

Hybrid Changeover

• Overview

• Both systems in use

• Functionality migrates in controlled stages

• Restricted functionality in old system

Hybrid Changeover

• Issues

• Greater end user knowledge [may be] needed

• Further complexity in what features where

• Provides flexibility

• Heavily dependent on read/write functionality

Migration Impacts

• Downtime

• Time just to extract data

• e.g. 2-400GB from Oracle

• 1 hr acceptable window for extract & load

Migration Impacts

• Optimization

• All database products operate differently

• A well tuned Oracle/SQL Server system does not imply a well tuned MySQL system

• May not just be indexes or tuning

• Rework may be necessary

Critical Components

• Data Consistency

• Current Features

• Performance

• Growth Capacity

• Storage

• Features

• Physical Resources

Summary

• Data is the easy part of migration

• Staff/Skills can be the greatest unknown

• No User Interface changes is best

Workshop Agenda




4.Migration Assistance Tools




Session 4 Agenda

• MySQL Tools

• Third Party Tools

• Support Tools

MySQL Migration tool kit

• GUI Based

• Ideal first discovery tool

• Great for syntax, small data volume tests

• Not scriptable, (i.e. GUI tool)

• Not parallelizable

http://dev.mysql.com/downloads/gui-tools/

MS Access

MySQL

SQLServer Script files

Oracle

JDBC 3.0

Sybase

Informix DB2

Teradata

Migration Report

GUI

Scripting Interface

Migration Tool Kit Engine

Migration Toolkit – How Does it Work?

1. Connect to and Reverse Engineer Source2. Connect to MySQL Target3. Select Schemas/Objects to Migrate4. Customize MySQL Target Schemas/Objects5. Transform database Schemas/Objects to MySQL Schemas/Objects6. Move Data from database to MySQL

SQL Server/Oracle-to-MySQLObject Translation

Migration ToolkitSchema Database

Migration ToolkitTable Table

Migration ToolkitIndex Index

Migration ToolkitView View

manualStored

ProcedureStored

Procedure

manualFunction Function

manualTrigger Trigger

Golden Gate

• Transactional Data Management (TDM)

• High availability & heterogeneous

• Market 'Zero-downtime' Migrations

• Data, not application

http://www.goldengate.com/solutions/

Continuent

• Tungsten Enterprise

• Open Source

• Data Replication between sources

• Near real time rather then conversion

• No indication about initial conversion

http://www.continuent.com/solutions/mysql-solutions/replication-to-oracle

Continuent Tungsten

Inspirer

• SQLWays - Migration to MySQL

• Oracle & SQL Server

• DB2, SyBase, Informix

• Reported to support SP & Triggers migration

• Trial version, restricted to 'n' objects

http://www.ispirer.com/download/

DBMoto

• Provides Real-Time data replication

• Refresh

• Mirror

• Synchronization

• 30 day trial download

http://www.hitsw.com/products_services/dbmoto

Others

• Nob Hill Database Compare/Migration

• http://nobhillsoft.com

• WisdomForce - Fast Reader

• http://www.wisdomforce.com/products-FastReader.html

Review of Tools

• MySQL Migration Toolkit

• Golden Gate

• Continuent

• Inspirer

• DBMoto

Tools Review

• No tool will do a 100% migration

• Start with a detailed plan

• Try a simulated migration

• Database Objects

• Data (subset)

• Capture/Replay all SQL

• Evaluate Application impact necessary

MySQL in Oracle

• You can access MySQL directly from Oracle

• Oracle Heterogeneous Services

• Connect via MyODBC

• Read/Write with MySQL

Data Checksum

• Maatkit - Has number of support tools

• Table Checksum

• Checksum of MySQL data

http://www.maatkit.org/

Maatkit Internals

• Checksum Algorithm

• CHECKSUM - Build in MySQL

• BIT_XOR - cryptographic hash

• ACCUM - cryptographic hash

http://www.maatkit.org/doc/mk-table-checksum.html#--algorithm

Maatkit Internals

• Hash Function

• SHA1

• MD5

• CRC32

• FNV_64 - Custom C UDF

http://www.maatkit.org/doc/mk-table-checksum.html#--function

Support Tools

• Quest Toad Oracle/MySQL

• Quest Spotlight Oracle/SQL Server/MySQL

http://www.quest.com/database-management/

Support Tools

• Navicat Oracle/MySQL

• SQL Maestro Oracle/SQL Server/MySQL

http://www.navicat.com/

http://www.sqlmaestro.com/

MySQL End User Tools

• MySQL Administrator

• MySQL Query Browser

• SQLyog

• phpMyAdmin

• MySQL Workbench

Summary

• Migration Assistance of syntax/data

• Data consistency options between sources

• End User IDE/Developer Tools

• One tool does not do all

• A detailed plan is a necessary start

Workshop Agenda





5.Specific Migration Issues


7. Best practices round table discussion

Session 5 Agenda

• Oracle/SQL Server Specific Functionality

• Terminology

• MySQL Specifics

• MySQL Gotchas

Oracle Functionality

•Schema

•Millisecond/Microsecond

•BFILE

•Check Constraints

• MySQL supports FK’s (SE specific)

• MYSQL supports Range via ENUM

•CHECK (loc IN ('NEW YORK', 'BOSTON', 'CHICAGO')

Oracle Functionality

•Database Objects

•Materialized Views

•Synonyms

•System Triggers

•Multiple Triggers per type

• e.g. BEFORE|AFTER [table]

ANSI Standard SQL

• Oracle is not the ANSI Standard

• Neither is SQL Server

• Neither is MySQL

• e.g. (+) for outer joins is an Oracleism

• Developers often don't know this

Object Names

• Every product has different length characteristics

• MySQL - 64 bytes

• Oracle - 30 bytes

• SQL Server - 128 bytes

• MySQL by default is Case Sensitive Table Names

• e.g. Customer is not CUSTOMER

• Reserved words are allowed (when quoted)

http://dev.mysql.com/doc/refman/5.1/en/identifiers.html

Terminology

• Realize a term in one DB may have a different meaning

• e.g.

• Database

• User

MySQL Specifics

Indexes

• No function based indexes

• Covering indexes (performance)

• Partial length indexes (performance)

Optimizer

• Cost Based

• Rather different

• Query Execution Plan (QEP) may differ

• Statistics

• Sampling

MySQL Replication

• A strength

• MySQL Slaves can be used (e.g. Read Only) unlike Data Guard

• A weakness

• Promotion while possible is not seamless nor supplied feature

• Asynchronous

Data Dictionary

• INFORMATION_SCHEMA

• SELECT @@global_variable_name

• SHOW commands

• PROCESSLIST

• [GLOBAL|SESSION] STATUS

• [GLOBAL|SESSION] VARIABLES

Schema Admin

• ALTER TABLE is a blocking operation

Sequences

• MySQL does not have sequences

• Uses AUTO_INCREMENT

AUTO_INCREMENT

• Tied to a given table

• No native application wide scope

• Must be part of primary key

• Only one per table

• Easier to insert (simply use NULL)

• Harder to get current/next val

MySQL Gotchas

SQL_MODE

• Silent but deadly

• By default MySQL truncates invalid data

• SQL_MODE=STRICT_ALL_TABLES

• There is a ORACLE & MSSQL Mode

http://dev.mysql.com/doc/refman/5.1/en/server-sql-mode.html

Collations

• By default - latin1_swedish_ci

• 'a' = 'A'

• i.e. Not case sensitive

• Character/Collation available at

• Server / Schema / Table / Column

http://www.collation-charts.org/http://dev.mysql.com/doc/refman/5.1/en/charset.htmlhttp://dev.mysql.com/doc/refman/5.1/en/charset-charsets.html

Auto Commit

• Default is ON

• Not all SQL works in a transaction anyway

MySQL Storage Engines

• One of most difficult new concepts

• Transactional

• Non Transactional

• Best Practice is to stick with Transactional

Interesting Storage Engines

• CSV (Drop in CSV)

• BLACKHOLE (Very fast)

• 5.1 Pluggable

• FILE (Read File System, e.g. /proc/meminfo)

Similar looking Syntax

• For Example: Isolation Level

• Oracle Default - READ_COMMITTED

• MySQL Default - REPEATABLE_READ

• MySQL READ_COMMITTED != Oracle

http://www.oracle.com/technology/oramag/oracle/05-nov/o65asktom.html

http://www.tomlauren.com/weblog/archives/000019.html

http://dev.mysql.com/doc/refman/5.1/en/set-transaction.html

Data Types

• MySQL has 9 Numeric Data Types

• More efficient storage

• Can be significant savings (e.g. 25%)

CHAR Example

• 'A_ _ _ _' CHAR(5) with 4 trailing spaces

• Oracle stores with trailing spaces

• SQL Server stores with trailing spaces

• MySQL does not

• Oracle length is 5 - LEN()

• SQL Server length is 1 - LEN()

• MySQL length is 1 - LENGTH()

Dates

• No NLS_DATE_FORMAT

• Format is 'YYYY-MM-DD hh:mm:ss'

• MySQL has timezone support

String Functions

• LENGTH() not LEN()

• CONCAT(a,b) not a || b

Date Functions

• CURDATE() instead of SYSDATE

• NOW()

• DATE_PART()

• DATE_FORMAT()

Math Functions

• No MEDIAN()

• But you can write your own

• User Defined Functions (UDF)

• Written in C

• Loadable modules

SQL Commands

• Not in MySQL

• No INTERSECT or MINUS

• In MySQL

• REPLACE

• Multi Row INSERT VALUES

• DROP IF EXISTS

• SELECT INTO OUTFILE

Summary

• Don't get caught up in semantics

• Identify the actual roadblocks

• Functionality not supported

• Guaranteed to be differences

Workshop Agenda





5. Specific Migration Issues


7. Best practices round table discussion

Session 6 Agenda

• Migration Tips

Some Tips

Use an API

• Application Programming Interface

• Model, View, Controller (MVC) Design

• Every data access is via controlled means

• Provides centralization

• Enables better isolated testing

Testability

• Using an API

• Better control for regression

• Can be automated

Write Once Data

• Separation of data and attributes

• Generally largest volume

• Can be pre migrated

Language Agnostic

• Consider coding practices/styles

• SQL usage & syntax

Workshop Agenda





5. Specific Migration Issues


7.General MySQL Information

Session 7 Agenda

• MySQL Versions

• MySQL Enterprise

• MySQL Community

• MySQL Conference

• MySQL Cheat Sheets

• MySQL References

About MySQL Versions

• General Availability (GA)

• 5.1.31

• 5.0.67 (Community)

• 5.0.78 (Enterprise)

• Free to download (no registration reqd)

http://dev.mysql.com/downloads/

MySQL Enterprise

• Subscription Model (4 levels)

• Commercial Support

• Certified Binaries (not necessary for support)

• Monthly Critical Patch Releases

• Quarterly Service Packs

• MySQL Enterprise Monitor

• MySQL Query Analyzer

http://www.mysql.com/products/enterprise/

MySQL Community

• Blogs - http://planetmysql.org

• Forums - http://forums.mysql.com

• Forge - http://forge.mysql.com

• Lists - http://lists.mysql.com

• Developer Zone - http://dev.mysql.com

• Local User Groups

Local MySQL Community

• Columbia, MD

• http://mysql.meetup.com/145/

• Pikesville, MD


• Upper Malboro, MD


• Sterling, VA

• http://www.meetup.com/The-Sterling-MySQL-Meetup-Group/

New 3/14New 3/14

MySQL Conference

• April 20-23, 2009

• Santa Clara, Silicon Valley, CA

• Single largest MySQL Event

• 20% Discount code

http://www.mysqluc.com

MySQL Cheat Sheets

• Large consolidate cards

• MySQL Statements

• MySQL Functions

http://www.visibone.com/sql/

http://www.visibone.com/sql/

Oracle DBA References

• Learning MySQL as an Oracle DBA

• 1 Day Seminar

http://ronaldbradford.com/blog/learning-mysql-as-an-oracle-dba-2007-08-19

Best Practices for Migrating to MySQL -...

Documents

Transcript of Best Practices for Migrating to MySQL -...