Migrating between Proprietary and Open Source Databases Between...IBM DB2, Sybase ASE, Informix,...

22
Migrating between Proprietary and Open Source Databases Calculating your ROI Wednesday, May 29, 2019

Transcript of Migrating between Proprietary and Open Source Databases Between...IBM DB2, Sybase ASE, Informix,...

Migrating between Proprietary and

Open Source Databases

Calculating your ROI

Wednesday, May 29, 2019

© The Pythian Group Inc., 2018 2© The Pythian Group Inc., 2019 2

● About my employer and I

● A simple statement about database Migration

● My experiences in database Migration

● Some specific things to consider in modern database migration

● That simple statement again

Agenda

© The Pythian Group Inc., 2018 3© The Pythian Group Inc., 2019 3

PYTHIAN

A global IT company that helps businesses leverage disruptive technologies to better compete.

Our services and software solutions unleash the power of cloud, data and analytics to drive better

business outcomes for our clients.

Our 20 years in data, commitment to hiring the best talent, and our deep technical and business expertise

allow us to meet our promise of using technology to deliver the best outcomes faster.

© 2017 Pythian. Confidential 4

Years in Business

20Pythian Experts

in 35 Countries

400+Current Clients

Globally

350+

© The Pythian Group Inc., 2018 5© The Pythian Group Inc., 2019 5

DIVERSIFIED, BLUE CHIP CLIENT BASE

Media/Information

Services Retail E-commerce SaaSFinancial

Services

© The Pythian Group Inc., 2018 6© The Pythian Group Inc., 2019 6

45 years of working with data

Started with flat files on Tape drives

Over the years have worked with:

ISAM, Codasyl Network databases, IBM IMS, Tandem SQL/MP

IBM DB2, Sybase ASE, Informix, MySQL, Cassandra, MongoDB

PostgreSQL and many others

Helped to migrate thousands of databases at AOL from Sybase to MySQL

Today I travel around the world helping Pythian’s customers get the best return on their

database investments.

About me

© The Pythian Group Inc., 2018 7© The Pythian Group Inc., 2019 7

The fun stuff:

I like to travel

I love animals, Nature, National parks, Museums, Zoos and Gardens

I take a lot of pictures of flowers and landscapes. They are patient subjects

I take very few pictures of people. They are almost never patient subjects

About me Part II

© The Pythian Group Inc., 2018 8© The Pythian Group Inc., 2019 8

● Database migrations are usually hard work

● SQL is a language with many dialects

● Database servers are each designed with different design philosophies

● Operational tooling around each database varies in maturity and

functionality between proprietary and open source

● Documentation is often badly lacking in the open source community

● So…

● Look for an easy out

● Or Consider a complete re-write

● Or…. Well migrate

A Simple Opening Statement

© The Pythian Group Inc., 2018 9© The Pythian Group Inc., 2019 9

● We had a customer with 100+ SQL Server replicated clusters

● They asked us to help them migrate to an Open Source Database to

save on license costs

● We did an initial evaluation and offered them two options

● 1 – Spend 2 years and about 1 Million $ to migrate to PostgreSQL

or MySQL

● 2 – Create 10 up sized server clusters and migrate all of the 100+

databases to one for a few thousand $ from us to do the work. Then

re-negotiate with Microsoft for the new smaller footprint.

● They took choice two.

The Easy out

© The Pythian Group Inc., 2018 10© The Pythian Group Inc., 2019 10

● 15 years ago AOL having adopted Linux as their operating system

of choice decided to pick an Open Source database

● We picked MySQL to replace approximately 30 thousand Sybase

Server licenses

● We had a proxy we had developed with its own SQL dialect and it

looked like a fairly simple task of implementing a MySQL version

then putting it into place.

● We Overlooked Stored procedures.

● Stored Procedures in MySQL were a new thing then and we had

something like 15 thousand Sybase Stored procs

● It didn’t go well for many reasons.

● At the same time….

Some ancient history

© The Pythian Group Inc., 2018 11© The Pythian Group Inc., 2019 11

● All of our code at that point was pretty monolithic and written in

C/C++

● Most of our new hires had mostly Java backgrounds and

management was buying into the service oriented development

story.

● Time to market for new features was of course the big feature.

● So… We re-wrote about 300 applications into a few thousand

services

● Replacing all those Sybase databases with MySQL.

● How long did it take? Six years but the results were pretty good.

Ancient history continued

© The Pythian Group Inc., 2018 12© The Pythian Group Inc., 2019 12

● I have worked with several customers who decided to migrate from

either Oracle or SQL Server to an open source database.

● The happiest ones by far are the ones who decided do to a

complete re-write in the process.

● The ones doing migration have ended up in two categories:

● 1 – Fairly simple database with simple queries and no or few stored

procedures

● 2 – Large database, lots of tables, lots of queries, stored procs and

specialized functions.

● Group 1 completed their migrations

● Group 2 are still working on theirs or have given up.

Over the last Five years

© The Pythian Group Inc., 2018 13© The Pythian Group Inc., 2019 13

● Make sure you really want to go down this road

● Analyze your schema, queries, STPs etc

● Use SQLines (general purpose), Commercial package, AWS DMS/SCT,

Ora2pg (oracle to PG only)

● Work out a strategy to convert your schema and queries based on the analysis.

● Convert your schema and queries and test them against a test database

● Chang your applications if needed, to meet missing features in the new

database technology

● Copy your database from the old proprietary to the open source one. This

will be a point in time logical backup and reload.

● If you need a zero downtime migration then use a tool like Goldengate or

Continuent tungsten replicator to bring the new DB up to date.

So how do you do a migration

© The Pythian Group Inc., 2018 14© The Pythian Group Inc., 2019 14

● Cost

● Staff time to do the analysis and conversion

● Tools to help with the migration Golden Gate, Tungsten proprietary

conversion tool etc.

● Lost opportunity cost while you are doing the conversion and not

implementing new features

● Training on the new Database and tooling

● Increased risk

● Savings

● License cost

What are your your costs and savings

© The Pythian Group Inc., 2018 15© The Pythian Group Inc., 2019 15

● Oracle has some significant common features with PostgreSQL

● But… There are some major things you need should probably know

● Stored Procedures STPs

● PostgreSQL stored procedures aren’t Stored Procedures they are functions. May seem picky

point but…

● PostgreSQL STPs are not pre compiled when created. They are parsed and query planned at

first use on each new connection.

● PostgreSQL STPs are never really compiled hence they are significantly slower than Oracle

STPs almost always.

● Numeric

● Oracle and PostgreSQL both have the numeric type which allows an large number of

significant digits

So lets be honest this is probably about Oracle

© The Pythian Group Inc., 2018 16© The Pythian Group Inc., 2019 16

● Numeric

● Oracle is smart about how numeric is interpreted. If a numeric is defined less than 10 digits it

gets defined internally as a 4 byte integer.

● PostgreSQL always stores numeric fields as BCD value. So far as I know only IBM Z series

systems have “hardware” instructions to handle BCD. Otherwise it gets processed in

software.

● Operational tools.

● PostgreSQL operational tools are separate from the core server. Maintained by different

groups in the community. Often poorly documented.

● Example – Until PostgreSQL 9 replication was only available externally. Until PostgreSQL 10

only binary replication between the same versions of PostgreSQL could be used.

So lets be honest this is probably about Oracle

© The Pythian Group Inc., 2018 17© The Pythian Group Inc., 2019 17

● Partitioned tables

● In PostgreSQL 8 partitioned tables were introduced. The mechanism was elegant and also

kind of um… Stupid.

● In PostgreSQL 10 partitioned tables were re-introduced the right way and very similar to the

way they have worked in Oracle forever.

● Use the new way.

● Vacuum

● Oracle and PostgreSQL both are MVC engines but their implementations are very different.

● PostgreSQL uses a leave in place algorithm which requires cleanup using a tool called

vacuum. It is important to understand how it works and its impact on operations before the

database explodes or shuts down. See: https://www.percona.com/blog/2018/08/06/basic-

understanding-bloat-vacuum-postgresql-mvcc/

So lets be honest this is probably about Oracle

© The Pythian Group Inc., 2018 18© The Pythian Group Inc., 2019 18

● Connections

● PostgreSQL was developed before threads were invented.

● It still does not use them.

● Instead when a new connection request is made from a client a new

process is forked.

● You probably don’t want more than a few hundred connections to a

single PostgreSQL instance.

● Connection pooling tools most commonly pgBouncer are required to

allow for thousands of client connections.

So lets be honest this is probably about Oracle

© The Pythian Group Inc., 2018 19© The Pythian Group Inc., 2019 19

● If you are going to migrate from Oracle are you going to the cloud?

● You probably are:

● Do you want to deploy on base infrastructure or are you looking for DbaaS?

● Amazon you get RDS PostgreSQL, MySQL and others as well as Aurora flavors of

PostgreSQL and MySQL

● GCP offers CloudSQL with some fairly severe limitations including replication in and

out limitations

● Azure Database for PostgreSQL and MySQL

● All the DbaaS offerings offer built-in replication/HA and automatic backups all at the

push of a button or two.

● There are limits on some features of PostgreSQL including which extensions you

can install on all three cloud vendor DbaaS offerings.

And what about the cloud?

© The Pythian Group Inc., 2018 20© The Pythian Group Inc., 2019 20

● Database migrations are usually hard work

● SQL is a language with many dialects

● Database servers are each designed with different design philosophies

● Operational tooling around each database varies in maturity and

functionality between proprietary and open source

● Documentation is often badly lacking in the open source community

● So…

● Look for an easy out

● Or Consider a complete re-write

● Or…. Well migrate

A Simple Closing Statement

© The Pythian Group Inc., 2018 21

John Schulz

[email protected]

Thank You

© 2017 Pythian. Confidential 22