: Flexible Database Clustering Middleware

Usenix Annual Conference, Freenix track – June 2004 – [email protected] 1

: Flexible

Database Clustering Middleware

Emmanuel Cecchet – INRIA Julie Marguerite – ObjectWeb

Willy Zwaenepoel – EPFL


Internet

Database tier should be– scalable

– highly available

– without modifying the client application

– database vendor independent

– on commodity hardware

Motivations

JDBC

JDBCJDBC


Scaling the database tier – Alternative 1 (master-slave)

Internet

Web frontend

App. server

Cons– failover time on master failure– scalability

Master


Scaling the database tier – Alternative 2 (SMP)

Internet

Web frontend

App. server

Well-known database

vendor here

Database

Well-known hardware +database vendors here

Cons– Cost

– Scalability limit


Scaling the database tier – Alternative 3 (shared disks)

Internet

Web frontend

App. server Database Disks

Another well-knowndatabase vendor

Cons– still expensive hardware– availability


Outline

C-JDBC architectureHigh availabilityUse casesConclusion


architectural overview

JVM

Applicationserver

MySQL JDBC driver

MySQLdatabase

C-JDBC JDBC driver

C-JDBCcontroller

JVM


basic concepts

2 components– C-JDBC driver– C-JDBC controller– 100% Java implementation

Read-one, Write all approachTunable replication

– full partitioning– full replication– partial replication


C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log

Authentication Manager

MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver

Client application (Servlet, EJB, ...)


Functional overview

connect myDBconnect login, passwordexecute SELECT * FROM t


C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver



Functional overview

execute INSERT INTO t …


Outline



C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver



Failures

No 2 phase-commit– parallel

transactions– failed nodes are

automatically disabled

execute INSERT INTO t …


Restoring a backend

Updates stored in the recovery log

Database dumps associated to checkpoints

...

dump for last

checkpoint

PostgreSQL

C-JDBC Controller

PostgreSQL JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

dump for last

checkpoint

Recovery Log

Octopus

PostgreSQL PostgreSQLenabled

enabled

JDBC Recovery Log

enabledrestoring

dump for initial

checkpoint


Synchronization

Replay missing updates from log

dump for

checkpoint

Octopus...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



Recovery Log


enabled

JDBC Recovery Log

synchronizing enabled


Healed Cluster

Re-enable backend when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint


Vertical scalability

Addresses JVM scalability issues

Distributing large number of connections on many backends

DB 4DB 3

DB native JDBC driver

DB 5

C-JDBC driver

DB 1 DB 2


DB 6


DB7

C-JDBC controller Partial replication




C-JDBC driverJVM

Client program C-JDBC

driver

JVM

Client program

C-JDBC driver

JVM

Client program


Controller replication

Prevent the controller from being a single point of failure

Group communication for controller synchronization

MySQL PostgreSQL

C-JDBC Controller

MySQL JDBC driver


C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC Controller

JGroups

MySQLPostgreSQL

MySQL JDBC driver



Controller replication

C-JDBC Controller

Distributed Request Manager

C-JDBC Controller

Distributed Request Manager

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

Virtual database 2

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer


Oracle JDBC driver

OracleMySQL

MySQL JDBC driver

C-JDBC driver


C-JDBC driver



Request Manager

Query result cache

Scheduler

Load balancer


jdbc:cjdbc://node1,node2/myDB

Total order reliable multicast


Mixing horizontal & vertical scalability

DB 6DB 5


DB 7

C-JDBC driver

DB 1 DB 2


DB 3


DB 4

C-JDBC controller Full replication




C-JDBC driverJVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

C-JDBC driver


Current limitations

Replication granularity is tableNo distributed joinsNetwork partition with replicated

controllersJDBC only

– support of PHP, Perl, ODBC through wrappers or bridges– partial support of JDBC 3.0


Other features

SSL supportSupport for heterogeneous databasesSQL monitoringJMX based administration consoleRequest player


Outline



Budget High Availability

High availability infrastructure “on a budget”

Typical eCommercesetup

http://www.budget-ha.com


OpenUSS: University Support System

eLearningHigh availabilityPortability

– Linux, HP-UX, Windows

– InterBase, Firebird, PostgreSQL, HypersonicSQL

http://openuss.sourceforge.net

Firebird

C-JDBC Controller Full replication

Firebird JDBC driver

Firebird

C-JDBC driver

JVM

C-JDBC driver

JVM


Firebird JDBC driver

C-JDBC driver

JVM

JOnAS J2EE server

C-JDBC driver

JVM

Enhydra server

OpenUSS OpenUSS OpenUSS OpenUSS

Enhydra Director

Apache

JOnAS J2EE server

Enhydra server

JOnAS J2EE server

Enhydra server

JOnAS J2EE server

Enhydra server

Enhydra Director

Apache

Enhydra Director

Apache


Flood alert system

Disaster recovery Independent nodes synchronized with C-JDBC VPN for security issues

http://floodalert.org

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

> 1000 miles


J2EE benchmarking

Large scaleJ2EE clusters

http://jmob.objectweb.org

Internet

emulated users


Outline



Conclusion

C-JDBC: Flexible Database Clustering Middleware

–scalable–highly available –without modifying the client application–database vendor neutral–on commodity hardware

LPGL software hosted by ObjectWeb


Q&A_________

Thanks to all users and contributors ...

http://c-jdbc.objectweb.org


TPC-W benchmark(Amazon.com)

Nearly linear speedups with the shopping mix

0

200

400

600

800

1000

1200

1400

1600

0 1 2 3 4 5 6

Number of nodes

Th

rou

gh

pu

t in

req

ues

ts p

er m

inu

te

Single Database

Full replication

Partial replication


Result cache

Cache contains a list of SQL->ResultSetPolicy defined by queryPattern->Policy3 policies

– EagerCaching: variable granularities for invalidations– RelaxedCaching: invalidations based on timeout– NoCaching: never cached

RUBiS bidding mixwith 450 clients

No cache Coherentcache

Relaxedcache

Throughput (rq/min) 3892 4184 4215

Avg response time 801 ms 284 ms 134 ms

Database CPU load 100% 85% 20%

C-JDBC CPU load - 15% 7%


Recovery log

All updates are stored in the recovery log

Database dumps associated to checkpoints

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled


Making new checkpoints

Disable one backend to have a coherent snapshot

Mark the new checkpoint entry in the log

Use Octopus to store the dump

...

dump for last

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for last

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledcheckpointing

dump for initial

checkpoint



Replay missing updates from log

dump for

checkpoint

Octopus...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



Recovery Log


enabled

JDBC Recovery Log

synchronizing enabled



Re-enable backend when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint


Handling a backend failure

A node fails!Automatically

disabled but should be fixed or changed by administrator

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enableddisabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint


Fault tolerant recovery log

C-JDBC Controller

MySQL MySQL

Application virtual database

Database Backend

Database Backend

Request Manager

Request Cache

Scheduler

Load balancer

Recovery Log


MySQL

Database Backend

Recovery virtual database

Request Manager

SchedulerLoad balancer


C-JDBC driver


Checkpointing service

Database Backend

Database Backend

C-JDBC driver

MySQL

UPDATE statement

: Flexible Database Clustering Middleware

Documents

Transcript of : Flexible Database Clustering Middleware