Highly available web sites with Tomcat and Clustered JDBC Emmanuel Cecchet.

Highly available web sites with Tomcat and Clustered JDBC

Emmanuel Cecchet

http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 2 - 15/11/2004

Internet

Database tier should be scalable highly available without modifying the client application database vendor independent on commodity hardware

Motivations

JDBC

JDBCJDBC


Scaling the database tier – Alternative 1 (master-slave)

Internet

Web frontend

App. server

Consfailover time on master failurescalability

Master


Internet

Consatomic broadcast scalabilityno client side load balancingheavy modifications of the database engine

Scaling the database tier – Alternative 2 (atomic broadcast)

Atomicbroadcast


Scaling the database tier – Alternative 3 (SMP)

Internet

Web frontend

App. server

Well-known database

vendor here

Database

Well-known hardware +database vendors here

Cons Cost Scalability limit


Scaling the database tier – Alternative 4 (shared disks)

Internet

Web frontend

App. server Database Disks

Another well-knowndatabase vendor

Consstill expensive hardwareavailability


Outline

RAIDbC-JDBCTomcat and C-JDBCScalabilityHigh availability


RAIDb concept

Redundant Array of Inexpensive DatabasesRAIDb controller

gives the view of a single database to the clientbalance the load on the database backends

RAIDb levels offers various tradeoff of performance and fault tolerance


RAIDb levels

RAIDb-0partitioningno duplication and no fault toleranceat least 2 nodes

table 2 & 3 table ...

RAIDb controller

table n-1table 1 table n

SQL requests


RAIDb levels

RAIDb-1mirroringperformance bounded by write broadcastat least 2 nodes

Full DB Full DB

RAIDb controller

Full DBFull DB Full DB

SQL requests


RAIDb levels

RAIDb-2partial replicationat least 2 copies of each table for fault toleranceat least 3 nodes

table x table y

RAIDb controller

table x & yFull DB table z

SQL requests


Outline



C-JDBC overview

Middleware implementing RAIDb100% Java implementationopen source (LGPL)

Two componentsgeneric JDBC driver (C-JDBC driver)C-JDBC Controller

Read-one, Write all approachprovides eager (strong) consistency

Supports heterogeneous databases


architectural overview

JVM

Applicationserver

C-JDBC JDBC driver

C-JDBCcontroller

JVM

MySQL JDBC driver

MySQLdatabase


Inside the C-JDBC Controller

XML configuration file

C-JDBC Controller

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

XML engine

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Configuration &

administrationAdministration console

RMIRMI

Recovery Log

Authentication Manager

MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Recovery Log


Oracle

Oracle JDBC driver

Oracle JDBC driver

Oracle

SocketsSockets

JMX


Virtual Database

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

gives the view of a single database

establishes the mapping between the database name used by the application and the backend specific settings

backends can be added and removed dynamically

configured using an XML configuration file



MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Matches real login/password used by the application with backend specific login/ password

Administrator login to manage the virtual database


Scheduler

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Manages concurrency controlSpecific implementations for

RAIDb 0, 1 and 2Pass-throughOptimistic and pessimistic

transaction leveluses the database schema that is

automatically fetched from backends


Request cache

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

3 optional cachestunable sizes

parsing cacheparse request skeleton only onceINSERT INTO t VALUES (?,?,?)

metadata cachecolumn metadatafields of a request

result cachecaches results from SQL requeststunable consistencyfine grain invalidationsoptimizations for findByPk requests


Load balancer 1/2

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

RAIDb-0query directed to the backend having the

needed tables

RAIDb-1read executed by current threadwrite executed in parallel by a dedicated

thread per backendresult returned if one, majority or all commitif one node fails but others succeed, failing

node is disabled

RAIDb-2same as RAIDb-1 except that writes are

sent only to nodes owning the updated table


Load balancer 2/2

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Static load balancing policiesRound-Robin (RR)Weighted Round-Robin (WRR)

Least Pending Requests First (LPRF)

request sent to the node that has the shortest pending request queue

efficient even if backends do not have homogeneous performance


Connection Manager

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver provides transparent connection pooling

Connection pooling for a backend

no poolingblocking poolnon-blocking pooldynamic pool

Connection pools defined on a per login basis

resource management per logindedicated connections for admin


Recovery Log

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Checkpoints are associated with database dumps

Record all updates and transaction markers since a checkpoint

Used to resynchronize a database from a checkpoint

JDBCRecoveryLogstore log information in a databasecan be re-injected in a C-JDBC

cluster for fault tolerance


C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver

Client application (Servlet, EJB, ...)


Functional overview (read)

connect myDBconnect login, passwordexecute SELECT * FROM t


C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver



Functional overview (write)

execute INSERT INTO t …


C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver



Failures

No 2 phase-commitparallel

transactionsfailed nodes are

automatically disabled

execute INSERT INTO t …


Outline



Internet

Apache clustering L4 switch, RR-DNS, One-IP techniques, LVS, …

Tomcat clustering mod_jk (T4), mod_proxy/mod_rewrite (T5), session replication

Database clustering C-JDBC

Highly available web site

mod-jkRR-DNS

Parsing cacheResult cache

Metadata cache


Result cache

Cache contains a list of SQL→ResultSetPolicy defined by queryPattern→Policy3 policies

EagerCaching: variable granularities for invalidationsRelaxedCaching: invalidations based on timeoutNoCaching: never cached

RUBiS bidding mixwith 450 clients

No cache Coherentcache

Relaxedcache

Throughput (rq/min) 3892 4184 4215

Avg response time 801 ms 284 ms 134 ms

Database CPU load 100% 85% 20%

C-JDBC CPU load - 15% 7%


Internet

Multiple databaseschoosing RAIDb levelrecovery log for

adding nodes dynamically recovering from failures

Highly available web site


Configuring Tomcat 5

Same setting for all Tomcat nodes

Copy c-jdbc-driver.jar to $CATALINA_HOME/common/lib

Without connection pooling1. load driver

Class.forName("org.objectweb.cjdbc.driver.Driver").newInstance();

2. get a connectionConnection conn = DriverManager.

getConnection("jdbc:cjdbc://host/myDB","user","password");

Configuring Tomcat 5

With connection pooling

<ResourceParams name="jdbc/TestDB"> <parameter> <name>factory</name> <value>org.apache.commons.dbcp.BasicDataSourceFactory</value> </parameter>

 <parameter> <name>driverClassName</name> <value>org.objectweb.cjdbc.driver.Driver</value> </parameter>  <parameter> <name>url</name> <value>jdbc:cjdbc://localhost/myDB</value> </parameter>


Configuring C-JDBC (1/3)

Virtual database configuration file

<?xml version="1.0" encoding="UTF8"?><!DOCTYPE C-JDBC PUBLIC "-//ObjectWeb//DTD C-JDBC 1.0.5//EN"

"http://c-jdbc.objectweb.org/dtds/c-jdbc-1.0.5.dtd">

<VirtualDatabase name=“myDB"> <AuthenticationManager> <Admin…> <VirtualUsers> <VirtualLogin

vLogin="user“ vPassword=“password"/> </VirtualUsers> </AuthenticationManager>


Configuring C-JDBC (2/3)

<DatabaseBackend name="node1“ driver="org.gjt.mm.mysql.Driver“ url="jdbc:mysql://host1/backend1“ connectionTestStatement="select 1">

<ConnectionManager vLogin=“user" rLogin=“dbuser" rPassword=“dbpass">

<VariablePoolConnectionManager initPoolSize=“10" minPoolSize=“5" maxPoolSize="50“ idleTimeout="30" waitTimeout="10"/>

</ConnectionManager> </DatabaseBackend>

<DatabaseBackend name="node2" driver="org.gjt.mm.mysql.Driver" url="jdbc:mysql://host2/backend2" connectionTestStatement="select 1">

<ConnectionManager …> </DatabaseBackend>


Configuring C-JDBC (3/3) <RequestManager> <RequestScheduler> <RAIDb-1Scheduler level="optimisticTransaction"/> </RequestScheduler>

<RequestCache> <MetadataCache/> <ParsingCache/>

<ResultCache granularity="table" /> </RequestCache>

<LoadBalancer> <RAIDb-1> <RAIDb-1-LeastPendingRequestFirst/> </RAIDb-1> </LoadBalancer>

<RecoveryLog> <JDBCRecoveryLog

driver="org.gjt.mm.mysql.Driver" url="jdbc:mysql://host/recovery" login="user" password="">

</JDBCRecoveryLog> </RecoveryLog>

</RequestManager> </VirtualDatabase>


Internet

<VirtualDatabase name="myDB"><Distribution> <BackendRecoveryPolicy

backendName=“node1" recoveryPolicy="on"/> <ControllerName name=“controller2”/>

<BackendRecoveryPolicy/></Distribution>…

Controller replication


Outline



C-JDBC vertical scalability

allows nested RAIDb levelsallows tree architecture for scalable write

broadcastnecessary with large number of backendsC-JDBC driver re-injected in C-JDBC controller


DB 4DB 3

DB native JDBC driver

DB 5

C-JDBC driver

DB 1 DB 2


DB 6


DB7

C-JDBC controller Partial replication

C-JDBC controller Full replication



C-JDBC driverJVM

Client program C-JDBC

driver

JVM

Client program

C-JDBC driver

JVM

Client program

Vertical scalability

Addresses JVM scalability issues

Distributing large number of connections on many backends


TPC-W benchmark(Amazon.com)

Nearly linear speedups with the shopping mix

0

200

400

600

800

1000

1200

1400

1600

0 1 2 3 4 5 6

Number of nodes

Th

rou

gh

pu

t in

req

ues

ts p

er m

inu

te

Single Database

Full replication

Partial replication


Outline




Prevent the controller from being a single point of failure

Group communication for controller synchronization

C-JDBC driver supports multiple controllers with automatic failover

MySQL PostgreSQL

C-JDBC Controller

MySQL JDBC driver

PostgreSQL JDBC driver

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC Controller

JGroups

MySQLPostgreSQL

MySQL JDBC driver


jdbc:c-jdbc://node1:25322,node2:12345/myDB



C-JDBC Controller

Distributed Request Manager

C-JDBC Controller

Distributed Request Manager

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

Virtual database 2

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer


Oracle JDBC driver

OracleMySQL

MySQL JDBC driver

C-JDBC driver


C-JDBC driver



Request Manager

Query result cache

Scheduler

Load balancer


jdbc:cjdbc://node1,node2/myDB

Total order reliable multicast


Mixing horizontal & vertical scalability

DB 6DB 5


DB 7

C-JDBC driver

DB 1 DB 2


DB 3


DB 4





C-JDBC driverJVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

C-JDBC driver


Building initial checkpoint

Octopus is an ETL tool Use Octopus to store a dump of

the initial database state Currently done by the user using

the database specific dump tool

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

disabled

dump for initial

checkpoint

Recovery LogOctopus

disabled


Logging

Backend is enabled All database updates are logged

(SQL statement, user, transaction, …)

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



enabled

dump for initial

checkpoint

Recovery LogOctopus

JDBC Recovery Log

enabled


Adding new backends 1/3

Add new backends while system online

Restore dump corresponding to initial checkpoint with Octopus

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus

PostgreSQL PostgreSQLdisabled disabledenabled

enabled

JDBC Recovery Log



Replay updates from the log

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus

PostgreSQL PostgreSQLenabled

enabled

JDBC Recovery Log

disableddisabled



Enable backends when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled


Making new checkpoints (1/3)

Disable one backend to have a coherent snapshot

Mark the new checkpoint entry in the log

Use Octopus to store the dump

...

dump for last

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for last

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enableddisabled

dump for initial

checkpoint



Replay missing updates from log

dump for

checkpoint

Octopus...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



Recovery Log


enabled

JDBC Recovery Log

disabled enabled



Re-enable backend when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint


Fault tolerant recovery log

C-JDBC Controller

MySQL MySQL

Application virtual database

Database Backend

Database Backend

Request Manager

Request Cache

Scheduler

Load balancer

Recovery Log


MySQL

Database Backend

Recovery virtual database

Request Manager

SchedulerLoad balancer


C-JDBC driver


Checkpointing service

Database Backend

Database Backend

C-JDBC driver

MySQL

UPDATE statement


Demo – xPetstore/HSQL

backend1

HypersonicSQLin-memorydatabase

xPetstore Servlet

http://xpetstore.sourceforge.net/open source implementation of PetstoreServlet solution: Velocity, WebWork, Sitemesh, POJO

and Hibernate.


Demo 2 – xPetstore/C-JDBC/HSQL

backend1

backend2

HypersonicSQLin-memorydatabases

JMX

Recovery log

RAIDb-1

xpetstore Servlet

C-JDBC administration console


C-JDBC today

Web site~200.000 hits/month~ 26.000 downloadsEU (18 countries) 36%, US 28%, Japan 12%, China

5%, Canada 4%, Australia 4%, India 3%, Brazil 2%, …

Community27 committers both industrial & [email protected]: >200 subscribers, 200-300

msgs/monthtranslation in japanese, italian, chinese, turkish, french

RPM on JPackage.org


What’s next?

Tribe (.objectweb.org)replacement for JGroupsuniform total order broadcast optimized for clusters

LeWYS (.objectweb.org)hardware and software monitoringmonitoring repository

Distributed query executionX509 certificatesOptimized support for edge-side servers and

interconnected clusters


Conclusion

RAIDbRAID-like scheme for databases

C-JDBCopen source middleware for database replicationperformance scalabilityhigh availability

Tomcat & C-JDBCno application modification requiredRDBMS vendor independent


Q&A_________

Thanks to all users and contributors ...

http://c-jdbc.objectweb.org

Bonus slides


Current limitations

JDBC onlyDistributed joinsUpdatable ResultSetsXA support through XAPool onlytransparent controller failover not supported

when using horizontal scalability with JGroupsnetwork partition/reconciliation not supported

HORIZONTAL SCALABILITY


Horizontal scalability

JGroups for controller synchronizationGroups messages for writes only

DB1 DB2 DB3 DB4

Controller1

DB1 DB2 DB3 DB4

Controller1 Controller2

Client 1 Client... Client n Client 1 Client... Client n



Centralized write approach issues

Issues with transactions assigned to connections

DB1 DB2 DB3 DB4


commit

DB1 DB2 DB3 DB4


commit

commit commit

DB1 DB2 DB3 DB4

Controller2

? ? ? ?

DB1 DB2


DB1 DB2


T1(Read)

T1(R)

DB1 DB2


T1(Write)

T1(R)T1(W) T2(W)



General case for a write query3 multicast + 2n unicast

DistributedVirtualDatabase@controller2

DistributedRequestManager





execWriteRequest(AbstractWriteRequest)

Scheduler

Write query handler

Load Balancer

Scheduler

Load Balancer

Distributed scheduling handler

Wait for schedulers replies

Send request to all controllers

Send execute request order to all controllers

Wait for load balancer replies

Distributed load balancer handler

Scheduler

Load Balancer



Notify controllers of completion success or not



Notify scheduler of completion

Return result

return result





Solution: No backend sharing1 multicast + n unicast [+ 1 multicast]

DB1 DB2 DB3 DB4 DB5 DB6

Controller1 Controller2 Controller3






execWriteRequest(AbstractWriteRequest)DistributedRequestManager

Scheduler

Write query handler

Load Balancer

Scheduler

Load Balancer


Send request to all controllers

Wait for load balancer replies


Scheduler

Load Balancer



Notify controller(s) that have failed if other have succeeded



Wait if all backends failed

Return result

return result





Issues with JGroupsresources needed by a channelinstability of throughput with UDPperformance scalability

TCP better than UDP butunable to disable reliability on top of TCPunable to disable garbage collectionordering implementation is sub-optimal

Need for a new group communication layer optimized for cluster



JGroups performance on UDP/FastEthernet

USE CASES


Budget High Availability

High availability infrastructure “on a budget”

Typical eCommercesetup

http://www.budget-ha.com


OpenUSS: University Support System

eLearningHigh availabilityPortability

Linux, HP-UX, Windows

InterBase, Firebird, PostgreSQL, HypersonicSQL

http://openuss.sourceforge.net

Firebird

C-JDBC Controller Full replication

Firebird JDBC driver

Firebird

C-JDBC driver

JVM

C-JDBC driver

JVM


Firebird JDBC driver

C-JDBC driver

JVM

JOnAS J2EE server

C-JDBC driver

JVM

Enhydra server

OpenUSS OpenUSS OpenUSS OpenUSS

Enhydra Director

Apache

JOnAS J2EE server

Enhydra server

JOnAS J2EE server

Enhydra server

JOnAS J2EE server

Enhydra server

Enhydra Director

Apache

Enhydra Director

Apache


Flood alert system

Disaster recovery Independent nodes synchronized with C-JDBC VPN for security issues

http://floodalert.org

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

> 1000 miles


J2EE benchmarking

Large scaleJ2EE clusters

http://jmob.objectweb.org

Internet

emulated users

PERFORMANCE


TPC-W

Browsing mix performance


TPC-W

Shopping mix performance


TPC-W

Ordering mix performance

Highly available web sites with Tomcat and Clustered JDBC Emmanuel Cecchet.

Documents

Transcript of Highly available web sites with Tomcat and Clustered JDBC Emmanuel Cecchet.