Berkeley-Helsinki Summer Course Lecture #3: Middleware Architectures
Middleware for High Availability and Scalability in Multi-Tier and Service-Oriented Architectures
-
Upload
francisco-perez-sorrosal -
Category
Technology
-
view
15 -
download
1
description
Transcript of Middleware for High Availability and Scalability in Multi-Tier and Service-Oriented Architectures
Middleware for Middleware for High Availability and High Availability and
ScalabilityScalability in Multi-Tier and Service-in Multi-Tier and Service-
OrientedOrientedArchitecturesArchitectures
© Francisco Pérez-SorrosalAdvisor: Marta Patiño-Martínez
Distributed Systems Laboratory (DSL/LSD)
Universidad Politécnica de MadridMadrid, Spain
MotivationMotivation
Ph.D. Thesis © Francisco Pérez-Sorrosal
2
DataServices
Com
pany
A
Apps.
Stateful Data &Transactions
EXPENSIVE
May 13th 2009
MotivationMotivation
Ph.D. Thesis © Francisco Pérez-Sorrosal
3
Data
Services
Com
pany
B
Services
DataServices
Apps.
Apps.
Apps.
Data
StateConsistency
Cluster
Replica
May 13th 2009
MotivationMotivation
Ph.D. Thesis © Francisco Pérez-Sorrosal
4
Service A Com
pany
AC
ompa
ny BService B
Service C
Critical Service
CompositeApplication
May 13th 2009
OutlineOutline• High Availability (HA) and Scalability
in Multi-Tier Architectureso Protocols for High Availability in MTAso A Protocol for HA and Scalability in MTAs
• High Availability in Service-Oriented Architectureso WS-Replication Framework
• Conclusion• Publications
Ph.D. Thesis © Francisco Pérez-Sorrosal
5May 13th 2009
Multi-tier Multi-tier Architectures: Architectures:
MotivationMotivation• Great success of MTAs
o CORBA, .NET & J(2)EE
• Cache requires concurrency controlo Serializabilityo Synchronization with the
underlying database
• Many databases provideo Classical isolation levels +
Snapshot IsolationPh.D. Thesis © Francisco Pérez-Sorrosal
6
Cache
ApplicationServer
May 13th 2009
HA and Scalability in HA and Scalability in MTAs: ContextMTAs: Context
• J2EE application serverso Transactional Services:
• ACID Transactions (JTA)• Advanced Transactions (Activity Service) • Our implementation available at http:\\jass.objectweb.org
o Component Model: Enterprise Java Beans (EJBs)• Stateless (SLSB) and Stateful (SFSB) Session Beans, Entity
Beans (EB) & Message-Driven (MDB)
• When replicating EJBs:o SLSBs & MDBs don’t keep state => NOT Replicatedo SFSB beans keep client-related state across requests o EBs represent persistent data in a datasource
Ph.D. Thesis © Francisco Pérez-Sorrosal
7May 13th 2009
Horizontal Replication Horizontal Replication (DB Replication)(DB Replication)
Ph.D. Thesis © Francisco Pérez-Sorrosal
8
BOTTLENECK& SINGLE POINT
OF FAILURE
X X
(X)
X
ReplicationProtocol
May 13th 2009
Horizontal ReplicationHorizontal Replication(App. Server (App. Server Replication)Replication)
Ph.D. Thesis © Francisco Pérez-Sorrosal
9
BOTTLENECK& SINGLE POINT
OF FAILURE
(X)
X
(X) (X)
ReplicationProtocol
May 13th 2009
Horizontal ReplicationHorizontal Replication(AS and DB Replication)(AS and DB Replication)
Ph.D. Thesis © Francisco Pérez-Sorrosal
10
COMPLEX
(X) (X) (X)
X X X
ReplicationProtocols
May 13th 2009
Our Solution: Vertical Our Solution: Vertical ReplicationReplication
• No single bottleneck• No single point of failure• Only one replication protocol
Ph.D. Thesis © Francisco Pérez-Sorrosal
11
Unit ofReplication
ReplicationProtocol
May 13th 2009
OutlineOutline
• High Availability (HA) and Scalability in Multi-Tier Architectureso Protocols for High Availability in MTAso A Protocol for HA and Scalability in MTAs
• High Availability in Service-Oriented Architectureso WS-Replication Framework
• Conclusions• Publications
Ph.D. Thesis © Francisco Pérez-Sorrosal
12May 13th 2009
Protocols for HA in Protocols for HA in MTAsMTAs
• Consider session data (SFSBs) and persistent data (EBs)• Are transaction aware & mask failures transparently• Approach: Vertical Replication + Primary-Backup
Ph.D. Thesis © Francisco Pérez-Sorrosal
13
App.Server
DB
Primary
App.Server
DB
Backup
Client
ClusterPrimary
GCS
May 13th 2009
Our protocols offer...Our protocols offer...• Data consistency in all the replicas
o Vertical replication + transaction managemento 1-copy correctness
• Exactly-once executiono The client performs a request only once and gets the
results also only once
• High available transactionso The replication protocols are transaction-awareo Transactions are not aborted if the primary fails
• Different interaction patternso 1 Req/1 Tx, N Req / 1 Tx, 1 Req / N Txs and N Req / M Txs
Ph.D. Thesis © Francisco Pérez-Sorrosal
14May 13th 2009
N requests/1 transaction: N requests/1 transaction: GoalsGoals
• Support transactional conversations o Several client requests inside a single
transaction
• Upon failover, resume the conversation from the last interactionoDo not abort ongoing transactions
Ph.D. Thesis © Francisco Pérez-Sorrosal
15May 13th 2009
N requests/1 transactionN requests/1 transaction
Ph.D. Thesis © Francisco Pérez-Sorrosal
16
T1Client invocations inside T1
T2
Client invocation that requires a new TX T2
May 13th 2009
N Req / 1 TX Replication N Req / 1 TX Replication Protocol: Primary (Begin)Protocol: Primary (Begin)
Ph.D. Thesis © Francisco Pérez-Sorrosal
17
TM
Backup
Intercep
tors
Primary
beginTx
Backup
beginTx + TxId
Client
TxId
TxId
May 13th 2009
Replication Protocol: Replication Protocol: Backup (Begin)Backup (Begin)
Ph.D. Thesis © Francisco Pérez-Sorrosal
18
TxTable
Intercep
tors
Backup
beginTx + TxId
Store TxId
May 13th 2009
Replication Protocol: Replication Protocol: Primary (Invocation)Primary (Invocation)
Ph.D. Thesis © Francisco Pérez-Sorrosal
19
BackupIntercep
tors
Primary
EJB
EJB
EJB
EJB
Backup
TxId + Bean changes + Response
ClientTxIdResponse
May 13th 2009
Replication Protocol: Replication Protocol: Backup (Invocation)Backup (Invocation)
Ph.D. Thesis © Francisco Pérez-Sorrosal
20
Uncommitted Table
Intercep
tors
Backup
Response Table
Apply
Save
SFSB
TxId+ Bean changes + Response
EB
May 13th 2009
Replication Protocol: Replication Protocol: Primary (Commit/Abort)Primary (Commit/Abort)
Ph.D. Thesis © Francisco Pérez-Sorrosal
21
TM
BackupIntercep
tors
Primary
commit/ abort Tx
Backup
Commit/abort Tx + TxId
Client
EB DBEBEB
May 13th 2009
Replication Protocol: Replication Protocol: Backup (Commit/Abort)Backup (Commit/Abort)
Ph.D. Thesis © Francisco Pérez-Sorrosal
22
Uncommitted Table
Intercep
tors
Backup
Apply
SFSB
commitTx + TxId
EB
EBDBEB
EB
May 13th 2009
Replication Protocol: Replication Protocol: FailoverFailover
Ph.D. Thesis © Francisco Pérez-Sorrosal
23
Intercep
tors
BackupPrimary
TxTable
Uncommitted Table
Apply
SFSB EB
TM
For each non-completed Tx
...
Create
Response Table
Client
TxId
Response
NOT FOUND,
RE-EXECUTE REQUEST
May 13th 2009
Evaluation: ECPerfEvaluation: ECPerf
• Benchmark to evaluate the throughput and scalability of J2EE Application Servers
• Emulates the processes involved in a supply-chain management scenario
• The load is measured as the Injection Rate (IR)o # of clients = IR * 5
• Throughput is given in Benchmark Business Operations per Minute (BBOps/Min)
Ph.D. Thesis © Francisco Pérez-Sorrosal
24May 13th 2009
Experiment SetupExperiment Setup
• JBosso Non-replicated
• JBoss Primary-Backupo Only SFSB replicationo Shared DB
• Our replication protocolo Primary-Backup +
Vertical replicationo SFSB & EB replicationo Transaction awarePh.D. Thesis © Francisco Pérez-Sorrosal
25
J2EE App.Server
Host 2
Client
DB
Host 3
Host 1
J2EE App.Server
Host 2
Client
Cluster
DB
Host 3
Host 1
J2EE App.ServerBackup
Host 4
Cluster
J2EE App.Server
Host 2
Client
DB
Host 3
Host 1
J2EE App.ServerBackup
Host 4
DBBackup
Host 5
May 13th 2009
ECPerf: ThroughputECPerf: Throughput
Ph.D. Thesis © Francisco Pérez-Sorrosal
26
17
20%
21
May 13th 2009
ECPerf: Response TimeECPerf: Response Time
Ph.D. Thesis © Francisco Pérez-Sorrosal
27
ECPerf Limit
May 13th 2009
OutlineOutline
• High Availability (HA) and Scalability in Multi-Tier Architectureso Protocols for High Availability in MTAso A Protocol for HA and Scalability in MTAs
• High Availability in Service-Oriented Architectureso WS-Replication Framework
• Conclusions• Publications
Ph.D. Thesis © Francisco Pérez-Sorrosal
28May 13th 2009
Limitations of Current Limitations of Current Middleware for HA in Middleware for HA in
MTAsMTAs
• Mismatch between isolation at Application Server and DBMSo Current application servers do not work
correctly with SI databases
• Snapshot Isolation (SI) has become the “de-facto” standard isolation level
• Current middleware does not scale-out stateful applications consistently
Ph.D. Thesis © Francisco Pérez-Sorrosal
29May 13th 2009
Our Protocol for HA and Our Protocol for HA and Scalability in MTAs…Scalability in MTAs…
• Is consistent, high available and scalable• Includes a SI cache at the middleware
level for correctness and performance in a single replica
• SI cache is combined with replication for scalability and fault-tolerance in a cluster
• Vertical replicationo Only-one replication protocol coordinates the
execution of transactions and the propagation of changes in a cluster
Ph.D. Thesis © Francisco Pérez-Sorrosal
30May 13th 2009
Snapshot IsolationSnapshot Isolation
Ph.D. Thesis © Francisco Pérez-Sorrosal
31
Start Timestamp = 10Reads the version of TReads the version of TPrevious TX T has written X, so C=10
Start Timestamp = 11Reads X written by T2Validation succeedsIncrement C and set new version for X
New private version of XCounter of Committed TXs Conflict with T2Read-only Tx -> No validation
May 13th 2009
A Protocol for HA and A Protocol for HA and Scalability in MTAsScalability in MTAs
Ph.D. Thesis © Francisco Pérez-Sorrosal
32
App.Server
DB
Replica 1
App.Server
DB
Replica 2
Client
Cluster
Client
May 13th 2009
Protocol FeaturesProtocol Features• Transactions: Started at the same time in AS and DBS• SI Cache: Maintains a certain number of versions to
o Avoid accesses to the DBo Guarantee conflict detection
• Conflicts: o Locally: Detected on-the-fly (Pessimistic)o Remotely: Detected on a validation phase
• Other Issues: o Creation and Deletion of Components (CRUD Ops.)o Garbage Collectiono Session Replication o Failure Handling (Transparent failover of clients)o Replica Recovery
Ph.D. Thesis © Francisco Pérez-Sorrosal
33May 13th 2009
How the Multi-version How the Multi-version Cache WorksCache Works
Ph.D. Thesis © Francisco Pérez-Sorrosal
34
Y:bX:a
Update x
x = e
T1Update y Read xT2 Read yCommit
y = fx = a
X:e Y:f
y = b
T1: STS=10 T2: STS=10CTS=11
Ver =11 Ver =11Ver = -1 Ver = -1
Ccounter=10Ccounter=11
Start T1Start T2
Update y
conflict
Abort
abortcommit
Cache
Read x
May 13th 2009
Cache ReplicationCache Replication
Ph.D. Thesis © Francisco Pérez-Sorrosal
35
Replica 2Replica 1
X:a
Cache
Update x
x = e
T1 T2Commit
x= f
x = a
X:e
x = a
T1: STS=10 T2: STS=10CTS=11
Ver =11 Ver =11
Ver = -1 Ver = -1
Ccounter=10Ccounter=11
Start T1
Update x
Ccounter=10
Commit
Ccounter=11
X:a
X:e
Cachex = e
T1: STS=10
CTS=11GCS
T1: STS=10X=e
T2: STS=10 x = f
T2: STS=10 x = f
T1: STS=10 x = e
Conflict
Abort T2
Abort
Conflict
Commit Commit
Read x
May 13th 2009
Throughput Throughput (SPECjAppServer)(SPECjAppServer)
Ph.D. Thesis © Francisco Pérez-Sorrosal
36
Baselines
Our Protocol
May 13th 2009
Response Time: Read-only Response Time: Read-only TxnTxn
Ph.D. Thesis © Francisco Pérez-Sorrosal
37May 13th 2009
Response Time: Update Response Time: Update TxnTxn
Ph.D. Thesis © Francisco Pérez-Sorrosal
38May 13th 2009
OutlineOutline
• High Availability (HA) and Scalability in Multi-Tier Architectureso Protocols for High Availability in MTAso A Protocol for HA and Scalability in MTAs
• High Availability in Service-Oriented Architectureso WS-Replication Framework
• Conclusions• Publications
Ph.D. Thesis © Francisco Pérez-Sorrosal
39May 13th 2009
HA in SOA: MotivationHA in SOA: Motivation
• Some Web Services are critical for the interaction among organizations and should remain available despite failures
• WS-Replication Framework helps on replicating these critical Web Services
Ph.D. Thesis © Francisco Pérez-Sorrosal
40May 13th 2009
HA in SOA: The WS-HA in SOA: The WS-Replication FrameworkReplication Framework
• WS-Replication is a framework that eases the replication of WSso SOAP-based web services
• Properties:o Respects WS autonomyo Provides transparent fault-tolerance
• Components:o Deployer toolo WS-Multicast serviceo WS-Dispatcher
Ph.D. Thesis © Francisco Pérez-Sorrosal
41May 13th 2009
Background: Active Background: Active ReplicationReplication
Ph.D. Thesis © Francisco Pérez-Sorrosal
42
ServiceReplica 1
ServiceReplica 2
ServiceReplica 3
m2 m2 m2m1 m1 m1
Client Client
m1 m2
May 13th 2009
WS-Replication: WS-Replication: Invoking a Replicated Invoking a Replicated
Service IService I
Ph.D. Thesis © Francisco Pérez-Sorrosal
43
Replica 1
WS-Proxy
transport
WSWS-Dispatcher
WS Replica 2
WS-Proxy
transport
WS-Dispatcher
WS-Multicast
WS
WS-Multicast
WS
Client
May 13th 2009
WS-Replication: WS-Replication: Invoking a Replicated Invoking a Replicated
Service IIService II
Ph.D. Thesis © Francisco Pérez-Sorrosal
44
Replica 1
WS-Proxy
transport
WSWS-Dispatcher
WS Replica 2
WS-Proxy
transport
WS-Dispatcher
WS-Multicast
WS
WS-Multicast
WS
Client
May 13th 2009
WS-Replication WS-Replication Evaluation: SetupEvaluation: Setup
Ph.D. Thesis © Francisco Pérez-Sorrosal
45
Zurich
Bologna
Montreal
Madrid
Critical Service
Load Generator
May 13th 2009
WS-I & WS-CAF WS-I & WS-CAF IntegrationIntegration
Ph.D. Thesis © Francisco Pérez-Sorrosal
46
WS-CAF
WS-Rep.
Zurich
WS-CAF
WS-Rep.
Bologna
WS-CAF
WS-Rep.
Montreal
DB
Madrid Host 1 Madrid Host 2
WS-IApplic.
ClientEmulator
May 13th 2009
WS-CAF ReplicationWS-CAF Replication
Ph.D. Thesis © Francisco Pérez-Sorrosal
47
10%
May 13th 2009
WS-CAF ReplicationWS-CAF Replication
Ph.D. Thesis © Francisco Pérez-Sorrosal
48
May 13th 2009
OutlineOutline• High Availability (HA) and Scalability in
Multi-Tier Architectureso Protocols for High Availability in MTAso A Protocol for HA and Scalability in MTAs
• High Availability in Service-Oriented Architectureso WS-Replication Framework
• Conclusions• Publications
Ph.D. Thesis © Francisco Pérez-Sorrosal
49May 13th 2009
ConclusionsConclusions• We have developed a set of replication and
recovery protocols for providing consistent high availability and scalability to multi-tier applications
• Main contributions:o Transaction-aware replicationo Exactly-once execution of client requestso Deal with several interaction patternso Scalability through a replicated SI cache in the app.
servero Online recovery (Not presented because the lack of
time)• Results show that the proposed protocols are
affordablePh.D. Thesis © Francisco Pérez-Sorrosal
50May 13th 2009
ConclusionsConclusions• We have also developed a framework to
provide high availability to SOAs• WS-Replication provides seamless
replication to critical WSs• Adequate engineering proved to provide
affordable performance• Evaluation of a realistic application in WANs
has shown a quite reasonable overhead
Ph.D. Thesis © Francisco Pérez-Sorrosal
51May 13th 2009
OutlineOutline• High Availability (HA) and Scalability in
Multi-Tier Architectureso Protocols for High Availability in MTAso A Protocol for HA and Scalability in MTAs
• High Availability in Service-Oriented Architectureso WS-Replication Framework
• Conclusions• Publications
Ph.D. Thesis © Francisco Pérez-Sorrosal
52May 13th 2009
PublicationsPublications• Jorge Salas, Francisco Pérez-Sorrosal, Marta Patiño-Martínez and
Ricardo Jiménez-Peris. WS-Replication: a Framework for Highly Available Web Services. WWW, 2006.o Acceptance rate: 11 %o Percentile top 0 % in Microsoft’s Libra (WWW category)
• Francisco Pérez-Sorrosal, Marta Patiño-Martínez, Ricardo Jiménez-Peris and Bettina Kemme. Consistent and Scalable Cache Replication for Multi-tier J2EE Applications. Middleware, 2007. o Acceptance rate: 20 %o Percentile top 12 % in Microsoft’s Libra (Dist. And Parall. Computing
category)
• Francisco Pérez-Sorrosal, Marta Patiño-Martínez, Ricardo Jiménez-Peris and Jaksa Vuckovic. Highly Available Long Running Transactions and Activities for J2EE Applications. ICDCS, 2006.o Acceptance rate: 13 %o Percentile top 3 % in Microsoft’s Libra (Dist. And Parall. Computing category)
Ph.D. Thesis © Francisco Pérez-Sorrosal
53May 13th 2009
Thank You!Thank You!
QUESTIONS?
Ph.D. Thesis © Francisco Pérez-Sorrosal
54May 13th 2009