1.

• • SOAIS, Honolulu, HI, USA, July 2008

• • German Shegalov (ex-MPII, Oracle, USA)

• • Gerhard Weikum (MPI Informatik, Germany)

Formal Verification of a TransactionalInteraction Contract funded by

2. Outline

Background & Problem Statement

Interaction Contracts Framework

Formal Specification of theTransacted IC (TIC)

• STATEMATE State & Activity Charts

Verification of IC's with model checking

• Computational Tree Logic

• Model Checking

Demo of Exactly Once (Web) Service

Summary

3. E-Business Scenario Reviewand Your server command (process id #20) has been terminated. Re-run your command (severity 13) in /opt/www/your-reliable-eshop.biz/mb_1300_db.mb1 place your order! 4.

Non- idempotence(Math 1.0)

• , n>1

Non-idempotence (Web 2.0, ERP, etc.)

• "Request timeout""request failure"

• "Request send""request resend"

• Anecdotal evidence: Don't click more than once!

• • 8health insurance id's for a3member family

• • Orderone , getmany... pay formany

Problem Statement 5. Transaction Recovery

At most once semantics

BEGIN TRANSACTION

/* LSN= 1: log for undo and redo in MM buffer*/

• UPDATE Accounts SET balance = balance 100,00WHERE Number = 1

/* LSN = 2: log for undo and redo in MM buffer*/

• UPDATE Accounts SET balance = balance + 100,00 WHERE Number = 2

/* LSN = 3: log commit and force (5-6 orders slower)*/

COMMIT TRANSACTION

Transfer 100 from 1 to 2

Recovery: Redo Committed, Undo Uncommitted

• LSN test guarantees idempotence

(LSN=0) (LSN=3) 2000,00 2 1000,00 1 Balance Number Accounts 2100,00 2 900,00 1 Balance Number Accounts 6. Transactions are great. However, Web Client Web ApplicationServer DatabaseServer Timeline Non-idempotent execution ! ACK Purchase Request Order Confirmation Start Transaction SQL Request SQL Response SQL Request SQL Response Commit Transaction ACK Transaction Restart Purchase RequestResubmission 7. Traditional OLTP: Queued Transactions

2 forced client writes (I/O queues)

1 forced write (client request id)

4 forced writes for 2PC commit

3 extra messages

8. Real-Worldn -Tier ApplicationExpediaSabre Server Amadeus ExpediaApp ServerSabre App Server Amadeus App Server Client Web ServerDB 1 DB 2 DB 3 DB 4 9. IC Framework

ComponentsandGuarantees

• Persistent (Pcom): Persistent, testable state & messages

• External (Xcom) (e.g., humans): No recovery

• Transactional (Tcom): Persistance and testability on commit

Interaction Contracts

• Xcom&Pcom = External IC (XIC)

• Pcom&Pcom = Committed IC (CIC)

• Tcom & Pcom = Transacted IC (TIC)

Failure model: transient failures, e.g., Heisenbugs

Exactly-Once Semantics

• Forget rollbacks : exactly-once execution is guaranteed

10. Pcom Design

Redo Log & Recovery Managers

Piecewise determinism+ Logging = Full Determinism

Unique message idfor duplicate elimination

Deterministic replayrecovers Pcom's

Installation Pointsspeed up replay

PCom1 PCom2 C 2 C 2 C 2 11. TIC Design

Tcom

• Traditional Redo & Undo Log

• FaithfulReply

• • Persists commit state

• • Persists commit reply message

• • Resends commit reply on a second request

• • No commit reply logged->aborted

• Commit request duplicate elimination.

Pcom

• forces log to disk before commit

• Periodically resends commit request

12. CIC's Informal Design

CIC sender (Pcom) obligations

• Persist state before send

• Tag message with aMSN

• Resend on timeout untilstableack

• Resend on receiver's"get msg"

• Forget interaction oninstalledack

CIC receiver (Pcom) obligations

• Eliminates duplicates byMSN's

• Persists interaction beforestableack

• "gets msg"if msg is not in log after failure

• Ensures autonomous recovery beforeinstalledack

13. Committed IC Activities

Activitychart = Functional View

CIC_AC @CIC_SC FAILURE_PRONE_ENVIRONMENT RCVR_CRASHSNDR_CRASHLINK_OUTAGE CIC_SNDR_AC CIC_RCVR_AC SEND_MSG STABLE INSTALLED @CIC_SNDR_SC @CIC_RCVR_SC EXTERNAL_APP_LOGIC SNDR_TRIGGER MSG_PROCESSED GET_MSG SYSTEM_ADMINISTRATOR ICIC TIMEOUTS 14. Committed IC Monitor

Statechart= Behavioral View

• Finite State Automaton (FSA) +

• Nesting+Orthogonalsubstates +

• E [ C ]/ Atransitions: onE vent whileC ondition

• • Leave source, enter target, executeA ction

• • E.g.,A=E'means generate eventE'

• Configuration= set of entered states

• Execution context= variable valuation

• • Step i :conf i ctxt i conf i+1 ctxt i+1

CIC_SCSENDINGRECEIVING(not SNDR_CRASH) [not active(CIC_SNDR_AC) ]/ start!(CIC_SNDR_AC)SENDINGRECEIVING(not RCVR_CRASH) [not active(CIC_RCVR_AC)]/ start!(CIC_RCVR_AC) SNDR_S RCVR_S 15. Committed IC Sender *EVENT_OK = EVENT LINK_OUTAGE STABLE_SSENDINGINSTALLED_SRECOVERYMSG_LOOKUPPREPARE_PERSISTENCESNDR_MSG_TM and not (STABLE_OK orINSTALLED_OK)/ SEND_MSGSNDR_ND/ SEND_MSG SNDR_TRIGGER [SNDR_LAST_LOGGED=='']/ SNDR_ND MSG_RECOVERED_TM/ SEND_MSGGET_MSG_OK[SNDR_LAST_LOGGED=='INSTALLED']INSTALLED_OK/ SNDR_LAST_LOGGED:='INSTALLED' STABLE_OKSNDR_STABLE_TM and not (INSTALLED_OK or GET_MSG_OK)/ IS_INSTALLED CIC_SNDR_SCSTABLE_SSENDINGMSG_LOOKUPSNDR_MSG_TM and INSTALLED_OK)/ SEND_MSGSNDR_ND/ SEND_MSG [SNDR_LAST_LOGGED=='']/ SNDR_ND MSG_RECOVERED_TM/ SEND_MSGGET_MSG_OKINSTALLED_OK/ SNDR_STABLE_TM and not (INSTALLED_OK or GET_MSG_OK)/ IS_INSTALLED SNDR_CRASHTTSTABLE_SSENDINGMSG_LOOKUPSNDR_MSG_TM and INSTALLED_OK)/ SEND_MSGSNDR_ND/ SEND_MSG [SNDR_LAST_LOGGED=='']/ SNDR_ND MSG_RECOVERED_TM/ SEND_MSGGET_MSG_OKINSTALLED_OK/ SNDR_STABLE_TM and not (INSTALLED_OK or GET_MSG_OK)/ IS_INSTALLED CIC_SNDR_SCSTABLE_SSENDINGMSG_LOOKUPINSTALLED_OK/ SNDR_MSG_TM and INSTALLED_OK)/ SEND_MSGSNDR_ND/ SEND_MSG SNDR_LAST_LOGGED SNDR_ND MSG_RECOVERED_TM/ SEND_MSGGET_MSG_OKINSTALLED_OK/ SNDR_STABLE_TM and not (INSTALLED_OK or GET_MSG_OK)/ IS_INSTALLED TTSNDR_LAST_LOGGED:='INSTALLED' _TM means TIMEOUT 16. Committed IC Receiver MSG_RECOVERYSTABLE_RINSTALLED_RMSG_RECEIVEDRECOVERYMSG_PROCESSEDRCVR_INSTALL_TM/ RCVR_LAST_LOGGED:='INSTALLED'; INSTALLED[RCVR_LAST_LOGGED=='INSTALLED'][RCVR_LAST_LOGGED=='STABLE']SEND_MSG_OK[RCVR_LAST_LOGGED=='STABLE']/ GET_MSG [ICIC]/ RCVR_LAST_LOGGED:='INSTALLED'; INSTALLEDMSG_EXEC_TM/RECEIVED;( RCVR_STABLE_TM orRCVR_ND [MSG_ORDER_MATTERS]) [not ICIC and RCVR_LAST_LOGGED=='']/ RCVR_LAST_LOGGED:='STABLE'; SEND_MSG_OK [RCVR_LAST_LOGGED=='']not SEND_MSG_OK and GET_MSG_TM/ GET_MSGRCVR_CRASHTCIC_RCVR_SCMSG_RECEIVEDRECOVERYMSG_PROCESSED[RCVR_LAST_LOGGED=='INSTALLED'][RCVR_LAST_LOGGED=='STABLE']SEND_MSG_OK[RCVR_LAST_LOGGED=='STABLE']/ GET_MSG [ICIC]/ RCVR_LAST_LOGGED:='INSTALLED'; INSTALLEDMSG_EXEC_TM/RECEIVED;[not ICIC and RCVR_LAST_LOGGED=='']/ RCVR_LAST_LOGGED:='STABLE'; SEND_MSG_OK [RCVR_LAST_LOGGED=='']not SEND_MSG_OK and GET_MSG_TM/ GET_MSGRCVR_CRASHTSEND_MSG or IS_INSTALLED/ SEND_MSG or IS_INSTALLED/ INSTALLEDSTABLE_RINSTALLED_RMSG_RECEIVEDRECOVERYMSG_PROCESSED[RCVR_LAST_LOGGED=='INSTALLED'][RCVR_LAST_LOGGED=='STABLE']SEND_MSG_OK[RCVR_LAST_LOGGED=='STABLE']/ GET_MSG [ICIC]/ RCVR_LAST_LOGGED:='INSTALLED'; INSTALLEDMSG_EXEC_TM/RECEIVED;STABLESEND_MSG_OK [RCVR_LAST_LOGGED=='']not SEND_MSG_OK and GET_MSG_TM/ GET_MSGRCVR_CRASHTCIC_RCVR_SCMSG_RECEIVEDRECOVERYMSG_PROCESSED[RCVR_LAST_LOGGED=='INSTALLED'][RCVR_LAST_LOGGED=='STABLE']SEND_MSG_OK[RCVR_LAST_LOGGED=='STABLE']/ GET_MSG [ICIC]/ RCVR_LAST_LOGGED:='INSTALLED'; INSTALLEDMSG_EXEC_TM/RECEIVED;SEND_MSG_OK [RCVR_LAST_LOGGED=='']not SEND_MSG_OK and GET_MSG_TM/ GET_MSGRCVR_CRASHTSEND_MSG or IS_INSTALLED/ STABLE SEND_MSG or IS_INSTALLED/ INSTALLED*EVENT_OK = EVENT LINK_OUTAGE, _TM means TIMEOUT RCVR_LAST_LOGGED:='INSTALLED' 17. Execution Abstraction

Kripke structure K =( S , R , L )overP

• Pis a finite set of atomic propositions

• Software: P is a union of all memory bits

• Sfinite set of states

• R S Sstate transitions

• L S P { true, false } valuation

• Non-determinism to determinism Computation Tree vs. Sequence

p ,q P p p q p q 18.

Basic Syntax

• Atomic propositions PCTL( P )

• Ifp, q CTL( P ), then so are

• • Propositional logic formulas ( p ,pq, etc. )

• • Path quantifiersE xists,A ll +modalityne X t ,U ntil

• • EX p

• • { E, A } ( p U q )

Derived Syntax

• • AX p ( EX p)

• • A F inallyp A( true U p )

• • EF p E( true U p )

• • A G loballyp (E( true U p ) )

• • EG p (A( true U p ) )

Computation Tree Logic 19. CIC Verification

Safety

• For alllogvaluesv { 'stable', 'installed' }

• AG(written ( log ) log= v AX AG( written ( log ) log= v ) )

• i.e., a value is written at most once

Liveness for timeouts < 30 steps

• F

• • Script called5times

• • Other server reports:Script called 1000 times

25. EOS

• Exactly-once semantics with

• • Transparent browser recovery

• • Concurrent accesses to shared data

• • Nondeterm. functions:time ,curl_exec ,rand

• • Anyninn -tier, any fanout

• • Failure masking:no changes to app codeneither to PHP scripts, nor to the browser

• Performance enhancements (side effects)

• • Log structured data access (sequential I/O)

• • LRU buffers for state and log data

• • Latches (Shared/Exclusive)

• • session_start ( bool $read_only )

26. Experiment Setup BackendServer P4 3Ghz, 1GB FrontendServer P4 3Ghz, 1GB shared count 1234 1235 private count 2 3 private count 2 3 private count 2 1 private count 2 3 POST (ICIC) action=increment b2b=true 1235

Privatel Count: 3

Shared Count: 1235 POST (ICIC) action=increment Web Client

• eBay-like auction service

• User settings at frontend (private)

• Auction items at backend (shared)

• 5 concurrent end users, synthetic load

27. Run-Time Overhead Backend Server Frontend Server shared count 1234 1235 private count 2 3 private count 2 3 private count 2 1 private count 2 3 POST ( ICIC ) action=increment b2b=true 1235

Privatel Count: 3

Shared Count: 1235 POST ( ICIC ) action=increment Web Client 33% 36% 44% Overhead (backend CPU)[%] 0.1600 0.0750 0.0130 EOS-PHP backend CPU time [sec] 0.1200 0.0550 0.0090 PHP backend CPU time [sec] 102% 122% 109% Overhead (frontend CPU) [%] 1.1545 0.6000 0.0815 EOS-PHP frontend CPU time [sec] 0.5727 0.2708 0.0390 PHP frontend CPU time [sec] 93% 113% 101% Overhead (elapsed time) [%] 3.1000 1.6850 0.3140 EOS-PHP elapsed time [sec] 1.6100 0.7900 0.1560 PHP elapsed time [sec] 10 steps 5 steps 1 step Session 28. Outline

• Problem Statement and Background

• Interaction Contracts Framework

• • Formal Specification of theCommitted IC

• • Verification of IC's with model checking

• • Verification of Web Service IC Model

• Implementation: Exactly-Once Web Service (EOS)

• • Overview

• • EOS-PHP

• • Demo

• Summary

29. Summary

• Generic IC framework specification

• Formal verification at IC and app level

• • To do: Overcome "model checking" non-scalability

• Efficient implementation: EOS

• • Rigorous recovery guarantees

• • • Based on the formal verified models

• • Many enhancements to PHP

• • • LRU buffer management

• • • Mostly sequential disk accesses

• • • Concurrency control with latches

30. EOS Demo USER 1 Backend Server Frontend Server B2B_LINK B2C_LINK 31. Thank You! ? 32. 2PC Message Sequence Coordinator DB i force-log begin Timeline prepare force-log prepared commit force-log commit force-log commit force-log end ack yes 33. PA-2PC Coordinator 34. PA-PC Cohort 35. Transactional IC Server 36. Transactional IC Client 37. External IC