Invariant Safety for DistributedApplications

Sreeja S NairSorbonne

Université—LIP6 &Inria, Paris, Francesreeja.nair@lip6.fr

Gustavo PetriARM Research,Cambridge, UK

gustavo.petri@arm.com

Marc ShapiroSorbonne

Université—LIP6 &Inria, Paris, France

marc.shapiro@acm.org

ABSTRACTWe study a proof methodology for verifying the safety of data invariants of highly-available distributed applications that replicate state. The proof is (1) modular: one canreason about each individual operation separately, and (2) sequential: one can reasonabout a distributed application as if it were sequential. We automate the methodologyand illustrate the use of the tool with a representative example.

KEYWORDSReplicated data, Consistency, Automatic verification, Distributed application design,Tool supportACM Reference Format:Sreeja S Nair, Gustavo Petri, and Marc Shapiro. 2019. Invariant Safety for Distributed Applica-tions. In Proceedings of Principles and Practice of Consistency for Distributed Data (PaPoC’19).ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1 INTRODUCTIONA distributed application often replicates its data to several locations, and accessesthe closest available replica. Examples include social networks, multi-user games,co-operative engineering tools, collaborative editors, source control repositories,or distributed file systems. To ensure availability, an update must not synchroniseacross replicas; otherwise, when a network partition occurs, the system will block.Asynchronous updates may cause replicas to diverge or to violate the data invariantsof the application.To address the first problem, Conflict-free Replicated Data Types (CRDTs)[13] have

mathematical properties to ensure that all replicas that have received the same set ofupdates converge to the same state [13]. To ensure availability, a CRDT replica executesboth queries and updates locally and immediately, without remote synchronisation. Itpropagates its updates to the other replicas asynchronously.

PaPoC’19, March 25, Dresden, Germany© 2019 Association for Computing Machinery.This is the author’s version of the work. It is posted here for your personal use. Not for redistribution.The definitive Version of Record was published in Proceedings of Principles and Practice of Consistencyfor Distributed Data (PaPoC’19), https://doi.org/10.1145/nnnnnnn.nnnnnnn.

arX

iv:1

903.

0275

9v1

[cs

.DC

] 7

Mar

201

9

PaPoC’19, March 25, Dresden, Germany Sreeja S Nair, Gustavo Petri, and Marc Shapiro

There are two basic approaches to update propagation: to propagate operations, orto propagate states. In the former approach, an update is first applied to some originreplica, then sent as an operation to remote replicas, which in turn apply it to updatetheir local state. Operation-based CRDTs require the the message delivery layer todeliver messages in causal order, exactly once; the set of replicas must be known.In the latter approach, an update is applied to some origin replica. Occasionally,

one replica sends its full state to some other replica, which merges the received stateinto its own. In turn, this replica will later send its own state to yet another replica. Aslong as every update eventually reaches every replica transitively, messages may bedropped, re-ordered or duplicated, and the set of replicas may be unknown. Replicasare guaranteed to converge if the set of states, as a result of updates and merge, formsa monotonic semi-lattice [13]. Due to these relaxed requirements, state-based CRDTshave better adoption [1]. They are the focus of this work.As a running example, consider a simple auction system. The state of an auction

consists of status, a set of bids, and a winner. This state is replicated at multiple servers;CRDTs ensures that all replicas eventually converge. Users at different locations canstart an auction, place bids, close the auction, declare a winner, inspect the localreplica, and observe if a winner is declared and who it is. All replicas will eventuallyagree on the same auction status, same set of bids and the same winner.However, the application may also require to maintain a correctness property or

invariant over the data. An invariant is an assertion on application data that mustevaluate to true in every state of every replica. For instance, the auction’s invariant isthat: when the auction is closed, there is a winner; there is a single winner; and thewinner’s bid is the highest.Such an invariant is easy to ensure in a sequential system, but concurrent updates

might violate it. In this case, the application would need to synchronise some updatesbetween replicas, in order to maintain the invariant. For instance, in the absence ofsufficient synchronisation, a replica might close the auction and declare a winner,while concurrently a user at a different replica is placing a higher bid.This problem has been addressed before, by stating correctness rules and proof

obligations; however, previous work considers only the operation-based approach[6, 8, 10].In this paper, we propose a proof methodology for applications that use state-

based CRDTs. We exploit the properties of state-based CRDTs to reason about aconcurrent system in a sequential manner. We have also developed a tool namedSoteria, to automate our proof rule. Soteria detects concurrency bugs and providescounterexamples.

2 SYSTEM MODELAn application consists of state, some operations, a merge function, and an invariant.The state is replicated at any number of replicas. A client chooses any arbitrary replicafor its next operation, called the origin replica for that operation. A replica occasionallysends its state to some other replica, which the receiving replica merges into its ownstate. In summary, the state of any given replica changes, either by executing anupdate operation for which it is the origin, or by merging the state received from aremote replica. Each replica is sequential. A merge is the only point where a replicaobserves concurrent operations submitted to other replicas. Each replica executes

Invariant Safety for Distributed Applications PaPoC’19, March 25, Dresden, Germany

Figure 1: Evolution of state in an auction application

sequentially, one local update or merge at a time; equivalently, an update or mergeoperation executes atomically, even if it updates multiple data items.An application invariant is an assertion over state. The invariant must evaluate

true in every state of every replica. Despite being evaluated against local state, aninvariant is in effect global, since it must be true at all replicas, and replicas eventuallyconverge.Figure 1 depicts the evolution of state in our auction application. Each line represents

a replica, time progressing from left to right. A box represents local state. A curvedarrow represents an update operation, labelled with the operation name. A diagonalarrow shows propagation (labelled with the propagated state), merged at the receivingreplica.We assume here that application state is a composition of CRDTs. This is not a

limitation, since many basic CRDT types have been proposed, which extend familiarsequential data types with a concurrency semantics [12].The CRDT convergence rules, or lattice rules, are the following. The state of a

replica progresses monotonically with time. The set of states forms a semi-lattice, i.e.,is equipped with a partial order and a least-upper-bound function. A state transitionrepresents the execution of either an update operation or merge. An update is aninflation, i.e., the resulting state is no less (in the partial order) than the previous one.Merge computes a state which is the least-upper-bound of the current local state andthe received remote state.We write σ for the current state of a replica, σnew for the new state after an oper-

ation or merge, and σ ′ for the incoming state from a remote replica. In the Boogiespecification language (used by our tool, described in Section 4), we denote operationexecution with the keyword call, an assertion with assert, and assumptions withassume. An operation op executes only if its precondition is true. Thus, we can writethe above lattice conditions as follows:

• An update operation op is an inflation.call σnew = op(σ )assert σnew ≥ σ

• Merge is a least upper boundcall σnew = merge(σ ,σ ′)assert σnew ≥ σ ∧ σnew ≥ σ ′ #upper bound

PaPoC’19, March 25, Dresden, Germany Sreeja S Nair, Gustavo Petri, and Marc Shapiro

assert ∀σ ∗,σ ∗ ≥ σ ∧ σ ∗ ≥ σ ′ =⇒ σ ∗ ≥ σnew #least

Let us illustrate with the auction example (for simplicity we consider a singleauction). Its state is as follows:

• Status: the status of an auction can move from its initial state, INVALID (underpreparation), to ACTIVE (can receive bids) and CLOSED (no more bids accepted),such that INVALID < ACTIVE < CLOSED.

• Winner: The winner of the auction. It is either ⊥, initially, or the bid with thehighest amount. In case of multiple bids with same amount, the bid with thelowest id wins. It is ordered such that ∀b ∈ Bids,⊥ ≤ b.

• Bids: Set of bids placed (initially empty)– BidId: A unique identifier for each bid placed– Placed: A boolean flag to indicate whether the bid has been placed or not,ordered TRUE > FALSE. It is enabled once when the bid is placed. Once placed,a bid cannot be withdrawn.

– Amount: An integer representing the amount of the bid; this cannot bemodifiedonce the bid is created.

Figure 1 illustrates how the auction state evolves over time; status, bids and winnerare represented by a circle, rectangle and a star respectively. The application state isgeo-replicated at data centres in Australia, Belgium and China.We now specify the merge operation for an auction. We denote the receiving replica

state σ = (status, winner, Bids); the received state is denoted σ ′ = (status′, winner′,Bids′).

merge((status ,winner ,Bids),

(status ′,winner ′,Bids ′)):statusnew := max(status ,status ′)if winner ′,⊥ then winnernew := winner ′

else winnernew := winner

∀b, if b ∈ Bids ∩ Bids ′ then

Bidsnew .b.placed := Bids.b.placed

∨ Bids ′.b.placedelse if b ∈ Bids ′ then

Bidsnew .b.placed := Bids ′.b.placedelse Bidsnew .b.placed := Bids.b.placed

∀b, if b ∈ Bids then

Bidsnew .b.amount := Bids.b.amount

else Bidsnew .b.amount := Bids ′.b.amount

In the absence of any extra synchronisation, it is possible to violate the invariant, asthe following execution scenario illustrates. Alice from Australia starts an auction, bysetting its status to ACTIVE (green in the figure) in the Australian replica. Henceforth,the auction can receive bids. The Australian replica sends its updated state to the othertwo replicas, which merge it into their own states. Now Bob in Belgium places a bidfor $100 (blue). This update is sent to other replicas. Charlie, in China, sees the auctionand Bob’s bid. He updates the China state with a higher bid of $105 (dotted blue),which is sent to both other replicas. However, due to a network failure, the remotereplicas do not receive this update (red dotted lines). Meanwhile Alice, unaware ofCharlie’s bid, closes the auction and declares Bob’s bid as the winner. Later when thenetwork heals, the updated states are sent and merged. The auction is new closed,

Invariant Safety for Distributed Applications PaPoC’19, March 25, Dresden, Germany

and contains Bob’s $100 bid and Charlie’s $105 bid. Unfortunately, Bob’s bid is thewinner, violating the application invariant.In the next section, we can discuss how to ensure invariants of applications build

on top of the system model we described.

3 PROVING INVARIANTSAs explained earlier, each replica executes a sequence of state transitions, due eitherto a local update, or to a merge incorporating remote updates. Thus, concurrency canbe observed only through merge.Let us call safe state a replica state that satisfies the invariant. Assuming the current

state is safe, any update (local or merge) must result in a safe state. To ensure this,every update is equipped with a precondition that disallows any unsafe execution.1Thus, a local update executes only when, at the origin replica, the current state is safeand its precondition currently holds. Similarly, merge executes only with two safestates that together satisfy a merge precondition.Formally, an update u (an operation or a merge), mutates the local state σ , to a new

state σnew = u(σ ). To preserve the invariant, Inv, we require that

σ ∈ Preu =⇒ u(σ ) ∈ Inv

To illustrate local preconditions, consider an operation close_auction(w: BidId),which sets auction status to CLOSED and the winner to w. The developer may havewritten a precondition such as status = ACTIVE, because closing an auction doesn’tmake sense otherwise. In order to ensure the invariant that the winner has the highestamount, one needs to strengthen it with the clause is_highest(Bids, w), definedas ∀b ∈ Bids : b.Amount ≤ w.Amount.To illustrate merge precondition, consider a CRDTwhose state is the pair of integers,

σ = (n,m) ∈ N×N. It has two operations, incn and incm, that respectively incrementn orm by 1, and a merge function:

merge(σ ,σ ′) = (max(n,n′),max(m,m′))We wish to maintain the invariant that their sum is no more than 10:

Inv ≜ (n +m) ≤ 10

The precondition of incn is Preincn ≜ (n + m) ≤ 9; similarly for incm. Startingfrom a safe state (4, 5), two replicas may independently increment to states (5, 5) and(4, 6) respectively. Both are safe. However, merging them would violate the invariant.Therefore, merge(σ ,σ ′) must have precondition

Premerge ≜ max(n,n′) +max(m,m′) ≤ 10

Since merge can happen at any time, it must be the case that its preconditionis always true, i.e., it constitutes an additional invariant. Now our global invariantconsists of two parts: first, the application invariant, and second, the precondition ofmerge. We can now state our proof rule informally as follows::2

1 Technically, this is at least the weakest-precondition of the update for the invariant. It strengthens anya priori precondition that the developer may have set.2 We omit the full formalisation and the proof of soundness for brevity.

PaPoC’19, March 25, Dresden, Germany Sreeja S Nair, Gustavo Petri, and Marc Shapiro

State-Based Safety Rule. Define the precondition ofmerge to be theweakest-precondition of merge, for the application invariant. The initial state mustsatisfy, and each local update ormerge operationmust preserve, the conjunc-tion of: (i) the application invariant, and (ii) the precondition of merge. □

In our Boogie notation, each operation can be verified as follows:assume Inv ∧ Premerge ∧ Preopcall σnew = op(σ )assert Inv ∧ Premerge

The case of the merge function can be verified with the following condition:assume Inv ∧ Inv′ ∧ Premergecall σnew = merge(σ ,σ ′)assert Inv ∧ Premerge

Note that there are two copies of state, the unprimed local state of the replica applyingthe merge, and the primed state received from a remote replica. Inv′ denotes that σ ′

preserves the invariant Inv.

3.1 Applying the proof ruleLet us apply the proof methodology to the auction application. Its invariant is thefollowing conjunction:(1) A bid is placed only when status is ACTIVE.(2) And: Once a bid is placed, its amount does not change.(3) And: There is no winner until status is CLOSED.(4) And: There is a single winner, the bid with the highest amount (breaking ties

using the lowest identifier).Computing the weakest-precondition of each update operation, for this invariant, isobvious. For instance, as discussed earlier, close_auction(w: BidId) gets precon-dition is_highest(Bids, w), because of Invariant Term 4 above.Despite local updates to each replica preserving the invariant, Figure 1 showed

that it is susceptible of being violated by merging. This is the case if Bob’s $100 bidin Belgium wins, even though Charlie concurrently placed a $105 bid in China; thisoccurred because status became CLOSED in Belgium while still ACTIVE in China. Theweakest-precondition of merge for Term 4 expresses that, if status in either statesis CLOSED, the winner should be the bid with the highest amount in both the states.Therefore, merge(σ ,σ ′) must have the following additional precondition:

status=CLOSED =⇒ is_highest(Bids , winner)

∧ is_highest(Bids ′, winner)

∧ status ′=CLOSED =⇒ is_highest(Bids , winner ′)∧ is_highest(Bids ′, winner ′)

Furthermore, the code for merge uses Term 2, for which its weakest-preconditionis as follows:

∀b ∈ Bids∩Bids ′, Bids.b.amount = Bids ′.b.amount

These two merge preconditions now strengthen the global invariant, in order topreserve safety in concurrent executions. Let us now consider how this strengtheningimpacts the local update operations. Since starting the auction doesn’t modify anybids, this operation trivially preserves it. Placing a bid might violate it, if the auction

Invariant Safety for Distributed Applications PaPoC’19, March 25, Dresden, Germany

is concurrently closed in some other replica; conversely, closing the auction couldviolate it, if a higher bid is concurrently placed in a remote replica. Thus, the auc-tion application is safe when executed sequentially, but is unsafe when updates areconcurrent. This indicates the specification has a bug, which we now proceed to fix.

3.2 Concurrency Control for Invariant PreservationAs we discussed earlier, the preconditions of operations and merge are strengthenedin order to preserve the invariant. This provides a sequentially safe specification. Anapplication must also preserve the precondition of merge in order to ensure concurrentsafety. Violating this indicates the presence of a bug in the specification. In that case,the developer needs to strengthen the application by adding appropriate concurrencycontrol mechanisms, ie., the operations that fail to preserve the precondition of mergemight need to synchronise. The required concurrency control mechanisms are addedas part of the state in our model. The modified application state is now composed of theCRDTs that represents the state and the concurrency control mechanism. Together, itbehaves like a composition of state-based CRDTs. The whole state should now ensurethe lattice conditions described in section 2.Recall that in the auction example, placing bids and closing the auction were not

preserving the precondition of merge. This requires strengthening the specification byadding a concurrency control mechanism to restrict these operations. We can enforcethem to be strictly sequential, thereby avoiding concurrency at all. But this will affectthe availability of the application.A concurrency control can be better designed with the workload characteristics

in mind. For this particular application, we know that placing bids are very frequentoperations than closing an auction. Hence we try to formulate a concurrency controllike a readers-writer lock. In order to realise this we distribute tokens to each replica.As long as a replica has the token, it can allow placing bids. Closing the auction requiresrecalling the tokens from all replicas. This ensures that there are no concurrent bidsplaced and thus a winner can be declared, respecting the application safety.The entire specification of the auction application can be seen in Figure 2. The

shaded lines in blue indicate the effect of adding concurrency control to the state.An alternative approach to our treatment of concurrency control could be to consider

the invariant as a resource in the style of Concurrent Separation Logic [11]. In thiscase, access to the application state, described through a separation logic invariant,is guarded by a concurrency control mechanism (typically some form of a lock).However, this approach is tied to separation logic reasoning, where assertions act asresources, and allows one to distinguish local from global resources. We consider thatthis was not essential for the kind of proofs that we conduct, but it might be morepromising when verifying client programs of our data types.

4 AUTOMATING THE VERIFICATIONIn this section we present a tool to automate the verification of invariants as discussedin the previous sections. Our tool, called Soteria is based on the Boogie [2] verificationframework. The input to Soteria is a specification of the application written in Boogie,an intermediate verification language.A specification in Soteria will consist of the following parts:

PaPoC’19, March 25, Dresden, Germany Sreeja S Nair, Gustavo Petri, and Marc Shapiro

• State: a declaration of the state. It can be a single CRDT or a composition ofCRDTs.

• Comparison function: The programmer provides a comparison function (an-notated with keyword @gteq) that determines the partial order on states.

• Operations: The programmer provides the implementation of the operationsand their respective preconditions, Preop. Operations are encoded either imper-atively as Boogie procedures or declaratively as postconditions.

• Merge function: The special merge operation is distinguished from the otheroperations (with annotation @merge). The programmer must provide a precon-dition to merge that is strong enough to prove the invariant.

• Application Invariant: The programmer provides the invariant (with keyword@invariant) to be verified by the tool as a Boogie assertion over the state.

In addition, Boogie often requires additional information such as: • User-defined datatypes, • Constants to declare special objects such as the origin replicame, or to boundthe quantifiers, • Axioms for inductive functions over aggregate data structures, forinstance, to compute the maximum of a set of values, • Loop invariants.The specification of the auction application can be seen in Figure 2.

VerificationThe verification of a specification is performed in multiple stages; in order:(1) Syntactic checks: validates the specification for syntactical errors and checks

whether the pre/post conditions are sound.(2) Compliance check: checks whether the specification provides all the elements

explained earlier.(3) Convergence check: checks whether the specification respects the properties

of a state-based CRDT, ie., each operation inflates the state and merge is theleast upper bound.

(4) Safety check: verifies the safety of the application invariant, as discussed insection 3. This stage is divided further into two sub-stages:• Sequential safety: whether each individual operation (or merge) upholds theinvariant. If not, the designer needs to strengthen the precondition of thecorresponding operation (or merge)

• Concurrent safety: whether every operation (and merge) upholds the precondi-tion of merge. Note that, while this check relates to the concurrent behaviourof state-based CRDTs, the check itself is completely sequential, ie., it does notrequire reasoning about operations performed by other processes. This checkensures that the invariant remains safe during concurrent operation. If thischeck fails, the application needs stronger concurrency control.

Each check in Soteria 3 generates counterexamples when the verification fails. Thesecounterexamples might guide the developer in debugging the specification accordingto the verification steps.

5 RELATEDWORKSeveral works have concentrated on the formalisation and specification of eventuallyconsistent systems [3, 4, 14]. A number of works concentrate on the specification and3The tool alongwith some sample specifications can be accessed at https://github.com/sreeja/soteria_tool.

Invariant Safety for Distributed Applications PaPoC’19, March 25, Dresden, Germany

Initial state:

status = INVALID ∧ winner = ⊥∧ ∄ b, Bids.b.placed

∧ ∀ r, Tokens.r = true

Comparison function:

status1 ≥ status2 ∧ (winner1,⊥ ∨ winner2=⊥)

∧∀b, (Bids1.b.placed ∨ ¬Bids2.b.placed)∧ (∀r, ¬Tokens1.r ∨ Tokens2.r)

Invariant:

∀b, Bids.b.placed =⇒ status≥ACTIVE

∧ Bids.b.amount >0

status≤ACTIVE =⇒ winner=⊥status=CLOSED =⇒ Bids.winner.placed

∧ is_highest(Bids , winner)

status=CLOSED =⇒ ∀r,¬Tokens.r

{Premerge:

status=CLOSED =⇒ is_highest(Bids , winner)

∧is_highest(Bids ′, winner)

∧ status ′=CLOSED =⇒ is_highest(Bids , winner ′)∧is_highest(Bids ′, winner ′)

∧ ∀b, Bids.b.amount = Bids ′.b.amount∧ ∀r, Tokens.r.me =⇒ Tokens ′.r.me∧ ∀r,b, (¬Tokens.r ∧ ¬Bids.b.placed)

=⇒ ¬Bids ′.b.placed∧ ∀ r,b, (r,me ∧ ¬Tokens.r ∧ ¬Bids.b.placed)

=⇒ ¬Bids ′.b.placed∧ ∀ r,¬Tokens.r =⇒ winner ′=winner ∨ winner ′=⊥∧ ∀ r, Tokens.r =⇒ winner=⊥ ∧ winner ′=⊥}

merge((status , winner , Bids , Tokens),

(status ′,winner ′,Bids ′,Tokens ′)):<merge of status , winner , Bids as in section 2>

∀r, Tokensnew .r := Tokens.r∧Tokens ′.r

{Prestart_auction: status = INVALID ∧ winner = ⊥∧ ∀r, Tokens.r}

start_auction ():

statusnew := ACTIVE

winnernew := ⊥{Preplace_bid: ¬Bids.b_id.placed

∧ Bids.b_id.amount = value

∧ status = ACTIVE ∧ winner = ⊥∧ Tokens.me}

place_bid(b_id , value):

Bidsnew .b_id.placed := true

Bidsnew .b_id.amount := value

{Preclose_auction: status = ACTIVE ∧ winner = ⊥∧ Bids.w.placed ∧ is_highest(Bids , w)

∧ ∀r, ¬Tokens.r}close_auction(w):

statusnew := CLOSED

winnernew := w

Figure 2: Specification of auction application

PaPoC’19, March 25, Dresden, Germany Sreeja S Nair, Gustavo Petri, and Marc Shapiro

correct implementation of CRDTs [5, 7]. Our work also verifies the CRDT (lattice)conditions, but additionally verifies an arbitrary application invariant over a replicastate.Gotsman et al. [6] provides a proof methodology for proving invariants of CRDTs

that propagate operations. The associated tools [8, 10] performs the check using anSMT solver as the backend and Nair and Shapiro [9] discusses some concurrencycontrol suggestions by using the counterexamples generated by the failed proofs.Gotsman et al. [6] assume that the underlying network ensures causal consistency,and their methodology requires reasoning about concurrent behaviours. This requireschecks for each pair of operations in the application (reflected as stability verificationconditions). Gotsman et al. [6] uses an abstract notion of tokens as concurrency controlmechanisms. The operations acquire tokens in order to preserve the applicationinvariant.In contrast, Soteria focuses on state-based CRDTs. We check convergence by verify-

ing the lattice conditions of section 2 and that because of the rules shown in section 3,we can reduce the problem of verifying the invariant to sequential proof obligations.This is reflected by the fact that all of our proofs are standard pre/post conditionschecks using the Boogie framework. Boogie framework. In contrast with Gotsmanet al. [6], Soteria includes concrete specification of concurrency control as part of theapplication state.To the best of our knowledge, ours is the first attempt in automated verification of

invariants of state-based CRDTs.

6 CONCLUSIONWe have presented a proof methodology to verify invariants of state-based CRDTimplementations guaranteeing: (1) that the implementation satisfies the lattice condi-tions of state-based CRDTs [1], and (2) that the implementation satisfies programmerprovided invariant reducing the problem to checking that each operation of the datatype satisfies a precondition of the merge function of the state.We implemented Soteria, a tool sitting on top of the Boogie verification framework,

to specify the implementation, its invariant and validate it.In future work, we plan to automate concurrency control synthesis. The synthesised

concurrency control can be analysed and adjusted dynamically to minimise the costof synchronisation. Another direction for future work can be to decouple the updatepropagation mechanism of CRDT from the proof rule resulting in a generic proof ruleto verify distributed systems.

ACKNOWLEDGMENTSThe authors would like to thank the anonymous reviewers for their comments whichhelped in improving this paper. This research is supported in part by the RainbowFSproject (Agence Nationale de la Recherche, France, number ANR-16-CE25-0013-01) andby European H2020 project 732 505 LightKone (2017–2020).

REFERENCES[1] C. Baquero, P. S. Almeida, A. Cunha, and C. Ferreira. Composition in state-based replicated data

types. Bulletin of the EATCS, 123, 2017. URL http://eatcs.org/beatcs/index.php/beatcs/article/view/507.

Invariant Safety for Distributed Applications PaPoC’19, March 25, Dresden, Germany

[2] M. Barnett, B.-Y. E. Chang, R. DeLine, B. Jacobs, and K. R. M. Leino. Boogie: A modular reusableverifier for object-oriented programs. In Proceedings of the 4th International Conference on FormalMethods for Components and Objects, FMCO’05, pages 364–387, Berlin, Heidelberg, 2006. Springer-Verlag. ISBN 3-540-36749-7, 978-3-540-36749-9. doi: 10.1007/11804192\_17. URL http://dx.doi.org/10.1007/11804192_17.

[3] S. Burckhardt. Principles of Eventual Consistency, volume 1 of Foundations and Trends in ProgrammingLanguages. Now Publishers, Oct. 2014. doi: 10.1561/2500000011. URL http://research.microsoft.com/pubs/230852/final-printversion-10-5-14.pdf.

[4] S. Burckhardt, A. Gotsman, H. Yang, and M. Zawirski. Replicated data types: Specification, verifica-tion, optimality. In Symp. on Principles of Prog. Lang. (POPL), pages 271–284, San Diego, CA, USA,Jan. 2014. doi: 10.1145/2535838.2535848. URL http://doi.acm.org/10.1145/2535838.2535848.

[5] V. B. F. Gomes, M. Kleppmann, D. P. Mulligan, and A. R. Beresford. A framework for establishingstrong eventual consistency for conflict-free replicated datatypes. Archive of Formal Proofs, 2017,2017. URL https://www.isa-afp.org/entries/CRDT.shtml.

[6] A. Gotsman, H. Yang, C. Ferreira, M. Najafzadeh, and M. Shapiro. ’Cause I’m Strong Enough:Reasoning about consistency choices in distributed systems. In Symp. on Principles of Prog. Lang.(POPL), pages 371–384, St. Petersburg, FL, USA, 2016. doi: 10.1145/2837614.2837625. URL http://dx.doi.org/10.1145/2837614.2837625.

[7] R. Jagadeesan and J. Riely. Eventual consistency for CRDTs. In A. Ahmed, editor, ProgrammingLanguages and Systems - 27th European Symposium on Programming, ESOP 2018, Held as Partof the European Joint Conferences on Theory and Practice of Software, ETAPS 2018, Thessaloniki,Greece, April 14-20, 2018, Proceedings, volume 10801 of Lecture Notes in Computer Science, pages968–995. Springer, 2018. ISBN 978-3-319-89883-4. doi: 10.1007/978-3-319-89884-1\_34. URLhttps://doi.org/10.1007/978-3-319-89884-1_34.

[8] G. Marcelino, V. Balegas, and C. Ferreira. Bringing hybrid consistency closer to programmers.In W. on Principles and Practice of Consistency for Distr. Data (PaPoC), PaPoC ’17, pages 6:1–6:4,Belgrade, Serbia, 2017. Euro. Conf. on Comp. Sys. (EuroSys), ACM. doi: 10.1145/3064889.3064896.URL http://doi.acm.org/10.1145/3064889.3064896.

[9] S. Nair and M. Shapiro. Improving the “Correct Eventual Consistency” tool. Rapport de rechercheRR-9191, Institut National de la Recherche en Informatique et Automatique (Inria), Paris, France,July 2018. URL https://hal.inria.fr/hal-01832888.

[10] M. Najafzadeh, A. Gotsman, H. Yang, C. Ferreira, and M. Shapiro. The CISE tool: Proving weakly-consistent applications correct. InW. on Principles and Practice of Consistency for Distr. Data (PaPoC),EuroSys 2016 workshops, London, UK, Apr. 2016. Assoc. for Computing Machinery Special InterestGroup on Op. Sys. (SIGOPS), Assoc. for Computing Machinery. doi: 10.1145/2911151.2911160. URLhttp://dx.doi.org/10.1145/2911151.2911160.

[11] P. W. O’Hearn. Resources, concurrency, and local reasoning. Theor. Comput. Sci., 375(1-3):271–307,2007. doi: 10.1016/j.tcs.2006.12.035. URL https://doi.org/10.1016/j.tcs.2006.12.035.

[12] M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. A comprehensive study of Convergent andCommutative Replicated Data Types. Rapport de Recherche 7506, Institut National de la Rechercheen Informatique et Automatique (Inria), Rocquencourt, France, Jan. 2011.

[13] M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. Conflict-free replicated data types. InX. Défago, F. Petit, and V. Villain, editors, Int. Symp. on Stabilization, Safety, and Security of Dist. Sys.(SSS), volume 6976 of Lecture Notes in Comp. Sc., pages 386–400, Grenoble, France, Oct. 2011. Springer-Verlag. doi: 10.1007/978-3-642-24550-3_29. URL https://doi.org/10.1007/978-3-642-24550-3_29.

[14] K. Sivaramakrishnan, G. Kaki, and S. Jagannathan. Declarative programming over eventuallyconsistent data stores. In Assoc. for Computing Machinery Special Interest Group on Pg. Lang.(SIGPLAN), PLDI ’15, pages 413–424, Portland, OR, USA, 2015. Assoc. for Computing Machinery,Assoc. for Computing Machinery. doi: 10.1145/2737924.2737981. URL http://doi.acm.org/10.1145/2737924.2737981.

Invariant Safety for Distributed ApplicationsSreeja S Nair

Sorbonne Université—LIP6 & Inria,Paris, France

sreeja.nair@lip6.fr

Gustavo PetriARM Research, Cambridge, UK

gustavo.petri@arm.com

Marc ShapiroSorbonne Université—LIP6 & Inria,

Paris, Francemarc.shapiro@acm.org

ABSTRACTWe study a proof methodology for verifying the safety ofdata invariants of highly-available distributed applicationsthat replicate state. The proof is (1) modular: one can reasonabout each individual operation separately, and (2) sequen-tial: one can reason about a distributed application as if itwere sequential.We automate themethodology and illustratethe use of the tool with a representative example.

KEYWORDSReplicated data, Consistency, Automatic verification, Dis-tributed application design, Tool supportACM Reference Format:Sreeja S Nair, Gustavo Petri, and Marc Shapiro. 2019. InvariantSafety for Distributed Applications. In Proceedings of Principles andPractice of Consistency for Distributed Data (PaPoC’19). ACM, NewYork, NY, USA, 7 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1 INTRODUCTIONA distributed application often replicates its data to severallocations, and accesses the closest available replica. Exam-ples include social networks, multi-user games, co-operativeengineering tools, collaborative editors, source control repos-itories, or distributed file systems. To ensure availability,an update must not synchronise across replicas; otherwise,when a network partition occurs, the systemwill block. Asyn-chronous updates may cause replicas to diverge or to violatethe data invariants of the application.

To address the first problem, Conflict-free Replicated DataTypes (CRDTs)[13] have mathematical properties to ensurethat all replicas that have received the same set of updatesconverge to the same state [13]. To ensure availability, a

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACMmust be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Requestpermissions from permissions@acm.org.PaPoC’19, March 25, Dresden, Germany© 2019 Association for Computing Machinery.ACM ISBN 978-1-4503-6276-4. . . $15.00https://doi.org/10.1145/nnnnnnn.nnnnnnn

CRDT replica executes both queries and updates locally andimmediately, without remote synchronisation. It propagatesits updates to the other replicas asynchronously.

There are two basic approaches to update propagation: topropagate operations, or to propagate states. In the formerapproach, an update is first applied to some origin replica,then sent as an operation to remote replicas, which in turnapply it to update their local state. Operation-based CRDTsrequire the the message delivery layer to deliver messages incausal order, exactly once; the set of replicas must be known.

In the latter approach, an update is applied to some originreplica. Occasionally, one replica sends its full state to someother replica, which merges the received state into its own.In turn, this replica will later send its own state to yet anotherreplica. As long as every update eventually reaches everyreplica transitively, messages may be dropped, re-ordered orduplicated, and the set of replicas may be unknown. Replicasare guaranteed to converge if the set of states, as a resultof updates and merge, forms a monotonic semi-lattice [13].Due to these relaxed requirements, state-based CRDTs havebetter adoption [1]. They are the focus of this work.As a running example, consider a simple auction system.

The state of an auction consists of status, a set of bids, and awinner. This state is replicated at multiple servers; CRDTs en-sures that all replicas eventually converge. Users at differentlocations can start an auction, place bids, close the auction,declare a winner, inspect the local replica, and observe if awinner is declared and who it is. All replicas will eventuallyagree on the same auction status, same set of bids and thesame winner.However, the application may also require to maintain a

correctness property or invariant over the data. An invariantis an assertion on application data that must evaluate to truein every state of every replica. For instance, the auction’sinvariant is that: when the auction is closed, there is a winner;there is a single winner; and the winner’s bid is the highest.

Such an invariant is easy to ensure in a sequential system,but concurrent updates might violate it. In this case, the ap-plication would need to synchronise some updates betweenreplicas, in order to maintain the invariant. For instance, inthe absence of sufficient synchronisation, a replica mightclose the auction and declare a winner, while concurrently auser at a different replica is placing a higher bid.

arX

iv:1

903.

0275

9v1

[cs

.DC

] 7

Mar

201

9

PaPoC’19, March 25, Dresden, Germany Sreeja S Nair, Gustavo Petri, and Marc Shapiro

This problem has been addressed before, by stating cor-rectness rules and proof obligations; however, previous workconsiders only the operation-based approach [6, 8, 10].

In this paper, we propose a proof methodology for applica-tions that use state-based CRDTs. We exploit the propertiesof state-based CRDTs to reason about a concurrent system ina sequential manner. We have also developed a tool namedSoteria, to automate our proof rule. Soteria detects concur-rency bugs and provides counterexamples.

2 SYSTEM MODELAn application consists of state, some operations, a mergefunction, and an invariant. The state is replicated at anynumber of replicas. A client chooses any arbitrary replicafor its next operation, called the origin replica for that oper-ation. A replica occasionally sends its state to some otherreplica, which the receiving replica merges into its own state.In summary, the state of any given replica changes, eitherby executing an update operation for which it is the ori-gin, or by merging the state received from a remote replica.Each replica is sequential. A merge is the only point wherea replica observes concurrent operations submitted to otherreplicas. Each replica executes sequentially, one local updateor merge at a time; equivalently, an update or merge oper-ation executes atomically, even if it updates multiple dataitems.An application invariant is an assertion over state. The

invariant must evaluate true in every state of every replica.Despite being evaluated against local state, an invariant is ineffect global, since it must be true at all replicas, and replicaseventually converge.

Figure 1 depicts the evolution of state in our auction appli-cation. Each line represents a replica, time progressing fromleft to right. A box represents local state. A curved arrowrepresents an update operation, labelled with the operationname. A diagonal arrow shows propagation (labelled withthe propagated state), merged at the receiving replica.We assume here that application state is a composition

of CRDTs. This is not a limitation, since many basic CRDTtypes have been proposed, which extend familiar sequentialdata types with a concurrency semantics [12].The CRDT convergence rules, or lattice rules, are the fol-

lowing. The state of a replica progresses monotonically withtime. The set of states forms a semi-lattice, i.e., is equippedwith a partial order and a least-upper-bound function. Astate transition represents the execution of either an updateoperation or merge. An update is an inflation, i.e., the result-ing state is no less (in the partial order) than the previousone. Merge computes a state which is the least-upper-boundof the current local state and the received remote state.

We write σ for the current state of a replica, σnew for thenew state after an operation or merge, and σ ′ for the incom-ing state from a remote replica. In the Boogie specificationlanguage (used by our tool, described in Section 4), we denoteoperation execution with the keyword call, an assertionwith assert, and assumptions with assume. An operationop executes only if its precondition is true. Thus, we canwrite the above lattice conditions as follows:

• An update operation op is an inflation.call σnew = op(σ )assert σnew ≥ σ

• Merge is a least upper boundcall σnew = merge(σ , σ ′)assert σnew ≥ σ ∧ σnew ≥ σ ′ #upper bound

assert ∀σ ∗, σ ∗ ≥ σ ∧ σ ∗ ≥ σ ′ =⇒ σ ∗ ≥ σnew #least

Let us illustrate with the auction example (for simplicitywe consider a single auction). Its state is as follows:

• Status: the status of an auction can move from itsinitial state, INVALID (under preparation), to ACTIVE(can receive bids) and CLOSED (no more bids accepted),such that INVALID < ACTIVE < CLOSED.

• Winner: The winner of the auction. It is either ⊥, ini-tially, or the bid with the highest amount. In case ofmultiple bids with same amount, the bid with the low-est id wins. It is ordered such that ∀b ∈ Bids,⊥ ≤ b.

• Bids: Set of bids placed (initially empty)– BidId: A unique identifier for each bid placed– Placed: A boolean flag to indicate whether the bidhas been placed or not, ordered TRUE > FALSE. It isenabled once when the bid is placed. Once placed, abid cannot be withdrawn.

– Amount: An integer representing the amount of thebid; this cannot be modified once the bid is created.

Figure 1 illustrates how the auction state evolves over time;status, bids and winner are represented by a circle, rectangleand a star respectively. The application state is geo-replicatedat data centres in Australia, Belgium and China.We now specify the merge operation for an auction. We

denote the receiving replica state σ = (status, winner, Bids);the received state is denoted σ ′ = (status′, winner′, Bids′).

merge((status ,winner ,Bids),

(status′,winner′,Bids′)):statusnew := max(status ,status′)if winner′,⊥ then winnernew := winner′

else winnernew := winner

∀b, if b ∈ Bids ∩ Bids′ then

Bidsnew .b.placed := Bids.b.placed

∨ Bids′.b.placedelse if b ∈ Bids′ then

Bidsnew .b.placed := Bids′.b.placedelse Bidsnew .b.placed := Bids.b.placed

Invariant Safety for Distributed Applications PaPoC’19, March 25, Dresden, Germany

Figure 1: Evolution of state in an auction application

∀b, if b ∈ Bids then

Bidsnew .b.amount := Bids.b.amount

else Bidsnew .b.amount := Bids′.b.amount

In the absence of any extra synchronisation, it is possi-ble to violate the invariant, as the following execution sce-nario illustrates. Alice from Australia starts an auction, bysetting its status to ACTIVE (green in the figure) in the Aus-tralian replica. Henceforth, the auction can receive bids. TheAustralian replica sends its updated state to the other tworeplicas, which merge it into their own states. Now Bob inBelgium places a bid for $100 (blue). This update is sent toother replicas. Charlie, in China, sees the auction and Bob’sbid. He updates the China state with a higher bid of $105(dotted blue), which is sent to both other replicas. However,due to a network failure, the remote replicas do not receivethis update (red dotted lines). Meanwhile Alice, unaware ofCharlie’s bid, closes the auction and declares Bob’s bid as thewinner. Later when the network heals, the updated states aresent and merged. The auction is new closed, and containsBob’s $100 bid and Charlie’s $105 bid. Unfortunately, Bob’sbid is the winner, violating the application invariant.In the next section, we can discuss how to ensure invari-

ants of applications build on top of the system model wedescribed.

3 PROVING INVARIANTSAs explained earlier, each replica executes a sequence ofstate transitions, due either to a local update, or to a mergeincorporating remote updates. Thus, concurrency can beobserved only through merge.Let us call safe state a replica state that satisfies the in-

variant. Assuming the current state is safe, any update (local

or merge) must result in a safe state. To ensure this, everyupdate is equipped with a precondition that disallows anyunsafe execution.1 Thus, a local update executes only when,at the origin replica, the current state is safe and its precon-dition currently holds. Similarly, merge executes only withtwo safe states that together satisfy a merge precondition.

Formally, an update u (an operation or a merge), mutatesthe local state σ , to a new state σnew = u(σ ). To preserve theinvariant, Inv, we require that

σ ∈ Preu =⇒ u(σ ) ∈ Inv

To illustrate local preconditions, consider an operationclose_auction(w: BidId), which sets auction status toCLOSED and the winner to w. The developer may have writtena precondition such as status = ACTIVE, because closing anauction doesn’t make sense otherwise. In order to ensure theinvariant that the winner has the highest amount, one needsto strengthen it with the clause is_highest(Bids, w), de-fined as ∀b ∈ Bids : b.Amount ≤ w.Amount.

To illustrate merge precondition, consider a CRDT whosestate is the pair of integers, σ = (n,m) ∈ N × N. It has twooperations, incn and incm, that respectively increment n orm by 1, and a merge function:

merge(σ ,σ ′) = (max(n,n′),max(m,m′))

We wish to maintain the invariant that their sum is no morethan 10:

Inv ≜ (n +m) ≤ 10

1 Technically, this is at least the weakest-precondition of the update for theinvariant. It strengthens any a priori precondition that the developer mayhave set.

PaPoC’19, March 25, Dresden, Germany Sreeja S Nair, Gustavo Petri, and Marc Shapiro

The precondition of incn is Preincn ≜ (n +m) ≤ 9; similarlyfor incm. Starting from a safe state (4, 5), two replicas may in-dependently increment to states (5, 5) and (4, 6) respectively.Both are safe. However, merging them would violate theinvariant. Therefore, merge(σ ,σ ′) must have precondition

Premerge ≜ max(n,n′) +max(m,m′) ≤ 10

Since merge can happen at any time, it must be the casethat its precondition is always true, i.e., it constitutes anadditional invariant. Now our global invariant consists oftwo parts: first, the application invariant, and second, theprecondition of merge. We can now state our proof ruleinformally as follows::2

State-Based Safety Rule. Define the precondition ofmerge to be the weakest-precondition of merge, forthe application invariant. The initial state must sat-isfy, and each local update or merge operation mustpreserve, the conjunction of: (i) the application invari-ant, and (ii) the precondition of merge. □

In our Boogie notation, each operation can be verified asfollows:

assume Inv ∧ Premerge ∧ Preopcall σnew = op(σ )assert Inv ∧ Premerge

The case of the merge function can be verified with the fol-lowing condition:

assume Inv ∧ Inv′ ∧ Premergecall σnew = merge(σ , σ ′)assert Inv ∧ Premerge

Note that there are two copies of state, the unprimed localstate of the replica applying the merge, and the primed statereceived from a remote replica. Inv′ denotes that σ ′ preservesthe invariant Inv.

3.1 Applying the proof ruleLet us apply the proof methodology to the auction applica-tion. Its invariant is the following conjunction:(1) A bid is placed only when status is ACTIVE.(2) And: Once a bid is placed, its amount does not change.(3) And: There is no winner until status is CLOSED.(4) And: There is a single winner, the bid with the highest

amount (breaking ties using the lowest identifier).Computing the weakest-precondition of each update op-eration, for this invariant, is obvious. For instance, as dis-cussed earlier, close_auction(w: BidId) gets precondi-tion is_highest(Bids, w), because of Invariant Term 4above.2 We omit the full formalisation and the proof of soundness for brevity.

Despite local updates to each replica preserving the invari-ant, Figure 1 showed that it is susceptible of being violatedby merging. This is the case if Bob’s $100 bid in Belgiumwins, even though Charlie concurrently placed a $105 bid inChina; this occurred because status became CLOSED in Bel-gium while still ACTIVE in China. The weakest-preconditionof merge for Term 4 expresses that, if status in either statesis CLOSED, the winner should be the bid with the highestamount in both the states. Therefore, merge(σ ,σ ′)must havethe following additional precondition:

status=CLOSED =⇒ is_highest(Bids , winner)

∧ is_highest(Bids′, winner)

∧ status′=CLOSED =⇒ is_highest(Bids , winner′)∧ is_highest(Bids′, winner′)

Furthermore, the code for merge uses Term 2, for whichits weakest-precondition is as follows:

∀b ∈ Bids∩Bids′, Bids.b.amount = Bids′.b.amount

These two merge preconditions now strengthen the globalinvariant, in order to preserve safety in concurrent execu-tions. Let us now consider how this strengthening impactsthe local update operations. Since starting the auction doesn’tmodify any bids, this operation trivially preserves it. Placinga bid might violate it, if the auction is concurrently closedin some other replica; conversely, closing the auction couldviolate it, if a higher bid is concurrently placed in a remotereplica. Thus, the auction application is safe when executedsequentially, but is unsafe when updates are concurrent. Thisindicates the specification has a bug, which we now proceedto fix.

3.2 Concurrency Control for InvariantPreservation

As we discussed earlier, the preconditions of operations andmerge are strengthened in order to preserve the invariant.This provides a sequentially safe specification. An applica-tion must also preserve the precondition of merge in orderto ensure concurrent safety. Violating this indicates the pres-ence of a bug in the specification. In that case, the developerneeds to strengthen the application by adding appropriateconcurrency control mechanisms, ie., the operations that failto preserve the precondition of merge might need to syn-chronise. The required concurrency control mechanisms areadded as part of the state in our model. The modified applica-tion state is now composed of the CRDTs that represents thestate and the concurrency control mechanism. Together, itbehaves like a composition of state-based CRDTs. The wholestate should now ensure the lattice conditions described insection 2.

Recall that in the auction example, placing bids and closingthe auction were not preserving the precondition of merge.

Invariant Safety for Distributed Applications PaPoC’19, March 25, Dresden, Germany

This requires strengthening the specification by adding aconcurrency control mechanism to restrict these operations.We can enforce them to be strictly sequential, thereby avoid-ing concurrency at all. But this will affect the availability ofthe application.A concurrency control can be better designed with the

workload characteristics in mind. For this particular applica-tion, we know that placing bids are very frequent operationsthan closing an auction. Hence we try to formulate a concur-rency control like a readers-writer lock. In order to realisethis we distribute tokens to each replica. As long as a replicahas the token, it can allow placing bids. Closing the auctionrequires recalling the tokens from all replicas. This ensuresthat there are no concurrent bids placed and thus a winnercan be declared, respecting the application safety.

The entire specification of the auction application can beseen in Figure 2. The shaded lines in blue indicate the effectof adding concurrency control to the state.

An alternative approach to our treatment of concurrencycontrol could be to consider the invariant as a resource in thestyle of Concurrent Separation Logic [11]. In this case, accessto the application state, described through a separation logicinvariant, is guarded by a concurrency control mechanism(typically some form of a lock). However, this approach istied to separation logic reasoning, where assertions act asresources, and allows one to distinguish local from globalresources.We consider that this was not essential for the kindof proofs that we conduct, but it might be more promisingwhen verifying client programs of our data types.

4 AUTOMATING THE VERIFICATIONIn this section we present a tool to automate the verificationof invariants as discussed in the previous sections. Our tool,called Soteria is based on the Boogie [2] verification frame-work. The input to Soteria is a specification of the applicationwritten in Boogie, an intermediate verification language.

A specification in Soteria will consist of the followingparts:

• State: a declaration of the state. It can be a single CRDTor a composition of CRDTs.

• Comparison function: The programmer provides acomparison function (annotated with keyword @gteq)that determines the partial order on states.

• Operations: The programmer provides the implemen-tation of the operations and their respective precon-ditions, Preop. Operations are encoded either impera-tively as Boogie procedures or declaratively as post-conditions.

• Merge function: The special merge operation is dis-tinguished from the other operations (with annotation

@merge). The programmermust provide a preconditionto merge that is strong enough to prove the invariant.

• Application Invariant: The programmer providesthe invariant (with keyword @invariant) to be veri-fied by the tool as a Boogie assertion over the state.

In addition, Boogie often requires additional informationsuch as: • User-defined data types, • Constants to declarespecial objects such as the origin replicame, or to bound thequantifiers, • Axioms for inductive functions over aggregatedata structures, for instance, to compute the maximum of aset of values, • Loop invariants.The specification of the auction application can be seen

in Figure 2.

VerificationThe verification of a specification is performed in multiplestages; in order:

(1) Syntactic checks: validates the specification for syn-tactical errors and checks whether the pre/post condi-tions are sound.

(2) Compliance check: checks whether the specificationprovides all the elements explained earlier.

(3) Convergence check: checks whether the specifica-tion respects the properties of a state-based CRDT, ie.,each operation inflates the state and merge is the leastupper bound.

(4) Safety check: verifies the safety of the application in-variant, as discussed in section 3. This stage is dividedfurther into two sub-stages:• Sequential safety: whether each individual operation(or merge) upholds the invariant. If not, the designerneeds to strengthen the precondition of the corre-sponding operation (or merge)

• Concurrent safety: whether every operation (andmerge) upholds the precondition of merge. Note that,while this check relates to the concurrent behaviourof state-based CRDTs, the check itself is completelysequential, ie., it does not require reasoning aboutoperations performed by other processes. This checkensures that the invariant remains safe during con-current operation. If this check fails, the applicationneeds stronger concurrency control.

Each check in Soteria 3 generates counterexamples whenthe verification fails. These counterexamples might guidethe developer in debugging the specification according tothe verification steps.

3The tool along with some sample specifications can be accessed at https://github.com/sreeja/soteria_tool.

PaPoC’19, March 25, Dresden, Germany Sreeja S Nair, Gustavo Petri, and Marc Shapiro

Initial state:

status = INVALID ∧ winner = ⊥∧ ∄ b, Bids.b.placed

∧ ∀ r, Tokens.r = true

Comparison function:

status1 ≥ status2 ∧ (winner1,⊥ ∨ winner2=⊥)

∧∀b, (Bids1.b.placed ∨ ¬Bids2.b.placed)∧ (∀r, ¬Tokens1.r ∨ Tokens2.r)

Invariant:

∀b, Bids.b.placed =⇒ status≥ACTIVE

∧ Bids.b.amount >0

status≤ACTIVE =⇒ winner=⊥status=CLOSED =⇒ Bids.winner.placed

∧ is_highest(Bids , winner)

status=CLOSED =⇒ ∀r,¬Tokens.r

{Premerge:

status=CLOSED =⇒ is_highest(Bids , winner)

∧is_highest(Bids′, winner)

∧ status′=CLOSED =⇒ is_highest(Bids , winner′)∧is_highest(Bids′, winner′)

∧ ∀b, Bids.b.amount = Bids′.b.amount∧ ∀r, Tokens.r.me =⇒ Tokens′.r.me∧ ∀r,b, (¬Tokens.r ∧ ¬Bids.b.placed)

=⇒ ¬Bids′.b.placed∧ ∀ r,b, (r,me ∧ ¬Tokens.r ∧ ¬Bids.b.placed)

=⇒ ¬Bids′.b.placed∧ ∀ r,¬Tokens.r =⇒ winner′=winner ∨ winner′=⊥∧ ∀ r, Tokens.r =⇒ winner=⊥ ∧ winner′=⊥}

merge ((status , winner , Bids , Tokens),

(status′,winner′,Bids′,Tokens′)):<merge of status , winner , Bids as in section 2>

∀r, Tokensnew .r := Tokens.r∧Tokens′.r

{Prestart_auction: status = INVALID ∧ winner = ⊥∧ ∀r, Tokens.r}

start_auction ():

statusnew := ACTIVE

winnernew := ⊥{Preplace_bid: ¬Bids.b_id.placed

∧ Bids.b_id.amount = value

∧ status = ACTIVE ∧ winner = ⊥∧ Tokens.me}

place_bid(b_id , value):

Bidsnew .b_id.placed := true

Bidsnew .b_id.amount := value

{Preclose_auction: status = ACTIVE ∧ winner = ⊥∧ Bids.w.placed ∧ is_highest(Bids , w)

∧ ∀r, ¬Tokens.r}close_auction(w):

statusnew := CLOSED

winnernew := w

Figure 2: Specification of auction application

5 RELATEDWORKSeveral works have concentrated on the formalisation andspecification of eventually consistent systems [3, 4, 14]. Anumber of works concentrate on the specification and cor-rect implementation of CRDTs [5, 7]. Our work also verifiesthe CRDT (lattice) conditions, but additionally verifies anarbitrary application invariant over a replica state.

Gotsman et al. [6] provides a proof methodology for prov-ing invariants of CRDTs that propagate operations. The asso-ciated tools [8, 10] performs the check using an SMT solveras the backend and Nair and Shapiro [9] discusses some con-currency control suggestions by using the counterexamplesgenerated by the failed proofs. Gotsman et al. [6] assumethat the underlying network ensures causal consistency, andtheir methodology requires reasoning about concurrent be-haviours. This requires checks for each pair of operationsin the application (reflected as stability verification condi-tions). Gotsman et al. [6] uses an abstract notion of tokensas concurrency control mechanisms. The operations acquiretokens in order to preserve the application invariant.In contrast, Soteria focuses on state-based CRDTs. We

check convergence by verifying the lattice conditions of sec-tion 2 and that because of the rules shown in section 3, wecan reduce the problem of verifying the invariant to sequen-tial proof obligations. This is reflected by the fact that allof our proofs are standard pre/post conditions checks usingthe Boogie framework. Boogie framework. In contrast withGotsman et al. [6], Soteria includes concrete specification ofconcurrency control as part of the application state.To the best of our knowledge, ours is the first attempt in

automated verification of invariants of state-based CRDTs.

6 CONCLUSIONWe have presented a proof methodology to verify invariantsof state-based CRDT implementations guaranteeing: (1) thatthe implementation satisfies the lattice conditions of state-based CRDTs [1], and (2) that the implementation satisfiesprogrammer provided invariant reducing the problem tochecking that each operation of the data type satisfies aprecondition of the merge function of the state.

We implemented Soteria, a tool sitting on top of the Boogieverification framework, to specify the implementation, itsinvariant and validate it.In future work, we plan to automate concurrency con-

trol synthesis. The synthesised concurrency control can beanalysed and adjusted dynamically to minimise the cost ofsynchronisation. Another direction for future work can beto decouple the update propagation mechanism of CRDTfrom the proof rule resulting in a generic proof rule to verifydistributed systems.

Invariant Safety for Distributed Applications PaPoC’19, March 25, Dresden, Germany

ACKNOWLEDGMENTSThe authors would like to thank the anonymous reviewersfor their comments which helped in improving this paper.This research is supported in part by the RainbowFS project(Agence Nationale de la Recherche, France, number ANR-16-CE25-0013-01) and by European H2020 project 732 505LightKone (2017–2020).

REFERENCES[1] C. Baquero, P. S. Almeida, A. Cunha, and C. Ferreira. Composition

in state-based replicated data types. Bulletin of the EATCS, 123, 2017.URL http://eatcs.org/beatcs/index.php/beatcs/article/view/507.

[2] M. Barnett, B.-Y. E. Chang, R. DeLine, B. Jacobs, and K. R. M. Leino.Boogie: A modular reusable verifier for object-oriented programs. InProceedings of the 4th International Conference on Formal Methods forComponents and Objects, FMCO’05, pages 364–387, Berlin, Heidelberg,2006. Springer-Verlag. ISBN 3-540-36749-7, 978-3-540-36749-9. doi:10.1007/11804192\_17. URL http://dx.doi.org/10.1007/11804192_17.

[3] S. Burckhardt. Principles of Eventual Consistency, volume 1 of Foun-dations and Trends in Programming Languages. Now Publishers, Oct.2014. doi: 10.1561/2500000011. URL http://research.microsoft.com/pubs/230852/final-printversion-10-5-14.pdf.

[4] S. Burckhardt, A. Gotsman, H. Yang, and M. Zawirski. Replicated datatypes: Specification, verification, optimality. In Symp. on Principlesof Prog. Lang. (POPL), pages 271–284, San Diego, CA, USA, Jan. 2014.doi: 10.1145/2535838.2535848. URL http://doi.acm.org/10.1145/2535838.2535848.

[5] V. B. F. Gomes, M. Kleppmann, D. P. Mulligan, and A. R. Beresford. Aframework for establishing strong eventual consistency for conflict-free replicated datatypes. Archive of Formal Proofs, 2017, 2017. URLhttps://www.isa-afp.org/entries/CRDT.shtml.

[6] A. Gotsman, H. Yang, C. Ferreira, M. Najafzadeh, and M. Shapiro.’Cause I’m Strong Enough: Reasoning about consistency choices indistributed systems. In Symp. on Principles of Prog. Lang. (POPL), pages371–384, St. Petersburg, FL, USA, 2016. doi: 10.1145/2837614.2837625.URL http://dx.doi.org/10.1145/2837614.2837625.

[7] R. Jagadeesan and J. Riely. Eventual consistency for CRDTs. InA. Ahmed, editor, Programming Languages and Systems - 27th Eu-ropean Symposium on Programming, ESOP 2018, Held as Part of the

European Joint Conferences on Theory and Practice of Software, ETAPS2018, Thessaloniki, Greece, April 14-20, 2018, Proceedings, volume 10801of Lecture Notes in Computer Science, pages 968–995. Springer, 2018.ISBN 978-3-319-89883-4. doi: 10.1007/978-3-319-89884-1\_34. URLhttps://doi.org/10.1007/978-3-319-89884-1_34.

[8] G. Marcelino, V. Balegas, and C. Ferreira. Bringing hybrid consistencycloser to programmers. In W. on Principles and Practice of Consistencyfor Distr. Data (PaPoC), PaPoC ’17, pages 6:1–6:4, Belgrade, Serbia,2017. Euro. Conf. on Comp. Sys. (EuroSys), ACM. doi: 10.1145/3064889.3064896. URL http://doi.acm.org/10.1145/3064889.3064896.

[9] S. Nair and M. Shapiro. Improving the “Correct Eventual Consistency”tool. Rapport de recherche RR-9191, Institut National de la Rechercheen Informatique et Automatique (Inria), Paris, France, July 2018. URLhttps://hal.inria.fr/hal-01832888.

[10] M. Najafzadeh, A. Gotsman, H. Yang, C. Ferreira, and M. Shapiro.The CISE tool: Proving weakly-consistent applications correct. InW. on Principles and Practice of Consistency for Distr. Data (PaPoC),EuroSys 2016workshops, London, UK, Apr. 2016. Assoc. for ComputingMachinery Special Interest Group on Op. Sys. (SIGOPS), Assoc. forComputing Machinery. doi: 10.1145/2911151.2911160. URL http://dx.doi.org/10.1145/2911151.2911160.

[11] P. W. O’Hearn. Resources, concurrency, and local reasoning. Theor.Comput. Sci., 375(1-3):271–307, 2007. doi: 10.1016/j.tcs.2006.12.035.URL https://doi.org/10.1016/j.tcs.2006.12.035.

[12] M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. A comprehen-sive study of Convergent and Commutative Replicated Data Types.Rapport de Recherche 7506, Institut National de la Recherche en Infor-matique et Automatique (Inria), Rocquencourt, France, Jan. 2011.

[13] M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. Conflict-freereplicated data types. In X. Défago, F. Petit, and V. Villain, editors, Int.Symp. on Stabilization, Safety, and Security of Dist. Sys. (SSS), volume6976 of Lecture Notes in Comp. Sc., pages 386–400, Grenoble, France,Oct. 2011. Springer-Verlag. doi: 10.1007/978-3-642-24550-3_29. URLhttps://doi.org/10.1007/978-3-642-24550-3_29.

[14] K. Sivaramakrishnan, G. Kaki, and S. Jagannathan. Declarative pro-gramming over eventually consistent data stores. In Assoc. for Comput-ing Machinery Special Interest Group on Pg. Lang. (SIGPLAN), PLDI ’15,pages 413–424, Portland, OR, USA, 2015. Assoc. for ComputingMachin-ery, Assoc. for Computing Machinery. doi: 10.1145/2737924.2737981.URL http://doi.acm.org/10.1145/2737924.2737981.