Byzantine Failures -...
Transcript of Byzantine Failures -...
![Page 1: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/1.jpg)
Hardening Cassandra Against Byzantine FailuresRoy Friedman and Roni LicherTechnion - Israel Institute of Technology
OPODIS 2017
![Page 2: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/2.jpg)
Overview
In this research we:● Analyze the presence of byzantine failures in Cassandra● Suggest solutions to prevent them● Iterate to improve common case performance● Benchmark implementation
![Page 3: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/3.jpg)
Cassandra
● Distributed Database● Open Source● Column Families
![Page 4: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/4.jpg)
Cassandra
● Distributed Database● Open Source● Column Families● Tunable Consistency● Very Scalable
![Page 5: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/5.jpg)
Cassandra
● Distributed Database● Open Source● Column Families● Tunable Consistency● Very Scalable
YCSB comparison [1]
[1] Cooper, Brian F., et al. "Benchmarking cloud serving systems with YCSB." Proceedings of the 1st ACM symposium on Cloud computing. ACM, 2010.
● Great performance
![Page 6: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/6.jpg)
Cassandra
● Distributed Database● Open Source● Column Families● Tunable Consistency● Very Scalable● Great performance● Highly adopted:
![Page 7: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/7.jpg)
Cassandra - The Ring
A
B
C
D
E
F
G
● Distributed Hash Table
● Replication
Key K
Nodes B,C & D store keys in range A & B (RF=3)
● Full membership view (gossip based)
![Page 8: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/8.jpg)
Cassandra - Client operations
A
B
C
D
E
F
G
Client decides the number of nodes that have to acknowledge the operation Write(K,V)
hash(K)=2
Write(K,V)
Write(K,V)
Write(K,V)
![Page 9: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/9.jpg)
Cassandra - Handling Failures - Hinted Handoff
A
B
C
D
E
F
G
On a node failure, the proxy saves the value
Write(K,V)Write(K,V)
Write(K,V)
Write(K,V)Send K to C
C
![Page 10: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/10.jpg)
Cassandra - Handling Failures - Anti-Entropy
● If a node is unresponsive for long enough, the saved hint might get deleted
● Nodes can exchange Merkle Trees and sync (expensive)
● A value can be updated during a Read-Repair
![Page 11: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/11.jpg)
Assumptions for Byzantine Hardening
● Fewer than ⅓ of the nodes are Byzantine:
N = 3f+1
● Fully connected network● Public Key Infrastructure and SSL● Loosely synchronized clocks (not perfect)
![Page 12: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/12.jpg)
Replication
● On writes, waiting for all nodes is not possible● Quorums:
○ All read sets have to intersect with all write sets○ In Cassandra majority is used:
Write Read
N=2f+1 W=f+1 R=f+1
f=1
![Page 13: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/13.jpg)
Byzantine Replication?
Read 2 Read 6
f=1 2 2 6
![Page 14: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/14.jpg)
Byzantine Replication
Using Byzantine quorums:● Writes and reads intersect in at least one correct node
Write Read
f=1
N=3f+1 W=2f+1 R=2f+1
2 2 6
![Page 15: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/15.jpg)
Digital Signatures
● A proof for the origin of the data● Requires a shared key
Public Key Signatures:
A
Sign with a private key Verify with a public key
Symmetric Key Signatures:
A
Sign with a private key Verify with a private key
B
B cannot prove to a third party that he got a message from A
Slow Fast
![Page 16: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/16.jpg)
The Proxy - Plain Cassandra
The proxy cannot be trusted
![Page 17: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/17.jpg)
Write Algorithm - Plain Cassandra
ClientNode1(Proxy) Node2 Node3
Write KWrite K
Write KWrite KACK ACK
ACKACK
![Page 18: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/18.jpg)
Write Algorithm - Hardened Cassandra (Option 1)
ClientNode1(Proxy) Node2 Node3
Write K, Sign(K)Write K
Write KWrite KSign(Sign) Sign(Sign) Sign(Sign)
3 Signatures
Node4
Sign(Sign)Write K
Signatures are incorrect?Write K, Sign(K)
Signatures and verifications (using only PKI):
Proxy:
Verify: 2f+1
Nodes:Sign: 3f+1Verify: 3f+1
Client:Sign: 1Verify: 2f+1
Total:Sign: 3f+2Verify: 7f+1
![Page 19: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/19.jpg)
Write Algorithm - Hardened Cassandra (Optimization 1)
● Should the proxy verify the nodes signatures?
Node
Correct Signature?
Signature
● No, if client isn’t happy, contact the proxy again...
ProxySignature
![Page 20: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/20.jpg)
Write Algorithm - Hardened Cassandra (Optimization 1)
ClientNode1(Proxy) Node2 Node3
Write K, Sign(K)Write K
Write KWrite KSign(Sign) Sign(Sign) Sign(Sign)
3 Signatures
Node4
Sign(Sign)Write K
Signatures and verifications (using only PKI):
Proxy:
Verify: 2f+1
Nodes:Sign: 3f+1Verify: 3f+1
Client:Sign: 1Verify: 2f+1
Total:Sign: 3f+2Verify: 5f+1
Total:Sign: 3f+2Verify: 7f+1
![Page 21: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/21.jpg)
Write Algorithm - Hardened Cassandra (Optimization 2)
● Now, the nodes sign and only the client verify it…
● Switch to symmetric key signatures from nodes to client
Node
Sign for proxy and client
SignatureProxy
Signature
Sign for client
![Page 22: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/22.jpg)
Write Algorithm - Hardened Cassandra (Optimization 2)
ClientNode1(Proxy) Node2 Node3
Write K, Sign(K)Write K
Write KWrite KSign(Sign) Sign(Sign) Sign(Sign)
3 Signatures
Node4
Sign(Sign)Write K
Signatures and verifications:
Nodes:Sign: 3f+1(s)Verify: 3f+1(p)
Client:Sign: 1(p)Verify: 2f+1(s)
Total:Sign: 3f+1(s) & 1(p)Verify: 2f+1(s) & 3f+1(p)
![Page 23: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/23.jpg)
Write Algorithm - Hardened Cassandra (Optimization 3)
● Still, not fast enough ● Switch to symmetric key signatures from client to nodes?
ASignature
ProxySignature
ASignature
● If new nodes join, how can they verify the signature?● If a node misses a write, how can it trust his neighbours?● How can the client know which nodes are responsible for
each value?Using only symmetric signatures is tricky...
![Page 24: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/24.jpg)
Write Algorithm - Hardened Cassandra (Optimization 3)
● Still, not fast enough ● Switch to symmetric key signatures from client to nodes?
ASignature
ProxySignature
ASignature
● If new nodes join, how can they verify the signature?● If a node misses a write, how can it trust his neighbours?● How can the client know which nodes are responsible for
each value?Using only symmetric signatures is tricky...
![Page 25: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/25.jpg)
Write Algorithm - Hardened Cassandra (Optimization 3)
● A client signs the value with a public key signature
Value
Public_Sign(V)
● Then, covers the value and signature with symmetric signatures, one for each node
● A node will verify only the symmetric signature and store the public signature
Symmetric_Sign_Node1(V, PS)Symmetric_Sign_Node2(V, PS)Symmetric_Sign_Node3(V, PS)Symmetric_Sign_Node4(V, PS)
2Verify:
Symmetric_Sign_Node2(V, PS)Store:
Value, Public_Sign(V)
![Page 26: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/26.jpg)
Write Algorithm - Hardened Cassandra (Optimization 3)
● Existing nodes will use only the symmetric signatures● New nodes / outdated nodes will use the public key
signature
2 42
I am new
V, PublicSign(V)
![Page 27: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/27.jpg)
Write Algorithm - Hardened Cassandra (Optimization 3)
ClientNode1(Proxy) Node2 Node3
Write K, Sign(K)Write K
Write KWrite KSign(Sign) Sign(Sign) Sign(Sign)
3 Signatures
Node4
Sign(Sign)Write K
Signatures and verifications:
Nodes:Sign: 3f+1(s)Verify: 3f+1(s)
Client:Sign: 1(p) & 3f+1(s)Verify: 2f+1(s)
Total:Sign: 6f+1(s) & 1(p)Verify: 5f+2(s)
![Page 28: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/28.jpg)
Write Algorithm - Hardened Cassandra (Optimization 4)
● Left with only one public key signature● Can we do it fast?
ECDSA (The Elliptic Curve Digital Signature Algorithm) RSA (Rivest, Shamir, and Adleman)
Fast SigningSlow Verification
Slow SigningFast Verification
![Page 29: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/29.jpg)
Read Algorithm - Plain Cassandra
ClientNode1(Proxy) Node2 Node3
Read K
Data DigestIf all digests match, returns data
Read K (digest)Read K (data)
On Digest mismatchRead K (data)
Stale dataResolve
Ack
Write Resolved
Resolved Data
![Page 30: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/30.jpg)
Read Algorithm - Hardened Cassandra
ClientNode1(Proxy) Node2 Node3
Read K
Data, signature SignatureIf all digests match, returns data & signatures
Read K (digest)Read K (data)
On Digest mismatchRead K (data)
DataResolve
SignatureWrite Resolved
Resolved Data, Signatures & all versions
Node4
Read K (digest)Signature
Read K (data) Read K (data)Stale data Data
Write ResolvedSignature
On failure, retrying with new proxy
![Page 31: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/31.jpg)
Read Algorithm - Hardened Cassandra (Optimizations)
Same as in the write path:● Proxy does not verify, client contacts it again if necessary● Symmetric signatures from nodes to client
ClientNode1(Proxy) Node2 Node3
Read K
Data, signature SignatureIf all digests match, returns data & signatures
Read K (digest)Read K (data)
Node4
Read K (digest)Signature
Signatures and verifications:Nodes:Sign: 2f+1(s)
Client:Verify: 2f+1(s)
Total:Sign: 2f+1(s) Verify: 2f+1(s)
What about verifying the data signature?
![Page 32: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/32.jpg)
Performance - YCSB - Workload A - 50/50 Read/Writes - Achieved Throughput
Same as Cassandra, 5 years ago [YCSB paper]
![Page 33: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/33.jpg)
Performance - YCSB - Workload A - 50/50 Read/Writes - Latency
RSA?
![Page 34: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/34.jpg)
Performance - YCSB - Workload A - 50/50 Read/Writes - Achieved Throughput - More
![Page 35: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/35.jpg)
Performance - YCSB - Workload A - 50/50 Read/Writes - Latency - More
![Page 36: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/36.jpg)
Other issues in the Paper
● Byzantine clients
● Deleting values
● Column families
● Membership
![Page 37: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/37.jpg)
Future Work
● Improve performance○ Introduce real batching
● Support more functionalities○ Lightweight transactions○ Multi data-center operations
![Page 38: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/38.jpg)
?
??
?
Questions?
??
? ?
![Page 39: Byzantine Failures - ULisboaopodis2017.campus.ciencias.ulisboa.pt/presentations/Session7_paper2.pdf · Proxy does not verify, client contacts it again if necessary Symmetric signatures](https://reader030.fdocuments.in/reader030/viewer/2022040806/5e45819f45c14916b811d58e/html5/thumbnails/39.jpg)
Thank You