Cassandra & Scylla at Kenshoo

About Me

• Wrote “Basic” code when I was a kid

• 17 years in the internet industry

• Big data fanatic for the last 6 years

• Big data team leader At Kenshoo

• Our team is Big data DBA

• Programming: ETL & Administration tools

Kenshoo

• 10 years, Tel-Aviv based Startup

• Industry Leader in Digital Marketing

• 500+ employees

• Heavy data shop

Kenshoo Legacy Architecture

Bigdata at Kenshoo

Cassandra at Kenshoo

• Datastax Community

• 5 clusters

• 70 nodes

• Biggest cluster, ver 1.2.19▪ 40 physical nodes▪ 4TB compressed to 1TB per node▪ 14 Billion records▪ 1500 bytes values, IOPS: 5K avg, 30K burst▪ Processing clicks and conversion for user behavior

Cassandra Cost

• Cassandra is great at writes (At first)

• Postponing the “cost” to background processes (no free meals)▪ Compact▪ Repair▪ Cleanup▪ Add / Remove node

War & PeaceKnowing your Cassandra

Day to day

• Requires a lot of in depth knowledge

• Lack of documentation

• Tuning per application

• Lot’s of custom maintenance scripts

• Don’t run repair , only rebuild

• Compaction didn’t complete during maintenance window

• Leveled CS is a ColumnFamily cluster wide configuration

• Found a jmx “hack” for a per server config

Migration to Leveled Compaction Strategy

• It took us a month to manually switch

• Needed to reconfigure each server after restart

GC Hell

• Lot’s of GC under load

• GC causing 95th percentile performance▪ One problematic node affects the cluster performance

• Full restart to the cluster each week

• Change rpc server type to hsha

• Tuning is black magic, takes days to see effects

Maintenance is delicate

• Need to wait between adding and removing nodes for

cluster to rebalance

• Turn off a node thrift service

• Running partial cleanup

• Tuning of params▪ compaction_throughput_mb_per_sec▪ concurrent_compactors

Scylla Evaluation

Self tuning features

Lab setup

• i2.8xlarge

• Cassandra 2.1.15▪ compaction_throughput_mb_per_sec = 0 (16)▪ stream_throughput_outbound_megabits_per_sec = 10000 (200)▪ inter_dc_stream_throughput_outbound_megabits_per_sec = 10000 (200)

• Scylla 1.3▪ out of the box

GC at Scylla

Repair

• 3 x i2.8xlarge

• RF 3

• 72GB data per node• 10M rows

▪ 5 columns▪ 1500 bytes value

• Delete all the data from one node

Repair results

Cleanup at Scylla

• 4 x i2.8xlarge

• RF 3

• 270GB data per node• 50M rows

▪ 5 columns▪ 1500 bytes value

• Decommission a node and join it back

Cleanup results

Compact at Scylla

• 3 x i2.8xlarge

• RF 3

• 30 minute stress• 10M rows

▪ 5 columns▪ 1500 bytes value

Compaction result

Latency & Throughput

• 3 x i2.8xlarge

• RF 3

• 72GB data per node• 10M rows

▪ 5 columns▪ 1.5K value▪ 3 writes, 2 reads mixed

• 4 x cassandra-stress nodes▪ 30 minutes

• Scylla has lower cost▪ Compaction, repair & cleanup are much more efficient▪ Consistent in latency under much higher load

• Moving forward▪ Integrate it in an inner production monitoring system

Conclusion

Q&A

Thank You!

Contact: noam.hasson@kenshoo.com