DataStax Enterprise in the Field – 20160920

DataStax Enterprise in the Field

Daniel Cohen Solutions Engineer @ DataStax

© DataStax, All Rights Reserved.

But Enough About Me…

• Solutions Engineer at DataStax • LA ➜ SF ➜ NYC ➜ SF ➜ London • Previously at JP Morgan in London • Finance & digital media

2


1 Introductions

2 Top Customer Questions

3 Field Lessons: Big Irish Bank

4 Field Lessons: Big British Bank

3


Top Customer Questions

• What are all the other [banks] doing? • How many nodes do I need? • What do you mean SSDs? • How do I load data from [Oracle]? • We already have [MongoDB] for NoSQL.

What’s the difference? • What are all the other [banks] doing?

4

What are all the other [banks] doing?

“Tell me secrets about my competitors.”


Transform Legacy Infrastructure

6

…USA Equities

UK FX

UK Bonds

Global Users

Legacy Systems

USA FX

DataStax Enterprise ClusterDSE

User Interface / Application Services


Transition Legacy to Microservices

7

Users µServices

DC NY1A B

C D

DC LDN1A Z

B

Messages

DC NY1

DC LDN1

DC NY1

DC LDN1

USA Customers

Data

UK Accounts

Legacy

C

DSE

DSE

How many nodes do I need?

“How long is a piece of string?”


The Node Count Dance

9



• “How many nodes do I need?” is a natural question. – Large organizations buy hardware months in advance.

9




• Desires ➔ Storage, Throughput, Latency, SLAs

9




• Desires ➔ Storage, Throughput, Latency, SLAs• Realities

– Cost – Data center capacity (space) – Operational capacity (people) – Your hardware – Your use cases

9






• Lesson 1 ➔ Computer science is about trade-offs.

9






• Lesson 1 ➔ Computer science is about trade-offs.• Lesson 2 ➔ Test, iterate, test.

9






• Lesson 1 ➔ Computer science is about trade-offs.• Lesson 2 ➔ Test, iterate, test.• Lesson 3 ➔ Good news! DSE scales linearly.

9

What do you mean SSDs?

“We have an amazing SAN.”


Storage Matters

11

SSD (consumer grade)

• 10K – 1M IOPS • 400 MB – 3 GB bandwidth • < 200us latency

✴ Acknowledgements to my colleague Kathryn Erickson.

15K RPM HDD (spinning rust)

• ~ 200 IOPS • ~ 160 MB bandwidth • > 5 ms latency


Storage Interfaces Matter

12

Interface Transfer Rate

SATA III 6 Gb/s

SAS II 6 Gb/s

SAS III 12 Gb/s

PCIe Gen 2 x8 32 Gb/s


A Nondeterministic Path to Failure

13



• What about my incredible SAN?

13



• What about my incredible SAN?– Do not use network attached storage with DSE.

13




• But our SAN is awesome! We paid a lot of money for it.

13




• But our SAN is awesome! We paid a lot of money for it.– No! Do not use network attached storage with DSE.

13





• Fine. What about EBS?

13





• Fine. What about EBS?– Let’s discuss!

13


Starting Points

Workload CPU RAM Storage

DSE (Read Heavy) 8-24 cores 32-128 GB ✴ Local SSD (.5 - 2 TB)

DSE (Write Heavy) 12-32 cores 32-128 GB Local SSD (1-3 TB)

DSE + Search 16-32 cores 128 GB Local SSD (1-3 TB)

DSE + Analytics 16-32 cores 128+ GB Local SSD (1-3 TB)

✴ Got extra RAM? Cache is king.

✴✴ 1 Gb ethernet is fine. 10Gb is future-proof.

14

We already have [MongoDB] for NoSQL. What’s the difference?

“Behold the one true NoSQL database.”


NoSQL16


NoSQLFan

tasy16


1 Introductions




17


Proof of Technology @ Big Irish Bank

18

Initial Goals

• Deploy on AWS • Ingest ten years of (fake)

customer data efficiently • Fast retrieval & search

Synopsis

• Payment Services Directive (PSD II) and Open Banking

• Customer access to current and historical data via APIs

• Competitive PoT versus other database vendors


Hardware

19


Hardware

19

PoT Recommendation • 6 x i2.xlarge (AWS) • 4 vCPU, 30.5 GB RAM • 1 x 800 local SSD


PoT Mark 1 • c4.8xlarge (AWS) • 36 vCPU, 60 GB RAM • EBS only

Hardware

19




Hardware

19


PoT Final • 6 x i2.xlarge (AWS) • 4 vCPU, 30.5 GB RAM • 1 x 800 local SSD



Hardware

19


Production • 8 nodes across 2 data centers (4:4) • HP DL380 Gen9 ➔ 32 cores, 256 GB RAM, 3.2 TB SSDs on SAS III • 10 Gb ethernet, fiber between DCs

PoT Final • 6 x i2.xlarge (AWS) • 4 vCPU, 30.5 GB RAM • 1 x 800 local SSD


Lessons

20


Lessons

20

1) The Node Count Dance is iterative. • Initial node count estimates were low. • Early refusal to modify AWS setup. • Avoid rigidity. Test, iterate, test.


Lessons

20

2) Quis custodiet ipsos custodes? • Hit performance plateau at 5,000 ops/s. • Added second jMeter, performance

doubled to 10,000 ops/s. • jMeter was the bottleneck! • Who will test the testers?



Lessons

20




3) EBS is still network attached. • 99% Read Latency (milliseconds)

▫ 3.311 ➔ local SSD ▫ 35.425 ➔ EBS Provisioned SSD

• Competing vendor falsified numbers. • Lies, damned lies, and statistics.


Lessons

20




4) Not all data needs to be hot. • PoT Mark 1 ➔ 10 years of hot data

▫ ~ 20 billion transactions ▫ ~ 30 nodes to reach latency targets

• PoT Final ➔ 2 years of hot data • Do not architect by convenience.

3) EBS is still network attached. • 99% Read Latency (milliseconds)

▫ 3.311 ➔ local SSD ▫ 35.425 ➔ EBS Provisioned SSD

• Competing vendor falsified numbers. • Lies, damned lies, and statistics.


1 Introductions




21


Production Pilot @ Big British Bank

22

Initial Goals

• Transition from mothballed trials of OrientDB, Titan

• Ingest enormous quantities of data from legacy DB

• Prove graph at scale

Synopsis

• Customer 360° use case across banking group

• DSE Graph • Dissatisfied with other

graph databases


Hardware

23


Hardware

23

Pilot Mark 1 • “Private Cloud” • N x Hosted VM • 8 vCPU, 112 GB RAM • SAN only (for now)


Pilot Mark 2 • “Hadoop Leftovers” • 4 x HP DL380s • 24 cores, 512 GB RAM • 1 x 2.1 TB SSD • 14 x 2 TB HDDs

Hardware

23




Hardware

23


Pilot Final • 3 x Dell C6220 • 12 cores, 128 GB RAM • 6 x 1 TB SATA HDDs

▫ 2 x OS ▫ 1 x commit log ▫ 3 x data, caches



Hardware

23


Production Target 16 nodes across 2 data centers (8:8) HP DL380 Gen9 ➔ 24 cores, 528 GB RAM, 3.4 TB SSDs

Pilot Final • 3 x Dell C6220 • 12 cores, 128 GB RAM • 6 x 1 TB SATA HDDs

▫ 2 x OS ▫ 1 x commit log ▫ 3 x data, caches


Lessons

24


Lessons

24

1) DSE essentials are critical. • Great team but zero DSE experience. • Ad hoc education introduces risk. • Walk before you run.


Lessons

24

2) Node Count Dance applies to Graph. • Data size unknown due to privacy. • Load 5% of data, extrapolate. • Test, iterate, test.



Lessons

24



3) Hardware matters, of course. • Leftover Hadoop boxes, spinning rust. • Get creative with configuration & tuning. • “Under no circumstances should you do

load tests on these boxes.”


Lessons

24



4) Avoid surprises before deadlines. • Upgraded from RHEL 6.7 to 7.1. • CPU spikes made nodes unusably slow. • Revert! • Nobody move, nobody gets hurt.

3) Hardware matters, of course. • Leftover Hadoop boxes, spinning rust. • Get creative with configuration & tuning. • “Under no circumstances should you do

load tests on these boxes.”

Thank you!

Daniel Cohen Solutions Engineer @ DataStax

[email protected] @CodaAzzurra

DataStax Enterprise in the Field – 20160920

Technology

Transcript of DataStax Enterprise in the Field – 20160920