Post on 22-Jan-2018
DataStax Enterprise & Apache Cassandra Essentials for Financial Services
Daniel Cohen Solutions Engineer
Company Confidential© 2015 DataStax, All Rights Reserved. 2
1 Introduction to DataStax Enterprise
2 Technology Overview
3 Architecture in Enterprise
4 Use Cases in Finance
5 Questions?
Company Confidential© 2015 DataStax, All Rights Reserved.
DataStax Enterprise
3
A Purpose-Built Platform on
Apache Cassandra
Company Confidential© 2015 DataStax, All Rights Reserved.
Certified Apache Cassandra
4
• Certified for production environments • Rigorous certification process:
– Extensive quality assurance testing – Performance and scale tests with
1,000 node clusters – 3rd party software validation – Defect resolution
• Pass internal certifications and audits
Company Confidential© 2015 DataStax, All Rights Reserved.
Visual Admin
5Confidential
Company Confidential© 2015 DataStax, All Rights Reserved.
Visual Admin
6
• Visual, browser-based user interface • Installation, configuration, and
administration tasks carried out in point-and-click fashion
• REST API allows scripting of any OpsCenter function
• Works on any device (tablet, etc.)
Company Confidential© 2015 DataStax, All Rights Reserved.
Management Services
7
• Repair service – automatically keeps data consistent across a cluster.
• Capacity service – historical trend analysis and forecasts future resource needs.
• Performance service – provides insight into the performance of a cluster with queryable CQL-based diagnostic objects
• Best Practice service – enforces best practice rules across many areas (e.g. security, configuration, etc.) to ensure proper setup and optimization.
• Proactive alerts and external notifications
Company Confidential© 2015 DataStax, All Rights Reserved.
Enterprise Security
8
Company Confidential© 2015 DataStax, All Rights Reserved.
Enterprise Security
8
Inherited from Open Source Cassandra
Internal authentication (IDs, passwords)
Permission management via GRANT/REVOKE
Client-to-node, node-to-node encryption
Company Confidential© 2015 DataStax, All Rights Reserved.
Enterprise Security
8
Inherited from Open Source Cassandra
Internal authentication (IDs, passwords)
Permission management via GRANT/REVOKE
Client-to-node, node-to-node encryption
DataStax Enterprise Only
External authentication(eg Kerberos, LDAP/Active Directory)Transparent data encryption
Data auditing
Company Confidential© 2015 DataStax, All Rights Reserved.
Enterprise Security
8
Inherited from Open Source Cassandra
Internal authentication (IDs, passwords)
Permission management via GRANT/REVOKE
Client-to-node, node-to-node encryption
DataStax Enterprise Only
External authentication(eg Kerberos, LDAP/Active Directory)Transparent data encryption
Data auditing
Company Confidential© 2015 DataStax, All Rights Reserved.
Enterprise Support
9
• 24x7x365 Technical Support • Covers both production and
non-production environments • Scheduled health checks for
assistance on architecture, design, and tuning
• Hot-fix support for emergency maintenance situations
Company Confidential© 2015 DataStax, All Rights Reserved.
DSE Analytics
10
• Integrated Real-Time Analytics • Integrated Batch Analytics • External Batch Analytics
– external Hadoop connectivity, certified to Cloudera, HortonWorks
– link hot transactional data with cold data from data warehouse / lake
ETL
Company Confidential© 2015 DataStax, All Rights Reserved.
DSE Search
11
• DSE Search inherits all the power and capabilities of Solr and builds on top of it
• Built-in scale out and continuous availability for search operations
• Enabled across multiple data centers and cloud providers
• Search via native API or through CQL
ETL
Company Confidential© 2015 DataStax, All Rights Reserved.
DSE In-Memory
12
• Brings all the goodness of Cassandra to an in-memory database
• In-Memory tables look and act like any Cassandra table
• Great for use cases requiring low latency reads
• Combine with in-memory analytics for full in-memory transactional / analytical processing
Company Confidential© 2015 DataStax, All Rights Reserved.
DataStax Enterprise
13
A Purpose-Built Platform on
Apache Cassandra
Company Confidential© 2015 DataStax, All Rights Reserved. 14
1 Introduction to DataStax Enterprise
2 Technology Overview
3 Architecture in Enterprise
4 Use Cases in Finance
5 Questions?
Company Confidential© 2015 DataStax, All Rights Reserved.
• A Distributed NoSQL Database • Google BigTable ✖ Amazon Dynamo
• Continuously Available • Disaster avoidance, not disaster recovery
• Scale Out with Linear Performance • Just add nodes
• Run on Commodity Hardware • In the cloud, on premise, or hybrid
What is Apache Cassandra?
15
Company Confidential© 2015 DataStax, All Rights Reserved.
• Fully Distributed • Data spread over multiple nodes • All nodes participate in a cluster • Configurable data replication
• Masterless • All nodes are equal • Read from or write to any node
• No Single Point of Failure
Distributed Architecture
16
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Node 3 3rd copy
Company Confidential© 2015 DataStax, All Rights Reserved.
A Quick Vocabulary Lesson
17
Company Confidential© 2015 DataStax, All Rights Reserved.
A Quick Vocabulary Lesson
Replication Factor – RF • Specifies how many copies of a row of data C* distributes across nodes • Highly configurable for SLA and network topology (racks, data centers)
CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'NY1' : 3, 'NY2' : 3};
17
Company Confidential© 2015 DataStax, All Rights Reserved.
A Quick Vocabulary Lesson
Replication Factor – RF • Specifies how many copies of a row of data C* distributes across nodes • Highly configurable for SLA and network topology (racks, data centers)
CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'NY1' : 3, 'NY2' : 3};
17
Consistency Level – CL • Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation
ANY … ONE … QUORUM … EACH_QUORUM … ALL
Company Confidential© 2015 DataStax, All Rights Reserved.
A Quick Vocabulary Lesson
Replication Factor – RF • Specifies how many copies of a row of data C* distributes across nodes • Highly configurable for SLA and network topology (racks, data centers)
CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'NY1' : 3, 'NY2' : 3};
17
Consistency Level – CL • Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation
ANY … ONE … QUORUM … EACH_QUORUM … ALL
Company Confidential© 2015 DataStax, All Rights Reserved.
Multiple Data Centers
18
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Node 3 3rd copy
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Node 3
London DCNew York DC
Company Confidential© 2015 DataStax, All Rights Reserved.
• Need more storage? Add more nodes.
• Need greater throughput? Add more nodes.
• Predictable, linear performance gains
Scale Out Linearly
19
Company Confidential© 2015 DataStax, All Rights Reserved.
• Need more storage? Add more nodes.
• Need greater throughput? Add more nodes.
• Predictable, linear performance gains
Scale Out Linearly
19
Company Confidential© 2015 DataStax, All Rights Reserved.
• Need more storage? Add more nodes.
• Need greater throughput? Add more nodes.
• Predictable, linear performance gains
Scale Out Linearly
19
Company Confidential© 2015 DataStax, All Rights Reserved.
Linear Scale Out at Netflix
20
Source: The Netflix Tech Blog http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
Node 2 2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
Node 2 2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
WriteConsistency Level = QUORUMReplication Factor = 3 Node 2
2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
ParallelWrite
WriteConsistency Level = QUORUMReplication Factor = 3 Node 2
2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
ParallelWrite
WriteConsistency Level = QUORUMReplication Factor = 3
5 μs ack
Node 2 2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
ParallelWrite
WriteConsistency Level = QUORUMReplication Factor = 3
5 μs ack
12 μs ack
Node 2 2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
ParallelWrite
WriteConsistency Level = QUORUMReplication Factor = 3
5 μs ack
12 μs ack12 μs ack
Node 2 2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
ParallelWrite
WriteConsistency Level = QUORUMReplication Factor = 3
5 μs ack
12 μs ack
500 μs ack
12 μs ack
Node 2 2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Node 1 1st copy
Tunable Consistency
21
ParallelWrite
WriteConsistency Level = QUORUMReplication Factor = 3
5 μs ack
12 μs ack
500 μs ack
12 μs ack
Node 2 2nd copy
Node 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ReadConsistency Level = QUORUMReplication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ReadConsistency Level = QUORUMReplication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ParallelRead
ReadConsistency Level = QUORUMReplication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ParallelRead
ReadConsistency Level = QUORUMReplication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ParallelRead
ReadConsistency Level = QUORUMReplication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ParallelRead
ReadConsistency Level = QUORUMReplication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ParallelRead
ReadConsistency Level = QUORUMReplication Factor = 3
Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ParallelRead
ReadConsistency Level = QUORUMReplication Factor = 3
Hints Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Continuous Availability
22
Node 1 1st copy
Node 4
Node 5 Node 2 2nd copy
ParallelRead
ReadConsistency Level = QUORUMReplication Factor = 3
Hints Node 4
Node 2 2nd copy
Node 1 1st copy
Node 3 3rd copyNode 3 3rd copyNode 4
Node 5
Company Confidential© 2015 DataStax, All Rights Reserved.
Disaster Avoidance
23
Singapore
New York
London
Company Confidential© 2015 DataStax, All Rights Reserved.
Disaster Avoidance
23
Singapore
New York
London
Company Confidential© 2015 DataStax, All Rights Reserved.
Disaster Avoidance
23
Singapore
New York
London
Company Confidential© 2015 DataStax, All Rights Reserved.
Disaster Avoidance
23
Singapore
New York
London
Company Confidential© 2015 DataStax, All Rights Reserved.
Mixed Workload DSE Cluster
Cassandra Only DC
Transactions
Workload Isolation with DataStax Enterprise
24
Cassandra + Spark DC Analytics
Analytical Application
Online Application
Company Confidential© 2015 DataStax, All Rights Reserved.
CQL Example
CREATE TABLE market_prices ( symbol TEXT, date TIMESTAMP, price DECIMAL, side INT, PRIMARY KEY (symbol, date)) WITH CLUSTERING ORDER BY (date DESC);
CQL: Cassandra Query Language
• Syntax similar to RDBMS SQL • Create objects via DDL
• e.g. CREATE…
• INSERT, UPDATE, DELETE • GRANT, REVOKE • SELECT…WHERE • Command line and GUI tools
25
Company Confidential© 2015 DataStax, All Rights Reserved.
• Certified by DataStax: • Java • C / C# / C++ • Python • Node.js • Ruby • PHP • ODBC • Connector for Apache Spark
• Many community drivers
Drivers & Connectors
26
Company Confidential© 2015 DataStax, All Rights Reserved. 27
1 Introduction to DataStax Enterprise
2 Technology Overview
3 Architecture in Enterprise
4 Use Cases in Finance
5 Questions?
Company Confidential© 2015 DataStax, All Rights Reserved.
DataStax in the Enterprise Ecosystem
28
Offline Application
External Hadoop Distribution(e.g. Cloudera, Hortonworks)
OpsCenter Services
Monitoring Operations
Operational Application
Real Time Search
Real Time Analytics
Batch Analytics
RDBMS
Analytics Transformations
Certified Apache Cassandra – No single point of failure – Linear Scalability – Disaster Avoidance
Security In-Memory
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
App ServerCache
C* C*
App Server
Cache
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
App ServerCache
C* C*
App Server
Cache
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
App ServerCache
C* C*
App Server
Cache
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
App ServerCache
C* C*
App Server
Cache
session
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
C* C*
App Server
Cache
session
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
C* C*
App Server
Cache
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
C* C*
App Server
Cache
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
C* C*
App Server
Cache
session
Company Confidential© 2015 DataStax, All Rights Reserved.
Application Tier Resilience
Example: A Single Session
DC NY1 DC NY2
• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra
• Sessions replicate naturally • Recover on disaster / failure
29
C* C*
App Server
Cache
session
Company Confidential© 2015 DataStax, All Rights Reserved.
A Resilient Enterprise Architecture
30
DC NY1
C*
DC NY2
C*
DC LDN1
C*DC LDN2
C*
…DC NY1
App App
App App
DC LDN2
App App
App App
Company Confidential© 2015 DataStax, All Rights Reserved.
A Resilient Enterprise Architecture
30
DC NY1
C*
DC NY2
C*
DC LDN1
C*DC LDN2
C*
…DC NY1
App App
App App
DC LDN2
App App
App App
Company Confidential© 2015 DataStax, All Rights Reserved.
A Resilient Enterprise Architecture
30
DC NY1
C*
DC NY2
C*
DC LDN1
C*DC LDN2
C*
…DC NY1
App App
App App
DC LDN2
App App
App App
Company Confidential© 2015 DataStax, All Rights Reserved.
A Resilient Enterprise Architecture
30
DC NY1
C*
DC NY2
C*
DC LDN1
C*DC LDN2
C*
…DC NY1
App App
App App
DC LDN2
App App
App App
Company Confidential© 2015 DataStax, All Rights Reserved.
A Resilient Enterprise Architecture
30
DC NY1
C*
DC NY2
C*
DC LDN1
C*DC LDN2
C*
…DC NY1
App App
App App
DC LDN2
App App
App App
Company Confidential© 2015 DataStax, All Rights Reserved.
A Resilient Enterprise Architecture
30
DC NY1
C*
DC NY2
C*
DC LDN1
C*DC LDN2
C*
…DC NY1
App App
App App
DC LDN2
App App
App App
Company Confidential© 2015 DataStax, All Rights Reserved.
A Resilient Enterprise Architecture
30
DC NY1
C*
DC NY2
C*
DC LDN1
C*DC LDN2
C*
…DC NY1
App App
App App
DC LDN2
App App
App App
Company Confidential© 2015 DataStax, All Rights Reserved. 31
1 Introduction to DataStax Enterprise
2 Technology Overview
3 Architecture in Enterprise
4 Use Cases in Finance
5 Questions?
Company Confidential© 2015 DataStax, All Rights Reserved.
A Few Use Cases in Financial Services
32
• Payments & Transactions • Fraud Prevention & Detection • Real-time Market Risk • Regulatory Compliance & Reporting • Authentication & Authorization • Customer 360° & Portfolio Analysis • Recommendations & Personalization • Global Document Search • Geospatial Search
$ £€ ¥🔐 📈
Company Confidential© 2015 DataStax, All Rights Reserved.
• Cassandra dominates time series data • Extremely high ingest rates • Low latency retrievals • Efficient storage
• Capture every tick and trade • Also important metadata, eg volume
• Real-time risk analysis
Capture Financial Market Data
33
• Example: Blue Mountain Capital Management • http://www.planetcassandra.org/blog/5-minute-c-
interview-bluemountain-capital-management/
Company Confidential© 2015 DataStax, All Rights Reserved. 34
Authentication & Authorization
DC LDN1
C*
DC LDN2
C*
DC AMS1
C*
DC ZUR1
C*DSE
Cluster
Web User
App Service
DC LDN2
App App
App App
DC AMS1
App App
App App ……
Company Confidential© 2015 DataStax, All Rights Reserved. 34
Authentication & Authorization
DC LDN1
C*
DC LDN2
C*
DC AMS1
C*
DC ZUR1
C*DSE
Cluster
Web User
App Service
DC LDN2
App App
App App
DC AMS1
App App
App App ……session?
Company Confidential© 2015 DataStax, All Rights Reserved. 34
Authentication & Authorization
DC LDN1
C*
DC LDN2
C*
DC AMS1
C*
DC ZUR1
C*DSE
Cluster
Web User
App Service
DC LDN2
App App
App App
DC AMS1
App App
App App ……session?
Company Confidential© 2015 DataStax, All Rights Reserved. 34
Authentication & Authorization
session
DC LDN1
C*
DC LDN2
C*
DC AMS1
C*
DC ZUR1
C*DSE
Cluster
Web User
App Service
DC LDN2
App App
App App
DC AMS1
App App
App App ……
Company Confidential© 2015 DataStax, All Rights Reserved. 34
Authentication & Authorization
session
DC LDN1
C*
DC LDN2
C*
DC AMS1
C*
DC ZUR1
C*DSE
Cluster
Web User
App Service
DC LDN2
App App
App App
DC AMS1
App App
App App ……quote?
Company Confidential© 2015 DataStax, All Rights Reserved. 34
Authentication & Authorization
session
DC LDN1
C*
DC LDN2
C*
DC AMS1
C*
DC ZUR1
C*DSE
Cluster
Web User
App Service
DC LDN2
App App
App App
DC AMS1
App App
App App ……quote?
Company Confidential© 2015 DataStax, All Rights Reserved. 34
Authentication & Authorization
session
DC LDN1
C*
DC LDN2
C*
DC AMS1
C*
DC ZUR1
C*DSE
Cluster
Web User
App Service
DC LDN2
App App
App App
DC AMS1
App App
App App ……quote
Company Confidential© 2015 DataStax, All Rights Reserved. 34
Authentication & Authorization
session
DC LDN1
C*
DC LDN2
C*
DC AMS1
C*
DC ZUR1
C*DSE
Cluster
Web User
App Service
DC LDN2
App App
App App
DC AMS1
App App
App App ……quote
Company Confidential© 2015 DataStax, All Rights Reserved.
Unify & Scale Legacy Infrastructure
35
…USA Equities
UK FX
UK Bonds
Global Users
Legacy Systems
USA FX
DataStax Enterprise ClusterC*
User Interface / Application Services
Thank you!Questions?