Cyber Integration, Message Fabric and Streaming Analytics · Cyber Integration, Message Fabric and...

Cyber Integration, Message Fabric and Streaming Analytics

SCRE Workshop

November 17, 2015

Why a Message Fabric for Cyber Integration?

• Abstraction (Pub-Sub, Request / Response, Queuing) • Separate physical systems from communication;

use any infrastructure without changing system behavior • Single point of web-based management

• Modularity • Quickly add new technologies/algorithms to stay ahead

• Efficiency • Instant response needed? Maybe not, but latency matters!

• Functionality • Discovery, connectivity, reliable exchange of data • Guaranteed delivery, fault tolerance, load balancing • Commodity hardware = lower entry and O&M costs

2

2

Respond

Respond

The Race to Respond

why speed is critical for Cyber Defense

Fast

Slow

Starting Line

Recognize

Recognize

Attack failed

Attack succeeded

You lost!

Attack starts

Events

Events You won!

3

Execution

Execution

The Race to the Exchange

why speed is critical for Capital Markets

Fast

Slow

Starting Line

Feed Handler

Feed Handler

Got GE at 30.21

Got GE at 30.45

You lost!

GE at 30.16

Market Data

Market Data

Exchange You won!

4

Market Data Growth = Data Deluge

5 7 10 13 26 120 160 310 559 696

1,100

1,562

1,925

2,562

3,410

4,380

5,957

7,174

Total Options Equities

Aggregated One Minute Peak Messages Per Second Rates

Arca, CTS, CQS, OPRA, NQDS (in thousands)

> 1Terabyte of Data per Day

5

Legacy Messaging Architectures

6 Data Hops

4 Data Hops

Daemon Based Design

Broker Based Design

6

2004 – Need for a State Change

• Motivations / Challenges • Not scaling to today’s needs (yet alone tomorrow’s!) • Availability at risk due to single points of failure

• Brokers are a bottleneck • Broker is a source of contention that limits scaling • Broker failure disastrous to latency and stability

Remove the Broker from the Message Path!

7

50% reduction in hardware cost 75% lower latency Source: Direct Edge 2008-2009

Replaced brokers with peer-to-peer message fabric

Consistent Performance & Stability with Record Transaction Volumes

Case Study: Direct Edge 3rd Largest US Stock Exchange in 2008 (after NYSE and NASDAQ)

Increased resiliency Predictable performance 8

Near Real-Time Financial Data Analytics Framework Counterparty Risk Assessment

9

Market Risk

Interest rate risk Currency risk Equity risk

Commodity risk

Trading Transactions

Portfolio Market Data Pricing Products

Credit Risk

Counterparty risk Sovereign risk Concentration risk Securitization risk Credit derivative risk

Lending Transactions

Client Activity Credit Lines Limits History

Liquidity Risk

Asset risk Funding risk

Payment Transactions

Credit Lines Liquidity Controls Routing

Operational Risk

Legal risk IT risk

Reputational risk

External Transactions

Fraud Process Break External Events Market/Fiduciary

Risk Management App Development Front & Middle

Office Operations

End-of-day Extracts End-of-day Extracts End-of-day Extracts

Current State – Disparate Data Siloes End-of-day extracts and long load/processing times

10

Trading Transactions

Portfolio Market Data Pricing Products

Lending Transactions

Client Activity Credit Lines Limits History

Payment Transactions

Credit Lines Liquidity Controls Routing

External Transactions

Fraud Process Break External Events Market/Fiduciary

Market Risk Credit Risk Liquidity Risk Operational Risk

Interest rate risk Currency risk Equity risk

Commodity risk

Counterparty risk Sovereign risk Concentration risk Securitization risk Credit derivative risk

Asset risk Funding risk

Legal risk IT risk

Reputational risk

Low Latency Message Fabric

High Performance Data Analytics

Risk Ops Apps

Near Real-time Access

• Cross Silo Risk Mgmnt • Trade Lifecycle • Market Scenario

• Trade Latency Compliance • Trade Cost Analysis

• Portfolio Management

Desired State – Correlation across all Data An Open “Single Source of Truth” for Financial Data

11

Near Real-time Database Query

and Analytics

Real-time Data Acquisition &

Transformation Lo

w L

aten

cy M

essa

ge F

abric

Pub/Sub

Pro

gram

mab

le

Ana

lytic

s S

QL,

Map

Red

uce,

PL/

R

Historical Database Query

and Replay

FIX

FPML

OPRA

ARCA

CME

Historical & End-of-day

Data Acquisition & Transformation

ETL and Batch Upload

Core Components

• Positional Trade Strategy • Dashboard & Reporting • Risk and Compliance • Intraday Operations

Flat file

Massively Scalable Database

Raw Message Store

Message Ingest Transform/Replay

Near Real-Time Data Analytics Real-time & Historical Stock Data with Near Real-time Query

Real-Time Feeds

12

Sample Trade Workbench Real-Time Dashboard Not a Production View

13

What about real-time? Streaming Analytics and Processing at the Edge

14

Processing “at the Edge” (and elsewhere)

• Considerations • Aggregation and correlation necessary for “big picture” • More distributed processing power than centralized • Raw data is necessary for some types of analysis

• Is it more efficient to send raw + processed or process later? • Strategies

• Derive as much as possible as early as possible • Continuous computation – counters, distribution statistics • Enrich (tag/classify unstructured events, add provenance details

– origin, identity, versioning, chain of custody) • Exception monitoring – deviations from norm, trending up/down

to exceed thresholds • Filter, summarize, compress, transform, mask, encrypt

• Focus on state changes (111100001110011100) • No-change is data too, but heartbeats may be enough

15

15

Edge nodes provide: • Filtering • Classification • Streaming data masking • Real-time aggregates • In-line enrichment

Event Cloud

Core nodes provide: • Event Fusion • Content-based Routing • Stream interfaces

User nodes provide: • User-defined rules • Enrichment with data

warehouse and MDM • Event-driven analytics • Integration with alerting

channels and workflows

Scalable Deployment w/ Distributed Nodes AFOC, DCGS-A, DCGS-AF, NATO, IC

All nodes managed as a single topology from a central

console.

What we want in a Message Fabric

17

Application A

Comm

1

Network

“Nothing in the Middle” Data Hop

Just 2 steps to move from A to B!!! Less is more!!

Application B

Comm

2

Sending CPU Receiving CPU Functions handled by modern O/S, CPU, Network and API

• routing

• forwarding • filtering • fan-out

• persistence

Benefits

• efficient (single data hop) • maximizes performance • no single points of failure

• scalable and flexible • easier to administer

Peer-to-Peer Message Fabric

18

Sending Application Persistent Data Stores Receiving Applications

Parallel Persistence®

19


Zero System Downtime! Zero Latency Failover!


20



21


then rejoins the live stream!

Receiver recovers with no impact to live message stream,


22

Extended Enterprise

Considerations: • Availability • Authentication • Authorization • Bandwidth • Encryption • Filtering • Firewalls • Protocols • Routing

23

XS

YS

10 10 10

10 10

10

10 10

60 50

10 10

XR

YR

ZS ZR

= App Domain

= WAN / firewall

= Message Router

XS = Sender / Receiver XR

24

Dynamic Routing - Least Cost Path

XS

YS

10 10 10

10 10

10

10 10

60 50

10 10

XR

YR

ZS ZR

= App Domain

= WAN / firewall

= Message Router

XS = Sender / Receiver XR

25

Dynamic Routing - Least Cost Path

How can you combine a peer to peer message fabric with standardized interfaces and centralized management?

26

00:00:46: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up

00:00:47: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up


00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down 2

*Mar 1 18:46:11: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36)

18:47:02: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36)

*Mar 1 18:48:50.483 UTC: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36)

00:00:46: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up



00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down 2

*Mar 1 18:46:11: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36)

18:47:02: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36)

*Mar 1 18:48:50.483 UTC: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36)

Sense Making

Streaming data collection…

27

Sense Making

Sense Making

How do you manage this?

28

logserver1 logserver2 logserver3 logserver4 logserver5 logserver6 logserver7 logserver8

Centralized management, peer to peer data flow

29

Summary: Essential Characteristics

• No daemons or servers in delivery path • Maximize speed and scalability • No single points of failure

• Choice of protocols (data “payload” agnostic) • TCP, UDP, AMQP, unicast, multicast, shared memory, etc.

• Secure transports, handshakes and storage • Integrity, with or without confidentiality

• Secure message routing for extended enterprise • Intelligently bridge segmented networks and applications

• Centralized monitoring (with API) • Integrated insight from every endpoint (other layers too!)

30

30

Summary: Essential Characteristics (cont’d)

• Dynamic service and peer discovery • Move applications without changing configuration or code • Establish data flows out-of-band to minimize overhead

• Full range of qualities of service • From reliable (best-effort) to durable (guaranteed)

• Standards-based interfaces • Easily plug in third-party products and services

• Centralized management (with API) • Configure top-down; implement locally

• No custom hardware • Pure software to always run on best infrastructure

31

31

Thank You!

32

Gay Adams [email protected]

cell: 301-980-9148

Cyber Integration, Message Fabric and Streaming Analytics · Cyber Integration, Message Fabric and...

Documents

Transcript of Cyber Integration, Message Fabric and Streaming Analytics · Cyber Integration, Message Fabric and...