Don’t let this happen to you!Cloud, complexity and drift

James Urquhart@jamesurquhart http://gigaom.com/cloud

May 6, 2010

”[A] large fundamental trader chose to execute [a $4.1B] sell program via an automated execution algorithm ('Sell Algorithm')."

- Findings Regarding The Market Events of May 6, 2010http://www.sec.gov/news/studies/2010/marketevents-report.pdf

"...the Sell Algorithm…executed the sell program extremely rapidly in just 20 minutes.”

The market responded, and trading volume increased…

"... [The Sell Algorithm] responded to the increased volume by increasing the rate at which it was feeding the orders into the market."

- Findings Regarding The Market Events of May 6, 2010http://www.sec.gov/news/studies/2010/marketevents-report.pdf

Quick!What was the root cause?

Market AAutomatic

Trading Algorithm 1

Market AAutomatic

Trading Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market AAutomatic

Trading Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Market AAutomatic

Trading Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

Market AAutomatic

Trading Algorithm 1

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

Automatic Trading

Algorithm N

Automatic Trading

Algorithm N Market A

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

Automatic Trading

Algorithm N

“Root cause” is not an answer—it’s a clue.

"May 6 was…an important reminder of the interconnectedness of our derivatives and securities markets, particularly with respect to index products."

- Findings Regarding The Market Events of May 6, 2010http://www.sec.gov/news/studies/2010/marketevents-report.pdf

Interconnectedness

The Internet is about interconnectedness.

APIs are about interconnectedness.

“The cloud” is about interconnectedness.

Both cloud computing and stock markets are

complex adaptive systems.

COMPLEX?!?

What areComplex Adaptive

Systems?

A large number of individual agents

Agents

State

Rules

OtherAgents

OtherAgents

Information/Actions

Information/Actions

Learning

A large number of individual agents

+ dynamic interactions between agents

Dynamic Interaction

From NetLogo 4.1.3

A large number of individual agents

+ dynamic interactions between agents

+ rules for reacting to/interacting with other agents

Rules

Equals

A system that:

demonstrates emergent behavior,

Emergent Behavior

A system that:

demonstrates emergent behavior,

can be modeled,

Models

Time

Flocking model from NetLogo 4.1.3http://ccl.northwestern.edu/netlogo/

A system that:

demonstrates emergent behavior,

can be modeled,

but

makes precise prediction of future behavior impossible.

Unp

redi

ctab

ility

Unpredictability

Unpredictability

Unp

redi

ctab

ility

So…what can you do?

EMBRACE COMPLEXITY

Never!

EMBRACE COMPLEXITY

SYSTEMS

Embrace Systems Thinking

Practical advice?

Do your homework!

Design for resilience

“The goal of producing a maximum sustained yield may result in a more stable system of reduced resilience.”

“Command and Control and the Pathology of Natural Resource Management”, C. S. Holling and Gary K. Meffehttp://landscape.forest.wisc.edu/courses/Landscape565spr01/Holling_Meffe1996.pdf

“[T]he resilience-stability tradeoff is more than just a simple transformation in distribution. …[A]gents adapt to a prolonged period of stability in such a manner that the system cannot ‘withstand even modest adverse shocks.’”

“The Euro and the Resilience-Stability Tradeoff”, Ashwin Parameswaranhttp://www.macroresilience.com/2011/11/14/the-euro-and-the-resilience-stability-tradeoff/

Focus on relationships

Automatic Trading

Algorithm N

Automatic Trading

Algorithm N Market A

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

Automatic Trading

Algorithm N

Automatic Trading

Algorithm N

Automatic Trading

Algorithm N Market A

Automatic Trading

Algorithm 2

Automatic Trading

Algorithm 3

Automatic Trading

Algorithm 4

Market B

Automatic Trading

Algorithm 5

Automatic Trading

Algorithm 6

Automatic Trading

Algorithm N

Circuit Breaker

http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html

Avoid Drift

Drift

Avoiding Drift:Test production

Release the monkeys!!!• Failure• Degredation• Usage

Avoiding Drift:The agent is not the

system

App-centric• Monitor each app

separately• Dig for root cause• Attempt to stabilize

System-centric• Monitor system at many

levels, and use as feedback• Search for system weakness• Focus on resilience

Avoiding Drift:

YOUare part of the system

Don’t let this happen to you!

Cloud, complexity and drift