© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Large-Scale Load Testing

Amazon.com’s Traffic on AWS

Carlos Arguelles, Amazon.com

November 15, 2013

What I’d like you to get out of this

Load and performance issues cost

What I’d like you to get out of this

What I’d like you to get out of this

How you can leverage AWS for load and stress tests

About me

Amazon.com retail site

Amazon.com receives a LOT of traffic

Amazon.com retail site

Significant fluctuation throughout the day

(not to scale)

Amazon.com retail site

Significant fluctuation throughout the year

(not to scale)

(not to scale)

Amazon.com retail site

Significant growth year to year

(not to scale)

(not to scale)

Some load-related issues

can

regular day (off-peak)

1st test (cancelled)

2nd test (successful)

50%

100%

85%

CPU Usage on our fleet

Some load-related issues

can only

Ingestion

Fleet

Database

Output

Fleet Amazon S3

Hadoop

Database Amazon

DynamoDB

Some load-related issues

cannot

5%

Disk

Usage

20%

5 hours

Start load…

What do you really want to do?

Resilience

Testing

Load

Testing

Stress

Testing

Performance

Testing

Load Testing

Stress Testing

Resilience Testing

Performance Testing

How does AWS help us?

Generating load

Replays from real-world traffic Artificial rate, blend of operations

Most useful AWS design pattern, ever

Distributing load, the hard way

Slave

Slave

Slave

Slave

Master

12,000 TPS

3000 TPS

3000 TPS

3000 TPS

3000 TPS

4000 TPS

4000 TPS

4000 TPS

0 TPS

Distributing load, the easy way

Controller

Job

Job

Job

Job

Job

Job

Job

Worker Worker

Controller Worker Worker Worker Worker Worker

Metrics &

Dashboards

Replaying traffic to generate load

Test Data

Repository

Controller

Job

Job

Job

Job

Job

Job

Job Controller

Worker Worker Worker Worker Worker Worker Worker

Service

under test

Metrics &

Dashboards Test Data

Repository

Controller

Job

Job

Job

Job

Job

Job

Job Controller

Worker Worker Worker Worker Worker Worker Worker

Amazon S3 for storing data

Amazon DynamoDB for

indexing

Amazon SQS

for state,

resilience

Amazon EC2 & Auto Scaling

for hardware

Amazon

CloudWatch

Reactive

auto scaling

based on

queue size

Generating load

Replays from real-world traffic Artificial rate, blend of operations

Artificial traffic to generate load

• Why?

– You do not have

real-world data

– You expect a

change in traffic

• How?

– Control rate

– Control blend

– Control duration

Artificial traffic to generate load

50,000 TPS

for 20 minutes 99% Read, 1% Writes

95,000 TPS

for 3 hours 80% Read, 20% Writes

85,000 TPS

for 45 minutes 90% Read, 10% Writes

Minute#1: 50,000 TPS, 99% 1%

Minute#20: 50,000 TPS, 99% 1%

…

Minute#1

10 TPS

for 1 minute,

99% R 1% W

10 TPS

for 1 minute,

99% R 1% W

10 TPS

for 1 minute,

99% R 1% W

…

1

2

5000

Artificial traffic to generate load

Controller

Job

Job

Job

Job

Job

Job

Job

Worker Worker

Controller Worker Worker Worker Worker Worker

Amazon EC2 Spot Instances

• A great way to inexpensively test – Up to 90% off regular price (name your price)

– Interruption-tolerant, time-flexible tasks

• Approaches – Combine with on-demand instances (burst)

– Try Spot Instances first, then fallback to on-demand

Takeaways

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

CPN102