Understanding your system

Andreas Bjärlestam 2016-03-30

_____ .__.__ / _ \ ____ |__| | ____ / /_\ \ / ___\| | | _/ __ \ / | \/ /_/ > | |_\ ___/ \____|__ /\___ /|__|____/\___ > \//_____/ \/ _____ ______ ____________/ ____\___________ _____ _____ ____ ____ ____ \____ \_/ __ \_ __ \ __\/ _ \_ __ \/ \\__ \ / \_/ ___\/ __ \ | |_> > ___/| | \/| | ( <_> ) | \/ Y Y \/ __ \| | \ \__\ ___/ | __/ \___ >__| |__| \____/|__| |__|_| (____ /___| /\___ >___ > |__| \/ \/ \/ \/ \/ \/ __ __ .__ _/ |_ ____ _______/ |_|__| ____ ____ \ __\/ __ \ / ___/\ __\ |/ \ / ___\ | | \ ___/ \___ \ | | | | | \/ /_/ > |__| \___ >____ > |__| |__|___| /\___ / \/ \/ \//_____/

Do you know on top of your head:• How many req/s your system can handle?

• What response time it has?

• How it scales?

• What the bottlenecks are?

• How stable it is over time?

Many still do big bang performance testing

Stop seeing performance tests as verification

It should be an integrated part of your development cycle

Continuously analyze your system

You should do it all the time!

I’m lazy, so performance testing must be quick and simple

isolation

Your test client should do nothing but loadtesting, no interference

Amazon AWS + tmux=

win!

You can leave it on and come back to check every now and then

Combine with monitoring:NewrelicKibana

Graphite

> sudo yum install siege

> siege -c10 -t30s -d1 -i -f urls.txt

siege

• Quick and simple• Instant visual feedback• Good summary of most important metrics• Good enough for most scenarios

= You can work in quick iterations

Create a Traffic Model

Your best guess of how the system will be used

When replacing a system

Get access logs from the old system

Analyze which parts of the system that generate the most load

Plot them over time to get a feeling for peaks and average load

If you replay access logs against your system with siege you can

gain a lot of insights and find problems like unhandled urls,

unnecessary redirects etc

Build a urls.txt file

Based on traffic model

Fill a file with urls that represent your expected traffic, one url per line

Think about the proportions of different types of urls

Build a urls.txt file

Expecting broad traffic -> many urls

Expecting narrow traffic -> not so many

Build a urls.txt file

http://example.comhttp://example.com/user/stinahttp://example.com/user/olofhttp://example.com/user/svenhttp://example.com/user/sivhttp://example.com/user/ellenhttp://example.com/country/swedenhttp://example.com/country/norway

Build a urls.txt file

curl cut jq grep etc are your friends

Build a urls.txt file

You can send POST as well

http://example.com/user/stina POST age=23http://example.com/user/stina POST a=1&b=2http://example.com/user/stina POST <./stina.txt

Finding the system limits

10 req/s OK100 req/s OK500 req/s Slooow300 req/s OK

Scale up with more CPU or processors and try again

Does it scale linearly?

This is a good time to think about the bottlenecks of your system

I/OCPU

Sync / Async

Adjust and try again

Keep an eye on system metrics

Response timeCPU load

Memory usageError rates

etc

Stability testing

Start a linux machine on AWSSet up a session with tmuxStart siegeLeave it running

> siege -c10 -t24h -i -d1 -f urls.txt

Put your system under continuous load from your first commit

Keep an eye on system metrics every now and then

Good to know: siege counts error responses and will stop if it

encounters more than 1024 errors

TLDR

Stop doing performance tests like its 1999Put your system under load from day 1

Run tests interactively, be creativeGain understanding

AWS + tmux + siege is awesome!

tmux cheat sheet

> tmux new -s loadtest starts a new session named loadtest

ctrl-b + d exits session

> tmux list-sessions lists all current sessions

> tmux attach -t loadtest attaches to the session named loadtest

If siege does not fit your use case

You could try- gatling- locust- wrk

They are bigger (more complex) hammers