©2013 DataStax Confidential. Do not distribute without consent.

@rustyrazorblade

Jon HaddadTechnical Evangelist, DataStax

Python Performance Profiling

1

What are our goals?• Understand potential bottlenecks in dev • Testing • Call graphs

• Understand code once it's in production • Micro benchmarks • Automatic logging of slow DB queries / api calls

• Gather evidence •No guessing •We need insight into both environments

Why do we need it in dev & prod?• Dev != production •No network latency on our desktops • Round trips are cheap in dev • Rarely hitting disk (DB fully in memory) • Zero CPU contention • Failure / failover rarely tested

Before Production

Approaches in Dev• Unit / functional tests • Code coverage is important • if you’re not testing it, it’s probably broken

•Must be reliable, repeatable • Always keep production in mind • Know your hardware • Load test regularly • Jenkins performance plugin

Finding slow tests is easy

Sometimes it's unavoidable…•Make sure you mark tests that are

expected to be slow • These are frequently testing offline tasks

in functional tests

Profiler - Hotshot

pycallgraph• Understand code structure and flow • Summarize times • Darker colors represent more time

spent

Blocking I/O• Usually the problem with web servers • Apps can be CPU bound but it's less frequent

Moving past blocking I/O• Event libraries! • libev most stable • gevent is a beautiful wrapper • Pool.map() is your friend • async can hide issues & make code

harder to profile

Profiler - GreenletProfiler• Takes into account greenlets • Generates callgrind files •Mac Users: qcachegrind

In Production

Profile with minimal overhead•We need something really lightweight! • Our applications can time EVERYTHING • api requests • database queries • individual functions • small blocks of code

• statsd is our friend •microtimers, counters • Integrates w/ librato, graphite

statsd + graphite / grafana

Logging• Log slow database queries / api calls

automatically • Log & aggregate errors •What table was hit? • Read or write? •What was the query? • Can we duplicate? • Logstash / splunk / etc

©2013 DataStax Confidential. Do not distribute without consent. 17