Broken Promises and Shattered DreamsNo more gut driven development! Demand decision making based on empirical evidence!

Kenneth Kalmer

Chief Rocket Scientist @ ValuationUP.com

Utopian @ Zero-One.io

@kennethkalmer

github/kennethkalmer

www.opensourcery.co.za

kennethkalmer

#186What We Actually Know About Software Development and Why We Believe It's True with Greg Wilson and Andreas Stefik

We’re Eternal Optimists

We don’t handle errors

We don’t instrument our code

It works on my machine!

We trust our gut

Hindsight is the only 20/20 vision

With more experience we look better, but do we actually know better?

- Morris Kline, Mathematics: The Loss of Certainty

“The most fertile source of insight is hindsight.”

How do we change that?

- Peter Drucker

“What gets measured, gets managed.”

We measure!

New Relic RPM & RUM

Skylight

Librato

Zabbix

Many more

Microservices

We’re all building them, right?

We’re all consuming them, right?

HTTP!

Your challenge

Configure the network timeout for your HTTP requests

Exhibit A - GitHub API

Exhibit B.1 - Twitter API

Exhibit B.2 - Twitter API

Easy? No?

Each service behaves completely different

Within a service, each endpoint differs substantially

Ruby has us covered!

1 require 'benchmark'2 3 time = Benchmark.benchmark do4 benchmark_lovingly5 end6 7 puts time

1 require 'benchmark'2 3 time = Benchmark.benchmark do4 benchmark_the_service_lovingly5 end6 7 puts time #=> 0.010000 0.000000 0.010000 (0.003298)

Falls a bit short

Works great with p-driven development

Gnarly to parse

Tricky to scale

ActiveSupport::Notification

Secret weapon lying dormant in our apps

Separates instrumentation from publishing/reporting

Instrument

1 AS::Notifications.instrument('foo-service', {}) do2 fooish_things3 end

Subscribe

1 AS::Notifications.subscribe 'foo-service' do |*args|2 event = AS::Notifications::Event.new(*args)3 event.duration #=> 10 (in miliseconds)4 end

Instrumenting Faraday

1 conn = Faraday.new( host: 'foo-service' ) do |c|2 c.use :instrumentation3 4 c.adapter Faraday.default_adapter5 end

Instrumenting Faraday 1 AS::Notifications.subscribe('request.faraday') do |*args| 2 event = ActiveSupport::Notifications::Events.new(*args) 3 url = event.payload[:url] 4 http_method = event.payload[:method].to_s.upcase 5 duration = event.duration 6 7 Rails.logger.debug '[%s] %s %s (%.3f s)' % [ 8 url.host, http_method, url.request_uri, duration 9 ]10 end

How do we visualise/report?

InfluxDB

Open source, distributed, time series databases with no external dependencies

Created by Paul Dix (amongst others)

Grafana

An open source, feature rich metrics dashboard and graph editor for InfluxDB, Graphite & OpenTSDB

Client side application

Pesky dependencies!

Easy to use docker image with InfluxDB & Grafana

docker run [lotsa options] tutum/influxdb

InfluxDB has binaries for brew, Debians & RedHats

Grafana has no dependencies, client-side app

Easy to plug in!

Drop influxdb-rails into your project

Out the box metrics for controller, view & db runtimes

Configured client ready use with ActiveSupport::Notifications

InfluxDB client supports async write operations

Already using statsd?

There is an InfluxDB backend available

Simple, fast

UDP, so the client is resilient to failures

Only a transport, needs to ship the stats to…

Lo-Fi RPM

Track what really matters to you and your team

Virtually unlimited data retention

Long tail of information to establish solid baselines

Hi-Fi ain’t always best

More intrusive tools with higher fidelity are most useful for local debugging

Huge variety of data can lead to analysis paralysis

Share the data

Lo-Fi is easier to share

Don’t assume the provider of a service measures the same things you do

Provide them with your consistent view of their service, watch them improve

Inspire them to science too

Engineering creeping in

Aaron’s Adequate Record work

Charles’ JRuby 9000 work

Just use double quoted string - viget.com blog

Experimentally verified - “Why client-side rendering is wrong” - onebigfluke.com

[YOUR WORK HERE]

Stop gut driven development

Demand evidence

Think critically

Thanks!