Setting up New Relic's Infrastructure: A bunch of things we did wrong and a few we did right.DAVE PETERSON & NIC BENDERS2013-10-25

David Peterson(dave@newrelic.com)

Nic Benders(nic@newrelic.com)

[“star” graph of code base at launch]

100,000req/sec

2Gbps

200TB

Who Does What?

Site Operations != IT

You are not Batman

Do your work during the day

Lowest hour is still 60% of peak hour!

Put your serverson the moon

Never open the box

Domain names are cheap

10.0.0.0/31 10.0.0.0-10.0.0.1 255.255.255.254 2

10.0.0.0/30 10.0.0.0-10.0.0.3 255.255.255.252 4

10.0.0.0/29 10.0.0.0-10.0.0.7 255.255.255.248 8

10.0.0.0/28 10.0.0.0-10.0.0.15 255.255.255.240 16

10.0.0.0/27 10.0.0.0-10.0.0.31 255.255.255.224 32

10.0.0.0/26 10.0.0.0-10.0.0.63 255.255.255.192 64

10.0.0.0/25 10.0.0.0-10.0.0.127 255.255.255.128 128

10.0.0.0/24 10.0.0.0-10.0.0.255 255.255.255.0 256

10.0.0.0/23 10.0.0.0-10.0.1.255 255.255.254.0 512

10.0.0.0/22 10.0.0.0-10.0.3.255 255.255.252.0 1024

10.0.0.0/21 10.0.0.0-10.0.7.255 255.255.248.0 2048

10.0.0.0/20 10.0.0.0-10.0.15.255 255.255.240.0 4096

10.0.0.0/19 10.0.0.0-10.0.31.255 255.255.224.0 8192

10.0.0.0/18 10.0.0.0-10.0.63.255 255.255.192.0 16384

10.0.0.0/17 10.0.0.0-10.0.127.255 255.255.128.0 32768

10.0.0.0/16 10.0.0.0-10.0.255.255 255.255.0.0 65536

Private IPsare FREE

Hardware

System Management

X6550 E5-2450 E5-2643 E5-2690

Average response time

137

ms

167

ms

158

ms

189

ms

25% performance improvement

2x 6-Core X6550@ 2.67 GHz

2x 8-Core E5-2690@ 2.90 GHz

Storage

√

Databases

32-bits isn't enoughclass CreateTribbles < ActiveRecord::Migration def change create_table :tribbles do |t| t.boolean :hungry t.string :color t.references :parent, index: true end endend

32-bits isn't enough`id` int(11) NOT NULL AUTO_INCREMENT

`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT

`id` SERIAL

Your database is not a queue

Long running querieswill kill you

And now, a thanks to the people who are out there every day fixing the mistakes that we made:

Jonathan Owens

Jonathan Thurman

Aaron Bento

Jonathan OwensJonathan Owens

Sean Kane

Paul Showalter

Karl Matthias