PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge....

44
PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE Michael E Fisher JUNE 28, 2017 Or, how we learned to stop worrying and love the cloud

Transcript of PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge....

Page 1: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE

Michael E Fisher JUNE 28, 2017

Or, how we learned to stop worrying and love the cloud

Page 2: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY
Page 3: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY
Page 4: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Frankenbump

Page 5: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

hello

Page 6: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

MONTH DD, YYYY

BUT FIRST, HOW DID WE GET HERE?

6

Page 7: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

feel: intreague

Page 8: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

feel: intreague

attention

Page 9: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

feel: intreague

attention

Page 10: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

feel: intreague

attention

Page 11: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY
Page 12: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

EDGEEDGE

STATIC FRONTENDS

API GATEWAY

SERVICESERVICESERVICE

SERVICE DISCOVERY

LOGS

DD

Page 13: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

feel: intreague

attention

Page 14: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

60 elections

80 tech staff

100 immutable backends

150 serverless frontends

577 days

2,500 max QPS

82,759,676 votes

Page 15: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Chaos, right? Immutable to the rescue.

Page 16: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Immutable as a handshake

Page 17: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

feel: realization

root@really-important-production-instance:~# service nginx restart

==> amazon-ebs: Authorizing SSH access on the temporary security group...==> amazon-ebs: Launching a source AWS instance...==> amazon-ebs: Waiting for instance to become ready...==> amazon-ebs: Connecting to the instance via SSH...==> amazon-ebs: Stopping the source instance...==> amazon-ebs: Waiting for the instance to stop...==> amazon-ebs: Creating the AMI: packer-example 1371856345==> amazon-ebs: AMI: ami-19601070==> amazon-ebs: Waiting for AMI to become ready...==> amazon-ebs: Terminating the source AWS instance...==> amazon-ebs: Deleting temporary security group...==> amazon-ebs: Deleting temporary keypair...==> amazon-ebs: Build finished.

==> Builds finished. The artifacts of successful builds are:--> amazon-ebs: AMIs were created:

us-east-1: ami-19601070

Page 18: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Easy:

• Build tooling • Deployment • Resiliency

Hard:

• Making it work for everything • Getting everyone on board

Page 19: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

MONTH DD, YYYY

BUILD TOOLS

19

• Travis CI • Packer • Continuum • Ansible

Page 20: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Let's focus in on our edge.

Page 21: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

EDGEEDGE

STATIC FRONTENDS

API GATEWAY

SERVICESERVICESERVICE

SERVICE DISCOVERY

LOGS

DD

Page 22: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Everyone is coming for you.

Page 23: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Scriptkiddies and DDoS

Page 24: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

sub vcl_recv { if (table.lookup(edge_settings, "non_us_ban", "disabled") == "enabled") {if (geoip.country_code != "US") { error 503; } }

if (client.ip ~ cc_blacklist && req.url.path == "/api/hamm/donations" && req.request == "POST") { if (randombool(1, 2)) { error 201; } else { error 402; } }

Confusion as a defense strategy

Page 25: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

if (table.lookup(THEWALL, client.ip) && !req.http.Fastly-FF) { error 819 "Bad Taco."; }

if (req.http.user-agent ~ "^WordPress") { error 819 {"Forbidden 🌮"}; }

Just ban them

Page 26: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

feel: curiosity / excitement

solution

Page 27: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY
Page 28: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

EDGE

STATIC FRONTENDS

API GATEWAY

SERVICESERVICESERVICE

SERVICE DISCOVERY

LOGS

DD

Page 29: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Onesie

Page 30: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

hillaryclinton.com/calls onesie-web.s3-aws-us-east-1.amazonaws.com/calls

• Caching

• Paths and query params

• Regional failover

Page 31: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

EDGES3

TRAVISGIT

Onesie

Page 32: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Let's assume the remaining traffic are legitimate folks making API calls.

And they really, really want to go to Philadelphia.

Page 33: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

EDGE

STATIC FRONTENDS

API GATEWAY

SERVICESERVICESERVICE

SERVICE DISCOVERY

LOGS

DD

Page 34: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Frankenbump

Page 35: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

About that infrastructure diagram...

Page 36: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

EDGE

STATIC FRONTENDS

API GATEWAY

SERVICESERVICESERVICE

SERVICE DISCOVERY

LOGS

NODEJS HOMEPAGE

DD

Page 37: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

REDIS

SERVICE DISCOVERY

LOGS

WordPress

NodeJS

EDGESecretary of Stateless

NodeJS

NodeJS

NodeJS

NodeJS

DD

Page 38: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Biggest problem to solve:

What's the state of your state?

Page 39: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

WORDPRESS

SERVICE DISCOVERY

MYSQL REDIS NODEJS

Aurora EC2 ElasticCache EC2

CONSUL

TEMPLATES

S3

Page 40: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

Biggest problem to solve:

What's the state of your state?

Abuse S3, Consul, and ELB health checks to find out.

Page 41: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

export default function(servers, dog = new DD) { return async (ctx) => { let path = '/health';

// only run template version checks on the ELB version of the health check if (ctx.url === '/health-elb') { try { const templateRelease = await getCurrentTemplateRelease(); path = `/health?templateRelease=${templateRelease}`; } catch (err) { // For the initial release consul will be empty, so if the consul check errors with a 404 // there's nothing wrong, just continue on with the normal health check if (err.status === 404) { log.warn('Skipping templateRelease health check since consul is empty'); } else { dog.increment('sos-template-deploy.consul-error', 1); log.error({err}, 'Error reading data from consul'); } } }

try { await runHealthChecks(servers, path); } catch (err) { const message = 'Health check failed'; log.error({err}, `${message} : ${err.message}`); ctx.throw(message); } ctx.status = 200; };

Page 42: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

42

Takeaways • Immutable infrastructure was key to our technical success

• We moved quickly but were resilient against failure (most of the time)

• It takes more effort to apply immutable to everything you're doing, but it's worth it

• Ultimately, developers like the handshake between SRE and dev

Page 43: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

43

Takeaways • On a presidential campaign, innovation is a necessity, and there aren't any hard and fast rules in tech

• It's difficult to imagine where infrastructure tech will be in four years, but the next campaigns will be leveraging the most exciting stuff out there

Page 44: PRESIDENTIAL CAMPAIGNS & IMMUTABLE INFRASTRUCTURE€¦ · • Ansible. Let's focus in on our edge. EDGE EDGE STATIC FRONTENDS API GATEWAY SERVICE SERVICE SERVICE SERVICE DISCOVERY

44

Takeaways • You can build cool shit and work in public service