Post on 06-Jan-2017
Terraform at ScaleHashiconf
Calvin French-OwenCo-Founder of Segment
@calvinfo
September 7, 2016
đ
Scaling vectors
Complexity
People
Complexity
People
Complexity â
People
Complexity
â
How do we move nimblyâwhile adding people?
This talk- Terraform at Segment- What makes âgoodâ Terraform- Whatâs next
Terraform at Segment
By the numbers- 16 developers working with Terraform- 94 microservices- thousands of AWS resources
A year with TerraformDecember 2012 â Launch dayApril 2015 â Terraform first attempt (v1)November 2015 â Terraform âreduxâ (v2)
Before Terraform
đą
Terraform
Migrating to TerraformApril 2015
Migrating to Terraform
Migrating to Terraform1. AWS accounts per environment
dev stage prod old prodvpc peering
dev stage prod old prodvpc peering
managed by Terraform
Separate accounts- confidence to apply âat willâ- test the waters without screwing up the old
account- any sort of âglobalâ configs are okay
Migrating to Terraform1. AWS accounts per environment2. Docker and ECS
Terraform: First Attempt
Terraform (our first attempt)âââ Makefileâââ README.mdâââ environments âââ dev âââ production âââ stage
Terraform (our first attempt)âââ Makefileâââ README.mdâââ environments âââ dev âââ production âââ stage
Terraform (our first attempt)environments/stageâââ api.tfâââ bastion.tfâââ dns.tfâââ elasticache.tfâââ elbs.tfâââ iam.tfâââ outputs.tfâââ redis.tfâââ s3.tfâââ terraform.tfstateâââ terraform.tfvarsâââ vpc.tf
Terraform (our first attempt)resource "aws_ecs_task_definition" "app" { family = "app"
container_definitions = <<EOF[ { "cpu": 1024, "memory": 768, "environment": [ { "name": "NODE_ENV", "value": "stage" } ], "image": "segment/app:1.54.14", "name": "app", "portMappings": [ { "containerPort": 8000, "hostPort": 8000 } ] }]EOF}
Life was better
Life was better!Life was betterâŚ
Life was better!Life was betterâŚ
but notgood.
1. environment drift
Terraform first attemptâââ Makefileâââ README.mdâââ environments âââ ops âââ production âââ stage
resource "aws_ecs_task_definition" "app" { family = "app"
container_definitions = <<EOF[ { "cpu": 1024, "memory": 768, "environment": [ { "name": "NODE_ENV", "value": "stage" } ], "image": "segment/app:1.54.14", "name": "app", "portMappings": [ { "containerPort": 8000, "hostPort": 8000 } ] }]EOF}
<= stage
resource "aws_ecs_task_definition" "app" { family = "app"
container_definitions = <<EOF[ { "cpu": 1024, "memory": 768, "environment": [ { "name": "NODE_ENV", "value": "stage" } ], "image": "segment/app:1.54.14", "name": "app", "portMappings": [ { "containerPort": 8000, "hostPort": 8000 } ] }]EOF}
<= stage
resource "aws_ecs_task_definition" "app" { family = "app"
container_definitions = <<EOF[ { "cpu": 1024, "memory": 3072, "environment": [ { "name": "NODE_ENV", "value": "productionâ, } ], "image": "segment/app:1.54.17", "name": "app", "portMappings": [ { "containerPort": 8000, "hostPort": 3000 } ] }]EOF}
prod =>
2. one massive local state
3. production drift
$ terraform plan âtarget=aws_elb.feels_so_easy
$ terraform plan âtarget=aws_elb.oh_no_what_have_we_done
Terraform Redux (v2)
Terraform v1 Problems1. massive shared state2. locally stored state3. drift between environments
Terraform v1 Problems1. massive shared state: split states2. locally stored state: remote state3. drift between environments: modules
v2: state management
core(vpc, networking, security groups, asgs)
auth api site db cdn
services
core(vpc, networking, security groups, asgs)
auth api site db cdn
servicesâ
read
onl
y â
/** * Remote state. */
resource "terraform_remote_state" "state" { backend = "s3" config { bucket = "segment-ops" key = "terraform/${var.environment}/terraform.tfstate" }}
data "template_file" âtest" { template = "${file("${path.module}/init.tpl")}"
vars { zone_id = "${terraform_remote_state.state.zone_id}" }}
/** * Remote state. */
resource "terraform_remote_state" "state" { backend = "s3" config { bucket = "segment-ops" key = "terraform/${var.environment}/terraform.tfstate" }}
data "template_file" âtest" { template = "${file("${path.module}/init.tpl")}"
vars { zone_id = "${terraform_remote_state.state.zone_id}" }}
read only!
/** * Remote state. */
resource "terraform_remote_state" "state" { backend = "s3" config { bucket = "segment-ops" key = "terraform/${var.environment}/terraform.tfstate" }}
data "template_file" âtest" { template = "${file("${path.module}/init.tpl")}"
vars { zone_id = "${terraform_remote_state.state.zone_id}" }}
read only!
reference
v2: modules
Modules enforce configuration parity.
What makes good* Terraform?
*for some definitions of good
Docker AMIs by Packer
Service Config by Terraform
1. Variables2. Composition3. State4. Versioning
1. Variables- anything a user might want to override should be
a variable- use defaults liberally
1. Variablesresource "aws_instance" "bastion" { ami = "${module.ami.ami_id}" source_dest_check = false instance_type = "${var.instance_type}" subnet_id = "${var.subnet_id}" key_name = "${var.key_name}" vpc_security_group_ids = ["${split(",",var.security_groups)}"] monitoring = true tags { Name = "bastion" Environment = "${var.environment}" }}
configurableconfigurable
configurableconfigurable
configurable
1. Variablesresource "aws_instance" "bastion" { ami = "${module.ami.ami_id}" source_dest_check = false instance_type = "${var.instance_type}" subnet_id = "${var.subnet_id}" key_name = "${var.key_name}" vpc_security_group_ids = ["${split(",",var.security_groups)}"] monitoring = true tags { Name = "bastion" Environment = "${var.environment}" }}
configurableconfigurable
configurableconfigurable
configurable
non-configurablenon-configurable
non-configurable
1. Variablesresource "aws_instance" "bastion" { ami = "${module.ami.ami_id}" source_dest_check = ${var.source_dest_check} instance_type = "${var.instance_type}" subnet_id = "${var.subnet_id}" key_name = "${var.key_name}" vpc_security_group_ids = ["${split(",",var.security_groups)}"] monitoring = ${var.monitoring} tags { Name = "bastion" Environment = "${var.environment}" }}
2. Composition- build modules as you need them- itâs okay if not everything fits the abstraction
2. Composition â âfull stackâmodule âstackâ { source = âgithub.com/segmentio/stackâ name = âmy-stackâ environment = âproductionâ}
2. Composition â inside stackmodule "vpc" { source = "./vpcâ âŚ}
module "security_groups" { source = "./security-groupsâ âŚ}
module "bastion" { source = "./bastionâ âŚ}
module "dhcp" { source = "./dhcpâ âŚ}
2. Composition â byo editionmodule âclusterâ { source = âgithub.com/segmentio/stack//ecs-clusterâ
environment = âprodâ name = âcdnâ vpc_id = âvpc-eff2eadaâ image_id = âami-204faaf3â}
3. State management- separate core from services- states per service- use atlas or s3- use binary plans
core(vpc, networking, security groups, asgs)
auth api site db cdn
servicesâ
read
onl
y â
4. Versioningmodule âstackâ { source = âgithub.com/segmentio/stack?ref=v1.xâ}
Whatâs next
Whatâs next- Applying in CI- Atlas- Data sources- Terraform generation
People
Complexity
â
Fin
Prior ArtStack: github.com/segmentio/stackAtlas Examples: github.com/hashicorp/atlas-examples