(ARC401) Black-Belt Networking for the Cloud Ninja | AWS re:Invent 2014

https://s3.amazonaws.com/reinvent-arc401/index.html

• Basic automation– Automating network build

• Intermediate automation– Going beyond initial network build

• Advanced automation– Dynamic component configuration

aws ec2 create-vpc --cidr-block 10.0.0.0/16

aws ec2 replace-route --route-table-id $ROUTE_TABLE_ID

--destination-cidr-block 0.0.0.0/0

--instance-id $INSTANCE_ID

aws ec2 attach-network-interface --network-interface-id $ENI

--instance-id $INSTANCE_ID

--device-index 1

aws ec2 assign-private-ip-addresses --network-interface-id $ENI

--private-ip-addresses 10.0.0.100

• AWS CLI

#!/bin/sh

export AWS_DEFAULT_REGION="us-east-1"

VPC_ID=`aws ec2 create-vpc --cidr-block 10.0.0.0/16 --output text | awk '{print $6;}'`

SUBNET_ID=`aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.1.0/24 --output

text | awk '{print $6;}'`

echo "Created $VPC_ID & $SUBNET_ID"

#Clean up

aws ec2 delete-subnet --subnet-id $SUBNET_ID

aws ec2 delete-vpc --vpc-id $VPC_ID

• Custom scripts

#Powershell script

Initialize-AWSDefaults -Region 'us-east-1'

#Create new VPC

$vpc = New-EC2Vpc -CidrBlock '10.0.0.0/16'

$subnet = New-EC2Subnet -VpcId $vpc.VpcId -CidrBlock '10.0.1.0/24' -AvailabilityZone 'us-east-1d'

Write-Host “Created VPC: " $vpc.VpcId " subnet: " $subnet.SubnetId

#Clean up VPC

Remove-EC2Subnet $subnet.subnetId -Force

Remove-EC2Vpc $vpc.VpcId -Force

• Amazon SDK

"Resources" : {

"VPC" : {

"Type" : "AWS::EC2::VPC",

"Properties" : {

"CidrBlock" : “10.0.0.0/16”,

"Tags" : [ { "Key" : “Name", "Value" : “VPCName“ } ]

}

},

"PublicSubnet" : {

"Type" : "AWS::EC2::Subnet",

"Properties" : {

"VpcId" : { "Ref" : "VPC" },

"CidrBlock" : “10.0.1.0/24”,

"Tags" : [ { "Key" : "Network", "Value" : "Public" } ]

}

}

• AWS CloudFormation

• Allows network build to be:– Automated

– Tracked

– Version controlled

• A great start!– Aspirational for many of my customers

• Control changes to the network

• Managing additional network components– Peering or VPN connections

– NAT or VPN instances

• Automate application-specific network components– EIPs, secondary IP assignments, routed VIPs

• Controlling network changes with CloudFormation– Templates can be version controlled

– UpdateStack

• Add/Remove resources

• Modify security group rules

– Events are tracked by CloudFormation

• In-region network expansion with VPC peering– Peering handshake can be scripted

– CloudFormation “AWS::EC2::VPCPeeringConnection” type

• Cross-region network expansion– VPC, routes, VPN instances can be scripted

– Check out vpc2vpc as an example

https://github.com/vinayselvaraj/vpc2vpc

vpc2vpc create 10.1.0.0/16 10.2.0.0/16 10.3.0.0/16

#!/bin/sh

NAT_ID=“i-12345”

NAT_RT_ID=“rtb-22574640”

REGION=“us-east-1”

…

# So we can monitor the other NAT instance

NAT_IP=`aws ec2 describe-instances --instance-id $NAT_ID --region $REGION |

grep PrivateIpAddress -m 1 | awk '{print $2;}' | sed -re 's/[",]//g'`

…

aws ec2 replace-route --route-table-id $NAT_RT_ID --instance-id $Instance_ID

--destination-cidr-block 0.0.0.0/0 --region $REGION

• Manage networking components (for example, HA)

https://aws.amazon.com/articles/2781451301784570

• Allows networks and components to be:– Automated

– Tracked

– Version controlled

• Not very dynamicInstance_ID=“ ”

Route_Table_ID=“ ”

Virtual_IP=“ ”

EIP_Alloc_ID=“ ”

• Dynamic network automation scripts

• Automation that responds appropriately when network or application conditions change – Without changing the scripts

• Examples– VIP reassignment in response to Auto Scaling

– New subnets get appropriate dynamic routing rules

– Create VPN tunnels when new regions are brought online

• Dynamic network automation approaches– Instance bootstrapping

– Store dynamic information in an external store

– Change polling, detection, and response

• Standard external stores– Amazon S3, Amazon DynamoDB, Configuration Management Tool

• What about using resource tags?– AWS resources can be tagged

• Tags are great for identifying resources– In the console

– By project or environment

– For billing

• Network tagging scheme– Route table tags

• NAT = true

• NATAZ = [any, us-east-1a]

Public Subnet 1

AWS Region

Availability Zone 1 Availability Zone 2

NAT

Public Subnet 2

Private Subnet 1 Private Subnet 2

TAG

NATAZ

any

Auto Scaling Group

TAG

NATAZ

anyThis subnet

needs NAT

This subnet

needs NAT

Public Subnet 1

AWS Region


NAT

Public Subnet 2

NAT


This subnet

needs AZ-

specific NAT

This subnet

needs AZ-

specific NAT

TAG

NATAZ

AZ1

TAG

NATAZ

AZ2

--filters "Name=tag-key,Values=NAT"

--filters "Name=tag:NATAZ,Values=any,us-east-1a"

• Filtering AWS resources by tag--filters are awesome!

#!/bin/bash

INSTANCE_ID=`curl --silent http://169.254.169.254/latest/meta-data/instance-id`

AZ=`curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone`REGION="${AZ%?}"

MAC=`curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/`

VPC_ID=`curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-id`

#!/bin/bash

INSTANCE_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/instance-id`

AZ=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone`

REGION="${AZ%?}"



ROUTE_TABLES=`aws ec2 describe-route-tables --region $REGION --output text

--filters "Name=tag:NATAZ,Values=any,$AZ" | grep ROUTETABLES | awk '{print $2}'`

#!/bin/bash

INSTANCE_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/instance-id`

AZ=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone`

REGION="${AZ%?}"



ROUTE_TABLES=`aws ec2 describe-route-tables --region $REGION --output text

--filters "Name=tag:NATAZ,Values=any,$AZ" | grep ROUTETABLES | awk '{print $2}'`

# Parse through RouteTables that need to be modified

for MY_RT_ID in $ROUTE_TABLES; do

aws ec2 replace-route --route-table-id $MY_RT_ID --destination-cidr-block 0.0.0.0/0 --instance-id $INSTANCE_ID` --region $REGION

done

Public Subnet 1

US-West-2


Public Subnet 2


US-East-1

TAG

VPN

EIP

TAG

VPN

true

TAG

VPN

true

TAG

VPN

EIP

• Network automation doesn’t require hard-core

development

• Simple scripts can be very powerful– Create dynamic, resilient networks and network components

– Respond to application or business requirements

• Network automation

• Manipulating IPs

• Network monitoring

• Virtual IP addresses– Support less cloud-friendly use cases

• Multicast– Support legacy multicast use cases

• Floating networks– Support overlapping network use cases

• Elastic IP

10.0.0.55

72.44.63.250

10.0.1.79

AWS Region

aws ec2 associate-address --private-ip-address 10.0.1.79

--allocation-id [EIP Allocation ID] --allow-reassociation

Availability Zone Availability Zone

• Secondary private IP

10.0.0.55

72.44.63.250

10.0.0.79

AWS Region

aws ec2 assign-private-ip-addresses --private-ip-addresses 10.0.0.10

--network-interface-id eni-123abcde --allow-reassociation

10.0.0.10

Availability Zone

• Routed virtual IP

10.0.0.55

192.168.0.10

10.0.1.79

AWS Region

#ifconfig eth0:1 192.168.0.10/32 up

aws ec2 replace-route --route-table-id [Route Table ID]


--instance-id [Instance ID]

Availability Zone Availability Zone

• Configure your instance OS with another IP

• Disable SRC/DST checking

• Use a replace-route API call to direct traffic

# ifconfig eth0:1 192.168.0.10/32 up

aws ec2 replace-route --route-table-id [Route Table ID]


--instance-id [Instance ID]

aws ec2 modify-instance-attribute --instance-id [Instance ID]

–no-source-dest-check

Approach Pros Cons

EIP Multi-AZ Public IP only

(split-brained DNS within

VPC for private IP)

Secondary IP Public and/or

private IP

Single AZ

Routed VIP Multi-AZ Private IP only

Only accessible from

instances within the VPC

• Each VIP option supports application-specific

requirements

• Embrace automation– Build this into application deployment

• Build your network as an integral part of the

application it supports

• By and large, a disappointment

• Some legacy apps/app servers require multicast– Node discovery

– Session management

– Automated failover

• Not directly supported

• Can be implemented with an overlay network• GRE or L2TP tunnels, Ntop’s N2N

10.0.0.54

10.0.0.79

10.0.1.132

Subnet 10.0.0.0/24 Subnet 10.0.1.0/24

10.0.1.18310.0.0.41

• This technically isn’t multicast– Multicast packets are wrapped in unicast packets

• Not a solution for unicast scaling

– Additional packet overhead

• GRE approach adds 38 bytes (MTU of 1500 will effectively be 1462)

• Unicast traffic can also traverse the overlay network– Not subject to security group filtering

• Decent option for small, legacy clusters– App server node discover

– Low traffic volumes

• GRE configuration can be automated– Multicast configuration stored in tags

• Periodically check for new members (60 seconds)

172.31.16.124

172.31.28.164

172.31.47.71

Subnet 172.31.16.0/20 Subnet 172.31.32.0/20

TAG: multicast

App1,192.168.0.12/24

TAG: multicast

App1,192.168.0.11/24

TAG: multicast

App1,192.168.0.10/24

192.168.0.0/24 Overlay

Community: App1

• Trend: We have less control over the network

ranges our networks must interact with– SaaS

– Mergers

– DevOps

Customer ‘n’

192.168.0.0/16

Customer 3

10.0.0.0/16

10.0.0.0/16

Customer 2

10.0.0.0/16

Customer 1

172.16.0.0/16

172.16.0.268

Service 1

Service 2

Service EIPs

Amazon

Route 53

Service ‘n’

...

Service load

balancers

• Most straightforward approach

• Great when your services do

not need to initiate connections

10.0.0.0/22

Customer 2

10.0.0.0/16

10.0.0.58

10.0.0.105

192.168.0.0/22

192.168.0.124

Customer 1

172.16.0.0/16

172.16.0.268

• Fairly straightforward

• Viable option with AWS

automation

• Typically best for single tenants– Shared systems can be challenging

10.0.0.0/16

10.0.0.0/16

192.1

68.0

.0/1

6

10.0.0.58

10.0.0.105

SRC: 10.0.0.58

DST: 192.168.0.105

SRC: 192.168.0.58

DST: 10.0.0.105

192.168.0.0/16

10.0.0.0/16

10.0.0.0/1610.0.0.58

10.0.0.105

SNAT: 192.168.0.0/16

DNAT: 10.0.0.0/16

# iptables -t nat -A PREROUTING -d 192.168.0.0/16 -j NETMAP --to 10.0.0.0/16

# iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -j NETMAP --to 192.168.0.0/16

Customer #

Customer #

10.0.0.0/16

Customer #

Customer #

• Slightly more complicated

• Consider a shared service, HA VPN tier

• Viable option with VPN appliance automation

• May be better for multitenant solutions

• Network automation

• Manipulating IPs

• Network monitoring

• Monitoring has traditionally been siloed– Application

– Server

– Network

• Limited access to networking devices– NAT/VPN instances

– No access to cloud switches/routers

• Need to think about monitoring differently

• Lorien Wood School– OpenDNS, SNMP (cacti)

• New network monitoring requirements– More visibility into student activity

• Urlsnarf

• Logrotated

• Cron

• SCP

• Python script

• PHP/MySQL app

t2.micro

• Focus on requirements– “Network monitoring” can mean just about anything

– Plan for scale

• Find the right tool to meet the requirements– Gateway-based vs. host-based

– Simple scripts

– OS tools (job schedulers, SSH/WMI, logging capabilities, and log file rotators)

– AWS partner solutions

• Leverage reusable architectural patterns

AWS_Regions

CloudWatch_Region

cw ec2.cloudwatch.connect_to_region

region AWS_Regions

vpcconn vpc.connect_to_region

vpns vpcconn.get_all_vpn_connections

vpn vpns

vpn.state available

active_tunnels

vpn.tunnels[0].status UP

active_tunnels

vpn.tunnels[1].status UP

active_tunnels

cw.put_metric_data VPNStatus vpn.id

VGW vpn.vpn_gateway_id CGW vpn.customer_gateway_id

• Log network events locally and ship them to: – CloudWatch logs

– Syslog / Windows logs

– AlertLogic, Cloudlytics, Logentries, Loggly, Splunk, SumoLogic …

• Leverage existing log monitoring infrastructure– Should already be built for scale

• Add network requirements and admins to log monitoring infrastructure development

Orchestrator Configure

MonitoringDistributed

Monitoring

Store ResultsAnalyze and

Report

Monitoring

Orchestrator

SCP

SSHtcpdump -G 60 pcap

collectorpcap

analyzer

curl

• Network monitoring gateways– Barracuda Networks, Checkpoint, Palo Alto Networks,

Sophos

• Host-based network intrusion detection– AlertLogic, TrendMicro, ThreatStack

• Network traffic analysis– Boundary, CloudShark

• Application network performance– Compuware

• Focus on requirements

• There are actually a lot of tools out there to help– Some you might not have thought of before

• Leverage automation to create application-specific

network monitoring solutions

• Plan for scale from the start

• Don’t fear doing things a little differently

Please give us your feedback on this session.

Complete session evaluations and earn re:Invent swag.

http://bit.ly/awsevals

https://s3.amazonaws.com/reinvent-arc401/index.html

http://bit.ly/awsevals

(ARC401) Black-Belt Networking for the Cloud Ninja | AWS re:Invent 2014

Technology

Transcript of (ARC401) Black-Belt Networking for the Cloud Ninja | AWS re:Invent 2014