AWS re:Invent 2016: From One to Many: Evolving VPC Design (ARC302)
-
Upload
amazon-web-services -
Category
Technology
-
view
1.086 -
download
1
Transcript of AWS re:Invent 2016: From One to Many: Evolving VPC Design (ARC302)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rob Alexander, Principal Solutions Architect
December 2, 2016
ARC302
From One to ManyEvolving VPC Design
Disclaimer:
Do Try This at Home!
Assuming you’ve heard of…
Route Table
Elastic
Network
Interface
Amazon VPC
Internet
Gateway
Customer
Gateway Virtual
Private
Gateway
VPN
Connection
VPC subnet
Network ACL
Security group
Enhanced
Networking
VPC
Peering
AWS Direct
Connect
Related Sessions
NET201 – Creating Your Virtual Data Center: VPC
Fundamentals and Connectivity Options
NET305 – Extending Datacenters to the Cloud:
Connectivity Options and Considerations for Hybrid
Environments
From one…
Subnet
Availability Zone A
Subnet
Availability Zone B
VPC
us-east-2
VPCVPC
VPC
VPCTransit VPC
VPC
us-west-2
VPC
VPC
VPC
eu-west-1
VPCVPC
VPC
VPCTransit VPC
VPC
Branch Branch
NA
HQ
VPC
VPC
VPC
VPC
VPC
VPC
Chicago DX
AP
HQ
London DX
ap-northeast-1
VPC
VPC
VPC
VPCTransit VPC
VPC
EU
HQ
Tokyo DX … to many
VPC/16
Choose a CIDR
• CIDR fixed on VPC
creation
• /16 down to /28
• Go Big
VPC IPv4 space design
• Plan for expansion to additional Availability
Zones or regions
• Consider connectivity to corporate networks
• Don’t overlap IP space
• Save space for the future
• IPv4 space is required, but …
IPv6 now supported in VPC
• Optionally enable IPv6 on VPC
• /56 of Amazon’s Global Unicast Address (GUA) per VPC
• /64 CIDR block per subnet
• IPv6 completely independent from IPv4
• Enabled per subnet or per instance (per ENI)
• Supported by Security Groups, Route Tables, NACLs, VPC
Peering, IGW, DX, Flow Logs and DNS Resolution
Availability Zone A
VPC
• Even distribution of IP
space across AZs
• Use at least 2 AZs
• Subnets are AZ
specific
• How big? How many?
Create subnets
Subnet
Availability Zone B
Subnet
Availability Zone C
Subnet
/16
Availability Zone A
Subnet
VPC
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet Subnet Subnet Subnet Subnet Subnet
Subnet Subnet Subnet Subnet Subnet
Subnet Subnet Subnet Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
/16
VPC subnet design
• Traditional switching limitations do not apply
• Consider large, mixed use subnets
• Use security groups to enforce isolation
• Use tags for grouping resources
• Use subnets as containers for routing policy
Related Sessions
NET401 – Another Day, Another Billion Packets
Availability Zone A
Public subnet
Private subnet
Availability Zone B
Public subnet
Private subnet
VPC/16
Availability Zone C
Public subnet
Private subnet
/22 /22 /22
/20 /20 /204091 IPs
1019 IPs
VPC/16
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
Availability Zone C
Private subnet
Public subnet
Private subnet
/22 /22 /22
/20
/20
/20
/20
/20
/20
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
.1
VPC
.1
.1 .1
.1 .1
Routing Policy
Main Route Table
Destination Target
10.1.0.0/16 Local
VPC CIDR 10.1.0.0/16
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Public Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 IGW
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Private Route Table
Destination Target
10.1.0.0/16 Local
Corp CIDR VGW
Availability Zone A
Public subnet
Private subnet
Availability Zone B
VPC/54
Availability Zone C
/64
/64
18 MILLION,
Public subnet
Private subnet
Public subnet
Private subnet
What about IPv6?
/64
/64
/64
/64
TRILLIONIPs
Availability Zone A
Private subnet
Public subnet
Availability Zone B
Private subnet
Public subnet
VPC
IPv6
Routing Policy
AWS Region
Internet
Public Route Table
Destination Target
10.1.0.0/16 Local
2001:db8:1234:1a00::/56 Local
0.0.0.0/0 IGW
::/0 IGW
Availability Zone A
Private subnet
Public subnet
Availability Zone B
Private subnet
Public subnet
VPC
IPv6
Routing Policy
AWS Region
Internet
Public Route Table
Destination Target
10.1.0.0/16 Local
2001:db8:1234:1a00::/56 Local
Corp CIDR VGW
::/0 EIGW
Egress-Only IGW
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 ???
Corp CIDR VGW
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Why go outside?
• AWS API endpoints
• Regional services
• Third-party services
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT Instance
Corp CIDR VGW
NAT
Instance
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT Instance
Corp CIDR VGW
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 Black Hole
Corp CIDR VGW
NAT
Instance
Scalable and Available NAT
Evolving design requirements
• Public subnets for resources reachable from Internet
• Private subnets with egress only access to public network
• Scalable, highly available NAT
• One AWS account
• One VPC
• One region
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
NAT
Instance
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 Black Hole
Corp CIDR VGW
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
Deploy a
NAT Gateway
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT Gateway
Corp CIDR VGW
NAT
Gateway
Why a NAT Gateway?
10.1.1.112:54318 52.27.192.88:35678
NAT Instance
Source IP: Port NAT’d Source IP:Port
Security Updates
Package Repos
NTP
VPC
Public Network
Why a NAT Gateway?
10.1.1.112:54318 52.27.192.88:35678
Source IP: Port NAT’d Source IP:Port
VPC
Source IP is the same
Source Port must be
unique
Destination
IP and Port
are the same
NAT Instance
Public Network
52.27.192.88:33622
52.27.192.88:38438
52.27.192.88:48132
52.27.192.88:29754Security Update
Why a NAT Gateway?
10.1.1.112:54318 52.27.192.88:35678
Source IP: Port NAT’d Source IP:Port
VPC
Source IP is the same
Source Port must be
unique
Destination
IP and Port
are the same
Public Network
52.27.192.88:33622
52.27.192.88:38438
52.27.192.88:48132
52.27.192.88:29754
NAT Gateway
Security Update
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
Deploy a
NAT Gateway
NAT
Gateway
• Still need IGW
• Separate subnets
• Requires EIP
• AZ specific
• Burst to 10 Gbps
1
NAT Gateway: Securing Access
NAT Gateway ENI:
Network ACL
Public subnet
NAT
GatewayNetwork ACLs still apply
NAT Gateway: Securing Access
Use routing
policy to control
access to NAT
Gateway
Private subnet
Public subnet
Private subnet
NAT Enabled
no-NAT
no-NAT Private Route Table
Destination Target
10.1.0.0/16 Local
NAT Enabled Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT Gateway
NAT
Gateway2
NAT Gateway: Securing Access
Outbound Rules
Type Protocol Port Range Destination
All traffic All 0 - 65535 0.0.0.0/0
Use security groups
to restrict outbound
access for instances
Default VPC security group:
3
NAT Gateway: Securing Access
Outbound Rules
Type Protocol Port Range Destination
All traffic All 0 - 65535 10.2.0.0/16
Outbound Rules
Type Protocol Port Range Destination
All traffic All 0 - 65535 0.0.0.0/0
Use security groups
to restrict outbound
access for instances
Default VPC security group:
NAT Enabled VPC security group:
3
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
Deploy a
NAT Gateway
NAT
Gateway
NAT Enabled
no-NAT
NAT Enabled
no-NAT
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
Deploy a
NAT Gateway
NAT
Gateway
NAT
Gateway
NAT Enabled
no-NAT
NAT Enabled
no-NAT
• Drop in replacement for NAT instance
• Fully managed
• Highly available and fault tolerant
• Scalable to 10 Gbps burst per gateway
• Supports VPC Flow Logs
• No higher level functions like IPS, UTM,
URL Filtering, packet inspection, etc
• Cannot associate security group to
gateway
Pro & Con: NAT Gateway
AWS
Region
Considering multiple VPCs
Public-facing
web apps
Internal
company
apps
What’s next?
VPN
connection
VPC VPC VPC
Customer
network
One VPC, Two VPC
VPC
Why not 1 big VPC?
Why not 1 AWS Account?
• Blast radius
• Account Limits
• API Limits
Considerations for one or many VPCs
AWS Region
ProdNot
Prod
VPCVPC
Considerations for one or many VPCs
AWS Region
PCI
Apps
VPC VPC
Non
Regulated
Apps
Considerations for one or many VPCs
AWS Region
Prod
VPC
AWS Region
Disaster
Recovery
VPC
Considerations for one or many VPCs
AWS RegionVPC
Audit
Logging &
Analytics
AWS
CloudTrail
AWS
Config
VPC Flow
Logs
VPC
Legal
VPC
Finance
VPC
Sales
App Logs,
S3 Access Logs,
ELB Logs
Amazon
Redshift
Amazon
EMR
S3
AWS Region
Internal application to VPC
Public-facing
web app
Internal
company
app
VPN
connection
VPCVPC
Customer
network
Availability Zone A
Private subnet Private subnet
AWS Region
Virtual
Private
Gateway
VPN
connection
Intranet
app
Intranet
app
Availability Zone B
Internal customers
Private Route Table
Destination Target
10.1.0.0/16 Local
Corp CIDR VGW
VPC
Internal application to VPC
Customer
network
But apps will make heavy use of …
Amazon S3
…as a primary data store
VPC Egress Control
Evolving design requirements
• VPN connectivity to private-only VPC
• No egress in the VPC to public networks
• Private IP access to Amazon S3
• Content-specific access controls
• One AWS account
• One VPC
• One region
Availability Zone A
Private subnet Private subnet
AWS
Region
Virtual
Private
Gateway
VPN
connection
Intranet
app
Intranet
app
Availability Zone B
You really don’t want to do this:
Amazon
S3
Internet
Customer
border router
Customer VPN
Internet
VPC
Customer
network
Availability Zone A
Private subnet Private subnet
AWS
Region
Virtual
Private
Gateway
Intranet
app
Intranet
app
Availability Zone B
So do this instead:
Amazon
S3
VPC
VPN
connection
VPC Endpoints
• No IGW
• No NAT
• No public IPs
• Free
• Robust access
control
Customer
network
Creating S3 VPC endpoint
aws ec2 create-vpc-endpoint
--vpc-id vpc-40f18d25
--service-name com.amazonaws.us-west-2.s3
--route-table-ids rtb-2ae6a24f rtb-61c78704
Private subnet
VPCRoute Table
Destination Target
10.1.0.0/16 Local
Corp CIDR VGW
Prefix List for S3 us-west-2 VPCE
Creating S3 VPC endpoint
aws ec2 create-vpc-endpoint
--vpc-id vpc-40f18d25
--service-name com.amazonaws.us-west-2.s3
--route-table-ids rtb-2ae6a24f rtb-61c78704
Public subnet
VPCRoute Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 IGW
Prefix List for S3 us-west-2 VPCE
Creating S3 VPC endpoint
Private subnet
VPC
Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 NAT Gateway
Prefix List for S3 us-west-2 VPCE
Public subnet
NAT
Gateway
Prefix lists
aws ec2 describe-prefix-lists
PREFIXLISTS pl-68a54001 com.amazonaws.us-west-2.s3
CIDRS 54.231.160.0/19
CIDRS 52.218.128.0/18
• Logical route destination target
• Dynamically translates to service IPs
• S3 IP ranges change over time
• S3 prefix lists abstract change
Prefix lists
… and use them in your outbound security group rules!
Private subnet
Controlling VPC access to Amazon S3
AWS Identity & Access
Management (IAM) policy
on VPCE:
VPC
{
"Statement": [
{
"Sid": "vpce-restrict-to-backup-bucket",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:PutObject”
],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::backups-reinvent",
"arn:aws:s3:::backups-reinvent/*"]
}
]
}
Backups bucket?
Private subnet
Controlling VPC access to Amazon S3
S3 bucket policy:
VPC
From
vpce-bc42a4e5?
{
"Statement": [
{
"Sid": "bucket-restrict-to-specific-vpce",
"Principal": "*",
"Action": "s3:*",
"Effect": "Deny",
"Resource": ["arn:aws:s3:::backups-reinvent",
"arn:aws:s3:::backups-reinvent/*"],
"Condition": {
"StringNotEquals": {
"aws:sourceVpce": "vpce-bc42a4e5”
}
}
}
]
}
Controlling VPC access to Amazon S3
Recap on security layers:
1. Route table association
2. VPCE policy
3. Bucket policy
4. Security groups with prefix list
Private subnet
VPC
1.
2.
3.
4.
Private subnet Private subnet
AWS
Region
Intranet
apps
Compliance
app
Endpoints in action
VPC
Compliance Backups
VPCE1 VPCE2
Private subnet
Intranet
apps
Private subnet Private subnet
AWS
Region
Intranet
apps
Compliance
app
Endpoints in action
VPC
Compliance Backups
VPCE1 VPCE2
Private subnet
Intranet
appsPrivate subnet Private subnet
Private subnet
Logs Analytics
• Secure, highly scalable and highly
available access to S3
• Fine grained control of access to
content in S3 from VPC
• Control which VPCs/VPCEs can
access which S3 buckets
• No public IPs required, source IPs kept
private
• Bucket policy restricted to specific
VPCs (or VPCEs) will disable S3
Console access
• Requires Amazon DNS enabled on
VPC
Pro & Con: VPC Endpoints
AWS Region
Public-facing
web apps
Internal-
only
apps
What’s next?
VPN
connection
VPC VPC VPC
Customer
networkCustomer Gateway
(CGW)
Shared Service Hubs
AWS
Region
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPCVPC
Customer
network
Public
apps
Internal
apps
AWS Region
VPC
HA VPN Pair
Availability Zone A
HA VPN
To
VPC
iBGP
eBGP
Customer CIDRs or Default Route
eBGP
AWS ASN 7224
Re-advertise VPC CIDR via IGP
VGW
VPC CIDR
Customer ASN (Public or Private)
CGW1 CGW2
VPN1
Tun1
VPN1
Tun2
Availability Zone A
VPN2
Tun1
VPN2
Tun2
Reuse your CGW Public IP
to connect to more VPCs
Customer
network
MED
MED
AWS
Region
VPCVPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• SecurityShared services
Customer
network
Evolving design requirements
• Centralize network connectivity to and from cloud
• Centralize management, security, and common services
• Account owners in control of own VPC resources
• Many AWS accounts
• Many VPCs
• One region
AWS
RegionVPC
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• SecurityShared services
Hub and
Spoke
with
Peering
VPC
Shared
services
VPC
VPC
Customer
network
Spoke VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Spoke VPC
VPC
Customer
network
AWS Region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Public subnet
10.2.0.0/1610.1.0.0/16
Private subnet
Private Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.11.0/24 PCX-1
VPC peering
Shared services
10.2.22.0/24
10.1.11.0/24
AWS Region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Public subnet
10.2.0.0/1610.1.0.0/16
Private subnet
Private Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.11.0/24 PCX-1
172.16.0.0/16 PCX-1
Edge-to-edge routing
Shared services
10.2.22.0/24
10.1.11.0/24
172.16.0.0/16Customer
network
AWS Region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Proxy
subnets
10.2.0.0/1610.1.0.0/16
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/16 PCX-1
Edge-to-edge via proxy
PCX-1 10.2.22.0/24
Internal
ELB
Proxy
fleet
Internet
Public
services
S3VPC
Customer
network
Proxy Route Table
Destination Target
10.1.0.0/16 local
10.2.0.0/16 PCX-1
172.16.0.0/16 VGW
Proxy Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
172.16.0.0/16 VGW
0.0.0.0/0 IGW
S3 Prefix List VPCE
Customer
network
Availability Zone A
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
AWS Region
Internet
VPC
Auto Scaling
proxy
fleet
Public
servicesS3
PCX-1
Availability Zone B
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
Auto Scaling
proxy
fleet
Spoke VPC
VPC
Private subnet
Proxy in practice
Hub VPC
Availability Zone A
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
AWS Region
Internet
VPC
Auto Scaling
proxy
Fleet
Public
servicesS3
PCX-1
Availability Zone B
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Bastion
host
Auto Scaling
proxy
fleet
Spoke VPC
VPC
Private subnet
Proxy in practice
Hub VPC
Customer
network
Shared Services Hub: To-Do List
• Use IAM to restrict spoke AWS accounts from altering network
• Create a NetOps IAM role in all accounts:https://aws.amazon.com/blogs/security/how-to-assign-permissions-using-new-aws-managed-policies-for-job-functions/
• Enable AWS CloudTrail, AWS Config, and VPC Flow Logs for all accounts
• Integrate CloudTrail with CloudWatch Logs and create alarms:https://aws.amazon.com/blogs/aws/cloudtrail-integration-with-cloudwatch-now-available-in-four-more-regions
• Minimizes on premises network change
• Reduces latency, cost of cloud
applications accessing common services
• Provides spoke accounts control over
own resources
• But controls and secures egress traffic
from spokes
• Security Groups work across peers
• Cost and management of central proxy
layer
• Not a transparent proxy
• Configuring end devices to use proxy
• Restricted to HTTP/S
• No transitive networking
• Peering data transfer cost
Pro & Con: Shared Services Hub and Spoke
AWS Region
VPCVPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• Security
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
Shared services
Customer
network
Dev hubProd hub
Data
services
hub
AWS Region
VPCVPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• Security
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
Shared services
Customer
network
Dev hubProd hub
Data
services
hub
VPC
VPC
Customer
network
AWS Region
Availability Zone A
Private subnet
VPC
Availability Zone B
Private subnet
AWS Lambda
Amazon API Gateway
Elastic
Network
Interface
VPCVPC
VPC
VPC
Prod hub
VPC
Internet
Hybrid
Serverless
Amazon
AuroraReplica
Mobile Application VPC
Legacy
Apps
Customer
network
AWS Region
Availability Zone A
Private subnet
VPC
Availability Zone B
Private subnet
AWS Lambda
Amazon API Gateway
Elastic
Network
Interface
VPCVPC
VPC
VPC
Prod hub
VPC
Internet
Hybrid
Serverless
Amazon
AuroraReplica
Mobile Application VPC
us-east-2 region
VPC VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
eu-west-1 region
VPC VPC
VPC
VPC
VPC
VPC
VPC Mass Transit
Evolving design requirements
• Centralize and minimize network connections
• Allow end to end routing from cloud to existing networks
• Minimal operational overhead
• Leverage AWS network
• Many AWS accounts
• Many VPCs
• Many regions
Availability Zone A
Public subnet
VPC
Transit
VPC
Availability Zone B
Public subnet
AWS
Region
EC2 VPN EC2 VPN
Availability Zone A
Public subnet
VPC
Transit
VPC
Availability Zone B
Public subnet
EC2 VPN EC2 VPN
AWS
RegionVPC
Spoke VPC
Transit VPC
VPCSpoke VPC
VPCSpoke VPC
AWS
RegionVPC
VPC
VPC
VPC
VPC
VPC
VPCTransit VPC
Customer
network
Spoke VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Spoke VPC
Branches
Transit
VPC
https://aws.amazon.com/answers/networking/transit-vpc/
Transit VPC
Transit VPC
Built using Cisco Cloud Services Router (CSR) 1000V
• Available on the AWS Marketplace
• A virtualized ASR with full IOS-XE software stack
• BYOL or Pay-as-you-Go license models
Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
CSR1 CSR2
AWS Region
Transit VPC
S3 Bucket
for
VPN Config
Route Table
Destination Target
100.64.127.224/27 Local
0.0.0.0 IGW
Prefix List for S3 VPCE
100.64.127.224 / 27
Transit VPC:
Creation
What is EC2 Auto Recovery?
RECOVER Instance
Instance ID
Instance metadata
Private IP addresses
Elastic IP addresses
EBS volume attachments
Instance retains:
* Supported on C3, C4, M3, M4, P2, R3, T2, and X1 instance types with EBS-only storage
StatusCheckFailed_System
Amazon CloudWatch
per-instance metric alarm:
When alarm triggers?
Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
CSR1 CSR2
AWS Region
Transit VPC
S3 Bucket
for
VPN Config
VPCSpoke VPC
AWS Lambda
Cisco
Configurator
AWS Lambda
VGW Poller
transitvpc:spoke = true
Transit VPC:
Add Spoke
SSH Only to CSR Security Group
Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
AWS Region
Transit VPC
VPCSpoke VPCTransit VPC:
Preferred
Route
Spoke VPC Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 VGW
Transit VPC Route Table
Destination Target
100.64.127.224/27 Local
0.0.0.0 IGW
Prefix List for S3 VPCE
Active / Active
Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
AWS Region
Transit VPC
VPCSpoke VPC
transitvpc:preferred-path = CSR1
Transit VPC:
Preferred
Route
Spoke VPC Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 VGW
Transit VPC Route Table
Destination Target
100.64.127.224/27 Local
0.0.0.0 IGW
Prefix List for S3 VPCE
Spoke VGW Tag
Active / Passive
Transit VPC: Preferred route spoke configuration
From CSR2:
!
address-family ipv4 vrf vpn-8a23d2e3
neighbor 169.254.35.57 remote-as 7224
neighbor 169.254.35.57 timers 10 30 30
neighbor 169.254.35.57 activate
neighbor 169.254.35.57 as-override
neighbor 169.254.35.57 soft-reconfiguration inbound
neighbor 169.254.35.57 route-map rm-vpn-8a23c7e3 out
exit-address-family
!
route-map rm-vpn-8a23c7e3 permit 10
set as-path prepend 64512 64512
!
BGP AS override
configured by default
Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
CSR1 CSR2
AWS Region
Transit VPC
S3 Bucket
for
VPN Config
VPCSpoke VPC
AWS Lambda
Cisco
Configurator
AWS Lambda
VGW Poller
transitvpc:spoke = false
Transit VPC:
Remove Spoke
AWS
RegionVPC
VPC
VPC
VPC
VPC
VPC
VPC
Transit VPC
Customer
network
Spoke VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Spoke VPC
Branches
Transit
VPC
Internet
Public
services
Customer
network
VPCTransit VPC
us-east-2
us-west-2
VPC
VPC
Spoke VPC
Spoke
VPC
VPCTransit VPC
eu-west-1
eu-central-1
VPC
VPC
Spoke VPC
Spoke
VPC
AWS Network
Backbone
Internet
• End to End routing between VPCs in all
regions and any other non-AWS network
• Central transit routers can perform higher
level networking and security functions
• Spoke VGWs are HA by default
• Minimizes on premises networking changes
• Can minimize cost if replacing on premises
or colo networking hardware
• Availability and management of transit router
instances
• Licensing costs
• Cost of data transfer between transit, spokes
and other networks
Pro & Con: Transit VPC
AWS
Region VPC
VPC
VPC
VPC
VPCTransit VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Transit VPC
with
AWS Direct
Connect
(DX)
Detached
VGW
transitvpc:spoke = true
Customer
network
AWS Direct Connect
location
Private virtual interface (VIF) to
detached VGW• 1 PVI per VGW
• 1 BGP ASN
• 1 802.1Q VLAN Tag
• 1 BGP MD5 key
Private fiber connectionOne or multiple
50 – 500 Mbps,
1 Gbps or 10 Gbps pipes
AWS
Region VPC
VPC
VPC
VPC
VPCTransit VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Customer
network
AWS Direct Connect
location
Private DX VIF to
dedicated VGW
100.64.127.224 / 27
Private Virtual Interface 1
VLAN Tag 101
BGP ASN 7224
BGP Announce 100.64.127.224/27
Interface IP 169.254.251.5/30
Customer Interface 0/1.101
VLAN Tag 101
BGP ASN 65001
BGP Announce Customer Internal
Interface IP 169.254.251.6/30
AWS
Region VPC
VPC
VPC
VPC
VPCTransit VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Customer
network
AWS Direct Connect
location
Public DX VIF to
dedicated VGW
Public EIPs
Public Virtual Interface 1
VLAN Tag 501
BGP ASN 7224
BGP Announce AWS Regional
Public CIDRs
Interface IP Public /30 Provided
Customer Interface 0/1.501
VLAN Tag 501
BGP ASN 65501 (or Public)
BGP Announce Customer Public
Interface IP Public /30 Provided
NAT + Security layer
Equinix Chicago
Customer
network
us-west-2
VPC
VPC
VPCVPC
Transit VPC
VPC
us-east-2
VPC
VPC
VPCVPC
Transit VPC
VPC
AWS Direct Connect Inter-Region Connectivity
A single DX Public interface can reach all US regions
• Be selective in your public network announcements
• Filtering public prefix announcements if necessary
• Authoritative AWS public IP list available:
https://ip-ranges.amazonaws.com/ip-ranges.json
• For notification of IP changes, subscribe to SNS topic:
arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged
AWS Direct Connect Public Interface
Related Sessions
NET402 – Deep Dive: AWS Direct Connect and VPNs
Leverage corporate network
Headquarters
Branch
Branch
DX Location
Provider Edge (PE)Customer Edge (CE)
eBGP
Provider
MPLS
Network
PECE
PE
CE
eBGP
AWS Region
MPLS / IPVPN
PE DX
eBGP
CE PE
Headquarters
Branch
Branch
Chicago DX Location
eBGP
Provider
MPLS
Network
PECE
PECE
AWS
Ohio
region
Multi-region DX
PE DX
eBGP
CE PE
London DX Location
AWS
Ireland
region
PE DX
eBGP
Going global
AS 7224
AS 7224
100 BGP Route Max
100 BGP Route Max
• Private network, no Internet dependencies
• Predictable latency on DX connections
• Dedicated bandwidth to AWS
• Access to public networks of all US regions
over single US based DX connection
• Public DX BGP announcements may require
filtering
• For large networks, 100 route per VPC limit
may require summarization or default routes
• Cost of provider network and DX connections
Pro & Con: Transit VPC with DX
us-east-2
VPCVPC
VPC
VPCTransit VPC
VPC
us-west-2
VPC
VPC
VPC
eu-west-1
VPCVPC
VPC
VPCTransit VPC
VPC
AWS Network
Backbone
Provider
MPLS
Network
Branch Branch
NA
HQ
VPC
VPC
VPC
VPC
VPC
VPC
Chicago DX
AP
HQ
London DX
ap-northeast-1
VPC
VPC
VPC
VPC
Transit VPC
VPC
EU
HQ
Tokyo DX
Thank you!
Remember to complete
your evaluations!