OpenStack Network Design using Cisco

SolutionsShannon McFarland – CCIE #5245

Principal Engineer

@eyepv6

• Getting Started with OpenStack (DEVNET-1101)

• OpenStack Enabling DevOps (DEVNET-1104)

• OpenStack and OpenDaylight (DEVNET-1105)

• TECACI-2009 - ACI – The Policy Driven Data Centre

Other Sessions

• Introduction to OpenStack

• OpenStack Participation

• OpenStack Deployment Summary

• OpenStack Networking

• Cisco Integration into Neutron

• Advanced Neutron Considerations

• Conclusion

Agenda

Introduction to OpenStack

“OpenStack is a collection of open source

technologies delivering a massively scalable

cloud operating system” - openstack.org

OpenStack Projects

Compute (Nova) Dashboard (Horizon) Database (Trove)

Network (Neutron) Image (Glance) Orchestration (Heat)

Object Storage (Swift) Identity (KeyStone) Data Processing (Sahara)

Block Storage (Cinder) Telemetry (Ceilometer) Deployment (Triple O)

Bare Metal (Ironic) DNS (Designate) Application Catalog (Murano)

Containers (Magnum) Key Management (Barbican) Policy (Congress)

File System (Manila) Messaging (Zaqar) ….

OpenStack Participation

• Choice

• There is no one-size fits all option for cloud computing

• There is no single vendor who can fill all needs of a cloud stack – You will likely engage with multiple partners

• Community

• Open Source

• Community driven – Individual, organisational

• Better time-to-market and faster feature velocity

• Commercialisation

• Start with the ‘baseline’ OpenStack components

• Vendor opportunities for value-add integration on top of OpenStack baseline• Design, deployment, automation, operation, high-availability, applications, etc…

Why Does OpenStack Matter?

Cisco and OpenStack

• Cisco Validated Designs, UCSO

• Work closely and jointly with customers to design and build OpenStack environment

• OpenStack based Global Intercloud hosted across Cisco and partners data centres

• Metapod (Formerly MetaCloud)

• Neutron/Cinder/Ironic Plugins/Drivers for Cisco infrastructure – Nexus, APIC, CSR1K, ASR1K, UCS

• Cisco Applications on OpenStack

• Code contributions across several services – Network. Compute, Dashboard, Storage, Containers

Community Participation

Engineering

Partners/ Customers

Cloud Services

• Incubating new OpenStack related Projects – GBP, PlaceWise, AVOS, VMTP

http://www.cisco.com/c/en/us/solutions/data-center-virtualization/openstack-at-cisco/index.html

• Transport Layer Security

• Sub-ordinate certificate

feature

Kilo and Liberty release contributions lead by Cisco

Kilo + Liberty release

Gnocchi

Kolla

Magnum

Neutron

HorizonDevstack

Metering

Barbican

Heat

• Multiple IPv6 prefixes, IPv6 PD

• IPv6 router support

• VLAN trunking

• UCSM, Nexus driver

• ASR1000 driver

• CSR1Kv VPN driver

• Archive Policy per metric level

• New resources for Neutron PCI

Passthrough and Nova Flavor

• Heat template improvements

• Neutron IPv6 and L3

plugin support

• Kafka Publisher

• Alarms severity

• Network services notification

plugin

• Curvature panel

• Ceph panel

• Containers - Ceilometer, Mongo,

Neutron

• Container Sets - database-control,

messaging-control, service-control,

compute-control, compute-

operation-nova

• Kubernetes plugin

• Python API for k8s CLI

• Container Networking Model

OpenStack Deployment Summary

Common Enterprise Use Cases

• OpenStack, at least today, is targeted at modern day cloud native applications

• Sandbox environments

• A place to research, learn and test CI/CD processes

• PoC web applications along with ‘practicing’ the new DevOps methodology

• A place to learn the whole cloud deployment framework, document, train, move to production

• Development environments

• Using the lessons learned in the sandbox phase:• Build Dev, QA and production environments

• Apply CI/CD processes

• Slow-role Web application deployment either on ‘standard’ OpenStack or in conjunction with a PaaS deployment

• Data Processing environments – Big Data clusters, etc..

• Training systems – Cheap and fast to build and tear down for each class

• Revenue generating applications – Vertical applications

Telco’s are Turning to OpenStack for NFV

› Resource Allocation & Optimisation

› Resource Isolation

PLUGIN ESXi

OS NETWORK

FRAMEWORK

OS COMPUTE

FRAMEWORK

OS STORAGE

FRAMEWORK

NEUTRON

APINOVA API

SWIFT

API

PLUGIN

GLANCE

API

CINDER

API

PLUGIN

OS KEYSTONE

FRAMEWORK

KEYSTONE

API

Ce

ilom

ete

r

PLUGINLinux

COMPUTE STORAGENETWORK IDAM

Su

pp

ort fu

nc

tion

sPLUGINPLUGIN

Cloud Manager

Application Domain OSS

NFV Applications Enterprise Applications

› Real Time Response– Interrupt servicing

– OVS latency

› Networking– WAN orchestration

– VNF provisioning

› Carrier Grade Security– Multi-tenancy with end-to-end

isolation

› Software Management and Upgrade Support– Hitless & automated upgrades

› Backup and Restore– Automatic backup

› Audit and Trouble Shooting– Audit log, monitor

› Assurance:› High Availability– Mitigation of failures

– Fault monitoring and heath check

FirewallDPICDNWAN

AccelerationDNS

CarrierGrade NAT Session Border

Controller

PE RouterEPC

https://wiki.openstack.org/wiki/TelcoWorkingGroup

AIOController/Compute/

Storage

AIO Controller:

- MySQL, MariaDB, etc

- RabbitMQ, Qpid, etc..

- API Endpoints:

- Keystone

- Glance

- Nova

- Neutron

- Cinder

- Heat

- Swift

AIOController

Compute/

Storage

Compute/

Storage

Compute

Compute

Storage

Storage

StorageCompute

AIOController

All-in-One (AIO) – Getting Started

Data Centre

Infrastructure

OOB

Compute

Network

Node(s)

AIO

Controller

Compute

Network

Node(s)

AIO

Controller

Compute

Network

Node(s)

AIO

Controller

Spine/Agg Layer

TOR(s) TOR(s) TOR(s)

Spine/Agg Layer

Block

Storage

Block

Storage

Block

Storage

AIO Controllers:

- Galera/MySQL

- RabbitMQ

- API Endpoints:

- Keystone

- Glance

- Nova

- Neutron

- Cinder

- Heat

- Swift

OOB OOBSLB

Infrastructure

Services

Build/PXE

Automation

DNS

DHCP

NTP

Logging

Object

Storage

Object

Storage

Object

Storage

All-in-One (AIO) Compressed HA

OpenStack Networking

What Really Changes in my Data Centre?

• OpenStack components live South of the Top-of-Rack switch

• Your existing DC, Internet Edge and BN architecture stays the same

• It’s about the compute, storage and orchestration/management tiers

• Your apps go largely unchanged

Serv

ices

AccessLayer

AggLayer

CoreLayer

UC

S C

-Se

rie

s UC

S B

-Se

ries

Enterprise/Internet

OpenStack Lives Here

Neutron Networking for Layer 2 Tenant Isolation

Layer 2

Networks

Tenant

Networks

Admin

Provider

Networks

VLAN

VXLAN

GRE

vSwitch

ToR, Fabric

vSwitch, ToR

vSwitch

Network Type

Network Segmentation Scheme for tenant isolation

Device implementing Network Segmentation Scheme

Direct Device

Configuration

Device Configuration

through Controller

Neutron driver

Provisioned

Externally

Layer 2 Network Tenant Topologies

Compute

Node

Compute

Node

VM3 VM4 VM2

vswitch vswitch

Data Network

VM1

Fabric Leaf, Top of Rack

Compute

Node

Compute

Node

VM3 VM4 VM2

vswitch vswitch

Data Network

VM1

Fabric Leaf, Top of Rack

Host and Network based VLAN

Host based overlays

Compute

Node

Compute

Node

VM3 VM4 VM2

vswitch vswitch

Data Network

VM1

Fabric Leaf , Top of Rack

Network based overlays

VLAN Overlay

Neutron Networking for Layer 3 and Network Services

Layer 3

Networks

Tenant

Routers

Admin

Provider

Networks

Linux Host

Service

VM’s

Provisioned

Externally

Network Type Device implementing Network Service

Direct Device

Configuration

Device Configuration

through Controller

Neutron driver

Service

Node,

Fabric

Compute

Node

vswitch

Layer 3 Tenant Network Topologies

Linux Host

Compute

Node

VM1

Network

Node(s)

VM2

vswitchvswitch

Data Network

Namespace

Service VMs

Fabric, Top of Rack

VM1

Compute

Node

VM2

vswitch

Data Network

Service VMs

Fabric, Top of Rack

Compute

Node

VM1

Network

Node(s)

VM

vswitch

Data Network

Fabric, Service Node

Fabric or Service Node

vswitch

Cisco Integration into Neutron

Neutron Layer 2 Default Implementation

Neutron Server

Neutron Core plugin

(ML2)

Network REST API requests

Open vSwitch/Linux

Bridge

Mechanism Drivers

Compute Node

Network and

Compute Nodes

VM VM

vswitchRPC message to agent on nodes

• Implements Neutron Core Resources

• Open vSwitch and Linux Bridge Mechanism Drivers

• Agents on Network and Compute Nodes

• Host based VLAN or Overlay (VXLAN, GRE) Type Drivers

Nova HostNova HostNova Host

Neutron Reference – East-West L2 (Switched) Traffic

VM1Controller

Host(s)

Router

Neutron

Host(s)

DHCP ports

API NetworkExternal Network

Management Network

VM6VM5VM2 VM3 VM4

Internet

vswitch vswitch vswitchvswitch

Data Network

PKT

Packet path animation for packet

traveling from VM1 VM3.

VM on a Compute

Nodes

Neutron Cisco Nexus Driver Features

• Works with multiple Nexus platforms

• VLAN configuration

• VXLAN configuration

• Nexus_VXLAN Type Driver

• Multicast

• VLAN to VNI association

Benefits

• No need trunk all tenant VLANs on compute node interfaces on ToR

• Dynamic provisioning/deprovisioningon ToR

• Network based overlays

Neutron Server

Neutron Core

plugin (ML2)

Cisco Nexus Driver

Ncclient

Nexus

Nova

Compute Nodes

create/update port request sent to Neutron

Nexus ToR

VM VMnetconf

VMs on Compute

Node

Neutron Cisco UCSM Driver (KVM)Features:

• Nova and Neutron enhancements to support SR-IOV

• Supports VLAN configuration of SR-IOV ports (using port profiles) and vNIC ports (using Service Profiles)

• Enables configuration of VLAN profiles and automatic association with network ports

Benefits

• SR-IOV and non SR-IOV based UCS Fabric Interconnect configurations

Neutron Server

Neutron Core

plugin (ML2)

Cisco UCSM driver

UCS Fabric

Interconnect

UCSM SDK

Compute Nodes

Nova

create/update port

VM VM

Neutron Routing Implementation

Neutron Server

Neutron Service

plugin (L3)

Routing REST API requests

L3 agent on

Network Node

L3 agent on

Network Nodes

Default Gateway,

Namespace and

IPTables

Namespace maps to a Neutron logical router. IPTableshandle address translations

Agent Scheduler

Picks a L3 agent on a Network Node

Compute Node

Compute Nodes

L3 traffic goes through Network node

VM VM

Neutron router HA capabilities using VRRP

Nova HostNova HostNova Host

Neutron Reference – East-West L3 (Routed) Traffic

VM1Controller

Host(s)

Router

Neutron

Host(s)

API NetworkExternal Network

Management Network

VM6VM5VM2 VM3 VM4

Internet

vswitch vswitch vswitchvswitch

Data Network

PKT

Routing

Packet path animation for packet

traveling from VM1 VM4

Virtual Router

Nova HostNova HostNova Host

Neutron Reference – North-South L3 Traffic (NAT)

VM1Controller

Host(s)

Router

Neutron

Host(s)

API NetworkExternal Network

Management Network

VM6VM5VM2 VM3 VM4

Internet

vswitch vswitch vswitchvswitch

Data Network

PKT

NAT

Packet path animation for packet

traveling from VM1 Internet

Virtual Router

Issues in Neutron Reference L3 and ASR1K Solutions

• NAT for External Connectivity:

• Issue - Scale limitation in Linux iptables software NAT

• Solution - ASR1K can scale up to 4 million dynamic NAT entries and 16K static NAT entries

• Tenant Routing:

• Issue - Scale limitations in Linux namespaces based software tenant networking

• Solution - ASR1K uses Virtual Routing and Forwarding (VRF) instances for tenant routers. ASR1K can scale up to 4k VRFs (8k in upcoming release)

• Tenant Networks:

• Issue- Scale limitations in Linux software based interfaces

• Solution - ASR1K plugin maps tenant networks to sub-interfaces on ASR1K. ASR1K supports up to 64k sub-interfaces

• Data Throughput:

• Issue - Performance limitations with software packet forwarding and NAT on generic compute hardware

• Solution - ASR1K can perform packet forwarding and NAT at rates up to 230 Gbps

Neutron Cisco ASR1000 for Neutron L3 Service• Mapping of Neutron reference L3

implementation -

• Linux namespaces - ASR1K VRF

• Internal Router ports – ASR1K VLAN or Port Channel sub interfaces

• External Gateway ports – ASR1K VLAN or Port Channel sub interfaces

• Linux IPTables – ASR1K NAT

• Benefits

• Routing using physical infrastructure

• Support for HSRP and Port Channel

Neutron Server

Neutron Service plugin

(L3)

Routing Device Driver

(ASR1K)

Config AgentCisco Config Agent

NexusASR1K

netconf

OpenStack Neutron + Nexus + ASR : Physical Topology Example Layer-3

Network Core

ASR 1000

Routers

OpenStack Controller

Neutron Server with

Cisco Config AgentNova Compute Nodes

Nexus Layer-2 Fabric

Tenant VLANs and

External Traffic

Management Network (NETCONF provisioning)

ASR1K

Neutron

Host(s)Nova HostNova HostNova Host

ML2 Nexus and ASR1K - East-West L3 (Routed) Traffic

VM1Controller

Node(s)

RouterAPI NetworkExternal Network

Data Network

(L3 routed)

Management Network

VM6VM5VM2 VM3 VM4

Internet

ML2 Nexus Driver

vSW vSW vSW

Nexus TOR Nexus TOR

ASR1K

L3

Plugin

VRF with default GW and NAT (to global routing).

PKT

Note : Packet animation included –

VM1 VM4

Virtual Router

ASR1K

Neutron

Host(s)Nova HostNova HostNova Host

ML2 Nexus and ASR1K - North-South L3 Traffic (NAT)

VM1Controller

Node(s)

RouterAPI NetworkExternal Network

Data Network

(L3 routed)

Management Network

VM6VM5VM2 VM3 VM4

Internet

ML2 Nexus Driver

vswitch vswitch vswitch

Nexus TOR Nexus TOR

ASR1K

L3

Plugin

VRF with default GW and NAT (to global routing).

PKT

Note : Packet animation included –

VM1 Internet

Virtual Router

ACI Integration with OpenStack• Multiple OpenStack Driver Options:

• APIC native Group-Based Policy

• Neutron ML2

• APIC VMM Domain support for OpenStack

• Endpoint statistics, health, faults in APIC

• Hypervisor local enforcement security policies

• Security Groups (ML2 driver) via IP Tables

• Group-Based Policies via OpenFlow in Open vSwitch

• Distributed NAT support on each compute node

• Floating IP

• sNAT (via hypervisor host IP)

• Distributed Neutron services per compute node

• L3 / Anycast gateway, metadata, DHCP

• Multiple VRF support

• Support for VLAN / VXLAN to ACI fabric

OpFlex

Proxy

OpenStack Controller

Hypervisor

vm1

Project 1 Project 2 Project 3

vm2vm5

vm4

vm3

OpFlex

Agent

APIC Driver

V(X)LAN

Open

vSwitch

New ACI + OpenStack Design Guides

• Cisco ACI with OpenStack OpFlex Architectural Overview: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/openstack/b_ACI_with_OpenStack_OpFlex_Architectural_Overview.html

• Cisco ACI with OpenStack OpFlex Deployment Guide for Red Hat: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/openstack/b_ACI_with_OpenStack_OpFlex_Deployment_Guide_for_Red_Hat.html

• Cisco ACI with OpenStack OpFlex Deployment Guide for Ubuntu: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/openstack/b_ACI_with_OpenStack_OpFlex_Deployment_Guide_for_Ubuntu.html

Opflex Extends ACI to Hypervisor

Pre-OpFlex Implementation OpFlex + OVS

Native neutron approach using Open vSwitch Agent

OpFlex Agent directly manages Open vSwitch and integrates with APIC

• VLAN per Network /

Group to ToR

• VXLAN within ACI

• Physical domain in ACI

• No APIC GUI

integration

• Supports unmodified

OVS, OVS agent

• VLAN or VXLAN per

Network / Policy Group to

ToR

• OpFlex Proxy runs in

leaf, OpFlex agent

manages OVS

• Hypervisor-local traffic

has policy, switching,

routing handled locally

• VMM domain, GUI

integration with APIC

• Distributed support for

NAT, metadata server

proxies, DHCP

OpenStack Controller

Hypervisor

vm1

Project 1 Project 2 Project 3

vm2vm5

vm4vm3

OVS

Agent

APIC Driver

VLAN

Open

vSwitch

OpFlex

Proxy

OpenStack Controller

Hypervisor

vm1

Project 1 Project 2 Project 3

vm2vm5

vm4vm3

OpFlex

Agent

APIC Driver

V(X)LAN

Open

vSwitch

OVS Driver

ACI Plugin Options

OpFlex

Proxy

OpenStack Controller

Hypervisor

vm1

Project 1 Project 2 Project 3

vm2vm5

vm4

vm3

OpFlex

Agent

APIC

Driver

V(X)LAN

Open

vSwitch

Group-Based

Policy

GBP policy and

NAT enforced

via OpenFlow

OpFlex

agent

receives

policy from

ACI leaf

OpFlex proxy

receives policy

from APICVLAN or VXLAN

between

hypervisor and

ToR

APIC Driver

converts GBP

to APIC

policyACI fabric

provides

distributed L2,

L3, and security

enforcement

OpFlex

Proxy

OpenStack Controller

Hypervisor

vm1

Project 1 Project 2 Project 3

vm2vm5

vm4

vm3

OpFlex

Agent

APIC

Driver

V(X)LAN

Open

vSwitch

IP tables used

for security

enforcement

OpFlex agent

receives policy

from ACI leaf

OpFlex proxy

receives policy

from APICVLAN or VXLAN

between

hypervisor and

ToR

APIC Driver

converts

Neutron to

ACI PolicyACI fabric

provides

distributed L2

and L3

IPTables

APIC ML2 Driver Group-Based Policy

VMs on Compute

Nodes

Neutron Cisco Application Policy Infrastructure Controller (APIC) Driver

Neutron Server

Neutron Core

plugin (ML2)

Cisco L2

APIC Driver

APIC

VMs on Compute

Nodes

Cisco L3

APIC Driver

ACI Spine/Leaf

Switches

REST APINetwork:EPG, Router:Contract

Provides distributed L2,L3 functionality

Neutron L3

Plugin Neutron API: Network, Router,

Subnet, Security Group

L2 / L3 enforced in fabric,

security groups enforced on

hypervisor

Demo

OpenStack integration with Cisco Solutions SummaryPurpose Using Cisco Product Code Availability

Network Layer 2 Virtual Switch Nexus 1000v OpenStack Juno

SR-IOV UCS Fabric Interconnect Cisco OpenStack Juno Plus Tech Preview

Physical Switch Nexus OpenStack Juno

DHCP IPAM Prime Network Registrar Not upstream yet

Network Layer 3 Virtual RouterCloud Services Router 1000v

OpenStack Juno

Physical Router ASR 1000 Not upstream yet

Network ServicesVirtual Firewall and VPN

Cloud Services Router 1000v

Firewall - Cisco OpenStack Juno Tech PreviewVPN - Cisco OpenStack Juno Plus Tech Preview

Network Layer2, Layer3, Services

ControllerApplication Policy Infrastructure Controller

APIC L2 - OpenStack JunoAPIC L3 - OpenStack Juno

Declarative Policy ModelGroup Based Policy Framework

Group Based Policy StackForge Juno

Advanced Neutron Considerations

The Hard Stuff – IPv6 + Cloud• Inside of a private cloud stack you have a lot of moving parts and they all ride on IP:

• API endpoints

• Provisioning, Orchestration and Management services

• Boatload of protocols and databases and high-availability components

• Virtual networking services <> Physical networking

• IPv6 has been available with OpenStack for awhile but it has depended on a lot of backports and custom patches to be functional

• Kilo offers the best ‘out-of-box’ support yet – but still needs more work

• Tenant IPv6 Address Assignment via:

• SLAAC, Stateful DHCPv6, Stateless DHCPv6

• Two common approaches for IPv6 support:

• Dual-Stack everything (Service Tier + Tenant Access Tier [Tenant management interface along with VM network access])

• Conditional Dual stack (Tenant Access Tier only – API endpoints & DBs are still IPv4)

Cloud Stack – IP Version Options

API endpoints

Service Tier

Database(s)

Automation

Interface

(GUI, CLI)

VM Operating

System

Tenant

Access Tier

Virtual

Networking

(L2/L3)

Virtual

Network

Services

(SLB/FW)

Tenant

Interface

(GUI, CLI)

Dual-Stack Everything

IPv4/IPv6

IPv4/IPv6

IPv4/IPv6

IPv4/IPv6

IPv4/IPv6

IPv4/IPv6

IPv4/IPv6

IPv4/IPv6

API endpoints

Service Tier

Database(s)

Automation

Interface

(GUI, CLI)

VM Operating

System

Tenant 1

Access Tier

Virtual

Networking

(L2/L3)

Virtual

Network

Services

(SLB/FW)

Tenant

Interface

(GUI, CLI)

Conditional Dual-Stack

IPv4/IPv6

IPv4/IPv6

IPv4/IPv6

IPv4/IPv6

IPv4

IPv4

IPv4

IPv4/IPv6

Tenant 2

Access Tier

IPv6

IPv6

IPv6

IPv6

VM Operating

System

Virtual

Networking

(L2/L3)

Virtual

Network

Services

(SLB/FW)

Tenant

Interface

(GUI, CLI)

Network Function Virtualisation

Tenant ACompute

Node

Compute

Node

VM1

Network

Node(s)

VM2 VM1

vswitch vswitchvswitch

Data Network

Namespace

10.1.0.4 10.1.0.5

10.1.0.1 10.1.1.1

10.1.1.4

Admin provisioned Service

Compute

Node

Compute

Node

VM1 VM2 VM1

vswitch vswitch

Data Network

10.1.0.4 10.1.0.5

Tenant provisioned Service

Service

VM

10.1.1.4

• Issue

• Anti-spoofing rules to ensure traffic originates and terminates as expected

• Doesn’t work for NFV VNF use cases

• Solution

• Added Port Security Extension• Adds new “Port Security enabled” attribute to

Network and Port Resources

• Only tenant owner can set this attribute on the resources

• Security Group and Allowed Address Pair are not allowed to be set

• Issue

• VXLAN for tenant isolation and VLAN for app traffic isolation within the tenant

• No means to identify VLAN transparent networks

• Solution

• Added Network Resource Extension• Adds new “Vlan Transparent” attribute to Network

Resource

• Only tenant owner can set this attribute on the resources

• No firewalling on VLAN tagged packets

Neutron and NFV

Conclusion

• OpenStack is for real and maturing at a rapid pace

• Many different players involved and it is evolving rapidly

• Align yourself with market leaders who have strong partnerships

• Installation is still tough but getting better

• Keep your design simple and reduce/remove technical debt wherever possible – it can kill your upgrades and scale out plans

• Get involved in the community – open source enjoys the major advantage of feature velocity

Q & A

Complete Your Online Session Evaluation

Learn online with Cisco Live!

Visit us online after the conference

for full access to session videos and

presentations.

www.CiscoLiveAPAC.com

Give us your feedback and receive a

Cisco 2016 T-Shirt by completing the

Overall Event Survey and 5 Session

Evaluations.– Directly from your mobile device on the Cisco Live

Mobile App

– By visiting the Cisco Live Mobile Site http://showcase.genie-connect.com/ciscolivemelbourne2016/

– Visit any Cisco Live Internet Station located

throughout the venue

T-Shirts can be collected Friday 11 March

at Registration

Thank you