Cisco ACI with OpenStack OpFlex Architectural OverviewFirst Published: February 11, 2016

Last Modified: March 30, 2016

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706USAhttp://www.cisco.comTel: 408 526-4000 800 553-NETS (6387)Fax: 408 527-0883

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITEDWARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITHTHE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain versionof the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDINGANYOTHERWARRANTYHEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS"WITH ALL FAULTS.CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OFMERCHANTABILITY, FITNESS FORA PARTICULAR PURPOSEANDNONINFRINGEMENTORARISING FROMACOURSEOFDEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUTLIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERSHAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, networktopology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentionaland coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnershiprelationship between Cisco and any other company. (1110R)

© 2016 Cisco Systems, Inc. All rights reserved.

C O N T E N T S

P r e f a c e Preface v

Audience v

Document Conventions v

Related Documentation vii

Documentation Feedback ix

Obtaining Documentation and Submitting a Service Request ix

C H A P T E R 1 Overview 1

About OpenStack and Cisco ACI 1

Extending OpFlex to the Compute Node 1

C H A P T E R 2 Solution Architecture 3

ACI with OpenStack Physical Architecture 3

OpFlex ML2 Software Architecture 4

Logical OpenStack Topology 6

Distributed Neutron Services 7

Mapping OpenStack and ACI Constructs 8

OpFlex NAT Operations 9

IP Subnets Required for NAT 9

OVS NAT and External Routing 10

Optimized DHCP and Metadata Proxy Operations 10

Optimized DHCP Services 11

Optimized Metadata Services 11

APIC OpenStack VMM Integration 12

Cisco ACI with OpenStack OpFlex Architectural Overview iii

Cisco ACI with OpenStack OpFlex Architectural Overviewiv

Contents

Preface

This preface includes the following sections:

• Audience, page v

• Document Conventions, page v

• Related Documentation, page vii

• Documentation Feedback, page ix

• Obtaining Documentation and Submitting a Service Request, page ix

AudienceThis guide is intended primarily for data center administrators with responsibilities and expertise in one ormore of the following:

• Virtual machine installation and administration

• Server administration

• Switch and network administration

Document ConventionsCommand descriptions use the following conventions:

DescriptionConvention

Bold text indicates the commands and keywords that you enter literallyas shown.

bold

Italic text indicates arguments for which the user supplies the values.Italic

Square brackets enclose an optional element (keyword or argument).[x]

Cisco ACI with OpenStack OpFlex Architectural Overview v

DescriptionConvention

Square brackets enclosing keywords or arguments separated by a verticalbar indicate an optional choice.

[x | y]

Braces enclosing keywords or arguments separated by a vertical barindicate a required choice.

{x | y}

Nested set of square brackets or braces indicate optional or requiredchoices within optional or required elements. Braces and a vertical barwithin square brackets indicate a required choice within an optionalelement.

[x {y | z}]

Indicates a variable for which you supply values, in context where italicscannot be used.

variable

A nonquoted set of characters. Do not use quotation marks around thestring or the string will include the quotation marks.

string

Examples use the following conventions:

DescriptionConvention

Terminal sessions and information the switch displays are in screen font.screen font

Information you must enter is in boldface screen font.boldface screen font

Arguments for which you supply values are in italic screen font.italic screen font

Nonprinting characters, such as passwords, are in angle brackets.< >

Default responses to system prompts are in square brackets.[ ]

An exclamation point (!) or a pound sign (#) at the beginning of a lineof code indicates a comment line.

!, #

This document uses the following conventions:

Means reader take note. Notes contain helpful suggestions or references to material not covered in themanual.

Note

Means reader be careful. In this situation, you might do something that could result in equipment damageor loss of data.

Caution

Cisco ACI with OpenStack OpFlex Architectural Overviewvi

PrefaceDocument Conventions

IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before youwork on any equipment, be aware of the hazards involved with electrical circuitry and be familiar withstandard practices for preventing accidents. Use the statement number provided at the end of each warningto locate its translation in the translated safety warnings that accompanied this device.

SAVE THESE INSTRUCTIONS

Warning

Related DocumentationThe Application Centric Infrastructure documentation set includes the following documents that are availableon Cisco.com at the following URL: http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html.

Web-Based Documentation

• Cisco APIC Management Information Model Reference

• Cisco APIC Online Help Reference

• Cisco APIC Python SDK Reference

• Cisco ACI Compatibility Tool

• Cisco ACI MIB Support List

Downloadable Documentation

• Knowledge Base Articles (KB Articles) are available at the following URL: http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html

• Cisco Application Centric Infrastructure Controller Release Notes

• Cisco Application Centric Infrastructure Fundamentals Guide

• Cisco APIC Getting Started Guide

• Cisco ACI Basic Configuration Guide

• Cisco ACI Virtualization Guide

• Cisco APIC REST API User Guide

• Cisco APIC Object Model Command Line Interface User Guide

• Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

• Cisco APIC Faults, Events, and System Messages Management Guide

• Cisco ACI System Messages Reference Guide

• Cisco APIC Layer 4 to Layer 7 Services Deployment Guide

• Cisco APIC Layer 4 to Layer 7 Device Package Development Guide

Cisco ACI with OpenStack OpFlex Architectural Overview vii

PrefaceRelated Documentation

• Cisco APIC Layer 4 to Layer 7 Device Package Test Guide

• Cisco ACI Firmware Management Guide

• Cisco ACI Troubleshooting Guide

• Cisco APIC NX-OS Style CLI Command Reference

• Cisco ACI Switch Command Reference, NX-OS Release 11.0

• Verified Scalability Guide for Cisco ACI

• Cisco ACI MIB Quick Reference

• Cisco Nexus CLI to Cisco APIC Mapping Guide

• Application Centric Infrastructure Fabric Hardware Installation Guide

• Cisco NX-OS Release Notes for Cisco Nexus 9000 Series ACI-Mode Switches

• Nexus 9000 Series ACI Mode Licensing Guide

• Cisco Nexus 9332PQ ACI-Mode Switch Hardware Installation Guide

• Cisco Nexus 9336PQ ACI-Mode Switch Hardware Installation Guide

• Cisco Nexus 9372PX and 9372PX-E ACI-Mode Switch Hardware Installation Guide

• Cisco Nexus 9372TX ACI-Mode Switch Hardware Installation Guide

• Cisco Nexus 9396PX ACI-Mode Switch Hardware Installation Guide

• Cisco Nexus 9396TX ACI-Mode Switch Hardware Installation Guide

• Cisco Nexus 93128TX ACI-Mode Switch Hardware Installation Guide

• Cisco Nexus 9504 NX-OS Mode Switch Hardware Installation Guide

• Cisco Nexus 9508 ACI-Mode Switch Hardware Installation Guide

• Cisco Nexus 9516 ACI-Mode Switch Hardware Installation Guide

Cisco Application Centric Infrastructure (ACI) Simulator Documentation

The following Cisco ACI Simulator documentation is available at http://www.cisco.com/c/en/us/support/cloud-systems-management/application-centric-infrastructure-simulator/tsd-products-support-series-home.html.

• Cisco ACI Simulator Release Notes

• Cisco ACI Simulator Installation Guide

• Cisco ACI Simulator Getting Started Guide

Cisco Nexus 9000 Series Switches Documentation

The Cisco Nexus 9000 Series Switches documentation is available at http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/tsd-products-support-series-home.html.

Cisco Application Virtual Switch Documentation

The Cisco Application Virtual Switch (AVS) documentation is available at http://www.cisco.com/c/en/us/support/switches/application-virtual-switch/tsd-products-support-series-home.html.

Cisco ACI with OpenStack OpFlex Architectural Overviewviii

PrefaceRelated Documentation

Cisco Application Centric Infrastructure (ACI) Integration with OpenStack Documentation

Cisco ACI integration with OpenStack documentation is available at http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html.

Documentation FeedbackTo provide technical feedback on this document, or to report an error or omission, please send your commentsto apic-docfeedback@cisco.com. We appreciate your feedback.

Obtaining Documentation and Submitting a Service RequestFor information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a servicerequest, and gathering additional information, seeWhat's New in Cisco Product Documentation at: http://www.cisco.com/c/en/us/td/docs/general/whatsnew/whatsnew.html

Subscribe toWhat’s New in Cisco Product Documentation, which lists all new and revised Cisco technicaldocumentation as an RSS feed and delivers content directly to your desktop using a reader application. TheRSS feeds are a free service.

Cisco ACI with OpenStack OpFlex Architectural Overview ix

PrefaceDocumentation Feedback

Cisco ACI with OpenStack OpFlex Architectural Overviewx

PrefaceObtaining Documentation and Submitting a Service Request

C H A P T E R 1Overview

This chapter contains the following sections:

• About OpenStack and Cisco ACI, page 1

• Extending OpFlex to the Compute Node, page 1

About OpenStack and Cisco ACICisco Application Centric Infrastructure (ACI) is a comprehensive policy-based architecture that provides anintelligent, controller-based network switching fabric. This fabric is designed to be programmatically managedthrough anAPI interface that can be directly integrated intomultiple orchestration, automation, andmanagementtools, including OpenStack. Integrating ACI with OpenStack allows dynamic creation of networking constructsto be driven directly from OpenStack requirements, while providing additional visibility within the ACIApplication Policy Infrastructure Controller (APIC) down to the level of the individual VM instance.

OpenStack defines a flexible software architecture for creating cloud-computing environments. The referencesoftware-based implementation of OpenStack allows for multiple Layer 2 transports including VLAN, GRE,and VXLAN. The Neutron project within OpenStack can also provide software based Layer 3 forwarding.When utilized with ACI, the ACI fabric provides an integrated Layer 2 and Layer 3 VXLAN-based overlaynetworking capability that can offload network encapsulation processing from the compute nodes onto thetop-of-rack or ACI leaf switches. This architecture provides the flexibility of software overlay networking inconjunction with the performance and operational benefits of hardware-based networking.

Extending OpFlex to the Compute NodeOpFlex is an open and extensible policy protocol designed to transfer declarative networking policies suchas those used in Cisco ACI to other devices. Utilizing OpFlex, the policy model native to ACI can be extendedall the way down into the virtual switches running on OpenStack Nova compute hosts. This OpFlex extensionto the compute host allows ACI to use Open vSwitch (OVS) to support common OpenStack features such asSource NAT (SNAT) and Floating IP in a distributed manner.

The OpFlex-based OpenStack drivers support two distinct modes of deployment. The first approach is basedon the Neutron API and Modular Layer 2 (ML2), which are designed to provide common constructs such asnetwork, router, and security groups that are familiar to Neutron users. The second approach is native to theGroup-Based Policy abstractions for OpenStack, which are closely aligned with the declarative policy model

Cisco ACI with OpenStack OpFlex Architectural Overview 1

used in Cisco ACI. The content of this document is focused specifically on details of the OpFlex ML2-baseddriver.

See cisco.com for further information and specific deployment guidance for the OpFlex GBP-based driver.

Cisco ACI with OpenStack OpFlex Architectural Overview2

OverviewExtending OpFlex to the Compute Node

C H A P T E R 2Solution Architecture

This chapter contains the following sections:

• ACI with OpenStack Physical Architecture, page 3

• OpFlex ML2 Software Architecture, page 4

• Logical OpenStack Topology, page 6

• Mapping OpenStack and ACI Constructs, page 8

• OpFlex NAT Operations, page 9

• Optimized DHCP and Metadata Proxy Operations, page 10

• APIC OpenStack VMM Integration, page 12

ACI with OpenStack Physical ArchitectureA typical architecture for an ACI fabric with an OpenStack deployment consists of a Nexus 9000 Spine/Leaftopology, an APIC cluster, and a group of servers to run the various control and compute components ofOpenStack. OpenStack can be deployed in many ways, but a very basic test architecture would consist of atleast one OpenStack Controller server which is also acting as the Neutron network node, and two or moreOpenStack compute nodes to host Virtual Machine (VM) instances. An ACI External Routed Networkconnection as a Layer 3 connection outside of the fabric can be used to provide connectivity outside theOpenStack cloud.

Cisco ACI with OpenStack OpFlex Architectural Overview 3

The configurations validated for this deployment guide utilized Cisco UCS C-Series rack mount serversin standalonemode. Third-party standalone rack servers running OpenStack can also be supported. Supportfor systems running UCS Manager with Fabric Interconnects attached to the ACI fabric is planned for afuture release.

Figure 1: Example ACI with OpenStack Physical Topology

Note

OpFlex ML2 Software ArchitectureTheModular Layer 2 framework in OpenStack allows integration of networking services based on TypeDrivers,andMechanismDrivers. Common networking type drivers include local, flat, vlan, vxlan, etc. OpFlex is addedas a new network type through ML2, with an actual packet encapsulation of either VXLAN or VLAN on thehost defined in the OpFlex configuration. A mechanism driver is enabled to communicate networkingrequirements from the Neutron server(s) to the Cisco APIC cluster. The APIC mechanism driver translatesNeutron networking elements such as a network (segment), subnet, router, or external network into APICconstructs within the ACI Policy Model.

The OpFlex ML2 software stack also currently utilizes a modified Open vSwitch Package, and local softwareagents on each OpenStack compute host that communicate with the Neutron server(s) and OVS. An OpFlexproxy from the ACI leaf switch exchanges policy information with the Agent-OVS instance in each computehost, effectively extending the ACI switch fabric and policy model into the virtual switch. This results in acohesive system that can apply networking policy anywhere in the combined virtual and physical switchingfabric starting from the virtual port where a VM instance attaches to the network.The Figure below illustratesthe interaction of the OpFlex ML2 APIC Driver with the ACI Fabric, and the extension of the OpFlex proxydown into the Agent-OVS service on the compute host.

Cisco ACI with OpenStack OpFlex Architectural Overview4

Solution ArchitectureOpFlex ML2 Software Architecture

The OpFlex ML2 APIC Driver for integration into Neutron runs on the server running theneutron-server service. This server may be a controller node running other OpenStack softwareelements, or be dedicated to the Neutron function. High Availability configurations with multiple Neutronservers are also supported.

Note

Figure 2: OpenStack with ACI architecture with OpFlex ML2

On the compute node, the neutron-opflex-agent service receives information about OpenStack endpoints fromtheML2 Driver software on the Neutron server. This information is stored locally in the endpoint files locatedin/var/lib/opflex-agent-ovs/endpoints. Theagent-ovs service uses the endpoint informationto resolve policy for the endpoints through the OpFlex Proxy on the connected ACI leaf switch. Theagent-ovs then programs policy on OVS using Open Flow for policies that can be enforced locally.

Cisco ACI with OpenStack OpFlex Architectural Overview 5

Solution ArchitectureOpFlex ML2 Software Architecture

Non-local policies are enforced on the upstream leaf switch. The Figure below illustrates the interactionbetween the Opflex modules running on the compute node, and OVS.

Figure 3: OpFlex Agent Architecture on the Compute Host

Logical OpenStack TopologyOpenStack defines multiple network connection requirements for the server nodes providing cloud services.Communication paths need to be defined and provided for Management traffic, Tenant Data, and Externalnetworking requirements, as well as API communication between the various OpenStack services. Deploymentsmay also dedicate a network segment for storage traffic or other specific needs. An ACI switching fabric isable to provide networking services to meet all of these requirements. Server connectivity can consist of eitherseparate physical interfaces, virtualized network adapters such as the Cisco VIC, or a managed blade serversystem such as Cisco UCS B-Series.

• Management and API Network(s)—This network segment is for administrative secure shell access toOpenStack servers, as well as API communication directly to the servers and between OpenStackfunctions. The Management and API functions may also be broken out into different network segments,the example configurations in this guide use a single network segment for both.

• External Network—With an ACI fabric integrated with OpFlex, the External Network path is providedby an External Routed Network configuration in the APIC. An External network in a system runningthe Neutron L3-Agent is a network on the outside of a software-based routing function. An ExternalNetwork utilizes NAT services to allow hidden and overlapping IPv4 address space to be used by Tenants.

• Tenant Data Network—A Tenant network in OpenStack can be dynamically created by a Tenant toprovide connectivity between VM instances in the cloud, and also to connect to cloud-based routingservices to other Tenant or External networks. The segment ID's assigned to Tenant networks withOpFlex are tracked by the ACI fabric, and consist of VXLAN between leaf switches, with either VXLANor VLAN between leaf switch and server.

Cisco ACI with OpenStack OpFlex Architectural Overview6

Solution ArchitectureLogical OpenStack Topology

Distributed Neutron ServicesOpenStack Neutron defines common networking constructs and services required by VM instances operatingin the cloud environment. Both availability and scale of Neutron services can become a concern whenimplementing all of these functions on a single server, or a small cluster of servers. The OpFlex ML2 Driversoftware stack provides the ability to distribute these network services across the compute nodes in the cluster,using a scale-out approach to reduce the load on any single instance of a service while increasing overallservice availability.

The following OpenStack Neutron services may be distributed to compute nodes when using the OpFlexML2Driver software:

• NAT for External Networks—The Opflex ML2 Driver approach for supporting External networksdistributes Source NAT and Floating IP functions for OpenStack to the Open vSwitch of the computehosts. Packets destined for IP addresses not defined in the private OpenStack Tenant space areautomatically NATted prior to egressing a compute host. The translated packets are then routed to theexternal routed network defined in the APIC. Distributed NAT services are inherent in the solution.

• Layer 3 Forwarding—The Neutron reference software implementation of Layer 3 Agent is replaced bya combination of Layer 3 forwarding in the ACI fabric, as well as local forwarding within a computenode. If two VMs connected to the same OpenStack Tenant router reside on the same compute node,Layer 3 traffic between them will be forwarded by OVS and remain local to that physical server.Distributed Layer 3 for traffic local to a compute node is inherent in the solution.

• DHCP—Reference Neutron software implementations have a DHCP Agent service centralized on theNeutron server(s). The OpFlex ML2 driver software allows a distributed DHCP approach using theagent-ovs service. Distributing the DHCP function across the compute nodes keeps DHCP Discovery,Offer, Request and Acknowledge (DORA) traffic local to the host and ensures reliable allocation of IPaddressing to VM Instances. Central Neutron address management functions communicate DHCPaddressing and options to the local agent-OVS over the management network. This optimized DHCPapproach is enabled by default in the solution, but can be reverted to the traditional centralized mode ifdesired.

• Metadata Proxy—OpenStack VM's can receive instance-specific information such as instance-id,hostnames, and SSH keys from the Nova Metadata Service. This service is normally reached through aNeutron service acting as a proxy on behalf of OpenStack VM instances. The OpFlex ML2 softwareallows this proxy function to be distributed to each of the compute hosts. This optimized metadata proxyis disabled by default, and either a traditional centralized or the distributed approach may be configured.

Cisco ACI with OpenStack OpFlex Architectural Overview 7

Solution ArchitectureDistributed Neutron Services

The logical topology diagram in the Figure below illustrates the connections to OpenStack network segmentsfrom Neutron server(s) and compute hosts, including the distributed Neutron services.

Figure 4: Logical OpenStack Network Connectivity with Distributed Neutron Services

The Management/API network for OpenStack may be connected to servers using an additional VirtualNIC on a common uplink with tenant networking to the ACI fabric, or via a separate physical interface.

Note

Mapping OpenStack and ACI ConstructsCisco ACI uses a policy model to enable network connectivity between endpoints attached to the fabric.OpenStack Neutron uses more traditional Layer 2 and Layer 3 networking concepts to define networkingconfiguration. The OpFlex ML2 driver translates the Neutron networking requirements into the necessaryACI policy model constructs to achieve the desired connectivity. The Table below illustrates OpenStackNeutron constructs, and the corresponding APIC policy objects that will be configured when they are created.

APIC ObjectOpenStack Object

ACI Tenant(s), VMM DomainNeutron Instance

APP Profile or Separate ACI TenantTenant/Project

Endpoint Group + Bridge DomainTenant Network

Cisco ACI with OpenStack OpFlex Architectural Overview8

Solution ArchitectureMapping OpenStack and ACI Constructs

APIC ObjectOpenStack Object

SubnetSubnet

N/A (Linux iptables rules are maintained per-host)Security Groups/Rules

Contract+EPG+Bridge DomainRouter

Layer 3 Out / Outside EPGExternal Network

By default, the OpFlexML2Driver associates an entire instance of OpenStack Neutron to a single ACI tenant,and names this tenant according to the apic_system_id setting in the/etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file. This allows theACI administratorto manage each cloud instance connected to the fabric as a single entity, and not generate a large number ofACI tenants in the APIC for a fabric that is being used for multiple systems. In this mode, separate OpenStacktenants are defined in the APIC as different Application Profiles.

There is an alternative option that can be configured at installation time in theml2_conf_cisco_apic.inifile using the settingsingle_tenant_mode = Falsewhich tells the system to create a newACI Tenantfor each OpenStack tenant. This results in a 1:1 correlation between OpenStack tenant and ACI tenant, andgenerates ACI tenants named according to the convention_<apic_system_id>_<openstacktenant name> for eachOpenStack tenant. If multi-tenant mode is used the valueapic_name_mapping= use_uuid should also be set in the ml2_conf_cisco_apic.ini file for proper system function.

OpFlex NAT OperationsThe OpFlex ML2 Driver software brings the capability to support Network Address Translation (NAT)functions in a distributed manner using the local OVS instance on each OpenStack compute node. Thisdistributed approach increases the availability of the overall solution, and offloads the central processing ofNAT from the Neutron server L3-agent used in the reference implementation.

IP Subnets Required for NATThree distinct IP subnets are required to take full advantage of external network functionality with the OpFlexML2 driver. This is a different approach from the default Neutron external network behavior that typicallyuses a single external subnet for these functions.

• Link Subnet—This subnet represents the actual physical connection to the external next-hop routeroutside of the fabric. This will be assigned to a routed interface, sub-interface, or SVI depending on theconfiguration.

• Source-NAT Subnet—The term Source-NAT or SNAT in OpenStack is used to describe allowing VMinstances to initiate connections with destinations outside of the cloud by sharing an address on theexternal network. This subnet is used for Port Address Translation (PAT) allowing multiple VM’s toshare an outside-routable IP address. A single IP address is assigned to each compute host and Layer 4port number manipulation is used to maintain unique session traffic.

Cisco ACI with OpenStack OpFlex Architectural Overview 9

Solution ArchitectureOpFlex NAT Operations

• Floating IP Subnet—The term Floating IP in OpenStack is used when a VM instance is allowed to claima distinct static NAT address to support inbound connections to the VM from outside the cloud. TheFloating IP subnet will be the subnet assigned within OpenStack to the Neutron external network entity.

Traffic egressing the cloud will carry a source IP address of either the SNAT subnet or Floating subnet.The routing hops external to ACI needs to have routes back to these subnets, either through a dynamicrouting protocol or static configuration to allow return traffic to find its way back to OpenStack.

OVS NAT and External RoutingWith the NAT function itself occurring in the local OVS of the compute node, the physical switches in theACI fabric need only to route external traffic to and from the external next-hop router. This external routingis taken care of through a Virtual Routing and Forwarding (VRF) instance associated with the Layer-3 Out.This L3-Out VRF has an interface associated with the physical link to the external next-hop router. The sameVRF also has an interface with an IP address of the assigned Source NAT subnet, and the Floating IP subnet.A loopback interface is also present in this VRF for routing protocol interaction. A graphical representationof the subnet architecture to support this NAT approach is shown in Figure below.

Figure 5: OpFlex Local OVS NAT Subnet Architecture

The L3-Out VRF associated with the OpenStack Neutron external network processes NAT traffic that egressesOVS on the compute host. Non-NAT traffic is processed by a Tenant VRF based on the OpenStackTenant/Project association of the VM Instance.

Optimized DHCP and Metadata Proxy OperationsThe OpFlex ML2 Driver software stack provides optimized traffic flow and distributed processing to provideDHCP andMetadata Proxy services for VM instances. These services are designed to keep as much processing

Cisco ACI with OpenStack OpFlex Architectural Overview10

Solution ArchitectureOVS NAT and External Routing

and packet traffic local to the compute host. The distributed elements communicate with centralized functionsto ensure system consistency.

Optimized DHCP ServicesThe reference OpenStack Neutron architecture utilizes the neutron-dhcp-agent service running on theNeutron server(s) to provide all DHCP communication to VM instances over the OpenStack tenant networks.The neutron-dhcp-agent provides central IP address management, and also communicates with eachVM instance for DHCP Discovery, Offer, Request, and Acknowledgement (DORA) functions.

The OpFlex optimized DHCP approach instead provides all DORA services locally on the compute host viathe agent-ovs service. The distributed services communicate over the Management network to the Neutronserver(s) for allocation of IP addressing and DHCP options. This architecture keeps the bulk of the packettraffic required to issue a DHCP lease local to the compute host itself, while also offloading the processingof this interaction from the Neutron server(s). An illustration of this DHCP architecture is provided in theFigure below.

Figure 6: OpFlex based DHCP Architecture

•

Optimized Metadata ServicesThe reference OpenStack Neutron architecture for Metadata delivery to VM instances provides a centralizedproxy service running on the Neutron server(s). This proxy service looks up instance information in NovaAPI, then adds HTTP headers and redirects the Metadata request to the Nova Metadata service. The Metadatarequests from VM instances are transmitted on the OpenStack Tenant networks.

The OpFlex optimizedMetadata Proxy approach instead providesMetadata delivery using distributedMetadataProxy instances running on each compute host. The agent-ovs service reads the OpFlex service file andprograms a flow in OVS to direct Metadata service requests to the local neutron-metadata-agent.This local agent runs in a separate Linux namespace on the compute host. The Metadata Proxy function thenaccesses Nova-API and Nova Metadata Service running on the OpenStack Controller over the Management

Cisco ACI with OpenStack OpFlex Architectural Overview 11

Solution ArchitectureOptimized DHCP Services

network to deliver VM-specific metadata to each VM instance. An illustration of this Metadata Proxyarchitecture is shown in the Figure below.

Figure 7: OpFlex based Metadata Proxy Architecture

APIC OpenStack VMM IntegrationCisco ACI supports integration withmultiple VirtualMachineManager (VMM) systems, including OpenStack.This integration provides direct APIC visibility to the OpenStack compute nodes, including a detailed listingof all of the VM instances on each node along with the virtual interface information for each learned port. Anexample VM Networking view of OpenStack hypervisors is shown in the Figure below.

Figure 8: APIC VM Networking Hypervisors View

The VM Networking section of the APIC web interface also provides a view by Distributed Virtual Switch(DVS). Each DVS corresponds to an OpenStack network, which may be distributed across multiple computenodes. The listing includes details of where each compute node and ACI leaf each VM instance is connected.

Cisco ACI with OpenStack OpFlex Architectural Overview12

Solution ArchitectureAPIC OpenStack VMM Integration

This listing is instrumented with sort and filtering capabilities to locate any VM by IP or MAC address. Anexample VM Networking view of OpenStack DVS instances is shown in the Figure below.

Figure 9: APIC VM Networking DVS View

Cisco ACI with OpenStack OpFlex Architectural Overview 13

Solution ArchitectureAPIC OpenStack VMM Integration

Cisco ACI with OpenStack OpFlex Architectural Overview14

Solution ArchitectureAPIC OpenStack VMM Integration