vCloud NFV Reference Architecture - VMware vCloud NFV 3 · vCloud Director VMware Integrated...

vCloud NFV ReferenceArchitectureVMware vCloud NFV 3.0

vCloud NFV Reference Architecture

VMware, Inc. 2

You can find the most up-to-date technical documentation on the VMware website at:

https://docs.vmware.com/

If you have comments about this documentation, submit your feedback to

[email protected]

Copyright © 2017.2018 VMware, Inc. All rights reserved. Copyright and trademark information.

VMware, Inc.3401 Hillview Ave.Palo Alto, CA 94304www.vmware.com

https://docs.vmware.com/

mailto:[email protected]

http://pubs.vmware.com/copyright-trademark.html

Contents

1 About vCloud NFV Reference Architecture 5

2 Introduction to vCloud NFV 6

3 Acronyms and Definitions 8

4 Reference Environment 10

Key Customer Objectives 13

5 Architectural Framework and Components 15

Key Stakeholders 15

Conceptual Architecture 16

Logical Architecture and Components 18

vCloud NFV Infrastructure and Orchestration 19

Platform Services 21

Continuity 22

Operations Management 22

vCloud NFV Components 24

Design Principles 25

6 Core Reference Architecture 27

Core Building Blocks 27

Physical Building Blocks 28

Virtual Building Blocks 29

Management Pod 32

Edge Pod 36

Resource Pod 39

7 Deployment Options 44

Three-Pod Configuration 44

Design Considerations 46

Two-Pod Configuration 48


8 Next Generation Data Center Evolution 52

Private Data Center NFV Transformation 52

Scope 52

Design Objectives 54

VMware, Inc. 3

Workload Acceleration 58

Scope 58


Multi-Tenancy 66

Scope 66


Distributed Clouds 69

Scope 69


Workload On-Boarding 72

Scope 73


Availability and Disaster Recovery 76

Availability 76

Disaster Recovery 77

NSX Data Center for vSphere Coexistence with NSX-T Data Center 78

NSX Data Center for vSphere Interoperating with NSX-T Data Center in a Greenfield

Deployment 78

NSX Data Center for vSphere Interoperating with NSX-T Data Center in Existing vCloud NFV

Deployments 80

9 Virtual EPC for Mobile Broadband Internet Access 82


Transformation with vCloud NFV 84

10 Virtual IMS for Mobile Multi-Tenancy Services 87


Transformation with vCloud NFV 90

11 Analytics-Enabled Reference Architecture 94

Management Pod Extensions 96

Components 96

Enabling Analytics with vRealize 99

Monitoring the Infrastructure 101

Predictive Optimization 103

Capacity and Demand Planning 103

Cost Management 105

Closed Loop Automation for Higher Availability 106

12 Authors and Contributors 108


VMware, Inc. 4

About vCloud NFV ReferenceArchitecture 1This reference architecture provides guidance for designing and creating a Network FunctionsVirtualization (NFV) platform by using VMware vCloud® NFV™.

This document describes the high-level design principles and considerations when implementing anenvironment that is based on vCloud NFV. It also provides example scenarios to help you understand theplatform capabilities.

Intended AudienceThis document is intended for telecommunications and solution architects, sales engineers, fieldconsultants, advanced services specialists, and customers who are responsible for the virtualizednetwork functions (VNFs) and the NFV environment on which they run.

VMware, Inc. 5

Introduction to vCloud NFV 2VMware vCloud NFV combines a carrier grade NFV infrastructure with VMware vCloud Director forService Providers as the NFV Virtualized Infrastructure Manager (VIM). This version of the vCloud NFVplatform combines the vCloud Director for Service Providers API with stable and supportable vCloud NFVInfrastructure (NFVI). This way, vCloud NFV provides a platform to support Communication ServiceProviders (CSPs) in realizing the goal for network modernization and business transformation.

The vCloud NFV platform implements a modular design with abstractions that enable multi-vendor, multi-domain, and hybrid physical, and virtual execution environments. The IaaS layer that is exposed throughthis version of vCloud Director for Service Providers, provides a CI/CD environment for workload life cyclemanagement. The platform also delivers an automation framework to interoperate with external functionsfor service orchestration and management.

In addition to the core NFV infrastructure components for compute, storage, networking, and VIM, thevCloud NFV platform includes a fully integrated suite for operational intelligence and monitoring. Thissuite can be used to further enhance the runtime environments with workflows for dynamic workloadoptimization and proactive issue avoidance.

VMware, Inc. 6

Figure 2‑1. vCloud NFV Components

NFVI

NFV Solutions

OSS/BSS

EMS

VNF

NFVO

VNF-M

NFVI Ops

VMware vCloud NFV

vSphere NSXvSAN

Compute Storage Networking VIM Operations, Managementand Analytics

vCloud Director

VMware IntegratedOpenStack

vRealize Operations

vRealize Log Insight

vRealize Network Insight

EMS

VNF

EMS

VNF

The vCloud NFV components, their interactions with each other, and how they meet CSP requirements,are described in this reference architecture.


VMware, Inc. 7

Acronyms and Definitions 3vCloud NFV uses a specific set of acronyms that apply to the NFV technology and the telco industry.

Table 3‑1. General Acronyms

Abbreviation Description

BFD Bidirectional Forwarding Detection, for failure detection on the transport links.

DPDK Data Plane Development Kit, an Intel led packet processing acceleration technology.

MTTR Mean Time to Repair.

MTTU Mean Time to Understand.

Table 3‑2. NFV Acronyms


CCP Centralized Control Plane in the NSX-T Data Center architecture.

LCP Local Control Plane in the NSX-T Data Center architecture.

MANO Management and Orchestration components, a term originating from the ETSI NFV architectureframework.

NFVI Network Functions Virtualization Infrastructure.

NFV-OI NFV Operational Intelligence.

N-VDS (E) Enhanced mode when using the NSX-T Data Center N-VDS logical switch. This mode enables DPDKfor workload acceleration.

N-VDS (S) Standard mode when using the NSX-T Data Center N-VDS logical switch.

VIM Virtualized Infrastructure Manager.

VNF Virtual Network Function, executing in a virtual machine.

Table 3‑3. Telco Acronyms


HSS Home Subscriber Server in the mobile evolved packet core 4G architecture.

MVNO Mobile Virtual Network Operator.

PCRF Policy, Charging and Rating Function, in the mobile evolved packet core 4G architecture.

PGW Packet Gateway in the mobile evolved packet core 4G architecture.

VMware, Inc. 8

Table 3‑3. Telco Acronyms (Continued)


SGW Service Gateway in the mobile evolved packet core 4G architecture.

SBC Session Border Controller used in voice telephone for control and data plane communications betweenclients.


VMware, Inc. 9

Reference Environment 45G services will require a mixture of low-latency, high throughput, and high user densities andconcurrences. The distribution of functional components will require a more sophisticated service deliverymodel.

The network is transforming into a mixture of highly distributed functions together with centralizedfunctions. This way, the network is moving away from the typical centralized models in service delivery.There is also an emerging paradigm shift with employing third-party IaaS, PaaS, and SaaS offerings frompublic cloud providers.

Figure 4‑1. Reference Environment

vRAN cRAN

Multi-Access

uCPE Nano NFV SD-WAN Caching

U-PlanеCaching

RT Analytics

U/C-PlanеCaching

Analytics Apps

C-PlanеShared NF

Analytics AppsMgmt-Plane

SaaSCloud Bursting

Analytics

Edge Regional CorePublic

The highly distributed topology of the network will support the next-generation service characteristics indistribution and composition. It will also require a new way of managing the network and infrastructureresources. The number of services that are spanning the industry verticals is exploding exponentially.Today’s end point ranging in fixed and mobile offers will grow into the billions with IoT connections. Thehighly distributed edge sites are projected to be in the thousands, regional sites in the 100’s, core sites inthe 10’s, and a large variety of public cloud provider sites.

NFV and Software Defined Networking (SDN) transformations will introduce complex interconnectionsbetween end-points such as branches, small offices, connected cars, and IoT gateways to private datacenters and public cloud providers. This reference environment and its transformation are not only atechnical challenge but also impacts the business and operating processes.

VMware, Inc. 10

Reference Environment RequirementsThe reference environment places strict requirements for service placement and management to achieveoptimal performance.

n Federation options. The reference environment topology offers a diverse set of federation options forend-points, private and public clouds, each with distinct ownership and management domains.Virtualized end-points provide better control and manageability, however they are not suitable for alltypes of use cases. Likewise, service functions can be distributed and managed across private andpublic clouds.

n Disaggregated functions. Services are highly disaggregated so that control, data, and managementplanes can be deployed across the distributed topology. Edge clouds offer the performanceadvantages of low latency and data plane intensive workloads. While control and management planecomponents can be centralized with a regional and global scope.

n Functional isolation. Network slicing provides network and service isolation across different tenancymodels in the reference environment. However, resource management considerations need to bemade for shared network functions such as DNS, policy, authentication, and so on.

n Service placement. The highly distributed topology allows for flexibility in the workload placement.Making decisions based on proximity, locality, latency, analytical intelligence, and other EPA criteriaare critical to enable an intent-based placement model.

n Workload life cycle management. Each cloud is elastic with workload mobility and how applicationsare deployed, executed, and scaled. An integrated operations management solution can enable anefficient life cycle management to ensure service delivery and QoS.

n Carrier grade characteristics. Because CSPs deliver services that are often regulated by localgovernments, carrier grade aspects of these services, such as high availability and deterministicperformance are also important.

n NFVI life cycle (patching and upgrades). The platform must be patched and upgraded by usingoptimized change management approaches for zero to minimal downtime.

NFV Reference ModelNFV is an architectural framework that is first developed by the ETSI NFV Industry Specification Group.The framework provides a reference model where network functions are delivered through softwarevirtualization with commercial off-the-shelf (COTS) hardware. This way, NFV moves away from theproprietary, purpose-built hardware that is dedicated to a single service. The result is a network that isagile, resilient, and equipped to deliver high quality of services. The NFV framework defines functionalabstractions and interactions between the building blocks. Some of these abstractions are alreadypresent in current deployments, while others must be added to support the virtualization process andoperation.

The following diagram shows the reference model for an NFV environment with clear functionalabstractions and interactions in a tiered approach.


VMware, Inc. 11

Figure 4‑2. Layered Abstractions of the NFV Environment

OperationsManagement

Tier

PhysicalInfrastructureTier

NFViTier

ResourceOrchestrationTier

CloudAutomationTier

SolutionsTier CPE SD-WAN IMS EPC MEC IoT

Service Orchestration Global SDN Control OSS / BSS

vCloud Director(VIM)

NSX Manager(SDN Control) Policy Blueprints

Resource Pool Resource Pool

VNF VNF VNF VNF VNF

Shared Tenant Slices Tenant Slice 1 … N

vSphere vSAN NSX

vRealize Log Insight

vRealize Network Insight

vRealize Operations

Manager

vRealize Orchestrator

Physical Tier Represents compute hardware, storage, and physical networking as theunderlying pool of shared resources. In addition, there are numerous otherphysical network devices such as switches, routers, EMS, and so on,making the execution ecosystem a hybrid virtual and physical topology.

NFVI Tier The lowest tier of the vCloud NFV platform. It delivers the virtualization run-time environment with network functions and resource isolation for VMworkloads. In NFVI, virtualized compute, storage, and networking aredelivered as an integrated solution through vSphere, vSAN, NSX forvSphere, and NSX-T Data Center. Isolated resources and networks can beassigned to a tenant slice, which is a runtime isolated partition deliveringservices. Tenant slices can be dedicated to a tenant or shared acrosstenants. The NFVI is optimized and adjusted for telco-class workloads toenable the delivery of quality and resilient services. Infrastructure highavailability, performance, and scale considerations are built into this tier forperformance optimization.

Resource OrchestrationTier

It provides resource management capabilities to the NFVI tier. This way, theNFVI can deliver a flexible infrastructure-as-code for life cycle managementof workloads, network management, and resource management. Theresource orchestration tier is responsible for controlling, managing, andmonitoring the NFVI compute, storage, and network hardware, the software


VMware, Inc. 12

for the virtualization layer, and the virtualized resources. The VIM modulemanages the allocation and release of virtual resources, and theassociation of virtual to physical resources, including resource optimization.VIM also maintains the inventory of NFVI, including the linkage andrelationship between components as they relate to an instance of a VNF orCNF workload. This way, VIM allows for monitoring in the context of asingle VNF.

Cloud Automation Tier The service management and control functions which bridge the virtualresource orchestration and physical functions to deliver services andservice chains. It is typically a centralized control and managementfunction, including embedded automation and optimization capabilities.

Solutions Tier The multi-domain ecosystem of software virtual functions as native VMfunctions. Such functions are composed in complex solutions to enableservice offers and business models that CSP customers consume.Solutions can range from small branch office functions to a fully evolvedpacket core that is delivered as tenant slices across multiple clouds.

OperationsManagement Tier

An integrated operational intelligence for infrastructure day 0, 1, and 2operations that spans across all other tiers. The functional componentswithin the operations management tier provide topology discovery, healthmonitoring, alerting, issue isolation, and closed-loop automation.

Key Customer ObjectivesThe goal of network modernization is to drive greater classes of service innovation and timelyenablement. Following are key objectives that CSPs are considering as they transform their networks anddesign for new business and operational models.

Fixed MobileConvergence

As networks evolved through 2G and 3G generations, the voice and datanetwork architectures were separated, particularly circuit-switched, andpacket-switched networks. As networks evolved, the CSPs went towardsan all IP network, therefore the convergence of Fixed and Mobile


VMware, Inc. 13

networking. The environments for voice mostly share the same corenetworking components with different access networks. The scale,performance, and management of such converged networks are morecritical than before.

Data IntensiveWorkload Acceleration

The demand for throughput has increased exponentially with smart devicesand immersive media services. The networking and compute expenditurescontinue to grow to meet such demands in traffic throughput. Accelerationtechnologies like DPDK, VPP, hardware offload, for example, are at theforefront to reduce OpEx for data intensive applications.

Distributed Clouds To meet the increased bandwidth and low-latency requirements, networkdesigns are expanding the centralized compute models to distributed edgecomputing models. Certain level of distribution already exists in regionaland core data centers, however further edge distribution will be necessaryto control traffic backhauling and to improve latencies. In conjunction, VNFsare disaggregating to distribute data plane functions at the edges of thenetwork whereas control functions are centralized. Service distribution andelasticity will be vital part of the network design consideration.

Network Slicing Network slicing is a way for cloud infrastructure to isolate resources andnetworking to control the performance and security for workloads that areexecuting on the same shared pool of physical infrastructure. Withdistributed topologies, the concept of network slicing furthermore stretchesacross multiple cloud infrastructures, including access, edge, and corevirtual and physical infrastructures. Multi-tenancy leverages such resourceisolation to deploy and optimize VNFs to meet customer SLAs.

Dynamic OperationalIntelligence

The cloud infrastructures will have to become adaptive to meet the needsof workloads. Right-sizing the environment and dynamic workloadoptimizations, including initial placement, will be part of the continuousautomation orchestration. The cloud infrastructure environments will requireintegrated operational intelligence to continuously monitor, report, andaction in a timely manner with prescriptive and predictive analytics.

Policy-BasedConsistency andManagement

Model driven approaches will play a key role in the modern cloudinfrastructures. Resource modeling, runtime operational policies, or securityprofiles, declarative policies and movement of policies with workloads,onboarding, and so on, will ensure consistency and ease of management.

Carrier Grade Platform The cloud infrastructure environment will have to meet the strictrequirements for availability, fault tolerance, scale, and performance.Security will be necessary across the transport, data, and workloaddimensions. The mobility of workloads across distributed clouds introducesa new challenge for its authenticity and integrity.


VMware, Inc. 14

Architectural Framework andComponents 5This section explores the overall framework for the vCloud NFV platform architecture, including the keystakeholders, conceptual architecture environment, logical architecture, and components of the vCloudNFV platform. The reference architecture design principles set the frame for the core and analytics-enabled designs of the vCloud NFV platform.

This chapter includes the following topics:n Key Stakeholders

n Conceptual Architecture

n Logical Architecture and Components

n vCloud NFV Components

n Design Principles

Key StakeholdersThe reference architecture considers key stakeholders that are involved in the end-to-end servicemanagement, life cycle management, and operations.

Cloud provider The CSP operations personnel who are responsible for provisioning andon-boarding all day 0 and day 1 functions to enable services for targetcustomers and tenants.

Consumer Consumer. The end user who is consuming the services that the tenantsprovide. For example, IoT devices, mobile handsets, API consumers, andMVNO.

Customer The enterprise or entity who owns the business relationship with the CSP.The customer might be an internal line of business such as fixed line andmobile services and can also be an external enterprise.

VMware, Inc. 15

Tenant The various classes of services (offers) that a customer provides to theirconsumers. Each tenant is represented as a resource slice, hence acustomer can have one or more tenants. A mobile line of business can offerslices to a MVNO customer, for example a tenant for voice services and atenant for data services.

Operations Support The operations management process and team that ensure the servicesare operating to meet the promised stringent SLAs.

Network Planning The operations management planning function that is responsible for theresource and VNF capacity and forecasting, and new data center designs.

Security Operations Security operations function that is responsible for all aspects of security,network, application, and data.

Conceptual ArchitectureCloud infrastructure-based computing is the next-generation standard in modernizing the CSP networksas they evolve to 5G architectures, services, and agile delivery. The shared infrastructure with completesoftwarization of network functions and applications provide greater advantages in cost, performance,and agility.

The modernization of the CSP infrastructure requires a complex ecosystem of solutions and functionsdelivering to a pre-set business and operating model. The cloud infrastructure modernization changes notonly the business model in service agility and metered revenue models, but also challenges the silooperating model. The following figure shows the conceptual view of the various domains, capabilities, andtheir interactions that need consideration in the modernization of networks and business and operationalmodels.

Figure 5‑1. Conceptual Architecture

Cloud Infrastructure Domain

Physical Infrastructure Domain

Cloud Automation Domain

Compute Storage Network

Hardware and Networking

Data PlaneAcceleration

WorkloadPlacement

WorkloadManagement

Cross-CloudInterconnect

Cloud Operations Domain

InfrastructureAssurance

DemandManagement

Security andCompliance

Closed-LoopAutomation

DisasterRecovery

Availability andContinuity

Cloud Management Domain

WorkloadOnboarding

ResourceOrchestration

DynamicOptimization

InventoryManagement

Policy-BasedControl

Billing andUsage

Demand and CostPlanning

Tenant Self-Service

Data Protection

IncidentManagement

Service Orchestration Global SDN Control Operational / Business Support Workflows

Cloud Platform Enablement Domain

Analytics Security


VMware, Inc. 16

Cloud Automation DomainCentralizes the overall service management functions such as service definitions, composition,onboarding, life cycle management, and support. This domain hosts functions such as an NFV-O that isresponsible for service blueprinting, chaining, and orchestration across multiple cloud infrastructureenvironments. Next to NFV-O are the global SDN control functions that are responsible for stitching andmanaging physical and overlay networks for cross-site services. Real-time performance monitoring canbe integrated into the SDN functions to dynamically optimize network configurations, routes, capacity, andso on.

The successful cloud automation strategy implies full programmability across other domains.

Cloud Platform Enablement DomainExtends a set of platform capabilities from the cloud infrastructure that the automation domain, VNFs,their managers, and other core components can leverage. Example enablement capabilities include:

n Analytics to ingest VNF metrics that can be correlated with infrastructure metrics for smarter contextand insights.

n Workload placement to determine the right location for a workload depending on available resources,class of resources, and feature capabilities such as data intensive acceleration.

n Workload acceleration using DPDK and SR-IOV for data intensive VNFs.

n Security for network, data, and workloads.

Cloud Management DomainPlays a critical role across many different dimensions. More fundamentally, it provides a templated andprescriptive workload management set of capabilities that the automation layer can use to program andorchestrate service on-demand and with agility. Service onboarding models can be turned into fully zero-touch provisioning and exposed to tenants through a self-service portal. Business models such asmetered billing can be enabled as a catalog of services and tariffs.

Once services and workloads are onboarded, the management domain also needs to ensure dynamicoptimization such as workload rebalancing or capacity growth or shrink to maintain agreed SLAs. Suchoptimizations need to integrate with the cloud operations domain for real-time usage and performanceintelligence. Polices, including platform awareness, NUMA affinity, host affinity, restart-sequences, arenecessary for efficient optimization.

Cloud Operations DomainEnsures that the operational policies and SLAs are being met by continuous data collection, correlation,and analytics. Infrastructure assurance is a key component of this domain. Intelligence can be tied into aclosed-loop workflow that can be integrated in the automation domain for proactive issue avoidance, forexample, triggering a trouble ticket incident management system.


VMware, Inc. 17

In addition, other functions for day 2 operations such demand and capacity planning, security andcompliance, high availability, and disaster recovery are necessary to ensure availability and integrityacross the cloud infrastructure environments.

Cloud Infrastructure DomainThe core virtualization domain providing resource abstraction for compute, storage, and networking andtheir orchestration through a VIM to allocate, control, and isolate with full multi-tenancy and platform-awareness.

Logical Architecture and ComponentsThe vCloud NFV platform implements the conceptual architecture that is outlined and defined at a highlevel through the logical building blocks and core components.

The VMware vCloud NFV platform is an evolution of the VMware NFV solution, based on extensivecustomer deployment and the continued development of standards organizations such as the EuropeanTelecommunications Standards Institute (ETSI). The vCloud NFV platform provides a comprehensive,service-oriented solution, leveraging a cloud computing model that allows ubiquitous, programmatic, on-demand access to a shared pool of compute, network, and storage resources. The solution is integratedwith holistic operations management and service assurance capabilities, empowering the CSP to rapidlydeliver services while ensuring their quality. With a fully integrated VIM, the same vCloud NFVinfrastructure delivers a myriad of telecommunications use cases and facilitates reusability of the servicecatalog based VNFs.

The following diagram maps the conceptual architecture to a logical view for the vCloud NFV platform.

Figure 5‑2. Logical Architecture

Replication Network Edge Services Analytics API/SDK Policy

(PDP/PEP)

NFVInfrastructure

EMS

Compute Storage Network

Recovery

ContinuityPlatform Services

Recovery Automation Dashboard Analytics Administration

Host Management

Storage Management

Network Management

Resource Orchestration

Workload Inventory

vCloud NFVi and Orchestration

Topology & Discovery

Monitoring

Issue Isolation

Remediation

Closed- Loop

Operations

PNF


VMware, Inc. 18

https://www.etsi.org/

https://www.etsi.org/

The vCloud NFV platform delivers a complete integrated solution that has been rigorously tested toensure compatibility, robustness, and functionality. The components that build the solution are currentlydeployed across many industries and scenarios. The vCloud NFV software components can be used invarious ways to construct a comprehensive, end-to-end solution that meets the business goals of CSPs.This document discusses one way in which components can be used to create a vCloud NFVarchitecture.

Logical Architecture ComponentsThe vCloud NFV platform consists of the three core domains of functions, core NFV infrastructure,infrastructure orchestration, and operations management. At the core infrastructure, ESXi is used tovirtualize the compute resources, NSX for vSphere, NSX-T Data Center to provide virtual networking, andvSAN for storage. The core NFV infrastructure virtualization layer provides the following functions:

n Physical Resource Abstraction. By using the software component layers between the physicalhardware and the VNFs, physical resources are abstracted. This provides a standardized software-based platform for running workloads, regardless of the underlying hardware. As long as the CSPuses certified physical components, workloads can be deployed by the carrier at the point ofpresence (POP), distributed, or centralized data center.

n Physical Resource Pooling. Physical resource pooling occurs when vCloud NFV presents a logicalvirtualization layer to workloads, combining the physical resources into one or more resource pools.Resource pooling together with an intelligent scheduler facilitates optimal resource utilization, loaddistribution, high availability, and scalability. This allows for fine grained resource allocation andcontrol of pooled resources based on specific workload requirements.

n Physical Resource Sharing. To truly benefit from cloud economies, the resources that are pooled andabstracted by the virtualization layer must be shared between various network functions. Thevirtualization layer provides the functionality that is required for VNFs to be scheduled on the samecompute resources, collocated on shared storage, and to have network capacity divided among them.The virtualization layer also ensures fairness in resource utilization and usage policy enforcement.

vCloud NFV Infrastructure and OrchestrationThe infrastructure and orchestration domain contain the NFVI abstractions for compute, storage, andnetworking. It also contains the Virtualized Infrastructure Manager (VIM), which is the resourceorchestration component of the NFV platform.

Compute - VMware ESXiESXi is the hypervisor software that abstracts the physical x86 server resources from the VNFs. Eachcompute server is called a host in the virtual environment. ESXi hosts are the fundamental computebuilding blocks of vCloud NFV. ESXi host resources can be grouped to provide an aggregate set ofresources in the virtual environment that is called a cluster. Clusters logically separate the managementand VNF components and are discussed in details in the Core Reference Architecture section. ESXihosts are managed by the VMware vCenter Server Appliance that is part of the VIM components.


VMware, Inc. 19

Key new features introduced in ESXi:

n Single reboot upgrade. vSphere upgrades can now be completed with one single reboot.

n ESXi quick boot. Allows a system to reboot in less than two minutes as it does not reinitialize thephysical server.

n Instant Clone. Enables a user to create powered-on VMs from the running state of another powered-on VM without losing its state.

n NVDIMM devices. Support for next generation of storage devices that use persistent DRAM memory.

n Enhanced vMotion Capability. Define minimum CPU preferred features per VM to ensure CPU awaremigration.

n Hugepages. The Size of Hugepages has now been extended to 1GB, improving memory accessperformance due to lower TLB misses.

n Enhancements for NSX-T Data Center. Together with NSX-T Data Center, vSphere introduces a newN-VDS Enhanced mode for switching to deliver substantial switching performance.

n RDMA over Converged Ethernet. RDMA provides low latency and higher-throughput interconnectswith CPU offloads between the end-points.

n Higher security enhancements for TLS 1.2 and FIPS 140-2 cryptography.

n Increases performance and availability.

Host Management - VMware vCenter ServerVMware vCenter Server® is the centralized management interface for compute and storage resources inthe NFVI. It provides an inventory of allocated virtual to physical resources, manages inventory-relatedinformation, and maintains an overview of the virtual resource catalogs. vCenter Server collects dataabout the performance, capacity, and state of its inventory objects. It exposes APIs to other managementcomponents for fine-grained control, operation, and monitoring of the underlying virtual infrastructure.

Networking - VMware NSX-T Data CenterNSX-T Data Center is the software defined networking component for the vCloud NFV referencearchitecture. It allows CSPs to programmatically create, delete, and manage software-based virtualnetworks. NSX-T Data Center also leverages technologies for high performance workloads such asDPDK. These networks serve the communication between VNF Components, and provide customers withdynamic control over their service environments. Dynamic control is enabled through tight integrationbetween the resource orchestration layer and NSX-T Data Center.

Network multitenancy is implemented with NSX-T Data Center by assigning tenants their own virtualnetworking components and providing different network segments. A two-tiered architecture is used in theNSX-T Data Center design to implement a provider and tenant separation of control across the logicalswitching and routing fabric. Logical switching is supported in two modes, N-VDS Standard and N-VDSEnhanced, both of which support overlay and VLAN backed networks. The fully distributed routingarchitecture enables routing functionality closest to the source. This structure gives both provider andtenant administrators complete control over their services and policies.


VMware, Inc. 20

NSX-T Data Center also implements a separation of management, control, and data planes. The NSXManager, Controller, and Edge are components of this architecture that are discussed in the sections tofollow.

Storage - VMware vSANvSAN is the native vSphere storage component in the NFVI virtualization layer, providing a sharedstorage pool between hosts in the vSphere cluster. With vSAN, storage is shared by aggregating the localdisks and flash drives that are attached to the host. Other third-party storage solutions with storagereplication adapters that meet the VMware storage compatibility guidelines are also supported.

Resource Orchestration - VMware vCloud Director for Service ProvidersVMware vCloud Director for Service Providers is the VIM component that vCloud NFV exposes as theinterface for VNF life cycle management. It uses vCenter Server and NSX Manager to orchestratecompute, storage, network, and imaging infrastructure services from a single, programmable interface.

Platform ServicesThe platform services domain represents the capabilities that vCloud Director for Service Providersdelivers to the NFV platform. VNFs, VNF managers, and other components running within the vCloudNFV platform can use these capabilities.

Edge Services - VMware NSX-T Data CenterNSX-T Data Center provides two classes of routing capabilities, Distributed Router (DR) and ServiceRouter (SR). The Service Router capability provides services such as NAT, firewall, load balancer, and soon. It also provides a Tier-0 and Tier-1 router that VNFs can employ with stateful and stateless options.The Edge services cannot be distributed and require a centralized pool of capacity with high availabilityand scalability. The appliances that host the centralized services or SR instances are called Edge Nodes.These nodes also provide connectivity to the physical infrastructure.

Analytics – VMware vRealize Operations ManagerThe vCloud NFV platform is fully integrated with an operations management solution for day 1 and day 2operations for health monitoring, issue avoidance, and closed-loop automation. This way, the platformprovides infrastructure assurance out of the box. The analytics framework can be used by networkfunction and application developers to ingest and correlate their services-specific data in vRealizeOperations Manager and leverage its capabilities seamlessly. The framework provides variousmechanisms through management and content packs for data management and closed-loop automationwith workflows that are custom to their operations and planning needs.

Policy ConsistencyThe vCloud NFV platform components utilize policy-based frameworks that can be defined once andapplied at runtime to maintain the desired end state. The decision and enforcement split makes themanagement and operation of the platform and its services highly flexible, which operations andapplications can leverage. Policies in compute can be used for right-sizing the infrastructure to ensure


VMware, Inc. 21

capacity and performance. Workload placement and runtime optimization for DRS and vMotion can beprescribed with platform awareness. Policies in networking range in physical networking such as teaming,network management and control, and security for East-West and perimeter traffic control. Storagepolicies provide services such as availability levels, capacity consumption, and stripe widths forperformance. Policies in operations can help to ensure that SLAs are met with configurable alerting,recommendation, and remediation framework.

ProgrammabilityThe VMware vCloud NFV solution is a fully open platform supporting flexible APIs and SDKs. For moreinformation on the VMware APIs, see the VMware API Explorer.

ContinuityThis domain represents components for business continuity and disaster recovery solutions, which are anintegral part of the vCloud NFV platform.

VMware Site Recovery ManagerSite Recovery Manager works with various storage replication solutions, including vSphere Replication, toautomate the migration, recovery, testing, and failing back virtual machine workloads for disaster recoveryacross multiple sites.

VMware vSphere ReplicationvSphere Replication is a virtual machine data protection and disaster recovery solution. It is fullyintegrated with vCenter Server and VMware vSphere® Web Client, providing host-based, asynchronousreplication of virtual machines including their storage.

Operations ManagementThis domain represents the day 1 and day 2 functions to ensure that the infrastructure and servicecomponents are operating in a healthy state so that SLAs are met.

The operations management solution includes four components that together provide a holistic approachto the operations management for the NFVI of a CSP. Together vRealize Operations, vRealize LogInsight, and vRealize Network Insight monitor the health of the virtual environment, collect logs andalarms, correlate events across multiple data sources and components to predict future issues. Thesecomponents and vRealize Orchestrator use policy-based automation framework to conduct remediationand analyze data to help the operators with health prediction and issue avoidance.

The key tasks that the operations management components perform are the following:

n Topology and discovery. NFVI visibility is achieved by collecting essential performance and faultmetrics from the virtualization layer, the physical devices, and the VIM components. The elastic anddynamic nature of the NFVI layer requires keeping track of objects and maintaining correlations tohelp with decision tree accuracy and intelligence for infrastructure assurance.


VMware, Inc. 22

https://code.vmware.com/apis

n Monitoring. The components of the Operations domain continuously collect and analyze health, SLA,and planning metrics to ensure that services are meeting stringent QoS and to help avoiding issues ina predictable and prescriptive cadence.

n Issue isolation. The components of the NFV environment in the physical infrastructure, thevirtualization layer, or even the VNFs themselves, generate various log messages and alarms. vCloudNFV includes an integrated log collection system that correlates between alerts and log messages toquickly troubleshoot issues.

n Remediation. Ongoing management of performance and capacity across the NFVI is required foroptimal and economic use of the platform. The performance management capability helps identifydegraded performance before VNFs are affected. Issues pertaining to performance, capacity,congestion, and so on, can be proactively avoided, increasing the Mean Time To Failure (MTTF).

n Closed-loop optimization. The operations management components analyze the system usage andproactively provide optimization recommendations, including network topology modifications. Actionscan be tied into orchestrated workflows and automation to trigger VNFM, service orchestrators, andothers to ensure continuous optimization, balancing, and recover in the event of failures.

VMware vRealize Operations ManagerVMware vRealize® Operations Manager™ delivers operations management with full stack visibility acrossthe physical and virtual infrastructure. Through performance and health monitoring functions, vRealizeOperations Manager improves the system performance, avoids service disruption, and helps the CSP toprovide proactive management of the NFVI. The key capabilities include predictive analytics, smart andconfigurable alerts, and guided remediation.

vRealize Operations Manager exposes the information it gathers through an API that MANO and othercomponents can use.

With this release, vRealize Operations Manager enables intent-based business and operational policiesfor dynamic optimization and capacity targets. Cost analysis is embedded into the solution, as withimproved capacity planning and forecasting engines.

VMware vRealize Log InsightVMware vRealize Log Insight delivers heterogeneous log management with dashboards, analytics, andthird-party extensibility. It provides deep operational visibility and troubleshooting across physical, virtual,and cloud environments. Its indexing and machine learning based grouping provides log searches thathelp with troubleshooting issues.


VMware, Inc. 23

VMware vRealize Network InsightvRealize Network Insight collects metrics, flow, network topology, and event data to provide a detailedview of the network configuration and its health. Information is collected on all NSX-T managed networksincluding East-West traffic between VNF components, and North-South traffic in and out of the NFVinfrastructure. Broad layer 2 to layer 3 support means that vRealize Network Insight can visualize both theunderlay and the overlay networks, providing operators with a holistic view into all relevant networklayers. By using this information for visibility and analytics across all virtual and physical elements, theoperator can optimize network performance and increase its availability.

On the security aspects, vRealize Network Insight offers intelligence operations for SDN and securityacross virtual and physical infrastructure by using micro-segmentation planning and policy distribution intoNSX-T. The solution can be scaled, providing early warning on security policy violations.

VMware vRealize OrchestratorThe vCloud NFV platform provides closed-loop automation workflows to enable self-healing across VMs,hosts, and datastores at infrastructure level. It also allows the CSP to create VNF-specific customworkflows for faster time to resolution. vRealize Operations Manager integrates with vRealizeOrchestrator through a management pack that provides access to the Orchestrator workflow engine formore remediation actions and the ability to run Orchestrator workflows directly from the vRealizeOperations Manager user interface.

vCloud NFV ComponentsThe vCloud NFV bundle packages together the essential building blocks to deploy an NFVI and VIMplatform, featuring the newest releases of VMware solutions that are proven in production.

The reference architecture is separated into two sections comprising of the Core Reference Architectureand Analytics Enabled Reference Architecture. The Core Reference Architecture is the base buildingblock for the vCloud NFV reference architecture and the Analytics Enabled Reference Architecture buildson top of it.

Table 5‑1. vCloud NFV Components

ComponentIncluded in the vCloud NFVBundle

Required for the CoreReference Architecture

Required for the Analytics-Enabled ReferenceArchitecture

VMware ESXi™ Yes Yes Yes

VMware vCenter® ServerAppliance™

No Yes Yes

VMware vSphere®

Replication™Yes Recommended Recommended

VMware vSAN™ StandardEdition

Yes Recommended Recommended

VMware® vCloud Director forService Providers

Yes Yes Yes


VMware, Inc. 24

Table 5‑1. vCloud NFV Components (Continued)

ComponentIncluded in the vCloud NFVBundle

Required for the CoreReference Architecture

Required for the Analytics-Enabled ReferenceArchitecture

VMware NSX-T® Data Center No Yes Yes

VMware NSX® Data Centerfor vSphere

No Recommended Recommended

VMware Site RecoveryManager™

No Recommended Recommended

VMware vRealize®

Operations™Yes Yes

VMware vRealize® LogInsight™

Yes Yes

VMware vRealize®

OrchestratorYes Yes

VMware vRealize® NetworkInsight™

No Recommended

Design PrinciplesThe following are the design principles behind the core and analytics reference architecture.

Flexible Deployment OptionsvCloud NFV can be deployed to support either a Three-Pod (Management, Resource, and Edge Pods) ora Two-Pod (Management and combined Resource and Edge Pods) configuration. The Three-Podarchitecture provides the highest flexibility and performance, because Edge Nodes are dedicated toperform packet forwarding in and out of the virtual domain. CSPs can use the Two-Pod architecture forsmaller starting deployments, where it is acceptable to combine the resource and Edge functionality in thesame hosts.

Advanced NetworkingTo provide multitenant capabilities to distributed logical routers in the networking stack, vCloud NFV usesNSX-T Data Center to deploy multiple tiers of distributed routing through Tier-0 and Tier-1 routers.Providers can use Tier-0 routers, whereas tenants can use Tier-1 routers. Network virtualizationcapabilities that are enabled through Geneve encapsulation provide a flexible capability in line withindustry standards. In addition, NSX-T Data Center performance enhancements for the N-VDS and NSXEdge Nodes offer advanced network capabilities.


VMware, Inc. 25

Workload AccelerationvCloud NFV includes several features to support workloads that require high performance. Thesefeatures are delivered with N-VDS with Enhanced Data Path Mode that can be used for high-performanceworkloads. In addition, North-South traffic forwarding between logical and physical domains can benefitfrom bare metal NSX Edge Nodes. This high-performance capability is available through enhancementsthat come with NSX-T Data Center such as optimizations through poll mode drivers, CPU affinity andoptimization, and buffer management.

Integrated Operational IntelligenceBy using a framework that continuously collects data from local and distributed agents, vCloud NFV alsoprovides the capability to correlate, analyze, and enable day 2 operations. In addition, this analyticalintelligence can be used with existing assurance engines for closed-loop remediation.


VMware, Inc. 26

Core Reference Architecture 6The VMware vCloud NFV 3.0 platform is an evolution of the VMware NFV solution, based on extensivecustomer deployment and the continued development of standards organizations such as the EuropeanTelecommunications Standards Institute (ETSI). The vCloud NFV platform provides a comprehensive,service-oriented solution, leveraging a cloud computing model that allows ubiquitous, programmatic, on-demand access to a shared pool of compute, network, and storage resources. The solution is integratedwith holistic operations management and service assurance capabilities, empowering the operator torapidly deliver services while ensuring their quality. With a fully integrated VIM, the same vCloud NFVinfrastructure delivers a myriad of telecommunications use cases and facilitates reusability of the servicecatalog based VNFs.

The vCloud NFV platform delivers a complete, integrated solution that has been rigorously tested toensure compatibility, robustness, and functionality. Components used in creating the solution are currentlydeployed across many industries and scenarios. vCloud NFV software components can be used in avariety of ways to construct a comprehensive, end-to-end solution that meets the business goals ofCSPs. This document discusses how components can be used to create a vCloud NFV architecture.

This chapter includes the following topics:n Core Building Blocks

n Physical Building Blocks

n Virtual Building Blocks

Core Building BlocksArchitecting vCloud NFV by using well-defined modules allows the CSP to accelerate the deployment ofthe platform and reliably expand it when needed. The platform components are grouped into three distinctcontainments. The VMware vCloud NFV platform uses the term Pods as a mean to streamline the NFVenvironment operations and delineate between different roles. For example, a cloud management teamcan easily operate the Management Pod, whereas a network management team is likely to oversee theEdge Pod. VNFs are always deployed in the Resource Pod.

VMware, Inc. 27

Each Pod is identified by its functional designation - Management Pod, Edge Pod, and Resource Pod.The Pod functions are the following:

Management Pod Management functions are required to manage the NFV Infrastructure andthe VNFs and their components. Management, orchestration, analyticsfunctionality, and ancillary elements such as DNS, VNF-M, and NFV-O aregrouped into this category. Resource orchestration such as vCenter ServerAppliance, NSX Manager, NSX Controller, and vCloud Director for ServiceProviders are hosted in this Pod. The analytics components, which includevRealize Operations Manager, vRealize Network Insight, vRealize LogInsight, vRealize Orchestrator, and business continuity components suchas Site Recovery Manager and vSphere Replication are all located in thispod. Other management-related components such as NFV Orchestratorsrun in the Management Pod. OSS/BSS can be very large in sizing, andtherefore their placement depends on the system itself.

Edge Pod The Edge functions provide a logical networking delineation between VNFsand external networks. Network traffic transitioning between the physicaldomain and the virtual domain is processed by these functions. NSX Edgeis hosted in a VM or bare metal appliance in the Edge Pod and handles allconnectivity to the physical domain in the architecture. The type ofnetworking traffic that traverses the Edge Pod is called North-South traffic.

Resource Pod VNFs are placed in the Resource Pod. The VNFs then form the virtualnetwork service.

Physical Building BlocksTraditional data center network fabrics are often designed with three tiers of switches that are core,aggregation, and access. Access switches connect to aggregation switches, which in turn connect to thecore switches. The design topology of the physical layer can impact the efficiency and latencies.


VMware, Inc. 28

Figure 6‑1. Physical Network Design

Pod 2

Host

Host

Host

Pod 1

Host

Host

Host

Leaf LeafLeaf

TelcoNetwork

Spine

Layer 3

Layer 2

Layer 3 Fabric

vCloud NFV

Spine

Leaf Leaf Leaf

Pod 3

Host

Host

Host

Communication between two endpoints within the data center begins from the access switch, traversesthe aggregation switch to the core switch, and travels from there to the remote endpoint. This trafficpattern results in inefficiencies and increased latency.

A two-tier leaf-and-spine network architecture is the preferred approach for building newer data centerinfrastructure. The two-tier architecture uses an access switch, or leaf, which is connected to anaggregation switch, or spine. The leaf switch provides connectivity between endpoints in the data center,while the spine switch provides high-speed interconnectivity between leaf switches. The leaf-and-spinenetwork is connected in a full mesh, providing predictable communication and latency between endpoints.Ethernet connectivity is used from the host to the leaf switch, and the broadcast domain terminates at theleaf. External Border Gateway Protocol (eBGP) is often used for routing within the leaf-and-spinearchitecture.

Virtual Building BlocksThe virtual infrastructure design comprises the design of the software components that form the virtualinfrastructure layer. This layer supports running telco workloads and workloads that maintain the businesscontinuity of services. The virtual infrastructure components include the virtualization platform hypervisor,virtualization management, storage virtualization, network virtualization, and backup and disasterrecovery components.

This section outlines the building blocks for the virtual infrastructure, their components, and thenetworking to tie all the components together.


VMware, Inc. 29

Figure 6‑2. Virtual Building Blocks

Operations (data)

Operations (data)

Operations (data)

Edge Pod

ManagementPod

Management Network

Operations(master)

Network Insight(master)

Log Insight(master)

vCenter Sever(active)

Orchestrator(master)

NSX Controller

vCenter Sever(active)

vCloudDirector

NSX Manager

NSX Edge(1)

NSX Edge(n)

External NetworkVNF Network

Resource Pod VNF VNF

InfrastructureNetwork

Storage DesignThis reference architecture uses a shared storage design that is based on vSAN. vCloud NFV alsosupports certified third-party shared storage solutions, as listed in the VMware Compatibility Guide.

vSAN is a software feature built in the ESXi hypervisor that allows locally attached storage to be pooledand presented as a shared storage pool for all hosts in a vSphere cluster. This simplifies the storageconfiguration with a single datastore per cluster for management and VNF workloads. With vSAN, VMdata is stored as objects and components. One object consists of multiple components, which aredistributed across the vSAN cluster based on the policy that is assigned to the object. The policy for theobject ensures a highly available storage backend for the cluster workload, with no single point of failure.

vSAN is a fully integrated hyperconverged storage software. Creating a cluster of server hard disk drives(HDDs) and solid-state drives (SSDs), vSAN presents a flash-optimized, highly resilient, shared storagedatastore to ESXi hosts and virtual machines. This allows for the control of capacity, performance, andavailability through storage policies, on a per VM basis.

Network DesignThe vCloud NFV platform consists of infrastructure networks and VM networks. Infrastructure networksare host level networks that connect hypervisors to physical networks. Each ESXi host has multiple portgroups configured for each infrastructure network.


VMware, Inc. 30

https://www.vmware.com/resources/compatibility/search.php

The hosts in each Pod are configured with VMware vSphere® Distributed Switch™ (VDS) devices thatprovide consistent network configuration across multiple hosts. One vSphere Distributed Switch is usedfor VM networks and the other one maintains the infrastructure networks. Also, the N-VDS switch is usedas the transport for telco workload traffic.

Figure 6‑3. Virtual Network Design

ESXi Host

Resource Pod.

N-VDS (S)

ESXi Host

Edge Pod

VDS (Virtual Machine)

N-VDS (E)

ESXi Host

Management Pod

Teaming Teaming

Teaming

Teaming

VDS (Infrastructure)

Teaming Teaming Teaming

Teaming

Infrastructure networks are used by the ESXi hypervisor for vMotion, VMware vSphere Replication, vSANtraffic, and management and backup. The Virtual Machine networks are used by VMs to communicatewith each other. For each Pod, the separation between infrastructure and VM networks ensures securityand provides network resources where needed. This separation is implemented by two vSphereDistributed Switches, one for infrastructure networks and another one for VM networks. Each distributedswitch has separate uplink connectivity to the physical data center network, completely separating itstraffic from other network traffic. The uplinks are mapped to a pair of physical NICs on each ESXi host, foroptimal performance and resiliency.

VMs can be connected to each other over a VLAN or over Geneve-based overlay tunnels. Both networksare designed according to the requirements of the workloads that are hosted by a specific Pod. Theinfrastructure vSphere Distributed Switch and networks remain the same regardless of the Pod function.However, the VM networks depend on the networks that the specific Pod requires. The VM networks arecreated by NSX-T Data Center to provide enhanced networking services and performance to the Pod


VMware, Inc. 31

workloads. The ESXi host's physical NICs are used as uplinks to connect the distributed switches to thephysical network switches. All ESXi physical NICs connect to layer 2 or layer 3 managed switches on thephysical network. It is common to use two switches for connecting to the host physical NICs forredundancy purposes.

Following are the infrastructure networks used in the Pods:

n ESXi Management Network. The network for the ESXi host management traffic.

n vMotion Network. The network for VMware vSphere® vMotion® traffic.

n vSAN Network. The network for vSAN shared storage traffic.

n Backup Network. The network that is dedicated to offline storage such as NFS and used for workloadbackup and restore as needed.

n Replication Network. This is the network that is used for replicating data for data protection.

Management PodThis section describes the design for the Virtualized Infrastructure Management (VIM) components thatare vCenter Server Appliance, NSX Manager, and vCloud Director for Service Providers.

Figure 6‑4. Management Pod

VMware vCloud Director

vCenter Server +Platform Services

ControllerNSX Manager

Management Pod

NSX ControllervCenter Server +Platform Services

Controller

Management Network

In addition to these core components, the Management Pod also contains the operations managementcomponents. For more information , see the Analytics-Based Reference Architecture section.

ComponentsThe Management Pod contains the components that manage the vCloud NFV runtime environment.

vCenter Server

The Management Pod is implemented as a cluster that is managed by the first vCenter Server instance.To form the foundation of a carrier grade virtualized infrastructure, the components of the ManagementPod benefit from the cluster features such as resource management, high availability, and resiliency. Asecond vCenter Server is deployed in the Management Pod to oversee the Edge and Resource Pods.


VMware, Inc. 32

Each vCenter Server instance is a virtual appliance that is deployed with an embedded database and anembedded Platform Services Controller. The vCenter® Server Appliance™ is preconfigured, hardened,and fast to deploy. The appliance allows for a simplified design, eases management, and reducesadministrative efforts. vCenter Server Appliance availability is ensured by using a vCenter HighAvailability (vCenter HA) cluster. The vCenter HA cluster consists of one active node that serves clientrequests, one passive node as a backup in the event of failure, and one quorum node that is called awitness node. Replication between nodes by using a dedicated vCenter HA network ensures that vCenterServer Appliance data is always synchronized and up-to-date.

The Platform Services Controller contains common infrastructure security services such as VMwarevCenter® Single Sign-On, VMware Certificate Authority, licensing, service registration, and certificatemanagement services. The Platform Services Controller handles identity management for administratorsand applications that interact with the vSphere platform. The Platform Services Controller and its relatedservices are embedded within the vCenter Server Appliance. This eliminates the need for separatePlatform Services Controller VM instances and their corresponding load balancers, thus simplifying itsdeployment and administration and reducing the management components footprint.

Data backup and restore of each vCenter Server instance and its embedded Platform Services Controlleris provided by using the native backup service that is built in the appliances. This backup is performed toa separate storage system by using network protocols such as SFTP, HTTPS, and SCP.

When vCenter HA is used with an embedded Platform Services Controller, the environment setup is asfollows:

Figure 6‑5. vCenter Server High Availability

Management Pod

vCenter HA NetworkManagement Network

vCenter HA Network

Resource Cluster


Controller


Controller


Controller


Controller


Controller


Controller

Edge Cluster

VMware NSX-T Data Center

NSX Manager. The management plane for the NSX-T system. It provides the ability to create, configure,and monitor NSX-T Data Center components, such as logical switches, and NSX Edge Nodes. NSXManager provides an aggregated system view and is the centralized network management component ofNSX-T Data Center. It provides a method for monitoring and troubleshooting workloads that are attachedto the virtual networks that NSX-T Data Center creates. NSX-T Data Center provides configuration andorchestration of logical networking components such as logical switching and routing, networkingservices, Edge services, security services, and distributed firewall capabilities.


VMware, Inc. 33

NSX Manager is deployed as a single node that uses vSphere HA for high availability. NSX Managercommunicates with its controller and Edge clusters over a common management network. Themanagement components of the vCloud NFV platform communicate over the same management networkto request network services from NSX-T Data Manager.

Figure 6‑6. NSX Manager and Components

NSXManager

Management Pod

NSX Controller

Management Network

NSX Controller . An advanced distributed state management system that controls virtual networks andoverlay transport tunnels. NSX Controller is deployed as a cluster of highly available virtual appliancesthat are responsible for the programmatic deployment of virtual networks across the entire NSX-T DataCenter architecture. The control plane is split in two parts in NSX-T Data Center, the central control plane(CCP), which runs on the NSX Controller cluster nodes, and the local control plane (LCP), which runs onthe transport nodes, adjacent to the data plane it controls. The Central Control Plane computes someephemeral runtime state based on configuration from the management plane and disseminatesinformation reported through the local control plane by the data plane elements. The Local Control Planemonitors local link status, computes most ephemeral runtime state based on updates from data plane andCCP, and pushes stateless configuration to forwarding engines. The LCP shares fate with the data planeelement that hosts it. The NSX-T Data Center Central Control Plane (CCP) is logically separated from alldata plane traffic, therefore any failure in the control plane does not affect the existing data planeoperations. Traffic does not pass through the controller, instead the controller is responsible for providingconfiguration to other NSX Controller components such as the logical switches, logical routers, and edgeconfiguration. Stability and reliability of data transport are central concerns in networking.

To further enhance the high availability and scalability, the NSX Controller is deployed in a cluster of threeinstances in the Management cluster. Anti-affinity rules are configured to ensure that the controllerinstances reside on separate hosts to protect against host failures.

VMware vCloud Director for Service Providers

VMware vCloud Director for Service Providers is an abstraction layer that operates on top of the otherVIM components, vCenter Server, and NSX Manager. A highly available vCloud Director implementationthat uses multiple load balanced vCloud Director cells are deployed in a vCloud Director Server Group.All cells in the server group are stateless and use a shared highly available clustered database. Each cellcontains all the software components required for vCloud Director. A cell can run on its own, but multiplecells running in an active-active cluster are used for scalability and redundancy.


VMware, Inc. 34

Figure 6‑7. vCloud Director Management Components

VMware vCloud Director

NFS Server

VCD DB

Management Pod

Management Network

vCloud Director builds a secure, multitenant virtual environment by pooling virtual infrastructure resourcesto virtual data centers (VDCs) and exposing them to users through Web-based portals and APIs as fully-automated, catalog-based services.

A fundamental concept in vCloud Director is that of the tenant. A tenant is a logically isolated constructrepresenting a customer, department, network function, or service, used to carve out infrastructureresources and deploy VNF workloads. vCloud Director isolates administrative boundaries to NFVItenants. VNF workload resource consumption is therefore segregated from other VNF workloads, eventhough the VNFs can share the same resources.

vCloud Director implements the open and publicly available vCloud API, which provides compatibility,interoperability, and programmatic extensibility to network equipment providers (NEPs) and their VNFManagers. The vCloud Director capabilities can be extended to create adaptors to external systemsincluding OSS/BSS.

NetworkingThe Management Pod networking consists of the infrastructure and VM networks. The following diagramshows all the virtual switches and port groups of the Management Pod.


VMware, Inc. 35

Figure 6‑8. Management Pod Networking

VLAN 50

VLAN 30

VLAN 20

BackupNetwork

VCHANetwork

ManagementNetwork

VLAN 400

VLAN 500

VLAN 100

VLAN 200

VLAN 300

ESXiManagement

vMotionNetwork

BackupNetwork

vSANNetwork

ReplicationNetwork

VDS

VDS

VMKernel

Virtual Machine

Teaming

vSphere Distributed Switch Infrastructure Networks

VDS: vSphere Distributed Switch

vSphere Distributed Switch Virtual Machine Networks

Teaming

ESXi Host

Management Pod

VLAN 40

ExternalNetwork

Edge PodThe Edge Pod provides the fabric for North-South connectivity to the provider networks.


VMware, Inc. 36

ComponentsThe Edge Pod components provide the fabric for North-South connectivity to the provider networks.Multiple configurations can be used for performance, scale, and Edge services.

Edge Nodes

An Edge Node is the appliance that provides physical NICs to connect to the physical infrastructure andto the virtual domain. Edge Nodes serve as pools of capacity, dedicated to running network services thatcannot be distributed to the hypervisors. The network functionality of the Edge node includes:

n Connectivity to physical infrastructure.

n Edge services such as NAT, DHCP, firewall, and load balancer.

Edge nodes are available in two form-factors, VM and bare metal. Both use the data plane developmentkit for faster packet processing and high performance.

Figure 6‑9. Edge Pod Components

Management Network

Edge Node

Edge Cluster

Overlay Network

External Network

Edge Pod (Bare Metal)

Management Network

Edge Node

Edge Cluster

Overlay Network

External NetworkHost

Host Host Host

Host Host

Edge Pod (VM)

Tier-1Router

Tier-0Router

Tier-1Router

Tier-0Router

The NSX-T Data Center bare metal Edge runs on a physical server and is installed by using an ISO file orPXE boot. The bare metal Edge is recommended for production environments where services like NAT,firewall, and load balancer are needed in addition to Layer 3 unicast forwarding. A bare metal Edge differsfrom the VM form-factor Edge in terms of performance. It provides sub-second convergence, fasterfailover, and greater throughput.

The VM form-factor of NSX-T Data Center Edge is installed by using an OVA. Depending on the requiredfunctionality, there are deployment-specific VM form-factors.


VMware, Inc. 37

Edge Clusters

Edge nodes are deployed as pools of capacity (a cluster), dedicated to running network services thatcannot be distributed to the hypervisors. An Edge cluster can either be all VM or all bare metal form-factors.

The Edge cluster provides scale out, redundant, and high-throughput gateway functionality for logicalnetworks. Scale out from the logical networks to the Edge nodes is achieved by using ECMP. There istotal flexibility in assigning logical routers to Edge nodes and clusters. Tier-0 and Tier-1 routers can behosted on either same or different Edge clusters. Centralized services must be enabled for the Tier-1logical router to coexist in the same cluster.

There can be only one Tier-0 logical router per Edge node, however multiple Tier-1 logical routers can behosted on one Edge node.

In addition to providing distributed routing capabilities, the Еdge cluster enables Еdge services at aprovider or tenant scope. As soon as one of these Еdge services are configured or an uplink is defined onthe logical router to connect to the physical infrastructure, a Service Router (SR) is instantiated on theEdge Node. The Edge Node is also a transport node just like compute nodes in NSX-T, and similar tocompute node it can connect to more than one transport zone, one for overlay and other for N-S peeringwith external devices.

A maximum of eight Edge Nodes can be grouped in an Edge cluster. A Tier-0 logical router supports amaximum of eight equal cost paths, thus a maximum of eight Edge Nodes are supported for ECMP. EdgeNodes in an Edge cluster run Bidirectional Forwarding Detection (BFD) on both tunnel and managementnetworks to detect Edge Node failure. The BFD protocol provides fast detection of failure for forwardingpaths or forwarding engines, improving convergence. Bare metal form factors can support sub-secondconvergence.

NSX-T Data Center supports static routing and the dynamic routing protocol BGP on Tier-0 logical routerson interfaces connecting to upstream routers. Tier-1 logical routers support static routes but do notsupport any dynamic routing protocols.

See the VMware® NSX-T Reference Design Guide for more information.

NetworkingThe Edge Pod virtual network largely depends on the network topology that is required by the VNFworkloads. In general, the Edge Pod has the infrastructure networks, networks for management andcontrol plane connectivity, and networks for workloads.


VMware, Inc. 38

https://communities.vmware.com/docs/DOC-37591

Figure 6‑10. Edge Pod Networking

vSphere Distributed Switch Infrastructure Networks

VDS: vSphere Distributed Switch

vSphere Distributed Switch Virtual Machine Networks

ESXi Host

Edge Pod

VDSVMKernel

Trunk VLAN 0-4094

ExternalNetwork

VDSVirtual Machine

VLAN 100

VLAN 200

VLAN 300

ESXiManagement

vMotionNetwork

vSANNetwork

Teaming

Teaming

Trunk VLAN 0-4094

OverlayNetwork

ManagementNetwork

VLAN 20

Resource PodVNFs are placed in the Resource Pod that forms the virtual network services.

ComponentsThe Resource Pod provides the runtime environment for the network functions. The section covers thelogical tenancy and networking components.


VMware, Inc. 39

Provider VDC

The Provider VDC is a standard container for a pool of compute, storage, and network resources from asingle vCenter Server instance. During deployment, a VNF can use one or more Provider VDCs per site,or multiple VNFs can share a single Provider VDC with different organization VDCs. The requirements ofeach VNF are assessed as part of the onboarding process.

Organization

Organizations are the unit of multi-tenancy within vCloud Director for Service Providers. An NFVI tenantcan be defined in multiple ways in the vCloud NFV platform. An NFVI tenant can, for example, be a VNFprovider, but it can also be defined per Telco application consisting of multiple VNF providers who providea joint service. The necessary level of separation between different VNFs and the creation of NFVItenants should be identified during the onboarding process.

Organization VDCs

An organization VDC is a subgroup of compute, storage, and network resources that are allocated from aProvider VDC and mapped to an organization. Multiple organization VDCs can share the resources of thesame Provider VDC. An organization VDC is the environment where VNF workloads are deployed andexecuted.

Quotas on organizations set limits on the vCloud Director tenant resources, organization VDCs allowproviding resource guarantees for workloads from the resource quota available to the organization andavoid noisy neighbor scenarios in a multitenant environment.

vApps

A vApp is a container for multiple virtual machines and the standard unit for workloads in vCloud Director.vApps contain one or more virtual machines and networks and can be imported or exported as an OVFfile. In the vCloud NFV platform, a VNF instance can be a vApp.

NetworkingThe networking of the Resource Pod is highly dependent on the network topology that is required by thetelco workloads, which tenants deploy. Tenant workloads require a certain set of networking buildingblocks.

Logical Switch

Logical switches are the layer 2 networks created by NSX-T Data Center to provide connectivity betweenits services and the VMs. Logical switches form the basis of the tenant networks in the vCloud NFVplatform. The primary component in the data plane of the transport nodes is N-VDS. N-VDS forwardstraffic between components running on the transport node (that is between VMs) or between VMs and thephysical network. In the latter case, N-VDS must own one or more physical interfaces (physical NICs) onthe transport node. As with other virtual switches, an N-VDS cannot share a physical interface withanother N-VDS. It can coexist with another N-VDS each using a separate set of physical NICs.


VMware, Inc. 40

Figure 6‑11. Resource Pod Networking

ESXi Host

Resource Pod

VMKernel

VLAN 70

Virtual Machine

VLAN 100

VLAN 200

VLAN 300

vSAN Network

VLAN N

VLAN 20

ESXi Management

vMotion Network

Management Network

Overlay Network

Enhanced Data Path Network

VDS

N- VDS (S/E)

Teaming

TeamingN-VDS(E)

Teaming

ESXi Host

Resource Pod

VMKernel

VLAN 70

Virtual Machine

VLAN 100

VLAN 200

VLAN 300

vSAN Network

VLAN 20

ESXi Management

vMotion Network

Management Network

Overlay Network

VDS

N-VDS(S)

Teaming

Teaming

Data and Control Plane using NSX-T Control Plane using NSX-T

ESXi Host

Resource Pod

VMKernel

Virtual Machine

VLAN 100

VLAN 200

VLAN 300

vSAN Network

VLAN <N>

VLAN 20

ESXi Management

vMotion Network

Management Network

Enhanced Data Path Network

VDS

Teaming

TeamingN-VDS(E)

ESXi Host

Resource Pod

VMKernel

Virtual Machine

VLAN 100

VLAN 200

VLAN 300

vSAN Network

VLAN 20

ESXi Management

vMotion Network

Management Network

VDS

Teaming

Data Plane using NSX-T Data Plane using SR-IOV

VLAN <N>

SR-IOV Network

Teaming

VDS


VMware, Inc. 41

Logical Routing

The NSX-T Data Center platform provides the ability to interconnect both virtual and physical workloadsthat are deployed in different logical layer 2 networks. NSX-T enables the creation of network elementslike switches and routers as software logical constructs and embeds them in the hypervisor layer,abstracted from the underlying physical hardware.

East-West Traffic

Configuring a logical router through the NSX Manager instantiates a logical router on each hypervisor. Forthe VNFs hosted on the same hypervisor, the East-West traffic does not leave the hypervisor for routing.The logical router is also responsible for routing East-West traffic between hypervisors.

Figure 6‑12. East-West Traffic

Tier-1

ESXi

Tier-1

ESXi

LS1LS2

VNFC1 VNFC2

North-South Traffic

In addition to providing optimized distributed and centralized routing functions, NSX-T Data Centersupports a multi-tiered routing model with logical separation between the provider routing function and thetenant routing function. This way, the concept of multitenancy is built in the routing model. The top-tierlogical router is called a Tier-0 router, whereas the bottom-tier logical router is called a Tier-1 router.Northbound, the Tier-0 logical router connects to one or more physical routers or layer 3 switches andserves as an on/off ramp to the physical infrastructure. Southbound, the Tier-0 logical router connects toone or more Tier-1 logical routers.


VMware, Inc. 42

Figure 6‑13. North-South Traffic

Tier-1

Tier-0

VNFC1 VNFC2

LIF1 LIF2

PhysicalRouter

Edge Node

This model also eliminates the dependency on a physical infrastructure administrator to configure orchange anything on the physical infrastructure when a new tenant is configured in the data center. For anew tenant, the Tier-0 logical router simply advertises the new tenant routes that are learned from thetenant Tier-1 logical router on the established routing adjacency with the physical infrastructure.


VMware, Inc. 43

Deployment Options 7Two design configurations are optimal with the vCloud NFV platform, a compact Two-Pod configurationand a Three-Pod configuration. These configurations can be deployed in the data center to meet targetdesign and scale objectives.

This chapter includes the following topics:n Three-Pod Configuration

n Two-Pod Configuration

Three-Pod ConfigurationThe Three-Pod design completely separates the vCloud NFV functional blocks by using a ManagementPod, Edge Pod, and Resource Pod for their functions. The initial deployment of a Three-Pod designconsists of three vSphere clusters, respectively one cluster per Pod. Clusters can be scaled up by addingESXi hosts, whereas Pods can be scaled up by adding clusters. The separation of management, Edge,and resource functions in individually scalable Pods allows the CSPs to plan capacity according to theneeds of the specific function that each Pod hosts. This provides greater operational flexibility.

The following diagram depicts the physical representation of the compute and networking connectivityand the logical layers of the switching and routing fabric.

VMware, Inc. 44

Figure 7‑1. Three-Pod Conceptual Design

Tier-0

Router

Logical View

Switching

Routing

Networking Fabric

BGP with BFD

Tenant Network

Virtual Machine Network

Infrastructure Network

VMmanagementvDS

L2 Fabric

Infrastructure vDS

ESXi Server 01

ESXi Server 02

ESXi Server 03

ESXi Server 04

Resource Pod

vSAN

N-vDS Standard / Enhanced

N-VDSEnhanced

Infrastructure vDS

Edge vDS

Infrastructure vDS

TelcoNetwork

Tier-0

Router

ToR

ESXi Server N

ESXi Server 01

ESXi Server 02

ESXi Server 03

ESXi Server 04

Management Pod

vSAN

ESXi Server N

ESXi Server 01

ESXi Server 02

ESXi Server 03

ESXi Server 04

Edge Cluster (VM)

vSAN

ESXi Server N

Edge Pod

Bare Metal 1

Bare Metal 2

Edge Cluster(Bare metal NSX Edge)

Bare Metal N

Tier-1

Router

Tier-1(n)

Router


VMware, Inc. 45

The initial deployment of a Three-Pod design is more hardware intensive than the initial deployment of aTwo-Pod design. Each Pod in the design scales up independently from the others. A Three-Pod designconsists of the same components as in a Two-Pod design, as the way functions are combined to form thesolution is different in each design. Regardless of the Pod design that is used to create the NFVI, VNFsperform the same way.

Logical Viewn Management Pod. Hosts all the NFV management components. Its functions include resource

orchestration, analytics, BCDR, third-party management, NFV-O, and other ancillary management.

n Edge Pod. Hosts the NSX-T Data Center network components, which are the NSX Edge nodes. Edgenodes participate in East-West traffic forwarding and provide connectivity to the physical infrastructurefor North-South traffic management and capabilities. Edge nodes can be deployed in a VM and baremetal form-factor to meet capacity and performance needs.

n Resource Pod. Provides the virtualized runtime environment, that is compute, network, and storage,to execute workloads.

Routing and SwitchingBefore deploying the Three-Pod configuration, a best practice is to consider a VLAN design to isolate thetraffic for infrastructure, VMs, and VIM.

The vSphere Distributed Switch port groups have different requirements based on the Pod profile andnetworking requirements.

For example, two vSphere Distributed Switches can be used in the Management Pod, one for VMkerneltraffic and another one for VM management traffic. While the switching design is standardized for theManagement and Edge Pods, the Resource Pod offers flexibility with the two classes of NSX-T DataCenter switches that are N-VDS Standard and N-VDS Enhanced. The N-VDS Standard switch offersoverlay and VLAN networking and the N-VDS Enhanced switch provider workload acceleration.

The NSX-T Data Center two-tiered routing fabric provides the separation between the provider routers(Tier-0) and the tenant routers (Tier-1).

The Edge nodes provide the physical connectivity to the CSPs core and external networking. Bothdynamic and static routing are possible at the Edge nodes.

Design ConsiderationsA Three-Pod configuration provides flexibility, performance, and VNF distribution design choice for telcoworkloads.


VMware, Inc. 46

FootprintThe best practice when using vSAN as shared storage for all clusters, is to use a minimum of four hostsper cluster for the initial deployment, which sums up a total of 12 hosts for the entire configuration. Thiscreates balance between the implementation footprint and resiliency and maintains the operationalrequirements for each Pod. A mid-level class of servers can be used in this configuration, because thedesign maximizes the scale flexibility.

Scale FlexibilityThe separation of Pods provides maximum scalability to the design. Individual Pods can be scaled tomeet the target services, delivery, and topology objectives.

The Resource and Edge clusters are sized according to the VNFs and their respective networkingrequirements. CSPs must work with the VNF vendors to gather the requirements for the VNFs to bedeployed. This information is typically available in deployment and sizing guides. As more tenants areprovisioned, CSPs must provide additional resources to the Resource Pod to support the VNF growth.The increase in the VNF workloads in the Resource Pod can lead to an increase in the North-Southnetwork traffic, which in turn requires adding more compute resources to the Edge Pod so that to scale upthe Edge Nodes. CSPs must closely monitor and manage the resource consumption and capacity that isavailable to the Edge Pod when the Edge Nodes are scaled. For higher performance, an Edge Pod canalso comprise of bare metal Edges that are grouped in an Edge Cluster in NSX-T Data Center.

High level of resiliency is provided by using four-host clusters with vSAN storage in the initial deployment.Four-host clusters also ensure a highly available Management Pod design, because clusteredmanagement components such as vCenter Server active, standby, and witness nodes, can be placed onseparate hosts. This same design principle is used for clustered vCloud Director components such asdatabase nodes.

The initial number and sizing of management components in the Management Pod should be planned inadvance. As a result, the capacity that is required for the Management Pod can remain steady. Whenplanning the storage capacity of the Resource Pod, the CSP should consider the operational headroomfor VNF files, snapshots, backups, VM templates, OS images, upgrade and log files.

When the Resource Pod is scaled up by adding hosts to the cluster, the newly added resources areautomatically pooled resulting in added capacity to the compute cluster. New tenants can be provisionedto consume resources from the total pooled capacity that is available. Allocation settings for existingtenants must be modified before they can benefit from the increased resource availability. vCoud DirectorProvider VDCs are added to scale out the Resource Pod. Additional compute nodes can be added byusing the vCloud Director vSphere Web Client extension.

In the Edge Pod, additional Еdge Nodes can be deployed in the cluster in conjunction with the physicalleaf-spine fabric for higher capacity needs.


VMware, Inc. 47

PerformanceThe platform is optimized for workload acceleration and the Three-Pod design offers maximum flexibilityto achieve these goals. The configuration can support the needs of use cases with high bandwidthrequirements. A dedicated accelerated Resource Pod or a hybrid standard and accelerated ResourcePod can be considered when designing for workload distribution. The separate Edge Pod provides notonly the separation benefit but also alternatives with VM and bare metal options for the Edge Nodes.

Function DistributionNetwork function designs are evolving to disaggregated control and data plane function. The separationalso maximizes the distribution of data plane functions closer to the edge of the network with centralizedcontrol and management planes. A separated Resource or Edge Pod design allows for having differentcontrol plane versus data plane configurations and scale designs depending on the service offers.

Operations ManagementThe Management Pod provides centralized operation function across the deployment topology, be it asingle or distributed data centers.

Two-Pod ConfigurationThe vCloud NFV facilitates combining the Edge and resource functionality in a single, collapsed Pod thatprovides a small footprint. CSPs can use a Two-Pod design to gain operational experience with thevCloud NFV platform. As demand grows, they can scale up and scale out within the Two-Pod construct.

An initial Two-Pod deployment consists of one cluster for the Management Pod and another cluster forthe collapsed Edge & Resource Pod. Clusters are vSphere objects for pooling virtual domain resourcesand managing resource allocation. Clusters scale up as needed by adding ESXi hosts, whereas Podsscale up by adding new clusters to the existing Pods. This design ensures that management boundariesare clearly defined, capacity is managed, and resources are allocated based on the functionality that thePod hosts. vCloud NFV VIM components allow for fine grained allocation and partitioning of resources tothe workloads, regardless of the scaling method that is used.

The diagram shows a typical Two-Pod design with all management functions located centrally in theManagement Pod. Edge and resource functions are combined in the collapsed Edge & Resource Pod.During the initial deployment, two clusters of ESXi hosts are used, one for the Management Pod, andanother one for the collapsed Edge & Resource Pod. Additional clusters can be added to each Pod as theinfrastructure is scaled up.


VMware, Inc. 48

Figure 7‑2. Two-Pod Conceptual Design

Logical View

Switching

Routing

Networking Fabric

BGP with BFD

Tenant Network

Virtual Machine Network

Infrastructure Network

VMmanagementvDS

L2 Fabric

Infrastructure vDS

ESXi Server 01

ESXi Server 02

ESXi Server 03

ESXi Server 04

Resource and Edge Pod

vSAN

N-VDS Standard/ Enhanced

N-VDSEnhanced

Infrastructure vDS

TelcoNetwork

ToR

ESXi Server N

ESXi Server 01

ESXi Server 02

ESXi Server 03

ESXi Server 04

Management Pod

vSAN

ESXi Server N

Tier-1

Router

Tier-1(n)

Router

Tier-0

Router

Logical Viewn Management Pod. This Pod is responsible for hosting all NFV management components. These

functions include resource orchestration, analytics, BCDR, third-party management, NFVO, and otherancillary management.


VMware, Inc. 49

n Edge & Resource Pod. This Pod provides the virtualized runtime environment, that is compute,network, and storage, to execute workloads. It also consolidates the NSX Edge Node to participate inthe East-West traffic and to provide connectivity to the physical infrastructure for North-South trafficmanagement capabilities. Edge Nodes can be deployed in a VM form factor only.

Routing and SwitchingBefore deploying a Two-Pod configuration, a VLAN design needs to be considered as a best practice toisolate the traffic for infrastructure, VMs, and VIM.

The general design of the routing and switching fabric is similar to the Three-Pod design, with theresource and Edge fabric converged into a single pod.

Design ConsiderationsA Two-Pod configuration provides a compact footprint design choice for telco workloads.

FootprintHigh level of resiliency is provided by using four-host clusters with vSAN storage in the initial deployment.Four-host clusters also ensure a highly available Management Pod design, because clusteredmanagement components such as vCenter Server active, standby, and witness nodes can be placed onseparate hosts. The same design principle is used for clustered vCloud Director components such asdatabase nodes.

With high-end class servers, this design maximizes the resource utilization and densification of workloadsand Edge Nodes in the same Pod.

Scale FlexibilityThe initial number and sizing of management components in the Management Pod should be planned inadvance. As a result, the capacity requirement for the Management Pod can remain steady. Whenplanning the storage capacity of the Management Pod, the CSP should consider the operationalheadroom for VNF files, snapshots, backups, VM templates, OS images, upgrade and log files.

The collapsed Edge & Resource cluster sizing will change according to the VNF and networkingrequirements. When planning the capacity of the Edge & Resource Pod, tenants must work with VNFvendors to gather requirements for the VNF to be deployed. Such information is typically available fromthe VNF vendors as deployment and sizing guidelines. These guidelines are directly related to the scaleof the VNF service, for example to the number of subscribers to be supported. In addition, the capacityutilization of Edge Nodes must be taken into consideration, especially when more instances of EdgeNodes are deployed to scale up as the number of VNFs increases.

When scaling up the Edge & Resource Pod by adding hosts to the cluster, the newly-added resources areautomatically pooled, resulting in added capacity to the compute cluster. New tenants can be provisionedto consume resources from the total pooled capacity that is available. Allocation settings for existingtenants must be modified before they can benefit from the increased resource availability.


VMware, Inc. 50

Operations ManagementThe Management Pod provides centralized operation function across the deployment topology, be it asingle or distributed data centers.


VMware, Inc. 51

Next Generation Data CenterEvolution 8CSPs can implement a set of solutions and use case scenarios to modernize the their cloud infrastructureenvironment with vCloud NFV.

This chapter includes the following topics:n Private Data Center NFV Transformation

n Workload Acceleration

n Multi-Tenancy

n Distributed Clouds

n Workload On-Boarding

n Availability and Disaster Recovery

n NSX Data Center for vSphere Coexistence with NSX-T Data Center

Private Data Center NFV TransformationTransform the cloud infrastructure in the CSP private data centers with vCloud Director IaaS layertogether with the VMware core virtualization technologies.

ScopeThis release of vCloud NFV introduces vCloud Director together with NSX-T Data Center, which is thenext-generation advanced networking stack of the NFVI. vCloud NFV explores the transformation of theCSP private cloud environments to next-generation NFV networks by using multitenant design objectives.

Considering a Three-Pod design, the following diagram illustrates how the Resource Pod enables bothaccelerated and normal workloads with the N-VDS switch in Enhanced (N-VDS (E)) and Standard (N-VDS (S)) modes. A separate Edge Pod provides external connectivity with different options for scale andperformance.

VMware, Inc. 52

Figure 8‑1. Private Cloud with Multiple Tenants

Tier-0 Tier-0

TelcoNetwork

PhysicalRouters

ToR

eBGPwithBFD

vSphere Distributed Switch

LS2

VNFC3

Organization VDC B

LS1

VNFC5

LS2

VNFC3

Provider External Network

Tenant ExternalNetwork

Edge Nodes1...N

Edge NodesCluster

Edge Pod

Resource PodN-VDS (E)

Organization

Organization VDC A

vCloud Director Provider VDC

LS1 LS3

VNFC1 VNFC2 VNFC4

N-VDS (S) / N-VDS(E)


Tier-1 Tier-1

VNFC1 VNFC2 VNFC4

The diagram illustrates how a fully-integrated vCloud Director Provider VDC, Organization andOrganization VDC, as well as NSX-T logical switches and Tier-1 logical routers can be used to provide amultitenant environment for deploying VNFs.

The Edge Pod hosts both the Tier-0 and Tier-1 logical routers that provide North-South and East-Westnetwork connectivity for workloads. The NSX Manager is the management component that CSPs can useto provision these networks. Once North-South and East-West networks are mapped to provider networksin vCloud Director, tenants consume them for their workload connectivity requirements.


VMware, Inc. 53

Design ObjectivesTransform the private data center by using a common shared pool of resources with multitenant compute,storage, and network isolation. Use the NSX-T Data Center advanced networking and accelerationcapabilities for control and data plane workloads.

Site DesignThe Management, Resource, and Edge Pods are deployed in each data center that is part of a site. Podsthat belong to a site can be scaled to meet the use cases and requirements of their data centers.

Resource Pods in each data center are sized to meet the scale and traffic requirements of the VNFs andother data plane functions.

Respectively, a pair of availability zones should be created in each site for redundancy and faulttolerance. A group of homogenous hosts, also called host aggregates should be mapped across theavailability zones. A single tenant (Organization and Organization vDC) map to the host aggregates todeploy workloads for control and user plane functions.

Compute Resource IsolationResource pools represent a unit of CPU, memory, and storage that are portioned from the sharedinfrastructure pool. These resource pools are attached to Provider VDCs that in return are the abstractedpools of resources that tenants consume through Organization VDCs. The resources of a Provider VDC,and hence the resources available to tenants, can be scaled by scaling the underlying resource pool tomeet future resource demands.

By mapping the different Provider VDCs to different resource pools backed by different vSphere hostclusters, CSPs can meet different SLAs as per tenant resource requirements.

Shared StorageIn this design, VMware vSAN is used as a shared storage solution. Other storage options with NFS,iSCSI, and Fiber Channel are also supported. vSAN aggregates local or directly attached data storagedevices to create a single storage pool that is shared across all hosts in the vSAN cluster. vSANeliminates the need for external shared storage and simplifies the storage configuration and VMprovisioning.


VMware, Inc. 54

Figure 8‑2. Shared Storage by Using vSAN

vSphere with vSAN

HDDSSD

vSAN Datastore

Disk Group

HDDSSD HDDSSD

Using a shared datastore is recommend for high availability and fault tolerance. In case of host failure,vSphere HA can restart the VMs on another host.

A cluster in vCenter Server provides management of software-defined storage resources the same wayas with compute resources. Instead of CPU or memory reservations, limits, and shares, storage policiescan be defined and assigned to VMs. The policies specify the characteristics of the storage and can bechanged as business requirements change.

Design for TenantsThe diagram in the Scope section of this use case shows how the fully-integrated vCloud DirectorCompute Cluster, Organization and Organization VDC, and NSX-T Data Center logical switches, andTier-1 logical routers can be used to provide a multitenant environment for VNF deployments. The EdgePod hosts the NSX-T Data Center Tier-0 logical routers that are used as provider routers for externalnetwork connectivity. The NSX Manager is used to provision the Tier-0 Edge routers, and othernetworking components needed for organization VDCs like logical switches (East-West) and logicalswitches (North-South) connecting to Tier-1 Router. NSX Manager also creates the Tenant Tier-1 router.

The CSP can map a vSphere compute cluster to a Provider VDC, then for each tenant the CSP canallocate and reserves resources by using organization-based constructs. Every organization VDC isassociated with a resource pool within the compute cluster for resource guarantee.

The separation of network access between organizations is important for multitenancy. vCloud Directorintegrates with NSX Manager to create isolated layer 2 tenant networks. Tier-1 logical routers allowtenants to route traffic between their tenant networks.


VMware, Inc. 55

Dual-Mode Switching with N-VDSThe tenant external network can carry network traffic to NSX-T Data Center switches that can be either N-VDS in Standard or Enhanced mode or both. This dual-mode N-VDS switching fabric can be used todesign for accelerated and non-accelerated workloads within the same or separated compute clusters.

Both the N-VDS Standard and N-VDS Enhanced switches can carry the tenant Tier-1 to provider Tier-0traffic, as well as East-West overlay Geneve traffic between VNFCs. VNFCs that require high data pathtraffic can use the N-VDS Enhanced switch on a VLAN port group and connect to the ToR switch.Services such as the PGW can use this design in a mobile network for broadband connectivity, streamingapplications for audio and video distribution, and SBC for IP voice peer-to-peer communications.

Service Provider Edge Cluster ConfigurationBefore establishing the vCloud Director configuration, the CSP has to create an Edge Node providercluster. The NSX Edge Node cluster consists of logical Tier-0 routers. The Edge Node cluster can consistof either VM or bare metal form-factors. The bare metal Edge is installed on a physical server providinghigher throughput data rates.

Table 8‑1. Edge Node Options

Edge Node Type Use

VM form-factor n Production deployment with centralized services like NAT, Edge firewall, and load balancer.n Workloads that can tolerate acceptable performance degradation loss with virtual edges.n Can tolerate lower failure convergence by using BFD (3 seconds).n Lower cost options instead of dedicated bare-metal nodesn Test proof of concept and trial setups.

Bare metal form-factor

n Production deployment with centralized services like NAT, Edge firewall, and load balancer.n Higher throughput more than 10Gbps.n Faster failure convergence using BFD (less than 1 second).

Edge Node Active-Active

In an Edge Node Active-Active configuration, Tier-0 routers are hosted on more than one Edge Nodes ata time to provide high availability. In ECMP mode, the traffic is load balanced between the links to theexternal physical routers. A maximum of 8 Edge Nodes can be configured in ECMP mode to providescalable throughput that spreads across the Edge Node physical uplinks to the provider network. Statefulservices like NAT and Firewall cannot be used in this mode.


VMware, Inc. 56

Figure 8‑3. Edge Node Active-Active Design

ToR

Edge Node 1

Edge Pod

Resource Pod


N-VDS (S) / N-VDS(E)

Tier-0

Edge Node 2 Edge Node N

Tier-0Tier-0

Active

ECMPNSX Edge Node Cluster

Active Active

ECMPECMP


Edge Node Active-Standby

A high availability configuration where a Tier-0 router is active on a single Edge Node at a time. Thismode is required when stateful services like NAT, Firewall, and load balancer must remain in a constantstate of synchronization between the active and standby Tier-0 routers on the Edge Node pair.

Figure 8‑4. Edge Node Active-Standby Design

Edge Node 1

Edge Pod

Resource Pod


N-VDS (S) / N-VDS (E)

Tier-0

Edge Node 2

Tier-0

NSX Edge Node Cluster

Failover

Active

ToR


Standby


VMware, Inc. 57

Dynamic Routing

Tier-0 routers can be connected to physical routers by using BGP or static routes. If static routes areused, every newly created external network has to be added manually to the Tier-0 router that peers withthe physical routers.

The NSX-T Data Center Edge Node also supports fast failure recovery by using Bidirectional ForwardingDetection (BFD) that is integrated with BGP. VM form-factor edges support a minimum timeout of onesecond with three retries, providing a three second failure detection time between nodes. With bare-metalnodes, the detection or convergence timeout is less than one second.

For more information on NSX-T Data Center, see the NSX-T Reference design.

Workload Life Cycle ManagementOnce the compute, storage, and networking environment are set for the tenant, workloads can beonboarded to the organization vDC.

For more information on workload onboarding, see the VNF Onboarding section.

To enable dynamic optimization by using runtime operational intelligence, see the Analytics-EnabledReference Architecture section.

Workload AccelerationThis version of the vCloud NFV platform supports advanced packet processing acceleration, which is animportant enhancement for supporting data plane intensive workloads. Both control or user planeworkloads can take advantage of this capability and achieve higher throughput, reduced latency, andscale in performance.

ScopeVNF components that require high network throughput can achieve high data plane performance by usingthe advanced switching fabric that is introduced with the N-VDS logical switch of NSX-T Data Center andthe Edge cluster acceleration in the Three-Pod configuration.

Design ObjectivesThe VMware vCloud NFV platform includes NSX-T Data Center as the virtualized networking component.NSX-T Data Center leverages DPDK techniques to support data plane intensive VNFs efficiently. TheNSX-T Data Center networking stack increases the CPU efficiency while preserving the existingfunctionality of the VMware NFV infrastructure. The new high-performance switching and enhancedplatform awareness delivers the VMware advanced networking architecture that will be transparent toVNF providers and delivers kernel-based security, deterministic resource allocation, and linear scalability.

vSphere introduces support for a high-performance networking mode that is called N-VDS in Enhanceddatapath mode, which works together with NSX-T Data Center. NSX-T Data Center N-VDS provideslogical switching fabric that works in two modes, a Standard NSX-controlled Virtual Distributed Switch (N-VDS Standard) and Enhanced NSX-controlled Virtual Distributed Switch (N-VDS Enhanced). N-VDS


VMware, Inc. 58

https://communities.vmware.com/docs/DOC-37591

Enhanced implements key DPDK features such as, Poll Mode Driver, Flow Cache, optimized packet copy,and provides better performance for both small and large packet sizes that are applicable to NFVworkloads. N-VDS Enhanced mode provides three to five times faster performance compared to thevSphere Distributed Switch. VNF vendors can now use a high-performance virtual switch withoutsacrificing any of the operational benefits of virtualization such as vMotion and DRS.

Both modes of the N-VDS switch use dedicated physical NICs that are not shared with other switches.Because of this, they can exist on the same host and carry different traffic types.

Table 8‑2. NSX-T Data Center Logical Switch Mode Options

Switch Mode Use

N-VDS Enhanced n Recommended for data plane intensive workloads.n High transactional control plane VNFs.n Connectivity towards external networks can be overlay or VLAN backed.

N-VDS Standard n Suitable for control and management plane workloads.n VNFs that require overlay and VLAN backed connectivity.n VNFs that require stateful and stateless Edge services such as load balancer, firewall, and NAT

Acceleration with the N-VDS Enhanced Switch ModeThe N-VDS Enhanced mode uses vertical NUMA compute alignment capabilities to accelerate workloads.

Hosts Preparation for Data Plane Intensive Workloads

Before data plane intensive workload can attach to an N-VDS Enhanced switch, the hosts in theResource Pod need specific configuration.

n vNICs that are intended for use with the N-VDS Enhanced switch should be connected by usingsimilar bandwidth capacity on all NUMA nodes.

n An N-VDS Enhanced switch with at least one dedicated physical NIC should be created on each host.

n The same number of cores from each NUMA node should be assigned to the N-VDS Enhancedswitch.

n A Poll Mode Driver for the N-VDS Enhanced switch must be installed on the ESXi hosts for use of thephysical NICs that are dedicated to the N-VDS Enhanced switch. You can download the driver fromthe VMware Compatibility Guide. Identify the correct drivers by looking for N-VDS Enhanced DataPath in the feature column of the VMware Compatibility Guide.

The host where the N-VDS Enhanced switch resides should be configured similarly to the topologydepicted in the figure that follows. To ensure optimal performance, vertical alignment is establishedbetween the VNF-C, N-VDS Enhanced, and the CPU assigned to the NIC on the same NUMA node. TheCPU scheduler ensures that the VNF-C's vCPU maps to physical CPU cores on the same NUMA nodewhere the N-VDS Enhanced physical cores are mapped.


VMware, Inc. 59

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

Figure 8‑5. N-VDS Enhanced Switch Configuration

ESXi Host

ToR

NUMA 0/1

pNIC1 pNIC2

vNIC1 vNIC2 vNIC1 vNIC2

N-VDS (E)

VLAN Port Group 1 VLAN Port Group 2

CPUCores

CPUCores

VNFC1 VNFC2

Workload Dual-Mode Connectivity

Both N-VDS Standard and Enhanced switches support VLAN and overlay Geneve encapsulated traffic.Both switch types can be deployed on the same host with dedicated physical NICs to each switch. Basedon performance needs, VNFs can have a control plane traffic VIF connected to either N-VDS Standard orEnhanced switch. The Enhanced switch mode can be used to provide acceleration. The N-VDSEnhanced switch with VLAN backing can be used for data plane VIFs with direct connectivity to ToRfabric.

Figure 8‑6. N-VDS Dual-Mode Configuration

ToR

vNIC1

pNIC3 pNIC4pNIC1 pNIC2


vNIC1 vNIC2

N-VDS (E)

VNFC 1 VNFC 2

CPUCores

ControlPlane

DataPlane

ESXiHost

DataPlane

CPUCores

VLAN Port Group 2VLAN Port Group 1VLAN Port Group 1


VMware, Inc. 60

Acceleration by Using Bare Metal EdgeNSX Edge Nodes are also available in a bare metal form-factor with improved performance compared tothe VM-based Edge Nodes.

A Three-Pod configuration is required for a bare metal NSX Edge cluster deployment. The Edge clusteracts as an Edge router that connects to the CSP’s physical router.

A bare metal NSX Edge delivers improved performance and higher throughput. The bare metal NSXEdge also provides sub-second BFD convergence and faster failover.

Physical Design

When a bare metal NSX Edge Node is installed, a dedicated interface is retained for management. Twophysical NICs can be used for the management plane to provide high availability and redundancy.

Figure 8‑7. Bare Metal Edge Physical View

Bare MetalEdge Node

pNIC1 pNIC2 pNIC3 pNIC4 pNIC5

Host OS

fp-eth2eth0 fp-eth0 fp-eth1

Management N-VDS(Overlay)

N-VDS(External)

N-VDS(External)

fp-eth3

For each physical NIC on the server, an internal interface is created following the fp-ethX namingscheme. These internal interfaces are assigned to the FastPath and are allocated for overlay tunnelingtraffic or uplink connectivity to top-of-rack (ToR) switches. There is full flexibility in assigning fp-ethinterfaces to physical NICs for overlay or uplink connectivity. Physical NICs can be assigned to eachoverlay or external connectivity for network redundancy. Because there are four fp-eth interfaces on thebare metal NSX Edge, a maximum of four physical NICs are supported for overlay and uplink traffic inaddition to the primary interface for management.

The BFD protocol provides fast failure detection for forwarding paths or forwarding engines, improvingloss of connectivity detection and therefore enabling quick response. The bare metal NSX Edges supportBFD for both the interfaces towards the provider router and a BFD-like protocol operating between theNSX Edges and the resource hosts.

Logical Design

In a multitenant environment, vCloud Director for Service Providers is configured to use the N-VDSStandard or Enhnaced switch together with a bare metal NSX Edge for higher performance. Regardlessof the workload type, both control and data plane workloads can take advantage of this configuration.Tenant workloads can take advantage of bare metal router features like dynamic routing, firewall, NAT,and load balancer. With bare metal, there is no virtualization overhead, because it directly connects tophysical NIC.


VMware, Inc. 61

Figure 8‑8. Bare Metal Edge Logical View

LS2

VNFC3

Organization VDC B

LS1 LS2

VNFC3

Organization VDC A

LS1

VNFC1 VNFC2 VNFC4

Tier-1 Tier-1

VNFC1 VNFC2 VNFC4

ProviderExternalNetwork


Edge Pod

Edge NodesTier-0Tier-0

ToR

Resource Pod

Organization X


ProviderExternalNetwork


Edge Node Cluster


High Availability Options

n Active-Standby mode. A high availability configuration that requires a minimum of two Edge Nodeshosting the Tier-0 routers. This mode also enables stateful services like NAT, firewall, and loadbalancer to be in constant state of sync between the active and standby Tier-0 on the Edge Nodes.


VMware, Inc. 62

Figure 8‑9. Bare Metal Edge Active-Standby Mode

Edge Node 1

Edge Pod

Bare Metal

Resource Pod


Edge Node 2Edge Node Cluster

Failover

Active

ToR

Tier-0

Sandby

Tier-0


n Active-Active Mode. Edge Node Active-Active configuration provides high availability mode as Tier-0routers are hosted on more than one Edge Node at a time. In ECMP mode, traffic is load balancedbetween the links to external physical routers. A maximum of eight Edge Nodes can be configured inECMP mode to provide scalable throughput that spreads across the Edge physical uplinks towardsthe provider network. Stateful services like NAT and firewall cannot be used in this mode.


VMware, Inc. 63

Figure 8‑10. Bare Metal Edge Active-Active Mode

ToR

Edge Node 1

Edge Pod

Bare Metal

Resource Pod



Tier-0

Edge Node 2 Edge Node N

Tier-0Tier-0

Active

ECMPEdge Node Cluster

Active Active

ECMPECMP

Acceleration with SR-IOVSR-IOV is a specification that allows a single Peripheral Component Interconnect Express (PCIe) physicaldevice under a single root port to appear as multiple separate physical devices to the hypervisor or theguest operating system.

SR-IOV uses physical functions (PFs) and virtual functions (VFs) to manage global functions for the SR-IOV devices. PFs are full PCIe functions that are capable of configuring and managing the SR-IOVfunctionality. VFs are lightweight PCIe functions that support data flow but have a restricted set ofconfiguration resources.

The number of virtual functions provided to the hypervisor or the guest operating system depends on thedevice. SR-IOV enabled PCIe devices require appropriate BIOS and hardware support, and SR-IOVsupport in the guest operating system driver or hypervisor instance.

Prepare Hosts and VMs for SR-IOV

In vSphere, a VM can use an SR-IOV virtual function for networking. The VM and the physical adapterexchange data directly without using the VMkernel stack as an intermediary. Bypassing the VMkernel fornetworking reduces the latency and improves the CPU efficiency for higher data transfer performance.

vSphere supports SR-IOV in an environment with specific configuration only. Find a detailed supportspecification at the SR-IOV Support page.


VMware, Inc. 64

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.networking.doc/GUID-E8E8D7B2-FE67-4B4F-921F-C3D6D7223869.html#GUID-E8E8D7B2-FE67-4B4F-921F-C3D6D7223869

In the topology below, the vSphere SR-IOV support relies on the interaction between the VFs and the PFof the physical NIC port for higher performance. VM network adapters directly communicate with the VFsthat SR-IOV provides to transfer data. However, the ability to configure the VFs depends on the activepolicies for the vSphere Distributed Switch port group ports (VLAN IDs) on which the VMs reside. The VMhandles incoming and outgoing external traffic through its virtual ports that reside on the host. The virtualports are backed by physical NICs on the host. VLAN ID tags are inserted by each SR-IOV virtualfunction. For more information on configuring SR-IOV, see Configure a Virtual Machine to Use SR-IOV.

Figure 8‑11. SR-IOV Virtual Function Configuration

ESXi Host

ToR

VF Driver

pNIC

VF Driver

Port 1 Port 2

PF VF VF

VNFC2

vSphereDistributed Switch

Port Group

VNFC1

SR-IOV can be used for data intensive traffic, but it cannot use virtualization benefits such as vMotion,DRS and so on. Because of this reason, VNFs employing SR-IOV become static hosts. A special hostaggregate can be configured for such workloads.

SR-IOV Configuration by Using vCloud Director for Service Providers

vSphere Distributed Switch and N-VDS Standard can be deployed on the same host with dedicatedphysical NICs to each switch. VNFs can have a VIF connected to both the vSphere Distributed Switchport group ports for North-South connectivity and N-VDS Standard switches for East-West connectivity.


VMware, Inc. 65

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.networking.doc/GUID-EE03DC6F-32CA-42EF-98FC-12FDE06C0BE0.html

Figure 8‑12. SR-IOV Logical View

VF Driver

pNIC

ESXi Host

pNIC

VF Driver

Port 1 Port 2

PF VF VF

N-VDS (S) /N-VDS (E)

VNFC 2

DataPlane

ControlPlane

Port GroupAssociation

VLANPort Group

vSphereDistributed Switch

DataPlane

ToR

VNFC 1

Multi-TenancyThe CSP can facilitate the NFV transformation on a shared resource infrastructure environment withmultitenant consumption models. The design of those tenant environments is outlined in this section.

ScopeMultitenancy defines the isolation of resources and networks to deliver applications with quality. Becausemultiple tenants will share the same resource infrastructure, secure multitenancy can be enabled by usingVMware vCloud Director in a single cloud island and across distributed clouds. In addition to the built-inworkload and resource optimization capabilities, predictive optimization can be enabled with analytics byusing features like vSphere DRS.

CSPs can converge their resource infrastructures across their IT and Network clouds enabling amultitenancy IaaS realization over it. Consumption models can serve both internal and external tenantsover the common shared infrastructure to deploy and operate their respective workloads and services.Network, compute, and storage isolation, are the design objective discussed in this section.

Design ObjectivesA unit of tenancy is called an Organization VDC within the scope of an Organization. It is defined as acomposition of dedicated compute, storage, and network resources as well as workloads. The tenant isassociated with a set of operational policies and SLAs.The tenant can be bound to a single OrganizationVDC or can be composed of many Organization VDCs. Services such as HSS, DNS are examples ofshared tenancy.

The design objectives include considerations for the compute and network resource isolation andautomation.


VMware, Inc. 66

Figure 8‑13. Multi-Tenancy with vCloud Director

Project 2Project 1

Tenant VDC A

VNFC2

Tenant VDC B

VNFC3

N-VDS (S)N-VDS (S)

N-VDS (S)

A-SECMP

Tenant VDC C

ToR Switch(es)

Physical routers

Resource PoolTenant VDC A

Resource PoolTenant VDC B

Resource PoolTenant VDC C

NFV Infrastructure

vCloud Director

Edge Clusters

Core Network

Tier-0

Tier-1

Tenant A

Tier-1

Tenant A

VNFC1 VNFC4

Tier-0Tier-0

Tier-1

Tenant B

Tier-0

Management PlaneThe management plane functions reside in the Management Pod. They are responsible for theorchestration of resources and operations. The management plane functions are local to each cloudinstance providing the infrastructure management, network management, and operations managementcapabilities.


VMware, Inc. 67

Resource isolation for compute and networking design are enabled together with vCenter Server, NSXManager, and VMware vCloud Director. Irrespective of the pod deployment configuration, the VMwarevCloud NFV platform provides the abstraction layers for multi-tenancy. vCenter Server provides theinfrastructure for fine-grained allocation and partitioning of compute and storage resources, whereasNSX-T Data Center creates the network virtualization layer.

The concept of tenancy also introduces shared administrative ownerships. A cloud provider, that is theCSP admin, can create a resource pool allocation and overlay networking for a tenant who in turn wouldconsume these resources as per the workload requirements.. In VMware vCloud Director, multipletenants can be defined with assigned RBAC privileges to manage resources as well as VNF onboarding.

Compute IsolationAllocation of compute and storage resources ensures that there is an optimal footprint available to eachtenant that is used to deploy workloads, with room for expansion to meet future demand.

Organizations provide a secure multitenant environment to deploy VNFs. Compute resources are definedas resource pools when an Organization VDC is created. The resource pool is an allocation of memoryand CPU from the available shared infrastructure, assignable to an Organization VDC. More resourcescan be added to a pool as capacity needs to grow. The Organization VDC can also stretch acrossmultiple hosts residing in different physical racks.

Network IsolationThe advanced networking model of NSX-T Data Center provides a fully-isolated and secure traffic pathsacross workloads and tenant switch and routing fabric. Advanced security policies and rules can beapplied at the VM boundary to further control unwarranted traffic. Also for better traffic management, QoSswitching profile can be used to provide high-quality and dedicated network performance for preferredtraffic, that requires high bandwidth using class of service (COS) and Differentiated Services Code Point(DSCP) values for tenants.

NSX-T Data Center introduces a two-tiered routing architecture which enables the management ofnetworks at the provider (Tier-0) and tenant (Tier-1) tiers. The provider routing tier is attached to thephysical network for North-South traffic, while the tenant routing context can connect to the providerTier-0 and manage East-West communications. The Tier-0 will provide traffic termination to the cloudphysical gateways and existing CSP underlay networks for inter-cloud traffic communication.

Each Organization VDC will have a single Tier-1 distributed router that provides the intra-tenant routingcapabilities. It can be also enabled for stateful services such as firewall and NAT. VMs belonging to aparticular Tenant can be plumbed to multiple logical interfaces for layer 2 and layer 3 connectivity.

AutomationTo meet the operational policies and SLAs for workloads, a closed-loop automation is necessary acrossthe shared cloud infrastructure environment.

The vCloud NFV leverages vSphere DRS to optimize the initial and runtime placement of workloads toensure health and performance to the cloud infrastructure. Organization VDCs and workloads aremonitored to ensure resources are being tuned and balanced dynamically.


VMware, Inc. 68

Capacity and demand planning, SLA violations, performance degradations, and issue isolationcapabilities can be augmented with the analytics-enabled reference architecture. The analytics-enabledarchitecture provisions a workflow automation framework to provide closed-loop integrations with NFVOand VNFM for just-in-time optimizations.

For more information, see the Analytics-Enabled Reference Architecture section.

Distributed CloudsThe NFV transformation in the private cloud will span distributed topologies. For lower latency andcompute proximity, user plane functions such as EPC gateways and IMS media will move closer to theedge clouds. Control plane functions can be still centralized.

ScopeDistributed clouds in this reference architecture are examined from a management and operationsperspective.

Design ObjectivesMulti-cloud sites can be used for distributing workload functions as well as user plane and control plane.Multi-cloud sites can be also used to provide geographic redundancy for protected sites requiring faultand maintenance domains. Multiple administrative ownerships also need to be considered across thetopology and regions.

Site DesignIn a distributed cloud topology, VNFs are stitched across the different cloud islands to deliver services.Each site can also have multiple instances of the NFV environment for redundancy and scale.

Each site is a standalone island with its own instance of the Management, Resource, and Edge Pods.Multi-site designs are typically used for load distribution and for higher availability in case of catastrophicsite failures.

The design assumes an active-standby configuration for geographic redundancy. Two sites areselectedas the primary-secondary pair for failure protection. The remaining remote sites have local instances ofthe management components for seamless cross-site association capabilities . The geographicredundancy design can be implemented with proximal sites or with larger physical distances, as networklatency between sites should be considered.


VMware, Inc. 69

Figure 8‑14. Distributed Clouds

vCloud Director

vCenter Server

Resource Pod

Org VDC1

Org VDC 2

Edge Pod

vCloudDirector

vCenter Server

Availability Zone (Failover for Site 1)

Management Pod

Site 1

Management Pod vCloud Director

Resource Pod

Org VDC 1

Org VDC 2

Edge Pod

Site N

vCenter Server

Site Association

Availability Zone (Protected)

vSphere Replication

Cloud Management PlaneThe management plane functions reside in the Management Pod. The management plane functions arelocal to each site providing the virtual infrastructure and host management, network management, andoperations management capabilities.

Resource Orchestration

Each cloud island should have a dedicated vCloud Director instance for virtual infrastrcture managementof that island. Each site should also have deidcated resource and management vCenter Serverinstances. . Optionally, this design choice could be centralized as latency and site distances should beconsidered.

The vCloud Director site and Organization credential association feature federates the identitymanagement across the various distributed vCloud Director instances and their Organizations within eachof the distributed sites. A tenant user can have an OrganizationVDC created across multiple data centers.A tenant administrator can then login to any of the associated vCloud Director instances to manage theirtenancy from a single pane view.


VMware, Inc. 70

Figure 8‑15. Global Site Access Conceptual Overview

Site-B-cloud.example.com Site-C-cloud.example.comSite-A-cloud.example.com

Site-X-cloud.example.com

Portal.cloud.example.com

vCloud Director Site ‘A’

Data center ‘A’

vCloud Director Site ‘B’

Data center ‘B’

vCloud Director Site ‘C’

Data center ‘C’

User Login

For more information see, Architecting Multisite VMware vCloud Director.

Operations Management

Operations management functions by using the VMware vRealize suite can be deployed in a singlecentralized site which can be used to collect and process analytics. The latency between sites should beconsidered in the design.

For more details on the operations management design, see the Analytics-Enabled ReferenceArchitecture section.

Site AvailabilityFor site disaster resovery and maintenance, multiple sites can be used to implement geographicredundancy. This ensures that a secondary site can be used to complement the primary site.


VMware, Inc. 71

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/vcat/architecting-multi-site-vmware-vcloud-director.pdf

RedundancyChoice Design Considerations

Singe Site n For non-critical services where longer downtime is acceptable.n Services are stateless and traffic can be routed to another instance in case of failure.n Partial site loss that can be recovered by using vSphere HA and shared storage.n Backup-based approaches are acceptable for VM state recovery.

Active-Standby n Suitable for a recovery point objective with acceptable maximum of five minutes.n High speed and capacity links can be established between primary and secondary siten Higher capital costs required to maintain replicas in recovery site

Active-Active n Suitable for mission-critical services with almost no downtimen Service are deployed in active-active configuration across multiple sitesn Any site should be able to handle the load of both active sites in case of failuren Higher capital cost is required in each site in case either site failsn This configuration is currently not available in this release

Availability Zones

Redundancy can be built in a single site to protect against power outages, for example. However, thisdesign choice comes with duplicate hardware cost overhead. This design is out of scope of this referencearchitecture.

By design, multiple availability zones belong to a single region. The physical distance between theavailability zones is short enough to offer low, single-digit latency and large bandwidth between the zones.This architecture allows the cloud infrastructure in the availability zone to operate as a single virtual datacenter within a region. Workloads can be operated across multiple availability zones in the same regionas if they were part of a single virtual data center. This supports an architecture with high availability thatis suitable for mission critical applications. When the distance between two locations of equipmentbecomes too large, these locations can no longer function as two availability zones within the sameregion and must be treated as multi-site design.

Multi-Region

Multiple sites support placing workloads closer to the CSP's customers, for example, by operating onesite in US Northern California and one site in US Southern California. In this multi-site design, asecondary site can become a recovery site for the primary site. When components in the primary sitebecome compromised, all nodes are brought up in the failover site.

VMware vSphere Replication is used to replicate VM disks from the primary to the secondary site. In caseof failure, the Resource Pod workloads can be brought up on the secondary site and routing updated todivert subscriber traffic to the secondary site.

Workload On-BoardingThe VNF onboarding process is typically a collaborative effort between the VNF vendor and the CSP. Aprescriptive set of steps need to be followed to onboard and deploy a VNF. The VMware Ready for NFVprogram is a good vehicle to onboard and certify the VNFs on vCloud NFV platform to ensure smoothdeployment in CSP's environment.


VMware, Inc. 72

ScopeThe VNF life cycle involves provisioning of the cloud infrastructure, VNF packaging, resource assignmentand configuration, deployment, and its placement.

Design ObjectivesThe workload onboarding process is aimed at VNFs that are deployed as native VM applications. Thissection covers the process of packaging, instantiation, to placement.

vCloud Director Conceptual DesignThe conceptual design provides a high-level view of the roles, areas of responsibility, and tenant flow thatare required to upload an image, onboard, and deploy it.

The VNF onboarding process is typically a collaborative effort between the VNF vendor and the CSP.Before a VNF is onboarded, the VNF vendor must provide the CSP with all the prerequisites for thesuccessful onboarding of the VNF. This includes information such as the VNF format, number of therequired networks, East-West and North-South network connectivity, routing policy, security policy, IPranges, and performance requirements.

VNF Format and Packaging

vCloud Director supports importing VNFs as standard OVF/OVA packages.As a best practice, the VNFvendor must author the package in a separate environment identical to the target vCloud NFVenvironmentso that to match its features and capabilities.

VMware Ready for NFV is a program where VMware and VNF vendors collaborate to ensure that theVNF is interoperable with vCloud NFV . Based on experience, VMware provides best practices to helpVNF vendors in preparing their VNFs for consumption by vCloud Director.

CSPs expect their VNF supplier to deliver a package that is readyfor consumption by vCloud Director.The package must support:

1 Autonomous VNF Life Cycle Operations. CSPs must be able to install, upgrade, and manage theVNF themselves.

2 Compact. All components that are required to operate the VNF are packaged together. VNFCs areclearly defined, use unique and specific names, and are included in the package.

3 Authorized. The package provided to the CSP must be verifiable. For example, an md5 hash could beused to verify that the package provided by the vendor is the same package that is installed by theCSP.

4 Holistic. The packaged VNF must include all configuration that is required to deploy a healthy VNF.For example, if the VNF is data plane intensive, all performance-related configuration such as CPUand NUMA affinity, CPU and memory reservations must be included in the package.


VMware, Inc. 73

VNF Onboarding by Using VMware vCloud Director

Onboarding a VNF in vCloud NFV includes the following steps:

1 Prepare the VNF for consumption by vCloud Director.

2 Import the VNF images to vCloud Director catalog.

3 Ensure that the environment is configured correctly for the VNF deployment.

4 Deploy the VNF in the vCloud NFV environment.

The process can be divided in CSP operations and tenant operations.

Before a VNF is onboarded, the CSP administrator must collect prerequisite information from the VNFsupplier. The prerequisites includes configuration information such as the number of required networks,IP ranges, North-South network connectivity, and so on.

vCloud Director uses the concept of catalogs for storing content. Catalogs are containers for VNFtemplates and media images. CSPs can create global catalogs with master VNF templates that can beshared to one or more tenants, while tenants can retain their own private catalog of VNF templates. TheOVF/OVApackage of a VNF is directly imported to the tenant catalog. In addition to VNF templates, thecatalog can contain VNFC templates, used to scale deployed VNFs.

Tenant administrators deploy VNFs from the available templates in the self-service catalog. Tenants alsoprovision East-West connectivity by using Organization VDC networks that they import from NSX-T. VNFNorth-South connectivity is established by connecting Organization VDC networks to e xternal networkthrough an NSX-T Edge Router.

Figure 8‑16. VNF Onboarding Conceptual Design

Project 1

Tenant VDC A (VNF 1)

VNFC1 VNFC2 VNFC3

Provider Network

LIF 1 LIF 2

InstanceVNFPackage

LogicalSwitches

Administrator Tenant

Resource Pool

Tenant VDC A

VNF 1

VNFC1

VNFC2

VNFC3

Image Store

Tier-0

Tier-1


VMware, Inc. 74

Resource Allocation

This process is typically collaborative between the VNF vendor and the CSP. The VNF vendor providesguidance based on lab testing, regarding the amount of virtual resources needed to support a specificsize or scale of teleco service. For example, a virtual Evolved Packet Core (vEPC) is likely to specify thatto serve a certain number of subscribers with active features such as deep packet inspection (DPI) andquality of service (QoS), a specific number of vCPUs, memory, network bandwidth, and storage isrequired. The CSP accounts for near-future scale requirements, using the resource allocationmechanisms provided by vCloud Director such as resource allocation models and storage policies

VNF Networking

Based on specific VNF networking requirements, a CSP administrator can use NSX Manager to provisionEast-West connectivity, security groups, firewalls, micro-segmentation, NAT, and LBaaS for a tenant.Tenants import their respective networks created by the CSP administrator, to an Organization VDC. VNFNorth-South connectivity is established by connecting tenant networks to external networks through NSX-T Data Center routers that are deployed in Edge Nodes. External networks are created by CSPadministrators and backed by physical networks.

Tenant networks are accessible by all Organization VDCs within the Organization. Therefore, theimplementation of East-West connectivity between VNFCs in the same Organization VDC, and theconnectivity between VNFs in two different Organization VDCs belonging to the same Organization, isidentical. Tenant networks are implemented as logical switches within the Organization. The North-Southnetwork is a tenant network that is connected to the teleco network through an N-VDS Enhanced switchfor data-intensive workloads or by using N-VDS Standard through an NSX Edge Cluster.

vCloud Director exposes a rich set of REST API calls to enable automation. By using these API calls, theupstream VNFM and NFVO can automate all aspects of the VNF lifecycle. Examples include VNFdeployment from a catalog, tenant network consumption, power operations, and VNF decommissioning.

Host Affinity and Anti-Affinity Policy

A vCloud Director system administrator can create groups of VMs in a resource pool, then use VM-Hostaffinity rules to specify whether members of a VM group should be deployed on members of a vSpherehost DRS Group. vCloud Director VM-Host affinity rules provide vCloud Director system administratorswith a way to specify how vSphere DRS should place VMs on hosts in a resource pool. VM-Host affinityrules can be useful when host-based licensing requires VMs that are running certain applications to beplaced on hosts that are licensed to run those applications. They can also be useful when VMs withworkload-specific configurations require placement on hosts that have certain characteristics such asacceleration.

DRS Host Groups for Placement

A host group is a vSphere host DRS group. The vSphere administrator must create host DRS groups in aresource pool mapped to a Provider VDC before they can be used in vCloud Director VM Host affinityrules. This will help to place workload on particular hosts cluster at power on once they are onboarded.


VMware, Inc. 75

Availability and Disaster RecoveryBusiness continuity is supported by the vCloud NFV platform across its management, control, and dataplane components. This section discusses the available designs and recovery considerations.

AvailabilityAll vCloud NFV platform components implement a high availability design by default. In addition, VNF'scan take advantage of platform capabilities to extend their availability in compute, storage, andnetworking.

vSphere High AvailabilityRedundancy with vSphere uses VM level replicas in conjunction with the VNF high availabilityarchitecture. vSphere HA can restart a VM on another host in the event of host failure thus providingredundancy for a VNF and VNFC pair. vSphere HA can be fully automated without the need for manualintervention for failure and recovery.

Figure 8‑17. vSphere High Availability

ESXi

Host Host Host

ESXi

Restart

ESXi

VMware NSX-T Data Center AvailabilityNSX-T Data Center by default provides high availability to VNFCs in the overlay network. NIC Teamingand protocols such as Equal-Cost Multipath (ECMP), Graceful Restart, and Link Aggregation Group(LAG), provide redundant connectivity. The NSX-T architecture also separates management, control, anddata plane traffic to further optimize service availability.

VMware vSANvSAN is fully integrated in vSphere and provides policy-driven fault handling with platform awarenesssuch as chassis and rack affinity to store object replicas. The virtual storage is integrated with vRealizeOperations so that in case of failure, the failed VM can be cloned and spun up automatically. StoragevMotion can be used to perform live migration of virtual machine disk files (VMDK) within and acrossstorage arrays by maintaining continuous service availability and complete transaction integrity at thesame time.


VMware, Inc. 76

Disaster RecoveryIn the event of a failure, the site is recovered by executing an automated recovery plan. Data replicationacross protected and failover zones is necessary to recover the state of the site.

vSphere ReplicationvSphere Replication replicates virtual machine data between data center objects within a single site oracross sites. vSphere Replication fully supports vSAN. It is deployed as a virtual appliance in theManagement Pod to provide a Recovery Point Objective (RPO) of five minutes to 24 hours.

When executing a disaster recovery plan, RPO and Recovery Time Objective (RTO) are the mostimportant aspects that must be considered. RPO is the duration of the acceptable data loss and it isfulfilled by the replication technology. RTO is a target duration with an attached SLA, during which thebusiness process must be restored. It includes the time for the recovery and service readiness, in a stateof normal business operation.

vSphere Replication provides the ability to set the RPO, however RTO is application-dependent.

Site Recovery ManagerSite Recovery Manager provides a solution for automating the recovery and execution of a disasterrecovery plan in the event of a disaster in a data center. When a catastrophe occurs, components in theManagement Pod must be available to recover and continue the healthy operations of the NFV-basedservices.

To ensure robust business continuity and disaster recovery, network connectivity between the protectedand recovery sites is required, with enough bandwidth capacity to replicate the management componentsby using vSphere Replication. Each site must have an instance of vCenter Server that governs theManagement Pod and its ESXi hosts, and a Site Recovery Manager server and vSphere Replicationappliance to orchestrate the disaster recovery workflows and replicate content across the sites. Theprotected site provides business critical services, while the recovery site is an alternative infrastructure onwhich services are recovered in the event of a disaster.

Inventory Mappings

Elements in the vCenter Server inventory list can be mapped from the protected site to their vCenterServer inventory counterparts on the recovery site. Such elements include VM folders, clusters orresource pools, and networks. All items within a single data center on the protected site must map to asingle data center on the recovery site.

These inventory mapping details are used across both the protected and recovery sites:

n Resource mapping maps cluster objects on the protected site to cluster objects on the recovery site.

n Folder mapping maps the folder structures like data centers or VM folders on the protected site tofolder structures on the recovery site.

n Network mapping maps the management networks on the protected site to management networks onthe recovery site.


VMware, Inc. 77

Protection Groups

A protection group is a group of management components at the protected site that can failover togetherto the recovery site during testing and recovery. All protected management components are placed withina single protection group.

Recovery Plans

Recovery plans are the run books that are associated with a disaster recovery scenario. A recovery plandetermines which management components are started, what needs to be powered down, which scriptsto run, the startup order, and the overall automated execution of the failover.

A complete site failure is the only scenario that invokes a disaster recovery. There is no requirement forrecovery plans to handle planned migrations or to move a single failed application within the managementcluster. A single recovery plan is created for the automated failover of the primary site, and the placementof management components into priority groups ensures the correct startup order.

VNF Recovery ConsiderationsEvery VNF vendor must provide a specific strategy for disaster recovery for any VNF managed directly bythe VNF Managers.

NSX Data Center for vSphere Coexistence with NSX-TData CenterAs NSX evolves from NSX Data Center for vSphere to its successor NSX-T Data Center, CSPs candeploy the vCloud NFV platform with two software-defined networking stacks simultaneously. CSPs candeploy NSX Data Center for vSphere in conjunction with NSX-T Data Center as part of vCloud NFV 3.0 oras a complement to a pre-existing vCloud NFV 2.1 deployment.

NSX Data Center for vSphere Interoperating with NSX-T DataCenter in a Greenfield DeploymentThis deployment option allows CSPs to deploy the vCloud NFV 3.0 platform by using both NSX DataCenter for vSphere and NSX-T Data Center networking stacks. The CSP can create a dedicated vSpherecluster for each networking stack and map it to vCloud Director logical tenancy constructs. The CSPshould create a dedicated Provider VDC for each networking stack, respectively NSX Data Center forvSphere and NSX-T Data Center.


VMware, Inc. 78

Table 8‑3. Greenfield vCloud NFV 3.0 Deployment

Building Block Design Objective

Management Pod n Single vCenter Server instance for both NSX Data Center for vSphere and NSX-T Data Centerstacks.

n Separate management and control planes for NSX Data Center for vSphere and NSX-T Data Center.n Single vCloud Director instance for both NSX Data Center for vSphere and NSX-T Data Center,

network and workload management.

Resource Pod n Dedicated NSX Data Center for vSphere and NSX-T Data Center resource pods.n Common vSphere versions across both stacks.n Dedicated Provider VDC for NSX Data Center for vSphere and NSX-T Data Center vSphere clusters.

Edge Pod n Dedicated NSX Data Center for vSphere and NSX-T Data Center Edge pods.

Note To deploy clusters based on NSX Data Center for vSphere, use the vCloud NFV 2.0 ReferenceArchitecture design guidelines and principles.

Figure 8‑18. NSX-T Data Center and NSX Data Center for vSphere Coexistence in GreenfieldDeployments

VMware vCloud NFV 3.0

Organization

NSX for vSphere Provider VDC

ESXi ESXi

Management Cluster

NSX - T Provider VDC

ESXi ESXi

Resource Cluster

ESXi

Edge Node

Edge Node

Edge Node

ESXi ESXi

Edge ClusterNSX-T Edge Cluster

Management Pod

ESXi ESXi

Resource Cluster

ESXi

Controller

NSX for vSphere Controller Cluster

ESG ESG

Edge

VNF - C VNF - C

Org VDC1

VNF - C VNF - C

Org VDC2

vSphere Distributed Switch N - VDS (S) N - VDS (E)

vCloud Director

vCloud Director

Controller

NSX for vSphereManager

NSX - TManager

vCenter Server+ Platform Services Controller

vCenter Server+ Platform Services Controller

ESXi ESXi

Edge Cluster

NSX Data Center for vSphere NSX-T Data Center

NFVI components such as vCenter Server treat both stacks as separate entities in terms of host clusters,resource pools, and virtual switches. As the top level VIM component, vCloud Director allows tenants toconsume the resources of either or both stacks by configuring organization VDCs backed bycorresponding provider VDCs. Each stack has a dedicated Resource and Edge Pods for compute andNorth-South connectivity respectively.


VMware, Inc. 79

NSX Data Center for vSphere Interoperating with NSX-T DataCenter in Existing vCloud NFV DeploymentsThis deployment model can serve CSPs who have existing vCloud NFV 2.1 deployments and are not in aposition to upgrade their entire platform to vCloud NFV 3.0. Such CSPs can complement their vCloudNFV deployments with advanced networking capabilities available through NSX-T Data Center andfeatures described in this reference architecture.

In this scenario, the CSP should upgrade only some of the platform components so that to enable theplatform to support both networking stacks, the existing NSX Data Center for vSphere based networkingas well as the enhanced NSX-T Data Center networking. The objective is to have a common vCloudDirector instance in both existing and vCloud NFV 3.0 deployment stack (NSX-T based).

This deployment is also referred to as a brown field deployment model.

Table 8‑4. Brown Field vCloud NFV 3.0 Deployment

Building Block Design Objective

Management Pod n Dedicated vCenter Server instances for each NSX Data Center for vSphere and NSX-T Data Centerstacks.

n Separate management and control planes for NSX Data Center for vSphere and NSX-T Data Center.n Single vCloud Director instance for both NSX Data Center for vSphere and NSX-T Data Center

networking and workload management.

Resource Pod n Dedicated Resource Pods for NSX Data Center for vSphere and NSX Data Center.n Disparate vSphere version for each stack.n Dedicated Provider VDC for the vSphere clusters of NSX Data Center for vSphere and NSX-T Data

Center.

Edge Pod n Disparate NSX-V and NSX-T edge pods for North-South communications.


VMware, Inc. 80

Figure 8‑19. NSX-T Data Center and NSX Data Center for vSphere in BrownfieldDeployments

vCloud Director

VMware vCloudNFV2.1

Management Pod

VMware vCloudNFV3.0

vCenter Server

Platform Services Controller

NSX for vSphere Manager

Controller

vCenter Server + Platform Services

Controller

NSX -T Manager

Controller

vCenter Server

Platform Services Controller

NSX for vSphere Manager

The CSP can start from existing vCloud NFV 2.1 components then upgrade only those components thatare required to support the deployment of NSX-T Data Center alongside NSX Data Center for vSphere.Once the required components are updated, the functionality of the VIM becomes identical to thegreenfield deployment described in NSX Data Center for vSphere Interoperating with NSX-T Data Centerin Greenfield vCloud NFV 3.0 Deployments. Workloads continue to run on the vCloud NFV platform withminimal maintenance window. This is because only a subset of the components is upgraded as opposedto the entire suite.Refer to the vCloud NFV 2.0 Reference Architecture design guides for existingdeployment of vCloud NFV 2.1.


VMware, Inc. 81

Virtual EPC for MobileBroadband Internet Access 9By using the vCloud NFV platform, CSPs are transforming their voice and data services to nextgeneration software-driven architectures. This chapter explores the mobile broadband transformationsdiscussing the design considerations for physical, virtual, and VNF deployment from an end-to-endperspective.

Telco data services for 4G today are delivered over the 3GPP Evolve Packet Core (EPC). The EPCbridges the user equipment (UE) over a Radio Access Network (RAN) to the core network creating trafficbearers to deliver mobile broadband with differentiated class of services. Various technologies influencethe service quality of the broadband service, from the air interface to overlay networks and physicaltransports.

The following figure is a conceptual view of the mobile network and EPC that illustrates an NFVtransformation in a software-defined data center.

Figure 9‑1. Conceptual Architecture for Virtual EPC

VRF

PE Block

Core Block

ISP Block

NSX to (S1-AP)

NSX to (S1-U)

Perimeter Edge

Data Plane

Control Plane

HSS

PCRF

MME

SGW

PGW

RAN

APN

EPC-APN S1-AP, SI-U

UE LTE

MPLS

Internet/External Network

VLAN

Tenant Overlays

S6a

S11

Gx

S5/S8

ECMP eBGP BFD

This chapter includes the following topics:

n Design Considerations

n Transformation with vCloud NFV

VMware, Inc. 82

Design ConsiderationsFor UE to have data services enabled, for example access to the internet, an IP tunnel needs to beestablished across the core domains and components of the mobile network topology. The conceptualarchitecture assumes a single tenant isolation, targeting the CSP's mobile retail customers. This sectiondiscusses components and interfaces that are relevant for the design.

Access Point NameIn the mobile network, the Access Point Name (APN) is a representation of an IP packet data networkand service type. This can be perceived as a basic concept of network slicing core to 5G mobilenetworks. In our example, a single APN is defined for the mobile broadband service. Once the end-to-endIP tunnels are set up, the APN represents a set of dual-homed UE IP addresses which are NAT’ed by theP-GW or other Carrier-Grade NAT (CGNAT) solutions for internet connectivity and routing. The trafficbearers are set up by using the S1-AP control interfaces, typically implemented as multiple physicalinterfaces over SCTP for high availability. The ingress data plane interfaces are implemented over the S1-U protocols. Additionally, the QoS Class Identifier (QCI) for LTE is set to 9 (Internet) priority for non-Guaranteed Bit Rate Bearer (non-GBR).

Backbone TransportThe backbone transport is typically established over MPLS, which is owned and operated by the CSP.Other metro area networks and super backbone providers exist, which are not shown for simplicity.

PE and Core RoutersThe PE and Core routers implement the label-VRF-VLAN translations, routing, and peering. The MPLSclass of service can be set to apply a packet classifier priority to packets in case of congestion. Forinternet the CoS EXP bits are typically set to “best-effort” or “expedited-forwarding” classification.Dedicated VRF’s are defined for the S1-AP and S1-U traffic streams.

ISP RouterThe ISP routers are typically Network-to-Network Interfaces (NNI) providing connectivity to the internetand other external networks, including operator to operator.

Perimeter Edge RouterThe Perimeter router is an NSX Edge which bridges the data center interconnect with the NFVinfrastructure. The PE router terminates the VRF’s that map to different VLANs and routed to the NSX T0Edge Node perimeter router.


VMware, Inc. 83

WorkloadsWhile there are other workloads involved to realize the virtual EPC, the following core network functionsare discussed:

n HSS . The master database for a user. It is the entity containing the subscription-related informationto support the network entities actually handling calls or sessions.

n Policy and Charging Rules Function (PCRF). It acts as a policy decision point for policy and chargingcontrol of service data flows or applications and IP bearer resources.

n Mobility Management Entity (MME). A control plane entity within EPS supporting mobility and sessionmanagement functions and interoperability with the circuit switched domain. It acts as the controlplane function between the UE and the core network.

n Serving Gateway (SGW). The gateway that terminates the interface towards E-UTRAN. It acts as thelocal mobility anchor for all data bearers and facilities inter-eNodeB handovers. The SGW can be splitinto an SGW-C and SGW-U across its control and user plane capabilities.

n Packet Data Network Gateway (PGW). It terminates the SGi interface towards the PDN. It isresponsible for assigning IP address to the UE, managing traffic bearers, flow control QoS by settingDSCP mapped to the bearer QCI, and flow-based charging. The PGW can be split into a PGW-C andPGW-U across its control and user plane capabilities.

Transformation with vCloud NFVThe conceptual design principles discussed in Virtual EPC for Mobile Broadband Internet Access areapplied in this section to transform the next-generation virtual EPC data center using vCloud NFVplatform.


VMware, Inc. 84

Figure 9‑2. Use Case Realization for Virtual EPC

Physical Routers

Tier-1

Telco Network

Tier-0 Tier-0

PCRFHSSMME SGW

ToR

eBGP With BFD


LS- S6a

Organization VDC (DP)

LS-Gx

Edge Nodes 1...N

Edge Nodes 1...N

Edge Node Cluster

Edge Pod

Resource Pod

N-VDS (E)

Organization

Organization VDC (CP)

LS-SGi- External

N-VDS (S)

Tier-1

LS-S11LS-S5

PGW

Provider VDC Provider VDC

LS-S1-U- External

LS-S1-AP-External

N-VDS (E) Overlay

Tier-0 Tier-0


Edge Node Cluster

Edge Pod

Physical LayersThe vCloud NFV platform interacts with the physical transports both at the core and ISP routing planes fornorth-south connectivity. The NSX-T Tier-0 edge node routers are used as this perimeter edge. Theedges can be partitioned into different edge clusters for the control (S1-AP) and data plane (S1-U) tocommunicate with the UE for traffic bearer setup, mobility, and session management.

Local internet breakout at the data center sites achieved by interfacing with the ISP block of routers. Inthe example above, the SGi interface of the data plane workloads, that is the P-GW, is directly connectedthrough VLAN to the ISP block for external internet connectivity.


VMware, Inc. 85

Load distribution and scale is managed through ECMP at the perimeter. Dynamic routing using eBGP andfailure detection using BFD capabilities reside on the Tier-0 edge nodes and available on the uplink only.Stateful services such as load balancer can be run as well on the Tier-0 Edge Nodes, however requiresactive-standby configuration.

Virtual LayersThe virtual design of the EPC service dictates a separation of resources across the control and dataplanes. The deployment design strategy could assume a distributed CUPS cloud topology whereby thedata plane functions would be deployed closer to the UE (for example a telco far edge site) and acentralized control plane hosted at the core data center. However, for simplicity this design assumes asingle cloud data center. The overlay network segments are set up for east-west traffic and connected tothe respective NSX Tier- 0 Edge Node clusters for S1-AP and S1-U interfaces.

This transformation scenario uses a single tenant design where a dedicated PvDC is setup for thefollowing data and control plane class of workloads.

n Data Plane. The data plane functions of the SGW-U and PGW-U are deployed in a dedicated clusterof resource, which maximizes the availability and performance. The resource reservations aremaximized to ensure availability (CPU and Memory). The workloads are also backed by and NSX-TEnhanced switch for packet processing acceleration by using CPU pinning and NUMA verticalalignment. The cluster is VLAN backed for higher through-put.

n Control Plane. The control plane functions such as the MME, PCRF, HSS and control plane functionsof the SGW, PGW (not shown in the diagram) are deployed in a dedicated cluster with less stringentperformance characteristics. Resource reservations can have more elasticity to allow for more clusterheadroom and on-demand scale up and scale out optimizations. The control plane functions aredeployed on a N-VDS Enhanced and N-VDS Standard non-accelerated switching fabrics. A localinstance of the HSS is shown, acting as a fast-access cache of subscriber and subscriptioninformation stored in the global HSS, that is the control plane function.

Service LayersThe virtual EPC service for mobile broadband internet access is delivered over a set of core 3GPP-defined components and interfaces. The VNFs are distributed across the virtual resource isolationboundaries defined with segregated networking for east-west and north-south communications. Noticethat the data plane intensive workloads are connected to control plane functions on the east-westnetworks.

Security profiles using micro-segmentation are defined and deployed at the east-west plane of the VNFs.This ensures inter-component communications are secured in this reference-point architecture andminimizes the risk from any rogue messaging floods.

With an analytics-enabled design the service components can be monitored for health, capacity, andproactively optimized to ensure acceptable SLAs.


VMware, Inc. 86

Virtual IMS for Mobile Multi-Tenancy Services 10CSPs are transforming their voice and data services into next-generation software driven architectures bythe using vCloud NFV platform. This chapter explores the voice telephony transformations discussing theconsiderations for physical, virtual, and VNF deployment design from an end-to-end perspective.

The telco voice services transformed to an all-IP packet switch transport with the deployment of LTE/4Gnetworks. The IP Multimedia Subsystem (IMS) 3GPP specification is used as the common core fabricwhich is independent of the access network. This makes it possible to use the same telephonyinfrastructure for fixed, mobile, and WiFi access networks. The radio access network transformation toLTE is able to deliver the QoS necessary to deliver high-quality voice to mobile devices. The followingfigure is a conceptual view of the mobile network and IMS core used to illustrate an NFV transformation ata software-define data center.

VMware, Inc. 87

Figure 10‑1. Use Case Realization for Virtual IMS

VRF

PE Block

Core Block

ISP Block

NSX to

Perimeter Edge

NSX to

Tenant Edge

Tenant Region A

Data PlaneMx/Mn

Mg/Mi

ISC

MW

DNS

Sh

Cx

P-CSCF (MGW)

Tenant Region A

Control Plane

I/S-CSCF

M/BGCF

TASa

P-CSCF

ENUM

HSS

RAN

APN

EPC-APN S1-AP, SI-U

UE LTE

MPLS

Internet/External Network

VLAN backend

ECMP eBGP BFD

eBGP BFD

Tenant Overlays

Gm

RTP


n Design Considerations

n Transformation with vCloud NFV

Design ConsiderationsFor the mobile networks, the end-to-end IP transport tunnels are established by the EPC as discussed inVirtual EPC for Mobile Broadband Internet Access. The IP transport bearers are used to deliver IMSservices over it. The conceptual architecture assumes an enterprise or B2B business model deliveringIMS capabilities to its MVNO partners with full multi-tenancy and SLA controls. This section discussescomponents and interfaces that are relevant for the design.


VMware, Inc. 88

Access Point NameThe APN is a representation of an IP packet data network and service type in the mobile network. Thiscan be perceived as a basic concept of network slicing core to 5G mobile networks. In our example,multiple APNs are defined for each tenant consumer of IMS services. Dedicated traffic bearers are set upfor the Gm and RTP (media) control and data plane interfaces respectively. Additionally, the QoS ClassIdentifier (QCI) for LTE is set to 1 (Conversational Voice) or 2 (Conversations Video) priority forGuaranteed Bit Rate Bearer (GBR).

Backbone TransportThe backbone transport is typically established over MPLS, which is owned and operated by the CSPs.Which in turn connects to other metro area and super backbone network providers.

PE and Core RoutersThe PE and Core routers implement the label-VRF-VLAN translations, routing, and peering. The MPLSclass of service can be set to apply a packet classifier priority to packets in case of congestion. For voicethe CoS EXP bits are typically set to “assured-forwarding” classification. Dedicated VRF’s are defined forthe Gm and RTP traffic streams.

ISP RouterThe ISP routers are typically Network-to-Network Interfaces (NNI) providing connectivity to externalnetworks, including operator to operator for voice services.

Perimeter Edge RouterThe Perimeter router is an NSX Edge which bridges the data center interconnect with the NFVinfrastructure. The PE router terminates the VRF’s which map to different VLANs and routed to the NSXT0 Edge Node perimeter router.

Tenant Edge RouterThe Tenant router is an NSX Edge which provides network and resource isolation and management foreach tenant consuming the IMS services.

WorkloadsWhile there are other workloads involved to realize the virtual IMS, the following core network functionsare discussed:

n Call Session Control Functions (CSCF). CSCF are part of the session and control layer of the IMScore that are based on the Session Initiation Protocol (SIP). The proxy, interrogating, and servingCSCF functions together provide the call management, session management, routing, and securitycapabilities.


VMware, Inc. 89

n Media and Break Out Gateway Control Functions (M/BGCF) Used to bridge voice sessions betweenIMS and a circuit switched phone in the PSTN or PLMN domain.

n Telephony Application Server (TAS). It provides a control plane function extension to integrate third-party value-added services such as call forwarding, multi-party calling, caller-id, presence,messaging, and so on.

n ENUM. A system that translates standard phone number into internet address such as SIP and H.323. This allows for a user to be reachable in other ways (such as SIP) instead of circuit switchedphone call.

n HSS. The master database a given user. It is the entity containing the subscription-related informationto support the network entities actually handling calls or sessions.

Transformation with vCloud NFVThe vCloud NFV platform transforms the next-generation multi-tenant Virtual IMS data center .Thissection applies the design principles discussed to the design of Virtual IMS for Mobile Multi-TenancyServices


VMware, Inc. 90

Figure 10‑2. Use Case Realization for Logical IMS

Physical Routers

Telco Network

Tier-0 Tier-0

TASENUM M/BGCFI/S-CSCF P-CSCF

ToR

eBGP With BFD


Organization VDC (DP)

Edge Nodes 1...N

Edge Node Cluster

Provider External Network

Edge Pod

Resource Pod

N-VDS (E)

Organization

Organization VDC (CP)

LS-MGW

N-VDS (S)

Tier-1

Mx/Mn

LS-Mx/Mn

P-CSCF

Provider VDC Provider VDC

LS-DNS

DNS

DNS Sh

LS-Sh

Cx

LS-Cx

Mw

LS-Mw

Mg/Mn

LS-RTP

HSS

Physical LayersThe vCloud NFV platform interacts with the physical transports both at the core and ISP routing planes forNorth-South connectivity. The NSX-T Tier-0 Edge Node routers are used as this perimeter edge. Theedge is used to multipath Gm and RTP traffic from upstream core switching and routing fabric to thedownstream tenant-specific NSX-T Tier-0 Edge Node clusters.


VMware, Inc. 91

Local network-to-network interface (NNI) at the data center site is established by interfacing with the ISPblock of routers. In the above diagram, both the SIP signaling and RTP media streams would traverseover the mobile packet core bearer SGi interfaces (not shown in the diagram).

Load distribution and scale is managed through ECMP at the perimeter. Traffic from the perimeter edgesis routed to each tenant's NSX Tier-0 edge cluster. Dynamic routing using eBGP and failure detectionusing BFD capabilities reside on the Tier-0 edge nodes and is available on the uplink only. Statefulservices such as NAT, firewall , and load balancer can be run as well on the Tenant Tier-0 Edge Nodes,however this requires an active-standby configuration.

Virtual LayersThe virtual design of the EPC service requires a separation of resources across the control and dataplanes. The deployment design strategy could assume a distributed CUPS cloud topology whereby thedata plane functions would be deployed closer to the UE (for example a telco far edge site) and acentralized control plane hosted at the core data center. However, for simplicity this design assumes asingle cloud data center. The overlay network segments are setup for east-west traffic and connected tothe respective Tenant NSX T0 Edge node clusters for Gm and RTP interfaces.

This transformation scenario uses a multitenant tenant design with tenant dedicated PvDC setup for thefollowing data and control plane class of workloads.

n Data Plane. The data plane functions of the P-CSCF (media) are deployed in a dedicated cluster ofresource, which maximizes availability and performance. The resource reservations are maximized toensure availability (CPU and Memory). The workloads are also backed by and NSX-T Enhancedswitch for packet processing acceleration using CPU pinning and NUMA vertical alignment. Theexternal network connectivity for the cluster is VLAN backed for higher through-put.

n Control Plane. The control plane functions such as the P/I/S-CSCF, TAS, M/BGCF, HSS and ENUMare deployed in a dedicated cluster with less stringent performance characteristics. Resourcereservations can have more elasticity to allow for more cluster headroom and on-demand scale upand scale out optimizations. The control plane functions are deployed on a N-VDS Standard non-accelerated switch. A local instance of the HSS and ENUM is shown; acting as a fast-access cacheof subscriber and subscription information stored in the global HSS - the control plane function.

Using multiple MVNOs as an example, each would be a tenant with their own instance of IMS workloads.A dedicated set of PVDCs are created for each tenant to guarantee network and resource isolation, SLAagreements, and scale.

Service LayersThe virtual IMS service for mobile broadband internet access is delivered over a set of core 3GPP-defined components and interfaces. The VNFs are distributed across the virtual resource isolationboundaries defined with segregated networking for east-west and north-south communications.


VMware, Inc. 92

Security profiles using micro-segmentation are defined and deployed at the east-west plane of the VNFs.This ensures that inter-component communications are secured in this reference-point architecture andminimizes the risk from any rogue messaging floods.

With an analytics-enabled design the service components can be monitored for health, capacity, andproactively optimized to ensure acceptable SLAs.


VMware, Inc. 93

Analytics-Enabled ReferenceArchitecture 11CSPs can enable the vCloud NFV platform for day 1 and day 2 operations after the platform is deployedin the cloud provider topology. The platform is integrated with an operations management suite thatprovides capabilities for health monitoring, issue isolation, security, and remediation of the NFVI andVNFs.

The NFVI operations management framework defines and packages a five-step approach to make day 1and day 2 workflows operational.

n Onboard service operations.

n Service launch and monitoring.

n Dynamic optimizations.

n Issue isolation.

n Demand planning and expansion.

The integrated operational intelligence will adapt to the dynamic characteristics of the NFV infrastructureto ensure service quality and issue resolution. Some of the key characteristics include:

n Dynamic resource discovery. Distributed and complex topologies together with workloads in motionrequire dynamic resource and service discovery. The platform provides continuous visibility overservice provisioning, workload migrations, auto-scaling, elastic networking, and network-slicedmultitenancy that spans across VNFs, hosts, clusters, and sites.

n SLA management. Continuous operational intelligence and alert notifications enable proactive serviceoptimizations, capacity scale-out or scale-in, SLA violations, configuration and compliance gaps, andsecurity vulnerabilities.

n Remediation. Reduced MTTU and timely issue isolation for improved service reliability andavailability. Prioritized alerting, recommendations, and advanced log searching enable isolation ofservice issues across physical and overlay networks.

n Security and policy controls. Multivendor services operating in a shared resource pool can createsecurity risks within the virtual environment.

n Ability to profile and monitor traffic segments, types, and destination to recommend security rulesand policies for north-south and east-west traffic.

VMware, Inc. 94

n Identification of security policy and configuration violations, performance impacts, and trafficroutes.

n Capacity planning and forecasting. New business models and flexible networks demand efficientcapacity planning and forecasting abilities in contrast to the traditional approach of over-provisioningthat is costly and unrealistic.

The framework continuously collects data from local and distributed agents, correlating, analyzing andenabling day 2 operations. The analytical intelligence can be also queried and triggered by third-partycomponents such as existing assurance engines, NMS, EMS, OSS/BSS, and VNF-M and NFV-O forclosed loop remediation.

Figure 11‑1. Analytics-Enabled Reference Architecture

...

Topology Performance Optimization

Capacity Causality Compliance

Remediate Configuration Efficiency

Performance Security

Compliance Planning

Search Configuration API

Dashboard

NotificationvRealize Log Insight

Log Analysis Machine Learning RCA Clustering Search

Discovery Correlation Analytics Reporting Alerting

IPFIXRouterSwitch

PNF

vCenterServer

NSX

vCloudDirector

SNMPSyslog

vRealize Operations vRealize Network Insight

vSAN Collectors

CSPs can deploy the operations management components in the Management Pod and centralize themacross the cloud topology, assuming that inter-site latency constraints are met.

n vRealize Operations Manager collects compute, storage, and networking data providing performanceand fault visibility over hosts, hypervisors, virtual machines, clusters, and site.

n vRealize Log Insight captures unstructured data from the environment, providing log analysis andanalytics for issue isolation. Platform component logs and events are ingested and tokenized, andmined for intelligence so that they can be searched, filtered, aggregated, and alerted.

n vRealize Network Insight provides layer 2, 3, and 4 visibility into the virtual and physical networks andsecurity policy gaps. The engine is integrated with the NFVI networking fabric, ingesting data thatranges in performance metrics, device and network configuration, IPFIX flow, and SNMP. It discoversgaps in the network traffic optimization, micro-segmentation, compliance, security violations, trafficrouting, and performance.


n Management Pod Extensions

n Monitoring the Infrastructure


VMware, Inc. 95

n Predictive Optimization

n Capacity and Demand Planning

n Cost Management

n Closed Loop Automation for Higher Availability

Management Pod ExtensionsThe analytics-enhanced reference architecture enriches the Management Pod with the vRealizemanagement components to provide infrastructure assurance capability.

Figure 11‑2. Analytics Extension to Management Pod

vCenter (s)

Operations (data)

Operations(master)

Operations(replica)

Operations(data)

Log Insight(master)

Log Insight(worker)

Log Insight(worker)

Orchestrator(master)

Orchestrator(worker)

Network Insight(platform)

Network Insight(proxy)

Network Devices

NSX Controller

NSX ManagervCenter Server (s)

vCloudDirector

Logs

ComponentsThe operations management components are deployed as a centralized function that is capable of day 1and day 2 operations spanning the CSP's deployment topology. The data collection architecture isspecific to each operations management component with a centralized single pane for monitoring,reporting, troubleshooting, and closed-loop automation.

vRealize Operations ManagerThe virtual environment relies on a monitoring solution that is able to collect data regarding its health,capacity, availability, and performance. vRealize Operations Manager provides a robust and integratedmonitoring platform that resides at the center of the NFV environment. It monitors the virtual environmentand collects data about its health, capacity, availability, and performance. vRealize Operations Managerserves as the single pane of glass to the NFV environment.


VMware, Inc. 96

vRealize Operations Manager extends and collects information through management packs. Thecollected information is filtered for relevancy, analyzed, and presented in customizable dashboards. Itexposes an API that retrieves performance and health data about NFVI and the virtual resources of theVNF instance.

The design of the operations management components is based on centralized management andcollection, with optional remote collection for a distributed topology. vRealize Operations Managersupports HA across the various components. HA creates a master replica for the vRealize OperationsManager master node and protects the management functions. In smaller deployments, the master nodecan also act as a data node. In larger deployments, data nodes host adapters that are responsible forcollecting data and can be scaled to meet additional capacity. To enable HA, at least one more data nodemust be deployed in addition to the master node. Anti-affinity rules should be used to keep nodes onspecific hosts.

vRealize Operations Manager is installed in the Management Pod in both Two-Pod and Three-Poddesigns. Depending on the number of metrics that are collected over time, additional storage capacityand compute capacity might be required. Adding more hosts to the management cluster or more storageis sufficient to address the growing storage needs of vRealize Operations Manager.

By default, VMware offers Extra Small, Small, Medium, Large, and Extra Large configurations duringinstallation. The CSP can size the environment according to the existing infrastructure to be monitored.After the vRealize Operations Manager instance outgrows the existing size, the CSP must expand thecluster to add nodes of the same size. See the vRealize Operations Manager Sizing Guidelines .

vRealize Log InsightCSPs can use vRealize Log Insight to collect log data from ESXi hosts and data about server events,tasks, and alarms from vCenter Server systems. vRealize Log Insight integrates with vRealize OperationsManager to send notification events. Because vRealize Log Insight collects real-time unstructured data,the CSP can configure all elements in the NFV environment to send their log data to vRealize Log Insight.This log aggregation provides a single log collector for the entire NFV environment.

vRealize Log Insight ingests syslog data from the physical and virtual NFVI components to delivermonitoring, search, and log analytics. It builds an index for analytics purposes by automatically identifyingstructure from machine-generated log data including application logs, network traces, configuration files,messages, performance data, and system state dumps. Coupled with a dashboard for stored queries,reports, and alerts, vRealize Log Insight assists the CSP in root cause analysis and reduction in MTTR.All NSX-T Data Center Manager syslog information, distributed firewall logs, and NSX-T Data CenterEdge syslog information is sent to vRealize Log Insight.

The vRealize Log Insight API provides programmatic access to vRealize Log Insight functionality and toits datastore. As a result, the OSS/BSS systems or MANO components can integrate with vRealize LogInsight to gain further insight into the system events and logs.

vRealize Log Insight is deployed by using a single cluster configuration, which consists of a minimum ofthree nodes leveraging the Log Insight Integrated Load Balancer (ILB). A single log message is onlypresent in one location within the cluster at a time. The cluster remains up and available to ingest dataand serve queries during a temporary unavailability of a single node.


VMware, Inc. 97

https://kb.vmware.com/s/article/54370

vRealize Log Insight provides preset VM sizes that the CSP can select from to meet the ingestionrequirements of their environment, Extra Small, Small, Medium, Large, and Extra Large configurations.These presets are certified size combinations of compute and disk resources, though extra resources canbe added afterward. See the VMware Documentation for sizing details.

vRealize Network InsightvRealize Network Insight provides operations for software-defined networking and security across virtualand physical infrastructure with micro-segmentation planning that can be scaled to thousands of VNFs.vRealize Network Insight is installed in the Management Pod in both Two-Pod and Three-Pod designs.

The vRealize Network Insight architecture consists of a platform VM, a proxy VM, and data sources. Theplatform VM provides analytics, storage, and a user interface to the data. The proxy VM, or the collector,collects data by using various protocols such as HTTPS, SSH, CLI, and SNMP, depending on the sourceand the configuration. A variety of data sources are supported, including vCenter Server, NSX-T DataCenter, firewalls, and various switch vendors.

The platform VM is deployed as a single cluster to provide high availability and scale. A minimum of threeplatform VMs are required in the cluster. The proxy VMs are used to collect data and can be deployed ina single data center or distributed across sites. Depending on the amount of data that will be collected,typically CSPs need one or more proxy VMs.

New vRealize Network Insight provides the following enhanced capabilities for NSX-T Data Center:

n Support for Tier-0 and Tier-1 routing for multi-tenancy and micro-segmentation along the East-Westnetwork.

n Provides NSX-T Data Center inventory with a number of nodes, layer2 networks, firewall rules andlogical routers.

n Support for NSX Distributed Firewall (DFW) generated IPFIX flows and firewall rule recommendationsto the micro-segmentation applications.

n Support for monitoring overlay and underlay virtual machine networks managed by NSX-T DataCenter.

n End to End Virtual machine network visibility for multiple ECMP paths between routers.

Ensure that the system meets the minimum hardware configurations to install vRealize Network Insight.See VMware Documentation for sizing details.

vRealize OrchestratorVMware vRealize Orchestrator is a development-and process-automation platform that provides a libraryof extensible workflows. Orchestrator workflows run on objects that are exposed through plug-ins andcustom scripting to interact with any component that is reachable through an API. By default, a VMwarevCenter plug-in is provided to orchestrate tasks in the cloud infrastructure environment. The CSP can useOrchestrator workflows to define and run automated configurable processes thus creating a frameworkfor closed-loop automation.


VMware, Inc. 98

https://docs.vmware.com/en/vRealize-Log-Insight/4.6/com.vmware.log-insight.getting-started.doc/GUID-284FC5F4-B832-47A7-912E-D407A760CAE4.html#GUID-284FC5F4-B832-47A7-912E-D407A760CAE4

https://docs.vmware.com/en/VMware-vRealize-Network-Insight/3.7/com.vmware.vrni.install.doc/GUID-F7A2A1D5-E599-4870-ACDD-F9260AFE665B.html

vRealize Orchestrator is integrated with vRealize Operations Manager through a management pack thatprovides workflows for automating the cloud infrastructure environment and for orchestration of third-partycomponents as well as management functions.

The vRealize Orchestrator is highly available and configured as a single cluster of multiple virtualappliance instances. The appliance is registered with a vCenter Single Sign-On service by using thevSphere Authentication mode. The appliance also requires a shared database instance.

Ensure that the system meets the minimum hardware configurations to install vRealize Orchestrator. Seethe VMware Documentation for sizing details.

Enabling Analytics with vRealizeThe vRealize components of the vCloud NFV platform are integrated in the platform and configured toreport and trigger analytical intelligence. The various configurations and extensions that follow will helpthe CSP to get started with the analytics-enabled reference architecture.

vRealize OperationsvRealize Operations is configured with the adapters that are necessary to start collecting data from theinfrastructure. The following solutions should be considered:

n vCenter Adapter. vSphere connects vRealize Operations Manager to one or more Resource andManagement vCenter Server instances. The system collects data and metrics from those instances,monitors them, and runs actions in them.

n End-Point Operations Management. The End-Point Operations Management solution is used togather operating system metrics and to monitor availability of remote platforms and applications.

n vSAN Adapter. In a production environment, operations management for vSAN can be provided byusing dashboards to evaluate, manage, and optimize the performance of vSAN and vSAN-enabledobjects in the vCenter Server system.

n vCloud Director Adapter.The vRealize® Operations Management Pack™ for vCloud Director includesdashboards to provide visibility to vCloud Director deployments, such as Compute Infrastructure,Tenants, and Storage for easy monitoring and management.

NSX Adapter.This provides the CSPs with insight to the health of the network infrastructurecomponents such as NSX Manager and the network devices, both virtual and physical.

n vRealize Orchestrator Solution. This management pack enables an adapter to communicate with thevRealize Orchestrator workflow automation engine.

vRealize Network InsightvRealize Network Insight can be configured to monitor all networking-related components in the NFVI.vRealize Network Insight can connect to the NFV components that are related to networking and securityand provide insights into the networking segments that are used to deploy and manage the vCloud NFVEdition platform. The management VLANs, external VLANs, and Overlay segments are all available formonitoring and diagnostics.


VMware, Inc. 99

https://docs.vmware.com/en/vRealize-Orchestrator/7.4/com.vmware.vrealize.orchestrator-install-config.doc/GUIDEB0FB317-BB55-469A-9546-D274E2B46D70.html

vRealize Network Insight uses the following data sources:

n vCenter Server. Both Resource and Management vCenter Server instances.

n NSX Manager. The management plane of the NSX-T Data Center networking.

n IPFIX. For flow analysis, which can be enabled within vCenter Server.

n Network devices. Physical devices such as Dell switches, Cisco Nexus and Catalyst switches, Arista,Juniper Networks, Hewlett Packard Enterprise, Brocade, and Palo Alto Networks switches.

vRealize OrchestratorvRealize Orchestrator is integrated into vRealize Operations Manager as a management pack to providebidirectional communication with the Orchestrator workflow engine. The management pack is deployed invRealize Operations Manager and configured to point and authenticate to the vRealize Orchestratorinstance.

VMware vRealize Operations Management PackvRealize Operations Manager can be extended through management packs to monitor VNFs based onvirtual machines.

VMware vRealize Operations Manager collects structure data from various vCloud NFV Editioncomponents, including gathering data from adapters that are connected to external components. For thismechanism to work, vRealize Operations Manager is configured to communicate with data sources byusing an authorized user account for the respective components. If the user account has limited access toobjects in the source server, it sees only the data for which the account has permissions. At a minimum,the user account must have read privileges across the objects from which it will collect data. A collectionof management packs is available on VMware Solution Exchange.

To minimize the traffic between vCenter Server and the vRealize Operations Manager, the vCenter ServerAdapter is installed with a five-minute collection interval.

Out-of-the-box, vRealize Operations Manager does not monitor VNF service availability or VNF internalkey KPIs. The VNF Manager derives this information through direct interaction with the respective VNFs.To extend the operations management functionality to the VNFs, VNF vendors can create custommanagement packs for vRealize Operations Manager. Management pack development requires anunderstanding of the vRealize Operations Manager inventory model and the management functions thatmanagement packs implement. These include auto-discovery and monitoring. More information isavailable at Endpoint Operations Management Agent Plugin Development Kit .

VMware vRealize Log Insight Content PackvRealize Log Insight gathers log events natively from multiple syslog data sources and through specialcontent packs. Specific dashboards can be customized to perform log analytics and alerts. For additionalinformation about vRealize Log Insight solutions, see VMware Solution Exchange.


VMware, Inc. 100

https://marketplace.vmware.com/vsx/

https://docs.vmware.com/en/vRealize-Operations-Manager/6.5/endpoint-operations-management-agent-65-plugin-development-kit.pdf

https://marketplace.vmware.com/vsx/

Monitoring the InfrastructureThe vCloud NFV Edition operations management components are pre-configured with defaultdashboards, alerts, actions, and profiles. To make day 2 operations effective, additional steps foronboarding service operations are required so that to enhance the monitoring and remediation of thecloud infrastructure environment.

Consider the following topology as an illustration of a multi-tenant service offer of the mobile packet core:

Figure 11‑3. Infrastructure Monitoring Reference Topology

Common Management

Underlay Fabric

TenantOverlay

VMware vCloud NFV Ops Management

vRealizeOperationsManager

vRealizeNetworkInsight DNS

SGW PGW MME HSS PGW-CP

MME HSS PGW-CP

APP-1 APP-2

SGW PGW

Customer B

Customer A

DC G/W DC G/W

S1-MMES1-US5S11SGi

Edge Cloud

NSX Edge

Core Cloud

NSX Edge

vRealizeLog

Insight

vRealizeOrchestrator

The example topology places the data plane components of the packet core in the edge cloud whilecontrol plane functions in the core cloud, including the operations management components.

n Access controls (RBAC).The CSP administrator should start by creating the user profiles, roles, andpermissions within vRealize Operations Manager for their tenants. This way, tenant users are able toview dashboards and performance or fault information that the CSP administrator selects to expose.

n Service definition. Once the VNFs are onboarded for the tenant, the CSP administrator can createservice definitions within vRealize Operations Manager. In the example, an evolve packet coreservice definition is created by using a Service Group construct. If the VMs for the various VNFs areplaced in vCenter Server folders, the CSP administrator can map them to a service group or selectindividual VMs for that tenant. The underlying compute, networking, and storage infrastructurecomponents are auto-discovered for each VM. This includes parent containers such as clusters,datastores, and data center. Once the CSP administrator creates the service definitions, they canmonitor the various performance counters, health, risk, efficiency, anomalies, and faults for thedelivered services. Similarly, the CSP administrator can create service definitions in vRealize NetworkInsight to create the networking and security views of the tenant services.


VMware, Inc. 101

n Dashboards. All components of the operations management solution provide default and

customizable dashboards. By using the library of widgets, the CSP administrator can create views fortime-series, object-relationships, topologies, alerting, heat-maps, capacity, and so on.

n Default Alerts. A key component for effective monitoring and remediation or the virtual environment isthe rich customizable alerting framework that is available in the platform. Along with the numerousout-of-the-box alerts in the platform, the CSP administrator can also create service-centric alerts byusing complex rules and machine learning across multiple VNFs. The framework allows the CSPadministrator to create alerts on target object types and specify the impact depending on the risk,efficiency, and health. They can also create complex symptom definitions and a recommendation.The recommendation can be an informational message on how to remediated the alert, which can betied to a semi-supervised or automated action the system can take. Timely action is key to ensurethat QoS and SLA guarantees are met.


VMware, Inc. 102

Action workflows. Action workflows trigger remediation for issue avoidance. Workflows can be internalto the NFVI layer or trigger an external element such a VNF-M or NFV-O. A transient spike in thepacket core utilization might trigger a rebalance of the Packet Gateway (PGW) to mitigate congestionon the host. A workflow trigger to the DRS can be leveraged in such a scenario. For externaltriggering, vRealize Operations Manager is integrated with vRealize Orchestrator. Custom workflowsand scripts can be defined in vRealize Orchestrator. They appear as actions in vRealize OperationsManager that can then be mapped in the alerting framework.

Predictive OptimizationIn vCloud NFV Edition, resource contention is evaluated based on continuous monitoring. vRealizeOperations Manager uses dynamic thresholds to monitor and analyze the behavior of VNFs. Statisticalconfidence bands are created for each metric and its time-series, indicating an upper and lower boundaryfor it. The boundaries provide visibility on resource consumption of various VNFs operating in the NFVIenvironment. The VNF intelligence is utilized by DRS to predict future contention and capacity exhaustand rebalance the workloads in a timely manner.

In addition to the dynamic utilization and forecasting intelligence data, policies can be defined for differentclasses of workloads, for example:

n Host affinity/anti-affinity rules, ensuring if tagged VMs are placed together or not on the same host.

n Latency Sensitivity is set to high for data plane intensive workloads, pinning CPU.

n NUMA affinity to ensure that the VM is assigned to a dedicate node.

n Current and target host CPU profile alignment to avoid mismatch (Enhanced vMotion Capability).

Distributed DRS can be used to balance VNF components based on their performance needs andcapacity availability, eliminating resource contention that might occur. Forecasts are generated every fiveminutes, which allows DRS to predict and act on any possible imbalances. The optimization can beenabled in a supervised or non-supervised control.

Capacity and Demand PlanningTo avoid the risk of capacity exhaust and unplanned infrastructure costs, capacity and demand planningare a critical part of the data center operations. vRealize Operations Manager provides a capacitymanagement solution that continuously monitors and advises on the health of the NFVI environment.

Right-SizingOver-provisioning the infrastructure can be avoided by right-sizing the resources. Capacity risk tolerancecan be set for a trial environment that is different from the production environment. You can plan forinfrastructure procurement or migrations based on available capacity.


VMware, Inc. 103

As the CSP considers the importance of their environment and workloads, they can set the capacity risklevel to Conservative by using the upper bound projection of the forecast for high-performanceworkloads. In contrast, they can set the capacity risk to Aggressive policy by using the forecast for non-critical workload environment.

Head-room capacity can also be defined on the cluster to ensure that there is space for unexpectedspikes in traffic and performance for critical workloads.

ForecastingDemand planning might be challenging, especially if the CapEx spend should be balanced over time. Theforecasting engine can be used to extrapolate existing utilization of workload CPU, memory , and storageacross VMs, clusters, and data centers. The just-in-time capacity forecasts can be used to plan for newinfrastructure or rebalance with predictable intelligence.


VMware, Inc. 104

New Service OrderAs an operations manager, when a new tenant order needs to be fulfilled you need to make decisions onwhere to place the new workloads without impacting current operations. Various approaches can be usedin the data center clusters including making room for the new order by moving workloads, addingadditional resources to the existing workloads, and adding new hosts to the cluster. You can use vRealizeOperations What-If analysis tool to emulate such scenarios by using the existing workloads as the targetprofile. You can model additional CPU, memory, storage to existing workloads to calculate new capacityexhaust. You can emulate new instance of similar workloads to estimate if there is enough capacity in thecluster.

Cost ManagementvRealize Operations Manager provides visibility into cost of operations providing business managementvisibility on workloads, resources, and data centers that are used to deliver the services. Operations andplanning teams can better manage their resource cost and consumption based on network function toresource ratio's and effective ROI in service delivery.

n Reclaim Resources. Resource consumption is continuously monitored for utilization. This way VMsthat are idle, underutilized, powered-off and so on, can be reclaimed to free up resource for costavoidance.

n Densification. Capacity consolidation can be another way to recover cost offsets that are caused byover-provisioning. Certain class of non-performance workloads can be tagged, and policy enabled forhigher consolidation, allowing the operator to utilize the resource cluster with a dense capacity risk.


VMware, Inc. 105

n Optimize licenses. As with densification, VMs that have license restrictions that are tied to a cluster,can be tagged and enabled with policies. This way both the initial placement and on-going DRSbalancing will respect the policy to ensure that such VMs are placed on the same cluster.

vRealize Operations Manager additionally provides an integrated cost management function for datacenters. The cost of running workloads in a private data center can be compared to public cloud costs.Depending on the characteristics of the workload, you can consider the following options:

n Migrate non-mission critical workloads to a public cloud.

n Burst capacity into public clouds to meet transient spikes.

n Use public clouds for upgrades and other maintenance activities.

Closed Loop Automation for Higher AvailabilityAutomation is key to the cloud infrastructure platform, ensuring that it is dynamic and adaptive to theworkloads and consumption models. During operation, workloads experience an increase in trafficdemand, congestion across resources and virtualization layers, therefore a proactive optimizationapproach is very effective.

The vCloud NFV Edition analytics framework provides key capabilities to develop closed-loop automation.This includes the ability to define intent-based operational policies and alerts as well as actions to enforcethe alerts. You can use the framework to manage performance, resources and workload life cycle runningwithin the cloud infrastructure.

In the next generation networks, the workloads will undergo changes in demand and utilization requiringthe optimization of allocated resource pools. Infrastructure resources (CPU, memory, storage, network)might need to be added or reclaimed and their reservations modified. Workloads might need to bemigrated to a less loaded host or cluster that has available resources. Workload life cycle management isequally critical to ensure restart, including network function component boot sequence, scale, and healoperations in accordance to the operational policies that are defined. The closed loop automation for suchchallenges can be addressed at the NFV infrastructure layer and at the service orchestrator and VNF-Mlayers.

vRealize Orchestrator together with vRealize Operations Manager provides the framework for closed-loopautomation. They separate the policy, alert definition and management from the decision point andenforcement. Actions can be triggered back to the NFV infrastructure or directed to third-party externalsystems.


VMware, Inc. 106

Figure 11‑4. Closed-Loop Automation and Workflows

Status

SA EMS VNF-M NFV-O SA

vRealizeOperations

vRealizeOrchestrator

Alert Action

Analytics

WorkflowUtilization

Data

Trigger

Symptom: High CPUAlert: VNF-Auto-ScaleAction: {nfvo-scale} Workflow: {nfvo-scale}

Params

Script + Params

When the alert is raised, the vRealize Orchestrator workflow is triggered, notifying the north-boundcomponent recipient which will then carry out the remediation appropriate for that application it ismanaging.


VMware, Inc. 107

Authors and Contributors 12The following authors co-wrote this paper:n Arunkumar Kodagi Ramachandra, Solutions Architect, NFV Solutions Engineering, VMware

n Indranil Bal, Solution Consultant, NFV Solutions, VMware

n Pradip Kadam, Solution Consultant, NFV Solutions, VMware

n Tuan Nguyen, Senior Solutions Engineer, NFV Solutions, VMware

n Sumit Verdi, Director Lighthouse Solutions, NFV Solutions, VMware

n T. Sridhar, Principal Engineer & Chief Architect, NFV Solutions, VMware

Many thanks for contributions from:

n Ramkumar Venketaramani, Director Product Management, NFV Solutions, VMware

n Jambi Ganbar, Sr. Technical Solutions Manager, NFV Solutions Engineering, VMware

n Andrea Li, Lead Solutions Test Architect, NFV Solutions Engineering, VMware

n Michelle Han, Director, Solutions Testing and Validation, NFV Solutions Engineering, VMware

n Danny Lin, Sr. Director Product Management and Architecture, NFV Solutions, VMware

n Dobrinka Boeva, Sr. Staff Technical Writer, Information Experience, VMware

Special thanks for their valuable feedback to:

n Frank Escaros-Buechsel, Gary Day, Mikael Brihed, Henrik Oberg, Mauricio Valdueza, Neil Moore,Mike Richer, Mustafa Bayramov, Suresh Babu Nekkalapudi, Mahesh Pandurangan, SudeshTendulkar

VMware, Inc. 108

vCloud NFV Reference Architecture - VMware vCloud NFV 3 · vCloud Director VMware Integrated...

Documents

Transcript of vCloud NFV Reference Architecture - VMware vCloud NFV 3 · vCloud Director VMware Integrated...