Deploying Lots of Telco Clouds - object-storage-ca-ymq-1 ... · OpenStack Fuel As mentioned in...

Deploying Lots of Telco Clouds

© 2016 AT&T Intellectual Property. All rights reserved. AT&T, Globe logo, Mobilizing Your World and DIRECTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners.

Introduction- Introduction to AT&T- Introduction to the Presenters

Outline:

1. From loading dock to cloud2. How to hatch lots of clouds3. How to manage lots of clouds4. Lessons learned and Q&A


Telco clouds and our challenges● What are the Telco Cloud Challenges

Lots of locations

Complex Networks

Service Provider Uptime Requirements

NFV Requirements

Site configurations and the hardware in them can fluctuate

Many enterprise services and global network controllers need to be plugged in and deployed

Build many sites in a compressed time frame

● Introducing AT&T’s AIC OpenStack Cloud


Going From Data Center Loading Dock to Working Cloud

DESIGN

MANAGE DEPLOYMENT


DESIGN


Automating DesignWhat were we trying to fix?

Excel Spreadsheets

Data Entry Mistakes

Inventory updates via e-mail

A long and manual process

No single view


AIC Formation


AIC Formation - Zone info tab


AIC Formation - Rack Elevation Tab


AIC Formation - Cable Map


AIC Formation - Zone Summary


DEPLOYMENT


Bootstrap AIC Formation contains all of the information that we need for site configuration, like:

● Physical device information● Network configuration● Site config version

We feed this information to a simple bootstrap image based on debian minimal installer, from there the installer sets up a single foundation node with simple foundational components.


UndercloudSimple undercloud with 4-5 foundational components (containers + KVM):

● PXE Dust● MaaS● Fuel● Jump/Access host● vCenter (in certain sites)


Bare-metal conundrumIn our undercloud, we have three different pieces of software that want to provision bare-metal:

● OpenStack Fuel● Metal-as-a-Service● vCenter/Auto-deploy

Rather than move the bare-metal provisioning functionality to one service, we chose to enable all three.


PXE DustA very simple solution was to implement a sort of PXE router. We did this by:

● Writing a simple python REST API that allows for rules and boot targets to be set

● Another API call applies rules and computes a boot target


OpenStack FuelAs mentioned in previous slides, OpenStack Fuel was primarily used for installing OpenStack and non-OpenStack components.

However, there were several scenarios that Fuel was not built for:

● Datacenter racks in L3 islands● Micro-roles (AIC has almost 3x the roles of stock Fuel)● Many configuration options are immutable

AT&T added several enhancements to deal with these problems some of which are found in Mitaka, more that will be seen in Newton and beyond.


OpsSimple● Ansible based● Central config repo● Calls Fuel APIs● Calls MaaS and Libvirt APIs● Executes some commands on nodes for pre and post deployment tasks● Hooks into OSS and BSS systems● All site config and actions are checked into/out of a central git repo● Logical configs (VMs, pseudo net interfaces, etc) sent back to Formation.

Challenge: How to do this with Heat? (Without a full blown OpenStack undercloud) Murano maybe?


MANAGE


Independent Sites but Centralized ManagementAT&T’s massive scale demanded independent zones for performance, resiliency, and security.

This presented challenges:

- User Experience Consistency (what are my flavors)- Account Creation (who goes where and what quotas)- Upgrades (orchestrating control plane upgrades)- Tenant Resource Discovery (what can I access)

Operational Challenge


Resource Creation Gateway


Discovery API Services

Tenant


Lessons Learned - Future plans- Work upstream when developing customizations to Fuel, MaaS, etc- You can’t invest too much in CI/CD

- A jenkins/gerrit with deep testing is one of the foundations of our success

- Overlapping functionalities are okay, but be prescriptive/restrictive with your teams or they will use things in ways you did not intend.

- Always more opportunities for automation - going to do it twice? Automate it!

- This includes monitoring, config management, incident management, network setup, data center physical deployment (robots), image building, platform definition, etc.

- Think machine learning, deep insights, reporting- Let the people do the thinking and the automation do the repetitive work

- Infrastructure elements are NOT ISLANDS! We continue to transform our silo’d teams into cross-skilled, interactive and collaborative teams.


Thank You!

http://about.att.com/innovationblog/att_cloud_journey


QUESTIONS?

Alan Meadows ([email protected])Lee Rivere ([email protected])

Mike Wilson ([email protected])


mailto:[email protected]

mailto:[email protected]

Deploying Lots of Telco Clouds - object-storage-ca-ymq-1 ... · OpenStack Fuel As mentioned in...

Documents

Transcript of Deploying Lots of Telco Clouds - object-storage-ca-ymq-1 ... · OpenStack Fuel As mentioned in...