1© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.

Randy Levensalor

Lead Architect / Business Technologies

OPNFV - From The Trenches

2© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.

• Purpose of the Lab Install

• Challenges Encountered–Networking

–Build server

–Debugging

–Repeatability

Agenda

2

3© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.3

About CableLabs

CableLabs

• non-profit research and development consortium funded by MSO’s (Multiple System Operator)

• dedicated to creating innovativeideas that significantly impact our cable operator members’ business

• serves to define interoperable solutions among our members and their technology suppliers in order to

• drive scale

• reduce costs

• promote innovate solutions

.

4© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.

Lab Details

4

Virtual Network Functions

Compute Node #2

• Lab 1– Install

• Stable release• Foreman installer

– Planed use• VNF development• Regression testing

• Lab 2– Install

• Top of tree• APEX installer

– Planed use• Installer• Infrastructure

HA HA

ODL

Control Node #1 Control Node #2 Control Node #3

Build Server

Virtual Network Functions

Linux

Compute Node #1

Jump Server

5© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.5

1. Preparing the Lab

Foreman Fuel

Installers Initial Network Setup

30 +Cables & 6 VLANs

Knowledge of OPNFV

6© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.6

1. Preparing the Lab - Solution

• Detailed port and VLAN mapping spreadsheet and diagram

• Foreman Specific

• Suggestion: Pay extra for blades

7© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.7

2. Violating build server

• Install must run on bare metal

• Must run as root

• Install is not immutable

• Virtualbox failures

8© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.8

2. Build server solutions

• Separate build and jump server for security

• Updated install directions and .iso

• Automated workarounds

• Improve error reporting

• Suggestions: Report Bugs & Submit Patches

9© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.9

3. Install failed…somewhere

• Failures after Foreman VM starts are difficult to trace

Deploy.sh ->

vagrant up ->

bootstrap.sh ->

Ansible ->

run.sh ->

Puppet

Foreman ->

pxe ->

10© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.10

3. Solution

• Check release notes and FAQ

• Look at reports on Foreman

• Still can be difficult to trace

• Suggestion: Engage mailing list with problems

11© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.11

4. Reliability Concerns

• Install pulls scripts from multiple git & yum repos

• Failures are catastrophic

• Can’t install without Internet access

12© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.12

4. Suggestions for Reliability

• Check IRC, mailing list and Jenkins (you are not alone)

• Manually reproduce

• Still not ideal

13© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.13

5. It’s running…or is it?

• Installs are unstable

• Only one VM started

• Can’t delete networks

• Corrupt data bases

14© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.14

5. Suggestions for Unstable Install

• Check the release notes and FAQ

• Arno is based on old versions of OpenStack and OpenDaylight and not all critical patches have been back ported yet

• Reinstall / automate post install setup and have a backup

15© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.15

Takeaways

• Lots of help–Wiki https://wiki.opnfv.org/

–Dev mailing list https://wiki.opnfv.org/developer/getting_started#mailing_list

–Direct with the developers

• Engage users early & often

• Run from a clean install

• Standardize documentation

© Cable Television Laboratories, Inc. 2015. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.