Sponsored by the National Science Foundation

Systematic Experimentation in GENI

Sponsored by the National Science Foundation 2Systematic Experimentation – GEC 20 – June 23, 2014

Agenda

Motivation

Sarah Edwards, Xuan Liu, Niky Riga, Vic Thomas, GPO

Case Study: Virtual Router Failure Restoration

Xuan Liu, UMKC/GPO

Tools and Techniques

Custom images and install scripts

InstaGENI, Gary Wong, University of Utah

ExoGENI, Paul Ruth, RENCI

Porting ExoGENI Images to InstaGENI

Ezra Kissel, Indiana University

Manipulating RSpecs with geni-lib Nick Bastin, Barnstormer Softworks

How to Collaborate

Sarah Edwards, GPO

Q&A

Experimenter Drop-In (next session)

Sponsored by the National Science Foundation

Systematic Experimentation in GENI:

Motivation

Sarah Edwards, Xuan Liu,

Niky Riga, Vic Thomas

GENI Project Office

Sponsored by the National Science Foundation 4Systematic Experimentation – GEC 20 – June 23, 2014

GENI-FIRE 150 node topology

Courtesy of Thierry Rakotoarivelo, NICTA, GEC 20 Demo

Clear plan AutomationScale and

Visualization

Sponsored by the National Science Foundation 5Systematic Experimentation – GEC 20 – June 23, 2014

Virtual Computer Networking Lab

Courtesy of Cong Wang and Divya Bhat, UMass Amherst, GEC 20 Demo

Start small

Sponsored by the National Science Foundation 6Systematic Experimentation – GEC 20 – June 23, 2014

Intelligent Data Movement System

Courtesy of Ezra Kissel, Indiana University, GEC 20 Demo

Long-lived slicefor stitched, shared VLAN

Dynamically add/remove nodes

as needed

Sponsored by the National Science Foundation 7Systematic Experimentation – GEC 20 – June 23, 2014

Why doesn’t this work?(Anonymous) Past Bug Reports

repeatable

incremental

test controller

in one AM?

GRE works

Stitch

test controllerin one AM

Multi-site Hadoop

Used control plane to send high volume data between

racks

Interfere with rack !

OpenFlow multi-site

OpenFlow multi-site

Sponsored by the National Science Foundation 8Systematic Experimentation – GEC 20 – June 23, 2014

Systematic Experimentation

Running an experiment on a testbed requires best

practices and methodology from a combination of:– Science (Scientific method)

– Programming

– System Administration

Today, talking about strategies and techniques for

bringing up experiments in a systematic way.

Sponsored by the National Science Foundation 9Systematic Experimentation – GEC 20 – June 23, 2014

Why experiment systematically?

Debugging Easier to debug experiments. Easier to get help debugging. Especially for complex experiments.

Scalability and Repeatability A well designed experiment will be repeatable and systematically scalable

Valid Develop a solid understanding of and validate your experimental setup

Rigorous Perform a scientifically rigorous (and publishable) study

Sponsored by the National Science Foundation 11Systematic Experimentation – GEC 20 – June 23, 2014

General Principle #1: One thing at a time

Experimentation 101 and Debugging 101

Only change one thing at a time

Examples:• Software/image• Configuration• Number of nodes/links• Geographic distribution of nodes/links

Sponsored by the National Science Foundation 12Systematic Experimentation – GEC 20 – June 23, 2014

General Principle #2: Start Small

Start by building smallest possible topology by hand.

Automate as needed.

Test and measure as you go.

client server

host

OVSswitch

host

host

router

router router

worker

master

worker

node

Sponsored by the National Science Foundation 13Systematic Experimentation – GEC 20 – June 23, 2014

General Principle #3: Save what you do

• Log all of your experimental artifacts for every experiment that works– RSpec – image– install script– custom software– measurements– etc

• Use version control to store your artifacts• Always know the last configuration that worked

Sponsored by the National Science Foundation 14Systematic Experimentation – GEC 20 – June 23, 2014

Experiment WorkflowSummary Workflow

How often should I loop?

Sponsored by the National Science Foundation 15Systematic Experimentation – GEC 20 – June 23, 2014

Process Recommendation

A. Build (smallest possible) topology by hand at a single aggregate

B. Automate topology creation at a single aggregate

C. Orchestrate and Instrument

D. Increase scale

E. More nodes

F. More aggregates

Automate

Sponsored by the National Science Foundation 16Systematic Experimentation – GEC 20 – June 23, 2014

Process: Start Small

A. Build smallest possible topology by hand at a single aggregate

B. Automate topology creation at a single aggregate – Store RSpecs, software, etc in version control – TOOLS Use install scripts and custom images– Determine how to update software on your nodes

Repeat A/B as needed.

C. Orchestrate and Instrument your experiment procedureTOOLS GENI Desktop and LabWiki

Automate as needed.

worker

master

worker

node

Sponsored by the National Science Foundation 17Systematic Experimentation – GEC 20 – June 23, 2014

Process: Scale

D. Increase scale (number of nodes, volume of traffic, etc) TOOLS Flack for small scale. geni-lib for larger scale

client

server

client

client clientclient

client server

Sponsored by the National Science Foundation 18Systematic Experimentation – GEC 20 – June 23, 2014

Process: Split across aggregates

E. Split an existing topology across second aggregate – How connect across aggregates matters (stitching vs OF etc)

TOOLS stitcher for VLAN stitching and GRE tunnels. ExoSM for Intra-ExoGENI stitching. Flack for GRE tunnels.

F. Split across multiple aggregates

client client

server

client clientclient

Colors represent different aggregates

Inter-aggregate link

Sponsored by the National Science Foundation 19Systematic Experimentation – GEC 20 – June 23, 2014

Avoiding Common Pitfalls

• Don’t start with a large or inter-aggregate topology

• In general, if you are keeping your experiment up because you can’t recreate your setup, then something is wrong.

• GENI is not a simulator

• Common Issue: OpenFlow controllers used on an multi-aggregate topology with loop MUST be tested in a single aggregate topology with loop

Sponsored by the National Science Foundation 20Systematic Experimentation – GEC 20 – June 23, 2014

Case Study

Sponsored by the National Science Foundation 21Systematic Experimentation – GEC 20 – June 23, 2014

Process Summary

A. Build (smallest possible) topology by hand at a single aggregate

B. Automate topology creation at a single aggregate

C. Orchestrate and Instrument

D. Increase scale

E. More nodes

F. More aggregates

Automate

Sponsored by the National Science Foundation 22Systematic Experimentation – GEC 20 – June 23, 2014

Overview of Tools and Techniques

• Automation– Custom images on InstaGENI/ExoGENI– Converting ExoGENI images to InstaGENI– Install script on InstaGENI– Postboot script on ExoGENI– Measurement with LabWiki and GENI Desktop– Repeating and sharing experiments with apt

• Scaling Up– geni-lib

• Inter-aggregate experiments– stitcher– OpenFlow

Tutorials

Now

Now

Now

Drop-In

Now

Tutorial

Tutorial

Now

Sponsored by the National Science Foundation 23Systematic Experimentation – GEC 20 – June 23, 2014

Agenda

Motivation

Sarah Edwards, Xuan Liu, Niky Riga, Vic Thomas, GPO

Case Study: Virtual Router Failure Restoration

Xuan Liu, UMKC/GPO

Tools and Techniques

Custom images and install scripts

InstaGENI, Gary Wong, University of Utah

ExoGENI, Paul Ruth, RENCI

Porting ExoGENI Images to InstaGENI

Ezra Kissel, Indiana University

Manipulating RSpecs with geni-lib Nick Bastin, Barnstormer Softworks

How to Collaborate

Sarah Edwards, GPO

Q&A

Experimenter Drop-In (next session)

Sponsored by the National Science Foundation

How to Collaborate

Sponsored by the National Science Foundation 25Systematic Experimentation – GEC 20 – June 23, 2014

Project Membership example

Projects have 1 Lead and any number of Admins, Members, and Auditors

http://groups.geni.net/geni/wiki/GENIConcepts#Project

Typical Class

Expiration

Typical Research Project

Sponsored by the National Science Foundation 26Systematic Experimentation – GEC 20 – June 23, 2014

Working with multiple members in a slice

Research AsstSlice Lead Post-Doc

Slice MemberProfessor

Slice Admin

Members of all slices in a project:

• Project Leads (Professor)• Project Admins (Post-docs, researchers)

Other can be added manually

http://groups.geni.net/geni/wiki/GENIConcepts#Slice

Sponsored by the National Science Foundation 27Systematic Experimentation – GEC 20 – June 23, 2014

Slice Access

Being a member of a slice means you can act on a slice:– Add resources– Check status– Delete resources– Renew resources

With any tool!

Sponsored by the National Science Foundation 28Systematic Experimentation – GEC 20 – June 23, 2014

To ensure access in collaborator’s resources:Option 1: Make resource reservation from Portal or omni

Option 3: Use a common public key

Add slice member’s accounts to existing resources:$ omni -V 3 poa SLICE geni_update_users

--useSliceAggregates –-useSliceMembersAlternatively…

$ addMemberToSliceAndSlivers myslice username

Slice Access: Logging in to resources

Slice membership does not guarantee ability to login to resources!

Only works on InstaGENI/ProtoGENI

Sponsored by the National Science Foundation 29Systematic Experimentation – GEC 20 – June 23, 2014

Questions?

Sponsored by the National Science Foundation 30Systematic Experimentation – GEC 20 – June 23, 2014

Special Topic Experimenter Drop-InIn this room after the break

• Writing ExoGENI velocity scripts • New ExoGENI image snapshotting feature • Converting ExoGENI images to InstaGENI images• Manipulating RSpecs with `geni-lib`• Sharing experiments with `apt`

Thanks to:– Paul Ruth, RENCI– Nick Bastin, Barnstormer Networks– Rob Ricci, University of Utah