Sponsored by the National Science Foundation Train-the-Admin Sarah Edwards, GPO June 22, 2014.

Sponsored by the National Science Foundation

Train-the-Admin

Sarah Edwards, GPO

June 22, 2014

Sponsored by the National Science Foundation 2Train the Admin – GEC20 – June 21, 2014

Outline

• GENI in the classroom: Technical Issues• Who can use a GENI rack?• Rack Layout & Strengths• Inter-aggregate links• OpenFlow and FOAM• GENI Meta-operations (GMOC)• Tools for rack administration• Getting Help• Q&A


GENI IN THE CLASSROOM: TECHNICAL ISSUES


How can you help?

• Four areas where campus IT can help:#1 InCommon

#2 ssh (especially from Windows machines)

#3 GENI Client Tool access

#4 Known GENI Ports


Access via the GENI Portal

For many experimenters:• no new passwords• familiar login screens

The GENI Portal leverages InCommon for single sign-on authentication

Experimenters from 304 educational and research institutions have InCommon accounts

GENI Project Office runs a federated IdP to provide accounts for non-federated organizations.

Portal needs certain attributes (more in minute)

#1


What can you do?

• The GENI Portal gives access to real resources• Therefore, we need to be able to contact experimenters if

something goes wrong• GENI Portal requires:

– eppn (eduPersonPrincipalName)– e-mail address

• GENI Portal prefers to receive:– affiliation– given name– surname

• InCommon members can easily share these attributes by enabling the Research & Scholarship (R&S) category

R&S: https://spaces.internet2.edu/display/InCCollaborate/Research+and+Scholarship+Category


Try logging in

Want to know if your institution is an InCommon member which shares the needed attributes?• The GENI Portal is at:

https://portal.geni.net • Click “Use GENI”• Pick your institution from the list• Login using your usual username and password• Does this work? You’re done• If not, e-mail [email protected]

– We will contact the appropriate person at your institution


SSH from Windows

SSH with keypair from Windows is non-trivial– No built-in ssh client

Possible Solutions

http://groups.geni.net/geni/wiki/HowTo/LoginToNodes

Does your campus have a standard solution?

– BitVise – FireSSH – javascript plugin for Firefox– SecureCRT (not free)– cygwin– Linux VM – make use of a slim OS– PuTTy (private key format different)

#2


Student access to client tools

• Three options:– Use their personal laptop

• LabZero is a good way to get setup• There are Mac/Windows Binaries for Omni

– Use Lab computers• Go through the exercises in lab computers• stress-test the resources or split students

– Use a VM with all the software loaded• http://groups.geni.net/geni/wiki/HowTo/CreateTutorialVM

Work with professors to determine how students will access client tools

#3


Known GENI Ports

• GENI is: – highly distributed– runs many services on many unusual ports– clients sometimes demonstrate unusual behavior

• eg many SSL connections because contacting many aggregates, repeated ssh connections to the same host, etc

• As a result, we’ve found that some networks will block some legitimate GENI traffic– In particular, campus Guest WiFi frequently has this

problem

• Checkout the Known GENI Ports

Known GENI Ports: http://groups.geni.net/geni/wiki/KnownGENIPorts

#4


First Exercise in GENI

□ Bulk-add students to project

Email Prework:□ GENI account□ Computer setup*□ Other?

1 week before

Class Prep:□ GENI Account Access□ Project for the Class□ Test Exercises□ Notify GMOC□ Figure Student Setup□ Email [email protected]

2 weeks before

(or sooner)

* Include steps for testing the setup

Instructor Timeline/Checklist

Instructor Checklist: http://groups.geni.net/geni/wiki/GENIEducation/Resources

mailto:[email protected]


WHO CAN USE A GENI RACK?


Who can use a GENI rack?

• Rack users can have accounts from one of three places:– GENI Portal/Clearinghouse (most common)– Emulab – PlanetLab

• In all three cases:– experimenters must be vouched for in order to access

GENI resources the first time– The operations groups coordinate to contact

experimenters if needed


Slice credentials

GENI: Terms and Definitions• Clearinghouse

– Slice authority: Creates and registers slices– GENI slice authorities: GENI Portal, PlanetLab, ProtoGENI

• Aggregate: Provides resources to GENI experimenters– Typically owned and managed by an organization– Examples: GENI Racks, Internet2, Emulab, PlanetLab– Aggregates implement the GENI AM API

Create & Register Slice

Researcher

Portal/Clearinghouse

Aggregate Manager API - listResources - createSliver … Aggregate

ManagerAggregate Resources

Slice credentials


GENI Portal/Clearinghouse

• The clearinghouse is a set of services to track:– experimenters, – projects, – slices, and – authorization

• The portal is a web-based user interface for experimenters to access the clearinghouse services and GENI aggregates– Accounts used in tutorials

• Anyone can get an account, but you must be a member of a project to do anything interesting


Project Lead Policies

Projects organize research in GENI– Projects contain both

people and their

experiments– A project is led by a single

responsible individual: the project lead

– Who can be a project lead?• Academic Faculty

• Senior technical staff in non-academic environments

Project

Lead

Members

Slice

We can contact both the experimenter and the project lead


Identifiers and URNs

• Slice URNs identify the issuing authority and the slice name– Use (URN, UUID) to uniquely identify slices over time– Exampleurn:publicid:IDN+ch.geni.net:tutorial+slice+monitor

• User URNs– Example

urn:publicid:IDN+ch.geni.net+user+sedwards

authority (GENI CH : project) slice name

authority username


RACK LAYOUT & STRENGTHS


InstaGENI Rack ExoGENI Rack

GENI Racks


What is a GENI Rack?

A GENI Rack contains compute resources, an OpenFlow dataplane switch and speaks the GENI AM API.

Racks are connected to each other across campuses, regionals, and backbone providers

Production GENI racks are developed by two teams: ExoGENI (IBM/RENCI), InstaGENI (HP/University of Utah)

In addition, GENI racks are being prototyped by:• Dell (prototype at Clemson and GPO) and • Cisco (prototype at WVNet and NCSU)


Getting a GENI Rack

Interested in getting a GENI Rack?

Email [email protected]

What’s involved?– High Level checklist for new racks from delivery to production release http

://groups.geni.net/geni/wiki/GENIRacksHome/RacksChecklist– InstaGENI requirements for configuring a rack http

://www.protogeni.net/wiki/instageni/checklist

Who has a production GENI aggregate? – http://groups.geni.net/geni/wiki/GENIProduction

– All aggregates (including racks) that have been tested and released for production use by experimenters.

– Includes "dev" racks (GPO, RENCI, Utah, Utah DDC, and Kansas)• "dev" designation indicates that rack might be taken down with less warning than others,

and might be running newer versions of software under test than others

http://groups.geni.net/geni/wiki/GENIRacksHome/RacksChecklist

http://groups.geni.net/geni/wiki/GENIRacksHome/RacksChecklist

http://www.protogeni.net/wiki/instageni/checklist

http://www.protogeni.net/wiki/instageni/checklist

http://groups.geni.net/geni/wiki/GENIProduction



Sponsored by the National Science Foundation 24Train the Admin – GEC20 – June 21, 2014©2010 HP Created on xx/xx/xxxxof 222

InstaGENI Rack Diagram

Diagram from InstaGENI slide deckJuly 10, 2012

Control plane switch

Data plane switch (OF)

Control node

Experiment nodes (5)

Expansion space(empty/PDUs)

bossops

foamflowvisor


ExoGENI Rack Diagram

Diagram from ExoGENI cabling diagram

Front Back

Control plane switchData plane switch (OF)

Head node

Worker nodes (10)

VPN (Juniper SSG)iSCSI storage

Console

PDUs not shown


Details: Hardware specs

InstaGENI ExoGENI *

Nodes per rack 5 + 1 control node 10 workers + 1 head

Cores per rack 60 120

Network interfaces 4x 1Gbit (upgradeable to 10 Gbps)

2x 10Gbit with SR-IOV (upgradeable to 40 Gbps)

Storage 1 TB local 150GB+500 GB local + 6 TB SAN (upgradeable)

Dataplane Switch

HP ProCurve 5406 (VLAN-based OpenFlow)

IBM G8264R (Port-based OpenFlow)

* Listed are the specs for ExoGENI IBM-based racks.

There are 110V and 220V versions of each rack


Compute resources

• Bare-metal host:– “expensive” resource, limits the lifetime of your

experiment

– ExoGENI : 2 bare-metal hosts per rack

– InstaGENI : 2 bare-metal hosts per rack

Experimenters should use VMs unless they have a very good reason

• ExoGENI : KVM VMs

• InstaGENI, ProtoGENI: linux containers, XEN VMs


Networking

• Speed of the NICs:– ExoGENI: 10Gbps (2x per host) (upgradeable to 40

Gbps)– InstaGENI: 1Gbps (4x per host) (upgradeable to 10

Gbps)

• Sharing:– ExoGENI: Virtual NICs share physical 10G NICs and

are bandwidth-provisioned via OVS among VMs unless using bare-metal hosts

– InstaGENI: Can get dedicated NICs even on shared nodes, but only 1Gb


Tool support

• ExoGENI & InstaGENI both support:– Reservation tools: Omni, GENI Portal, Flack– I&M: GIMI, GEMINI

• InstaGENI, ExoGENI also have testbed specific tools


INTER-AGGREGATE CONNECTIVITY


Inter-aggregate Links

• GENI does not own any networks. Instead GENI connects racks via regional and backbone providers.

• Connect racks in one of four ways:– Stitching: dynamically configured inter-domain VLANs

• Stitcher: Distributed with gcf/omni• Flukes: ExoGENI-only (not via GENI AM API)

– OpenFlow Mesoscale & AL2S & some regionals• e.g. CENIC, SOX, MOXI are OpenFlow regionals

– Preconfigured VLANs (Shared and Unshared)• Some pre-configured inter-domain VLANs are available• Some are OpenFlow-enabled

– GRE tunnels over control interface• Use Flack to connect IG nodes via a GRE tunnel


GENI Stitching: Under the Hood

How does GENI Stitching Work?

1.Rack Configuration: Long. Done once in advance.

2.Tool & Aggregates make Reservations: Quick, Live, Easy


Rack Stitching Configuration

• Identify a path or paths from the rack to other GENI aggregates. – Typically a connection to a national backbone

• Identify the network providers – Typically a campus, a regional, and the backbone

• Identify the endpoints and VLAN tags that can be used to connect to the rack

Backbone Regional Campus

Rack

GENI Aggregates

Static VLANs


SCS

Tool

Experimenter View: Creating a Circuit

<link client_id=”mylink"> <component_manager… <component_manager…

Aggregate 2

Aggregate 1

1. Simple Request

2. Send Path Request to Stitching Computation Service (SCS)

3. Get Expanded Request

4. Send Request to Aggregate 1

5. Get Manifest

6. Repeat for Other Aggregates

<link client_id=”mylink"> <component_manager… <component_manager… …<stitching> <path id=“mylink”> <hop id=“1”> <link id=“switch1:port1”… <vlan…>3747</vlan…>

7. Manifest Back


OPENFLOW AND FOAM


GENI META-OPERATIONS (GMOC)


GENI Meta-operations

GMOC: GENI Meta-operation Center• Keeps track of outages• Notification system for resource reservations• Emergency Stop

How to notify GMOC: http://groups.geni.net/geni/wiki/HowTo/PreReserveGENIResourcesGMOC Calendar: http://globalnoc.iu.edu/gmoc/index/support/gmoc-operations-calendars.html

GMOC Google Calendar keeps track of reservations/outages


Monitoring

• Racks are being monitored• You may get email from someone if something

looks amiss.• Example:

– Can’t reach anything on the rack via ping or an AM API GetVersion call


GMOC Notifications

• GMOC sends many notifications• Three types of interest:

– Outages– Resources pre-reservation for a class, tutorial, or demo– Potentially disruptive experiment

• All notifications are tracked on the calendar:– http://globalnoc.iu.edu/gmoc/index/support/gmoc-

operations-calendars.html


GMOC Resource Pre-Reservation

What?– A class, tutorial, or experiment needs a well defined set of

resources at a specified time

Why does GMOC send notifications?– Help us deconflict use of GENI and maintenances,

planned outages, etc.

What should you do?– Is there something happening on your campus that would

conflict with this event? Contact GMOC and let them know.

– Conflicts are a rare event. Usually the rack teams will take care of this for you.

– Please report events on your campus that only you can know about. e.g a known power outage


GMOC Resource Pre-ReservationSUBJECT: Resource Reservation - GENI Spring Train the TA TutorialAFFECTED: GPO Portal and Clearinghouse InstaGENI racks: GPO, Clemson, Missouri, Illinois, Nysernet GENI Tools: Flack, OmniSCHEDULED START TIME: Friday, January 24, 2014, 7:00 PM (1900) UTCSCHEDULED END TIME: Friday, January 24, 2014, 10:00 PM (2200) UTCDESCRIPTION: At the 2014 Sprint Train the TA there are going to be multiple GENI tutorials that will use the GENI resources listed above. We would kindly ask operators to refrain for any outages during this period without coordinating with the tutorial organizers first at [email protected]. Only light traffic volumes are expected.TICKET NO.: 753:126TIMESTAMP: Thu Jan 23 20:04:53 2014 UTC

Message ID: gmoc.753.2.1Thank You,GENI Meta Operations CenterIndiana [email protected], 317-274-7783Visit the GMOC Home Page athttp://gmoc.grnoc.iu.edu/

_______________________________________________GENI-Ops mailing [email protected]://mail1.grnoc.iu.edu/mailman/listinfo/geni-ops

Name of Event

Who’s affected?

Duration

Description

GMOC Ticket #

GMOC Website for more information

Is your rack listed?If so, keep reading

Mention in e-mail to GMOC

STOP



http://gmoc.grnoc.iu.edu/



https://mail1.grnoc.iu.edu/mailman/listinfo/geni-ops


Potentially Disruptive Experiment

What?– An experiment that might cause problems for other

experimenters or resource owners.– Examples:

• Reserving a large amount of bandwidth between two sites

Why does GMOC send notifications?– Alerts other experimenters and resource owners that

something unusual might happen

What should you do?– If you see unexpected behavior related to your rack,

notify GMOC. We can ask the experimenter to stop.


Potentially Disruptive ExperimentSUBJECT: Disruptive Experiment Reservation for GENI Participants U. Missouri and U. Utah DDCAFFECTED: U. of Missouri InstaGENI Rack Utah DDC InstaGENI Rack Internet2SCHEDULED START TIME: Thursday, December 12, 2013, 3:00 PM (1500) UTCSCHEDULED END TIME: Thursday, December 12, 2013, 6:00 PM (1800) UTCDESCRIPTION: An experiment will be reserving a 1GB link between the U. of Missouri and Utah DDC InstaGENI racks.Traffic spikes and other potential disruptions may occur across the GENI OpenFlow Backbone during the experiment. The entire window has been

reserved.TICKET NO.: 714:126TIMESTAMP: Wed Dec 11 20:23:12 2013 UTC

Message ID: gmoc.714.1.1Thank You,GENI Meta Operations CenterIndiana [email protected], 317-274-7783Visit the GMOC Home Page athttp://gmoc.grnoc.iu.edu/

Name of Event

Who’s affected?

Duration

Description

GMOC Ticket #

GMOC Website for more information

Is your rack listed?If so, keep reading

Mention in e-mail to GMOC

STOP




GENI Emergency Stop process

48

http://gmoc.grnoc.iu.edu/gmoc/documents.html

• Emergency Stop is a well defined process for what to do if something goes wrong.

• There is 24/7 staffing to respond in case of emergency

Slide by Eldar Urumbaev

http://gmoc.grnoc.iu.edu/gmoc/documents.html


What is Emergency Stop?• One of the essential early operational requirements for the GENI

facility is the need to manage and coordinate the stop and/or containment of GENI resources among all GENI projects in the case of an urgent request or misbehavior of the GENI experiments.

• Emergency stop is the system used to respond to incidents of interference or resource exhaustion caused either unintentionally (misconfiguration), or intentionally (malware).

The emergency stop system has 3 main goals: • To give experimenters and other GENI stakeholders a single place

to go for notification of emergency stop issues• To facilitate emergency stop with GENI aggregates • To provide an easily understood, flexible service that minimizes

disruptions for GENI experiments or GENI aggregates

49Slide by Eldar Urumbaev


These goals are accomplished through two mechanisms:• A coordination process to identify related GENI

aggregates and/or slices for a given stop request (via GMOC-DB), notify, facilitate communication, verify resolution and report.

• A last resort isolation mechanism (via Internet2 and NLR Core) to effectively quarantine aggregates with issues from the rest of the GENI infrastructure.

What Is Emergency Stop?


• GMOC – GMOC • Aggregate Operators • GENI-Interconnected Networks & Experimenters • GENI LLR Representative

Emergency Stop Participants & Stakeholders



GMOC Emergency Stop Contact Information:

• Phone: (317) 274-7783• Email: [email protected] • Web:

http://gmoc.grnoc.iu.edu/gmoc/support/report-a-problem.html





TOOLS FOR RACK ADMINISTRATION


• Currently the rack teams do most actual rack administration

• However tools exist which you can get access to for your rack

• Tools are specific to your rack type• Request admin accounts from the

InstaGENI/ExoGENI teams


InstaGENI Administrative E-mail

• InstaGENI sends mail when:– Experimenters act on slivers (create/renew/delete/etc)– Routine non-alarming events occur– Something more serious is wrong– http://www.protogeni.net/wiki/rackmaillists has more

info

IG rack wiki: http://www.protogeni.net/wiki/instageni

http://www.protogeni.net/wiki/rackmaillists


InstaGENI Administrative E-mailFrom: [email protected]: [email protected]: [email protected]: [gpo-ops] BBNINSTAGENI: protogeni-wrapper.plDate: Wed, 15 Jan 2014 10:12:11 -0500 (EST)

URN : urn:publicid:IDN+ch.geni.net+user+jbsModule : amMethod : CreateSliverVersion : 2.0StartTime : 10:11:49:815689slice_urn : urn:publicid:IDN+ch.geni.net:gpo-infra+slice+ps103slice_idx : 13430slice_uuid : ec5dbd9d-cc1d-4c06-acb5-0ebd6b11f366EndTime : 10:12:11:184891Elapsed : 21.37LogURN : urn:publicid:IDN+instageni.gpolab.bbn.com+log+de38e4abc56126289876f71d6e130013LogURL : https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=de38e4abc56126289876f71d6e130013Code : 0

Who acted?

AM API Call

Time

Slice

Log URLCheck here for more details /

Mention in e-mail


InstaGENI: Red dot mode

The web interface for each rack allows administrators to see extra information about what is happening on their rack.

Press to enter red dot mode

… but only if you are an administrator


Red Dot Example

For example, see what slices are running on your nodes and who has reserved those resources


ExoGENI Administrative Tools

• ExoGENI does not send administrative email

• ExoGENI provides – pequod, a command line tool to access your rack– A monitoring webpage– Wiki documentation

Pequod documentation: https://geni-orca.renci.org/trac/wiki/orca-pequodExoGENI monitoring: https://control.exogeni.net/monitor/check_mk/ExoGENI wiki: https://wiki.exogeni.net/doku.php


$ ssh bbn-hn.exogeni.gpolab.bbn.comLast login: Wed Jan 29 15:50:04 2014 from dhcp89-081-159.bbn.com|-----------------------------------------------------------------|| ____ ____ ____ ____ ____ ____ ____ || ||E |||x |||o |||G |||E |||N |||I || || ||__|||__|||__|||__|||__|||__|||__|| || |/__\|/__\|/__\|/__\|/__\|/__\|/__\| || ||-----------------------------------------------------------------|

$ pequodPequod ORCA Shell v.4.0-SNAPSHOT.build-5300 built on 07/16/2013 18:38 (c) 2012-2013 RENCI/UNC Chapel Hill

help: Returns help for individual commands file: File-related commands set: Modify internal set variables show: Show the state of things aux: Auxiliary commands manage: Manage actor state history: show command history (!<command index> invokes the command) exit: Exit from the shell (Ctrl-D or Ctrl-C also works)Type the entire command, or enter the first word of the command to enter subcommand with intelligent auto-completion (Using TAB).<snip>pequod>show reservations for all actor bbn-vm-am state active 68fc1bb0-3085-4e91-baa7-330a9ac3a37b bbn-vm-am Slice: 6146e4bf-0ebf-4072-8238-18cd688aeb53 1 bbnvmsite.vm [ active, nascent] Notices: Reservation 68fc1bb0-3085-4e91-baa7-330a9ac3a37b (Slice urn:publicid:IDN+ch.geni.net:gpoamcanary+slice+sitemon) is in state [Active,None] Start: Tue Jan 14 16:40:22 EST 2014 End:Thu Jan 30 19:00:00 EST 20144be1e2f8-7be1-456e-862c-cffdbebcdc9c bbn-vm-am Slice: 733fc88a-14e7-44ed-a2c5-1ea7ec4da223 1 bbnvmsite.vm [ active, nascent] Notices: Reservation 4be1e2f8-7be1-456e-862c-cffdbebcdc9c (Slice urn:publicid:IDN+ch.geni.net:gpo-infra+slice+ps103) is in state [Active,None] Start: Wed Jan 15 10:11:44 EST 2014 End:Sun Feb 02 18:00:00 EST 2014<snip>Total: 19 reservationspequod>exit

Logging out of containersShutting down commands help file set show aux manage Resetting terminal and exiting. Goodbye.

Login to head node on your rack

Run pequod

Query state of resources


ExoGENI Admin Webpages (nagios)

ExoGENI wiki: https://wiki.exogeni.net/doku.phpExoGENI monitoring: https://control.exogeni.net/monitor/check_mk/For a single rack: https://<site>-hn.exogeni.net/rack_<site>/For example: https://bbn-hn.exogeni.net/rack_bbn/

Status of racks

interfaces


Getting Help


General bug reporting advice

Gather as much information as you can– What did you do? What did you expect to happen? What

actually happened?– Include what you see (screenshots, omni output errors)– Where relevant, include:

• type of account you are using (eg portal)• the tool you are using (eg Flack, omni, portal)• your slice name or URN • aggregates you are using• a detailed description of what's wrong including any error messages


Getting help: General mailing lists

[email protected]– General place to get help using GENI– More aimed at experimenters than admins– Any questions about using GENI are

certainly [email protected]

– At least one admin/ops person from each GENI site– You should probably subscribe (directly or a local list)– Mostly outage announcements these days

[email protected]– General catch-all list for getting help– geni-users is usually better, but this is an ok last resort

[email protected] (not a mailing list)– Send outage notifications to this list– Get help with Internet2 and AL2S

http://groups.geni.net/geni/wiki/NikySandbox/GENIExperimenter/GENICommunity








Getting help: Topical mailing lists

[email protected]– General list for people running Emulab– Site admins with InstaGENI racks should

join– Any questions about running an IG rack

are [email protected]

– RENCI list for their ExoGENI ops team– Any questions about running an EG rack are welcome

[email protected]– GPO list for their infrastructure team– The best place for FOAM questions for now– Also fine to ask about anything else






Getting help: Other resources

The GENI wiki:– http://groups.geni.net/– http://groups.geni.net/geni/wiki/GENIOperationsWelcome– http://groups.geni.net/geni/wiki/GENIRacksHome/InstageniRacks/Operators

– For operators– http://groups.geni.net/geni/wiki/ConnectivityHome

– IP Addresses, VLANs assigned to GENI sites– http://groups.geni.net/geni/wiki/GeniAggregate– http://groups.geni.net/geni/wiki/GENIRacksHome

IRC:– http://groups.geni.net/geni/wiki/HowTo/ConnectToGENIChatRoom– Various GPO and other GENI people are often there

Other mailing lists:– http://groups.geni.net/geni/wiki/NikySandbox/GENIExperimenter/GENICommunity– Many other topic-specific lists– Using one of the general lists is often better

http://groups.geni.net/geni/wiki/GENIExperimenter/GetHelp

http://groups.geni.net/

http://groups.geni.net/geni/wiki/GENIOperationsWelcome

http://groups.geni.net/geni/wiki/GENIOperationsWelcome

http://groups.geni.net/geni/wiki/GENIRacksHome/InstageniRacks/Operators


http://groups.geni.net/geni/wiki/ConnectivityHome

http://groups.geni.net/geni/wiki/ConnectivityHome

http://groups.geni.net/geni/wiki/GeniAggregate


http://groups.geni.net/geni/wiki/HowTo/ConnectToGENIChatRoom



Questions?

Sponsored by the National Science Foundation Train-the-Admin Sarah Edwards, GPO June 22, 2014.

Documents

Transcript of Sponsored by the National Science Foundation Train-the-Admin Sarah Edwards, GPO June 22, 2014.