XSEDE Campus Bridging Birds Of a Feather

Rich Knepperrich@iu.edu

Craig Stewartstewart@iu.edu

James Wade Fergusonjwf@utk.edu

Presented at TeraGrid ‘11, 20 July, Salt Lake City, UTAvailable from: http://hdl.handle.net/2022/13423

© Trustees of Indiana University; released under Creative Commons 3.0 unported license; license terms on last slide.

2

The beginnings of all this….• Cyberinfrastructure consists of computational systems, data and information

management, advanced instruments, visualization environments, and people, all linked together by software and advanced networks to improve scholarly productivity and enable knowledge breakthroughs and discoveries not otherwise possible.

• In early 2009 National Science Foundation’s (NSF) Advisory Committee for Cyberinfrastructure (ACCI) charged six different task forces to make strategic recommendations to the NSF in strategic areas of cyberinfrastructure (CI): – Data– Grand Challenges and Virtual Organizations– High Performance Computing– Software and Tools– Work Force Development– Campus Bridging

• Why Bridging? We need bridges because it can feel like you are falling off a cliff when you go from your campus CI to a state, regional, or national CI facility

3

• The goal of campus bridging is to enable the seamlessly integrated use among a scientist or engineer’s personal cyberinfrastructure; cyberinfrastructure on the scientist’s campus; cyberinfrastructure at other campuses; and cyberinfrastructure at the regional, national, and international levels; as if they were proximate to the scientist.

• When working within the context of a Virtual Organization (VO), the goal of campus bridging is to make the ‘virtual’ aspect of the organization irrelevant (or helpful) to the work of the VO.

4

Branscomb Pyramid

NSF Advisory Committee for Cyberinfrastructure Task Force on Campus Bridging. Final Report. March 2011. http://www.nsf.gov/od/oci/taskforces/TaskForceReport_CampusBridging.pdf

5

Commercial cloud (Iaas and Paas)

Volunteer computing

Workstations at Carnegie research

universities

Campus HPC/ Tier 3 systems

Track 2 and other major facilities

NSF Track 1

0 2000 4000 6000 8000 10000 12000

Not a Branscomb Pyramid

So that anyone may quibble, the data are published: Welch, V., R. Sheppard, M.J. Lingwall and C.A. Stewart. Current structure and past history of US cyberinfrastructure (data set and figures). 2011. Available from: http://hdl.handle.net/2022/13136

6

(In)Adequacy of Research CI

Never (10.6%)Some of the time (20.2%)Most of the time (40.2%)All of the time (29%)

• From survey of researchers funded as PIs by NSF 2005-2009• 10% never have enough CI resources; 70% have at least insufficient resources

on occasion• Campus bridging should be one mechanism by which we help bridge this

resource gap – making optimal use of aggregate US (open)cyberinfrastructure

Stewart, C.A., D.S. Katz, D.L. Hart, D. Lantrip, D.S. McCaulay and R.L. Moore. Technical Report: Survey of cyberinfrastructure needs and interests of NSF-funded principal investigators . 2011. Available from: http://hdl.handle.net/2022/9917

7

Key initial findings of ACCI Campus Bridging Task Force

• Finding 1: The cyberinfrastructure environment in the US is now much more complex and varied than the long-useful Branscomb Pyramid. This is largely due to the maturity of commercial cloud facilities, volunteer computing.

• Finding 3: The US open science and engineering research community is not using the existing cyberinfrastructure as effectively or efficiently as possible, primarily as a result of the current state of cyberinfrastructure software and the resulting barriers of migration among and between the many and varied campus and national cyberinfrastructure facilities.

• Finding 4: The existing, aggregate, national cyberinfrastructure is not adequate to meet current or future needs of the US open science and engineering research community.

Note: All of the ACCI Task Force on Campus Bridging recommendations quoted here are available online from http://www.nsf.gov/od/oci/taskforces/TaskForceReport_CampusBridging.pdf

8

ACCI Findings, continued

• Finding 5: A healthy national cyberinfrastructure ecosystem is essential to US science and engineering research and to US global competitiveness in science and technology. Federal R&D funding overall is not sufficient to meet those needs, and the NSF share of this funding is not sufficient to meet even the needs of basic research in those disciplines that the NSF supports.

• A key point of the entire Task Force’s work is that NSF funding alone is insufficient to solve the nation’s CI problems but that NSF leadership and use of funding to align expenditures nationally (a la NSFNet) can have tremendous impact on the nation.

• Similarity with EU large scale proposals is clear

9

XSEDE campus bridging vision• Create the software and training tools that will allow excellent interoperation

between XSEDE infrastructure, and excellent usability from the researcher’s standpoint for a variety of modalities and types of computing: traditional HPC as well as data and High Throughput Computing as well

• Promote better use, via XSEDE and campus bridging tools, of the nation’s aggregate CI resources – Recommending the use of InCommon for all authentication systems– By making it easier to use contribute campus systems (in whole possibly but

generally in part) to the aggregate capacity and capability of XSEDE– By making it easier to use systems that are not contributed to the aggregate of

XSEDE overall more effectively in the context of workflows and cyberinfrastructure that include resources within and beyond XSEDE in a well coordinated fashion

• Our goal is going to be to work with the various groups in XSEDE (particularly XAUS, Campus Champions, Documentation / Training) to align activities and communications so that XSEDE collectively does things in a way that achieves the goals above

10

XSEDE campus bridging activities

• To be conscientiously targeted at Data, HPC, and HTC – probably in that order

• Working closely with architecture team to help disseminate XSEDE's plans– XSEDE architecture plans => out to campus champions and

community– Funnel community response => back to XAUS

• Strategy: conscientiously make a small number of reasoned choices, pursue them with diligence, and reap economies of scale (if things go right) or clear learning experiences (otherwise)

11

XSEDE campus bridging tactics

• Tools for doing this:– Installers (thoroughly tested)– Documentation & training– Science Gateways (document by Surresh Marru and Marlon

Pierce) – Ability to contribute community resources for greater good– Call for pilot sites for data transfer installer coming soon

• Planning to visit campuses to raise awareness and work with campus personnel

12

More consistency in CI setups => economies of scale

• In reality, the four cluster admins depicted here being in agreement are all right.

• Experienced cluster admins all learned how to use what they learn when the tools were still developing, so the tool each sysadmin knows the best is the tool that lets that sysadmin do their work the best

• The only way to develop consistency is to provide installers that will make their work easier

• The XSEDE architecture group is developing installers for file management tools

• *A la Steven Colbert, the “4 out of 5…” comment is not intended to be a factual statement

Training and outreach

Image from TeraGridEOT: Education, Outreach, and Training 2010. https://www.teragrid.org/web/news/news#2010scihigh

• Consistency in system setups – local becoming more like XSEDE – should also lead to economies of scale in training

• Materials and trainer expertise will be more easily transportable and extensible

• The campus bridging group plans to work very closely with the campus champions

14

XSEDE Campus Bridging Staff

• Craig Stewart (reports to Scott Lathrop in Scott’s role leading outreach)

• Jim Ferguson (works 25% on campus bridging)• Therese Miller – IU overall project lead for XSEDE

activities (will be aiding, expected particularly in regards to campus champions)

• Rich Knepper – Manager, Core Services, RT/PTI (likely to be sucked into the vortex as well)

15

List of work products in addition to task force report available re Campus Bridging

• Report on Campus Bridging Technologies Workshop: Networking and Data Centric Issues.

• Report on Campus Bridging Technologies Workshop: Campus Bridging Software and Software Service Issues.

• Report on Campus Leadership Engagement in Building a Coherent Campus Cyberinfrastructure.

• A Roadmap for Using NSF Cyberinfrastructure with InCommon • A Roadmap for Using NSF Cyberinfrastructure with InCommon:

Abbreviated Version • Technical Report: Survey of cyberinfrastructure needs and interests of

NSF-funded principal investigators. • Technical Report: TeraGrid eXtreme Digital Campus Bridging

Requirements Elicitation Meeting. • All available from pti.iu.edu/campusbridging• More content coming to XSEDE site soon!

http://www.nsf.gov/od/oci/taskforces/

16

17

http://pti.iu.edu/campusbridging/

.

Acknowledgments• Many activities related to the general theme of Campus Bridging were supported by the

NSF through grant awards: 040777, 1059812, 0948142, 1002526, 0829462• The activities of the authors as part of the XSEDE project are supported through

subcontracts to IU from the NSF award to University of Illinois Urbana-Champaign through NSF grant award 0941686

• Thanks to the Lilly Endowment for its support of and the Pervasive Technology Institute• Thanks to those individuals who gave permission to use images presented in this talk• Any opinions presented here are those of the presenter and do not necessarily

represent the opinions of the National Science Foundation, the Lilly Endowment, the NSF ACCI, NSF ACCI Task Force on Campus Bridging, or any other funding agencies or organizations

• The following individuals deserve many thanks for their contributions to the materials on which these slides are based:– Guy Almes, Von Welch, Patrick Dreher, Jim Pepin, Dave Jent, Stan Ahalt, Bill

Barnett, Therese Miller, Malinda Lingwall, Maria Morris– Gabrielle Allen, Jennifer Schopf, Ed Seidel, all of the NSF program officers

involved. – All of the IU Research Technologies and Pervasive Technology Institute staff who

have contributed to this entire 2 1/2 year process of activities developing backround material and reports on the topic of campus bridging … and all RT and PTI staff who have helped IU be a good example of what a well connected campus can enable in scientific, scholarly, and artistic achievement.

License terms

• Please cite as: Knepper, R., C.A. Stewart and J.W. Ferguson. XSEDE Campus Bridging Birds of a Feather. (Presentation) TeraGrid '11 (Salt Lake City, UT, 18-21 Jul, 2011). Available from: http://hdl.handle.net/2022/13423

• Items indicated with a © are under copyright and used here with permission. Such items may not be reused without permission from the holder of copyright except where license terms noted on a slide permit reuse.

• Except where otherwise noted, contents of this presentation are copyright 2011 by the Trustees of Indiana University.

• This document is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.

19