A Grid Enabled Collaboratory for Scientific Research (GESCR) Charles Severance University of...
-
date post
21-Dec-2015 -
Category
Documents
-
view
216 -
download
1
Transcript of A Grid Enabled Collaboratory for Scientific Research (GESCR) Charles Severance University of...
A Grid Enabled Collaboratory for Scientific Research (GESCR)
Charles SeveranceUniversity of Michigan
Sakai Project, NEESGrid Project, OGCE Project
www.dr-chuck.com
Outline
• Collaborative Software
• Historical Context
• The Sakai Project
• NEESGrid Experience
• Comparing Civil Engineering and HENP
• The Concepts of GESCR
• Going Forward
Collaboration Happens
• As individuals, we are parts of many groups and have many roles in those groups
SakaiWLAP
OGCE
uPortal
Architecture
Tools
Board
Board
UM Sakai June MTG
Dec MTG Next Grant
Support
Support
Database
Next Ver
JSR-168 FusionGridNEESGrid
Data Model
Data Repo
Minnesota
Texas
Version 3Post Oct.
HENP
GESCRUM IssuesOne View of
Chuck’s Context Map
Another View
Another View
Maintaining the Map
• Read E-Mail and move to proper folders
• Copy attachments into folders
• Searching for information
• Making calendar entries from E-Mail
Imagine Software
• That could create a new “context” in a few clicks– Enroll/invite others to the context as necessary in a few more clicks
• Context capabilities– E-Mail list (automatically extracts attachments and places them in
folders which appear on your desktop)– Schedule (you can either see a “federated” schedule across all
contexts or look at one context)– Persistent browser-based chat - quite useful during meetings when
the Polycom or VRVS messes up :)– Resource area where anyone can upload files which appear on
everyone’s desktop at the same time (WebDAV) – Threaded discussion area for the context
• Problem: There are literally hundreds of solutions to portions of this problem.
More Software
• A single place to see new activity in your “contexts”• These contexts are stored on backed-up production
servers rather than your desktop for many years• A search across your contexts - that would be really
cool• The ability to customize each context in terms of look,
feel, and capabilities• The ability to build unique domain specific tools and
interfaces to extend the mechanism using Portlets, Servlets, or Applets
A 10-Year Collaborative Mission @ UM
19981991 - 1997 1999 2000 2001 2002 2003 2004 2005
SPARC
Science of Collaboratories
Sakai
Worktools (Notes Based) WTNG
Coursetools (Notes Based) CTNG
CHEF 1 CHEF 2
OGCE Grid Portal
NEESGrid
Portal TechnologyJetspeed 2.0uPortal 3.0
Websphere …
Channels, Teamlets
JSR-168 Portlets
CHEF Services
JSR-168 Technology
OKI Services
Legacy
SakaiTeamlet
OtherServices
Sakai GUI
SakaiTeamlet
Sakai GUI
Java Swing
SPARC
2/2001 600 users 800 data sources
CourseTools
Over 42,000 users at the end of 2003
WorkTools
Over 9000 users (2000 active) at the end of 2003
Digital libraries & documents
groups-to-information
groups-to-facilities
people-to-people
Communication,Collaboration
Services
Distributed,media-richinformationtechnology
Remote instruments
http://www.scienceofcollaboratories.org/
Science of Collaboratories
NSF Funded ITR
CHEF 1.0
• Fall 2001: CHEF Development begins – Generalized extensible framework for building collaboratories– “Best-of” CourseTools, SPARC, WorkTools
• Integrate across current UM projects and adopt relevant standards
• Funded internally at UM as replacement for CourseTools
• All JAVA - Open Source– Jakarta Jetspeed Portal– Jakarta Tomcat Servlet Container– Jakarta Turbine Service Container
• Build community of developers through workshops and outreach
Not “just” a portal
• Portals are a framework to deploy tools (aka rectangles) and focus on how the user wants to arrange their own “rectangles”
• While CHEF technically is a portal, the goal is for the tools to work together closely and seem to really be parts of a larger “tool”
• CHEF has a lot of features, (services, presence, notification, etc..) which bridge the gap between portal and application framework
CHEF Applications
• CourseTools Next Generation
• WorkTools Next Generation
• NEESGrid
• NSF National Middleware Grid Portal
CourseTools Next Generation
Over 5000 users at the end of 2003http://coursetools.ummu.umich.edu/
Worktools Next Generation
New WorkTools Sites being created in WTNG as of 12/2003Run on the same servers as CTNG.
NEESGrid - The EquipmentNetwork for Earthquake Engineering Simulation
NSF Funded. NCSA, ANL, USC/ISI, UM, USC, Berkeley, MSU
CHEF-Based NEESGrid Software
NMI / OGCE www.ogce.org
NSF National Middleware IniativeIndiana, UTexas, ANL, UM, NCSA
What we learned in 10 years.
• Portal technology is a good idea - forces component approach - functionality does not “smear”
• Portals are not just aggregators of independent information - but can be an application framework
• Many (but not all) tools can be used for both teaching and learning and research collaboration
• Separating functionality into lightweight GUI components and pluggable services with strong and well-specified APIs allows significant reusability
• GUI elements program to abstract service interfaces - not databases, file systems, LDAP, etc. - this allows great flexibility.
While we were building collaboratories…
• The Open Knowledge Initiative (OKI) at MIT was developing APIs for learning management systems - involving many universities (UM, Indiana, Stanford, and MIT were strong participants)
• Indiana, Stanford, MIT all developed learning management system
• Java Community Process (JCP) produced JSR-168 - The “unified” portal standard API
• Oasis developed the Web Services for Remote Portals (WSRP) standard
• The open-source uPortal portal project had quietly moved into the #1 open source portal (#4 including commercial vendors)
Jan 04 July 04 May 05
Michigan•CHEF Framework•CourseTools•WorkTools
Indiana•Navigo Assessment•Eden Workflow•Oncourse
MIT•Stellar
Stanford•CourseWork•Assessment
OKI•OSIDs
uPortal
SAKAI 1.0 Release•Tool Portability Profile•Framework•Services-based Portal•Refined OSIDs & implementations
SAKAI Tools•Complete CMS•WorkTools•Assessment
SAKAI 2.0 Release•Tool Portability Profile•Framework•Services-based Portal
SAKAI Tools•Complete CMS•Assessment•Workflow•Research Tools•Authoring Tools
Primary SAKAI ActivityArchitecting for JSR-168 Portlets,
Refactoring “best of” features for toolsConforming tools to Tool Portability Profile
Primary SAKAI ActivityRefining SAKAI Framework,
Tuning and conforming additional toolsIntensive community building/training
Activity: Ongoing implementation work at local institution…
Dec 05
Activity: Maintenance &
Transition from aproject to
a community
So we got together and drew an über collaboration picture…
Sakai Core Members
• Universities– Indiana– Michigan– MIT– Stanford
• Projects– Open Knowledge Initiative (OKI)– uPortal - JaSIG
• Funding ($6.8M - 2 Years)– Mellon Foundation– Hewlett Foundation– Partners Program– Core member match
KYOU / sakai
Boundary, Situation
Sakai Concepts
• It is neither research nor teaching, it is all “collaboration” - many common tools– Teaching: Courses, tools, drop-boxes– Research: Putting the user interface on the
Grid and Virtual Organizations
Teaching and Learning
CollaborativeResearch
Collaboration andLearning Environment
What we agreed to build…
• A Collaborative Learning Environment– Open Source– Uses OKI (Open Knowledge APIs)– Uses uPortal as its portal framework
• Similar to– Blackboard– WebCT
• And all four core institutions would deploy the commonly developed software
Collaboration and Learning Environment
• Learning management systems are really just a form of collaboration– Freshman Calculus– Chess Club– Group of 5 faculty members working on
curriculum– 2000 physics researchers collaborating
across the world on a 15-year physics experiment
Sakai 1.0
• Site based collaboration environment– Worksite management– E-Mail Lists– Threaded Discussion– Resources (folders) with WebDav support– Chat– No search yet :(– Many other tools
• Beta Release July 15, 2004• Production site available at ctools.umich.edu
More Sakai Beta Tools Admin: Alias Editor (chef.aliases) Admin: Archive Tool (chef.archive) Admin: Memory / Cache Tool (chef.memory) Admin: On-Line (chef.presence) Admin: Realms Editor (chef.realms) Admin: Sites Editor (chef.sites) Admin: User Editor (chef.users)Announcements (chef.announcements) Assignments (chef.assignment) C. R. U. D. (sakai.crud) Chat Room (chef.chat) Discussion (chef.discussion) Discussion (chef.threadeddiscussion) Dissertation Checklist (chef.dissertation) Dissertation Upload (chef.dissertation.upload) Drop Box (chef.dropbox)Email Archive (chef.mailbox)
Help (chef.contactSupport)Membership (chef.membership) Message Of The Day (chef.motd) My Profile Editor (chef.singleuser) News (chef.news) Preferences (chef.noti.prefs) Recent Announcements (chef.synoptic.announcement) Recent Chat Messages (chef.synoptic.chat) Recent Discussion Items (chef.synoptic.discussion) Resources (chef.resources) Sample (sakai.module) Schedule (chef.schedule) Site Browser (chef.sitebrowser) Site Info (chef.siteinfo) Web Content (chef.iframe) Worksite Setup (chef.sitesetup) WebDAV
Sakai Going Forward
• Focus on the”Learning” of Collaborative Learning Environment through 2Q05– Getting ready for production deployment at the four partner
sites– Improving the look and feel of the software– Many feature enhancements (to satisfy four + 60 schools)– New GUI Programming Environment based on Java Server
Faces– Building new set of Sakai APIs (Java)
• Based on OKI - Enabling RDF
• Move into OGCE and NEESGrid starting 3Q04• Release 2.0 - 2Q04
The George E. Brown, Jr. Network for Earthquake Engineering
Simulation (NEES)
NEES Founding
• George E. Brown, Jr. Network for Earthquake Engineering Simulation (NEES).
• Funded in 1999 - > $100M• Goal: Transform the nation’s ability to carry out
earthquake engineering research, to obtain information vital to develop improved methods for reducing the nation’s vulnerability to catastrophic earthquakes, and to educate new generations of engineers, scientists and other specialists committed to improving seismic safety.
• To be Completed: October 2004
• NEESgrid facilitates research capabilities previously unavailable• NEESgrid links earthquake researchers across the U.S. with
leading-edge computing resources and research equipment and allowing collaborative teams (including remote participants) to plan, perform, and publish their experiments
• NEESgrid is a coordinated and secure architecture/environment • NEESgrid is a modular and extensible environment with a
customizable user interface• NEESgrid provides common tools that allow leveraging resources
and experiences• Rather than having to worry about the required cyber
infrastructure, NEESgrid allows researchers to focus on the earthquake engineering challenges at hand
• The goal of the System Integrator (SI) is to develop NEESgrid as the Cyber Infrastructure that will facilitate this next generation of experimentation/simulation in earthquake engineering
NEES Components
• New experimental facilities (15) – Oregon State University, Rensselaer Polytechnic
Institute, University of Buffalo, University of Colorado at Boulder, University of Minnesota, University of Nevada at Reno, University of Texas at Austin, and the University of California campuses at Berkeley, Davis and Los Angeles
• Collaborative Software System: NEESGrid– Collaboration– Data capture and sharing– Tele-presense and Tele-operation– Simulation– Support for Hybrid Simulation and Physical Experiments
Shake table: Nevada, Reno
Reaction wall: Minnesota
Centrifuge: UC Davis
Wave basin: Oregon State
Field structural: UCLA
Field geotechnical: Texas
If we build it, they will collaborate
• Data and access to data represent fundamental barriers to dispersed collaboration
• Efficient movement of vast amounts of data is a prime rationale for cyberinfrastructure
• Federating, visualizing and mining data are principle challenges
Researchers
DataFacilities
Synchronized dataSynchronized data and imagesData discovery
Automatic archivingSimulation codesHybrid experiments
TeleoperationTeleobservation
Synchronous communicationAsynchronous communication
The collaboratory concept
NEES Resources
Field Equipment
Laboratory Equipment
Remote Users
Remote Users: (K-12 Faculty and Students)
Instrumented Structures and Sites
Leading Edge Computation
Curated Data Repository
Laboratory Equipment
Global Connections
(Faculty, Students, Practitioners) Simulation
Tools Repository
The Grid in NEESgrid
NEESpop A
Experimental Equipment
Video I/O
Audio I/O
Site A: Experimental Data Producer
Hub A
Telepresence Equipment
Active PI
Data Cache
Site B: Remote Lead Investigator
Hub B
Internet Fabric and Operations
Site C: Passive Collaborator
Teleobservation Equipment
Passive co-PI
Data Cache
Hub C
Grid Data Repository
Grid Operations
Center
Experimental Component
Campus Net Component
NEESgrid Component
NEESGrid Software
• Founding NMI Technologiess– Globus Toolkit– OGCE Collaboration Toolkit
• New Work– Data and Metadata Repository - NCSA– Data Acquisition, Storage, and Visualization– Simulation Portal– Synchronized data and video (live and stored)
NEESgrid Experiment Data FlowNEESgrid Experiment Data Flow
NEESGridData
Repository
ProjectBrowser
DataTurbine
DataIngestion
ExperimentControl
StreamingViewer
DAQC
D
SiteSpecific
ProjectRelated
ExperimentalSetup
ExperimentalElement
DataElement
Data Model
DAQDisk
StoredViewer
DT Main System
PTZ/USB
StillCapture
DT Client
BT848Video
Frames
DT Client
Capturing Video and Data
Camera ControlGateway
DAQData
CaptureDT Client
SimulationCoordinator
Site A Site B
DT Main System
Data Monitoring Tools
Still Image / Camera Control
~
< >^
^
< >
Camera ControlGateway
Creareviewers
Still imagecameracontrol
Thumb-nail
Video andData Tivo
Thumbnail + Audio + Data
< > +
SimulationCoordinator
NTCPServer
DAQ
CompSim
DAQ
NTCPServer
LiveExtractor
Quicktime
DAQ Data
DataRepository
SampleExperimentSetup
Data Models
• Data models are developed in RDF• Local repository supports multiple simultaneous data
models with cross-model linkages• Metadata browser (aka Project browser) becomes
the Project Browser, Notebook Browser, Site Specification Database Browser
• Metadata browser can federate multiple sources of Metadata
InstrumentationSetup
SensorGroup
Sensor
Specimen
EquipmentSetup
CalibrationSet
Equipment
CameraDataEquipment
Overall Data Modeling EffortsOverall Data Modeling Efforts
NEES
Site A Site CSite B
Equipment People
Experiments Trials
Equipment People
Experiments Trials
Data Data Data
TsnumaiSpecimen
Shake TableSpecimen
GeotechSpecimen
CentrifugeSpecimen
Units Sensors Descriptions
SiteSpecificationsDatabase
ProjectDescription
Domain Specificmodels
Common Elements
Data / Observations
Ref. Source: Chuck Severance
Models + Data Model
Repo
Models
Configure
Data
Load
Con
figur
e
RDF/OWL
RDF<owl:ObjectProperty rdf:ID="hasPublications"> <rdfs:domain> <owl:Class> <owl:unionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Project"/> <owl:Class rdf:about="#Task"/> </owl:unionOf> </owl:Class> </rdfs:domain> <rdfs:range rdf:resource="#Publications"/> </owl:ObjectProperty>
Protégé - 2K
Models + Data Model
Repo
Models
Configure
Data
Load
Con
figur
e
RDF/OWL
RDF
DOE ELN / Example
NEESgrid: Simulation
Lessons Learned in NEESGrid (Chuck’s views)
• A collaborative environment *must* have a user interface• Just asking about requirements is a tiny part of the problem -
CS developers must learn to “walk in the shoes” of the scientists.
• Scientists know a lot about Computer Science - listen to them and involve them
• The real work begins after software is “delivered” - Version 1.0 is usually just a conversation starter - but a very important step
• There are some things that are useful across fields - but the most valuable elements are field-unique
• Data models, data repositories, and long-term curation is difficult!
HENP/Civil Similarities
HENP/Civil: Differences
• Willingness to collaborate– CE: Not too keen on the idea - NSF forcing the issue
using “carrot and stick”– HENP: It is part of the fabric of the field (at least within
Atlas and CMS :) )
• Technology Savvy– CE: It is all about the “within lab” electronics– HENP: In lab electronics is difficult, important and
different. The internet is a tool to be used both for human communication and data manipulation - have tried everything - use simple reliable stuff because the work cannot wait.
NEES Going Forward
• Release 3.0 complete July 31, 2004– Improve documentation and bug fixes through
September 30, 2005
• As of October 1, 2004, NEESGrid will transition from development lead by NCSA to maintenance led by SDSC– Focus on deployment and support– UM will continue involvement - move to Sakai -
improving data modeling, internationalization (Japan)
GESCR* - The Vision
* A Grid Enabled Collaboratory for Scientific Research
HENP: Unique Aspects
• Already highly collaborative as a scientific field
• Globally distributed in space and time (both time-zones and years)
• Technology savvy but wary :)• Solutions underway (iVDGL, GriPhyn,
etc) without significant user interface component
GESCR in a sentence..
….. will combine the best open-source applications from within the HENP communities (and from allied Grid and National Middleware efforts) within a common portal interface.
Put another way: There is a lot of work going on in HENP that does not really have much of a user interface yet. We want to put a user interface on the Grid and other activities.
GESCR in a sentence..
….. will combine the best open-source applications from within the HENP communities (and from allied Grid and National Middleware efforts) within a common portal interface.
What I tell people:GESCR will be a tri-corder for the Atlas and CMS collaborations. Starting with inter-human communication and moving towards data processing and analysis.
Elements of GESCR
• Sakai with Grid Support (from OGCE / NEESGrid)
• Sakai Collaboration Tools• Well funded analysis of application,
requirements, and challenges throughout the project
• Additional new capabilities– VRVS Integration (rooms, record, playback)– Web Lecture Archive (WLAP)– VNC support
GESCR Elements (cont)
• MonaLisa integration• Collaborative LaTeX editing environment• Peer-to-Peer file sharing• Language of Access to handle “over-
collaboration”• Adaption of HEPBook• GESCR support office (Y3)• Education and outreach activities
GESCR Project
• Planned: Four years - Medium ITR (NSF)• Partners: Caltech, Fermilab, Florida International
University, Lawrence Berkeley, Maryland, University of Michigan, University of Texas Austin, University of Iowa
• Joint between Atlas and CMS• Has been submitted twice to the NSF ITR program at
the urging of the Physics directorate to create a CMS/Atlas joint IT effort
GESCR - Going Forward
• Waiting for funding decision on most recent submission– If funded, work on unique physics capabilities
(VRVS, MonaLISA, etc) will begin immediately– If not funded, leverage NEESGrid / Open Grid
Computing Environment / Sakai - focus on the generic tools - deploy in a physics context
• Immediately, we can begin some experimental usage for interested groups using the University of Michigan CTools service
University of Michigan: Ctools
• University of Michigan runs a large scale professional server cluster for teaching and learning and research collaboration– Research collaboration simply needs a single UMich
member of each group - the rest of the members have no-charge “friend” accounts based on E-Mail addresses.
– For multiple accounts, it is best to have a UM person to act as your “problem solver” familiar with your group. If there is interest, we can provide staff from the MGrid (www.mgrid.umich.edu) organization to support HEP collaborative activities.
• ctools.umich.edu
Sakai-Based CTools @ UM
In production August 1, 2004.
Summary
• We will deploy collaboration software for Atlas / CMS sooner or later
• If / when we get funding it will accelerate the process and provide Physics specific capabilities
• Collaborative portals is a well-funded and well-coordinated activity (Sakai 30+FTE, NEESGrid 10+FTE, OGCE 5+ FTE)
• It is (almost) a no-lose situation - the longer we wait, the more mature the software will be