Grid of Grids Information Management

Kick off Meeting

AnabasJune 30 2006

ACKNOWLEDGMENTS

We are grateful to:

• Bill McQuay, • Raymund Garcia•The Air Force Research Laboratory - AFRL IFSD• Indiana University CGL (Subcontractor Partner)• Ball Aerospace (Subcontractor Partner)

General Goal• Build Net Centric Core Enterprise Services

in fashion compatible with GGF/OGF and Industry

• Add key additional services including those for sensors and GIS

• Support Systems of Systems by federating Grids of Grids supporting a heterogeneous software production model allowing DoD greater sustainability and choice of vendors

• Build tool to allow easy construction of Grids of Grids

Gerstner and Commercialization of GoG Architecture• Reference [1]: Who Says Elephants Can't Dance, Louis Gerstner, Jr., 2002 is practically

a primer on IBM's vision and strategy to re-gain its influence to setting IT industry agenda.

• "The industry's disaggregation into thousands of niche players would make IT integration services a huge growth segment of the industry."  [1] – Grid of Grids builds the information management infrastructure for efficient

integration of components.• "Over the next decade, customers would increasingly value companies that could

provide solutions - solutions that integrated technology from various suppliers and more importantly, integrated technology into the processes of an enterprise."  and "The two companies that have enjoyed the highest market valuation in the IT industry in over much of the last decade have been component manufacturers -- Intel and Microsoft. There is no doubt that a strategy built around providing fundamental building blocks of the computing infrastructure has proven to be extremely successful in this industry."  [1] – Grid of Grids has huge commercial potential by offering both the integration

infrastructure for component manufacturers and providing Network-Centric Enterprise Services as essential building blocks .

• "The dazzlingly wired-world would also change the course of computing in profound ways. For one thing, it was virtually certain that world would be built on open industry standards. There would be no other way to fulfill the promise of ubiquitous connections among all the businesses users, devices, and systems that would participate in a truly networked world. If that standards-based world came to pass, it would represent a major shift in the prevailing competitive landscape." [1]  – Grid of Grids information management with its foundation on GGF/OGF open

standards is a key to integration.

Technical Advisory Board• Build a group of some 6-10 knowledgeable people

to guide project • Mainly from Industry including Boeing, Lockheed

Martin, Ball Aerospace, Aerospace Corporation, Mitre, DISA, and the core technology companies (IBM, Microsoft)

• Use to guide both technology and commercialization strategy

• Meet F2F, email and telecons.• Brings significant external expertise to Grid of

Grids team, and visibility of Grid of Grids to other world-renowned institutions.

Database

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

FS

FS

FS

FS

FS

FS

FS

FS FS

FS

FS

FS

FS

FS

FS

FS

FS FS

FS

FS

PortalFS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

MD

MD

MD

MD

MD

MD

MD

MD

MD

MetaDataFilter Service

Sensor Service

OtherService

AnotherGrid

Raw Data Data Information Knowledge Wisdom

Decisions

SS

SS

AnotherService

AnotherService

SSAnother

Grid SS

AnotherGrid

SS

SS

SS

SS

SS

SS

SS

SS

FS

SOAP Messages

Portal

The Grid and Web Service Institutional Hierarchy

OGSA GS-*and some WS-*GGF/W3C/….XGSP (Collab)

WS-* fromOASIS/W3C/Industry

Apache Axis.NET etc.

Must set standards to get interoperability

2: System Services and Features(WS-* from OASIS/W3C/Industry)

Handlers like WS-RM, Security, UDDI Registry

3: Generally Useful Services and Features(OGSA and other GGF, W3C) Such as

“Collaborate”, “Access a Database” or “Submit a Job”

4: Application or Community of Interest (CoI)Specific Services such as “Map Services”, “Run

BLAST” or “Simulate a Missile”

1: Container and Run Time (Hosting) Environment (Apache Axis, .NET etc.)

XBMLXTCE VOTABLECMLCellML

The Ten areas covered by the 60 core WS-* Specifications

WS-* Specification Area Examples

1: Core Service Model XML, WSDL, SOAP

2: Service Internet WS-Addressing, WS-MessageDelivery; Reliable Messaging WSRM; Efficient Messaging MOTM

3: Notification WS-Notification, WS-Eventing (Publish-Subscribe)

4: Workflow and Transactions BPEL, WS-Choreography, WS-Coordination

5: Security WS-Security, WS-Trust, WS-Federation, SAML, WS-SecureConversation

6: Service Discovery UDDI, WS-Discovery

7: System Metadata and State WSRF, WS-MetadataExchange, WS-Context

8: Management WSDM, WS-Management, WS-Transfer

9: Policy and Agreements WS-Policy, WS-Agreement

10: Portals and User Interfaces WSRP (Remote Portlets)

Activities in Global Grid Forum Working Groups

GGF Area GS-* and OGSA Standards Activities

1: Architecture High Level Resource/Service Naming (level 2 of slide 6),Integrated Grid Architecture

2: Applications Software Interfaces to Grid, Grid Remote Procedure Call, Checkpointing and Recovery, Interoperability to Job Submittal services, Information Retrieval,

3: Compute Job Submission, Basic Execution Services, Service Level Agreements for Resource use and reservation, Distributed Scheduling

4: Data Database and File Grid access, Grid FTP, Storage Management, Data replication, Binary data specification and interface, High-level publish/subscribe, Transaction management

5: Infrastructure Network measurements, Role of IPv6 and high performance networking, Data transport

6: Management Resource/Service configuration, deployment and lifetime, Usage records and access, Grid economy model

7: Security Authorization, P2P and Firewall Issues, Trusted Computing

Net-Centric Core Enterprise Services Core Enterprise Services Service Functionality

NCES1: Enterprise Services Management (ESM)

including life-cycle management

NCES2: Information Assurance (IA)/Security

Supports confidentiality, integrity and availability. Implies reliability and autonomic features

NCES3: Messaging Synchronous or asynchronous cases

NCES4: Discovery Searching data and services

NCES5: Mediation Includes translation, aggregation, integration, correlation, fusion, brokering publication, and other transformations for services and data. Possibly agents

NCES6: Collaboration Provision and control of sharing with emphasis on synchronous real-time services

NCES7: User Assistance Includes automated and manual methods of optimizing the user GiG experience (user agent)

NCES8: Storage Retention, organization and disposition of all forms of data

NCES9: Application Provisioning, operations and maintenance of applications.

Produce the Needed Core Services

• We can classify services in many ways and following 2 charts are one way; slightly changed from proposal as NCOW and our work changed a little.

• Green is “in hand”; our team leaders• Orange is “in hand” with outside but available

solutions• Red has problems – Security does not have

industry consensus while current Scheduling work does not address DoD real-time service and network requirements

The Core Features/Service Areas IService or Feature WS-* GS-* NCES

(DoD)Comments

A: Broad Principles

FS1: Use SOA: Service Oriented Arch.

WS1 Core Service Architecture, Build Grids on Web Services. Industry best practice

FS2: Grid of Grids Distinctive Strategy for legacy subsystems and modular architecture

B: Core Services

FS3: Service Internet, Messaging

WS2 NCES3 Streams/Sensors. Team

FS4: Notification WS3 NCES3 JMS, MQSeries.

FS5 Workflow WS4 NCES5 Grid Programming

FS6 : Security WS5 GS7 NCES2 Grid-Shib, Permis Liberty Alliance ...

FS7: Discovery WS6 NCES4 UDDI

FS8: System Metadata & State

WS7 Globus MDSSemantic Grid, WS-Context

FS9: Management WS8 GS6 NCES1 CIM

FS10: Policy WS9 ECS

The Core Feature/Service Areas IIService or Feature WS-* GS-* NCES Comments

B: Core Services (Continued)

FS11: Portals and User assistance

WS10 NCES7 Portlets JSR168, NCES Capability Interfaces

FS12: Computing GS3

FS13: Data and Storage GS4 NCES8 NCOW Data StrategyFederation at data/information layer major research area; CGL leading role

FS14: Information GS4 JBI for DoD, WFS for OGC

FS15: Applications and User Services

GS2 NCES9 Standalone ServicesProxies for jobs

FS16: Resources and Infrastructure

GS5 Ad-hoc networks

FS17: Collaboration and Virtual Organizations

GS7 NCES6 XGSP, Shared Web Service ports

FS18: Scheduling and matching of Services and Resources

GS3 Current work only addresses scheduling “batch jobs”. Need networks and services

Additional Services• Sensors have low level support listed as FS3; higher

level integration using SensorML and Filters well understood. Some work in phase I

• GIS Grid services pioneered by team and already shown in phase I

• Mediation (Interoperability) Services needed to link Grids (defined as a collection of ≥ 1 Services)– Need to generalize existing solutions for Sensor Grids

and for MQSeries-SOAP Mediation– View NaradaBrokering as a SOAP Intermediary

Grid of Grids Building Tool• This will provide a graphical interface to build grids

from existing libraries of Services and Grids• Meta-data (provenance) needs to be specified• Grids (services) need to be linked• This will be built by extending an existing workflow

engine which is aimed at a more tightly couple version of the builder problem– We will consider HPSearch (CGL), Taverna (Open

source from UK OMII), BPEL with user interface (OMII or IU LEAD project)

– Semantic Grid provenance “add-ons”– We have experience with all these approaches

Applications• Indiana University will apply to Earthquake Science

and Emergency Response• Ball Aerospace will apply to Matrix• Anabas will work with Technical Advisory Board to

identify commercialization plan and possible prototype applications in first 12 months

• Anabas will implement appropriately in last half of project

Out of Scope for Phase II• Many areas are still evolving significantly

– Mediation/Interoperation– Security– Scheduling of non-compute Resources– Data/Information Federation– Semantic Grid and management

• We will not test scalability on large number of services, sensors and component Grids

• Integrating legacy systems not addressed• Grid of Grids building tool is “new idea” – can expect will benefit from

further work

Research Tasks

The R&D Effort is divided into five major tasks:

• Task 1: Implementation of Collaboration Grid Middleware

• Task 2: Enhanced NCOW Core Enterprise Services (NCES) with Enterprise Control Services and Metadata Services

• Task 3: Design and implementation of Grid of Grids mediation algorithms and NCOW services

• Task 4: Design and Implementation of Net-Centric Collaboration Grid Builder Tool

• Task 5: Technology Demonstration

Task Organization

1. Requirements2. Analysis3. Design4. Implement5. Test & Measure6. Repeat

Task Assignments

Task Anabas Ball CGLRequirementsCollaboration Grid Middleware (NCGGCM) (P) (S) (S)NCOW, GiG (P) (S)CES+ (11 service layers) (P) (S) (S)GWSA (P) (S) (S)Grid of Grids (GoG) (P) (S) (S)Mediation Algorithms (P) (S) (S)K2 service (S) (P)NCOW services (e.g., JBI publish, etc.) (S) (P)Net-Centric Collaboration Grid Builder Tool (P) (S) (S)CES+ (P) (S) (S)Wizards for each service (P) (S) (S)Collaboration and Messaging (P) (S) (S)GIS, Sensor and additional services (P) (S) (S)GIS and sensor (P) (S) (S)K2 service (S) (P)NCOW services (e.g., JBI publish, etc.) (S) (P)C2IEDM service (S) (P)Demonstration/Commercial Products (C) (C) (S)Crisis Management for Earthquake (C) (S) (C)Customer provided suggestions for DoD product (C) (C)

Task Assignments

Task Anabas Ball CGLResearch and Concept AnalysisCollaboration Grid Middleware (NCGGCM) (P) (S) (S)Grid of Grids (GoG) (P) (S) (S)Net-Centric Collaboration Grid Builder Tool (P) (S) (S)GIS, Sensor and additional services (P) (S) (S)Demonstration/Commercial Products (P) (S) (S)

Task Assignments

Task Anabas Ball CGLDesignCollaboration Grid Middleware (NCGGCM) (P) (S) (S)Grid of Grids (GoG) (P) (S) (S)Mediation Algorithms (P) (S) (S)K2 service (S) (P)NCOW services (e.g., JBI publish, etc.) (S) (P)Net-Centric Collaboration Grid Builder Tool (P) (S) (S)GIS, Sensor and additional services (P) (S) (S)GIS and sensor (P) (S) (S)K2 service (S) (P)NCOW services (e.g., JBI publish, etc.) (S) (P)C2IEDM Services (S) (P)Demonstration/Commercial Products (P) (S)

Task Assignments

Task Anabas Ball CGLImplementCollaboration Grid Middleware (NCGGCM) (P) (S) (S)Grid of Grids (GoG) (P) (S) (S)Mediation Algorithms (P) (S) (S)K2 service (S) (P)NCOW services (e.g., JBI publish, etc.) (S) (P)Net-Centric Collaboration Grid Builder Tool (P) (S) (S)GIS, Sensor and additional services (P) (S) (S)GIS and sensor (P) (S) (S)K2 service (S) (P)NCOW services (e.g., JBI publish, etc.) (S) (P)C2IEDM service (S) (P)Demonstration/Commercial Products (P) (S)

Task Assignments

Task Anabas Ball CGLTesting & EvaluationCollaboration Grid Middleware (NCGGCM) (P) (S) (S)Grid of Grids (GoG) (P) (S) (S)Mediation Algorithms (P) (S) (S)K2 service (S) (P)NCOW services (e.g., JBI publish, etc.) (S) (P)Net-Centric Collaboration Grid Builder Tool (P) (S) (S)GIS, Sensor and additional services (P) (S) (S)GIS and sensor (P) (S) (S)K2 service (S) (P)NCOW services (e.g., JBI publish, etc.) (S) (P)C2IEDM service (S) (P)Demonstration/Commercial Products (P) (S)

Task Assignments

Task Anabas Ball CGLRefine and RefactorCollaboration Grid Middleware (P) (S) (S)Grid of Grids (GoG) (P) (S) (S)Net-Centric Collaboration Grid Builder Tool (P) (S) (S)GIS, Sensor and additional services (P) (S) (S)Demonstration/Commercial Products (P) (S)

Documentation

Monthly Report

Final Report

Reviews &

Demonstrations

Refine & Refactor

Testing & Evaluation

Implement

Design

Research & Analysis

Requirements

242322212019181716151413121110987654321Task and Event

Month from Contract Start

Milestones & Timelines