Evolutionary Pressures on Grid Technology and MarketsEvolutionary Pressures on Grid Technology and...

© 2006 Open Grid Forum

Evolutionary Pressures onGrid Technology and Markets

Craig A. Lee, President, OGF

Grid World 2008Tokyo International ForumTokyo, Japan -- June 25, 2008

2© 2008 Open Grid Forum

Innovation “no man’s

land”OGF

Driving Innovation

Research push Market pull

Phase 1Solution proposal

Phase 2Prototype

Phase 3Pre-commercial product/service

Phase 4Commercial

product/service

Phase 0Research

Market Pull

Managing the Technology Maturation Process

Technology Readiness Levels

Ulf Ulf DahlstenDahlsten, Director , Director ‘‘EmergingEmergingTechnologies and InfrastructuresTechnologies and Infrastructures’’


Technology Readiness Levels9. Actual system 'flight proven' through successful

operational use

8. Actual system completed and 'flight qualified' through test and demonstration

7. System prototype demonstration in a operational environment

6. System/subsystem model or prototype demonstration in a relevant environment

5. Component and/or breadboard validation in relevant environment

4. Component and/or breadboard validation in laboratory environment

3. Analytical and experimental critical function and/or characteristic proof of concept

2. Technology concept and/or application formulated

1. Basic principles observed and reported

Req

uire

men

ts

Capabilities


Evolutionary Pressures

•• Technology Technology ---- and Markets and Markets ---- Never Stand StillNever Stand Still• The tension between Requirements and

Capabilities creates pressure on many fronts• Data Center Demand• Green IT• Clouds and Virtualization• Sustainability

• OGF is addressing all of these areas


Demand outstripping supply 3:1Demand outstripping supply 3:1

Rich media and social computing

New York City London

Source: www.Bastionhost.com

Global Demand for Data Centers

Bill St. Arnaud

“It is estimated that 1/3 of all servers shipped go to data centers for Google, Amazon, Yahoo, eBay, Microsoft and a few others.”


Today’s Data Center

• Utility & Cloud Services• Large-scale infrastructures• Scheduling Services• Provisioning• Discovery & Inventory Management• Rapid Diagnosis & Repair • Patch Management• Compliance Measurement, Enforcement, &

Reporting• Server & Application Provisioning• Server & Application Configuration Changes• Ad-Hoc Tasks & Remote Administration• Application Performance & Availability• Network Management & Performance• Service Level Management• Business Service Management

• IT-Driven

• Commoditization

• Availability

• Undetected Changes

• Unscheduled Downtime

• Over Provisioned

• Complex Sprawl

Data Center Pressures Available SolutionsData Center

Enterprise Datacenter

InfrastructureServices

& Service Oriented

Infrastructure

Shawn Shawn FindlanFindlan,, West Avenue West Avenue Management Partners,Management Partners, LLCLLC


OGF Data Center Reference Model:a Grid Component Instantiation Dependency Graph

OGFOGF Reference Model, v2.0E Draft, D. Reference Model, v2.0E Draft, D. SnellingSnelling, Fujitsu, P. Strong, eBay, Fujitsu, P. Strong, eBay


Grid Management Entities

OGFOGF Reference Model, v2.0E Draft, D. Reference Model, v2.0E Draft, D. SnellingSnelling, Fujitsu, P. Strong, eBay, Fujitsu, P. Strong, eBay


Spaceship Earth,Climate Change & Green IT

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.





Huge data center demand and finite resourcescreate clear motivation for a

Zero Carbon Strategy for CyberZero Carbon Strategy for Cyber--InfrastructureInfrastructure


Example:Grid in a Box at a Windmill


Bill St. Arnaud

• Extensive use of virtualizationvirtualization and fiber opticsfiber optics to put the computing where the power is

• Enable the workload to “follow the wind”or

“follow the sun”


OGF-23 Workshop onOptimization of Energy Efficiency

• Organized by Grid Computing Now!• Some Scenarios (Dave Berry, GCN!)

• Optimize data center utilization, cooling and UPS• Use existing resources as a desktop grid (no air

conditioning or UPS)• Buy compute capacity from an energy efficient

cloud provider• Chase the "night-rate" -- move jobs among

international sites according to lower diurnal energy costs

• Off-peak trading -- trade your excess capacity with others


Some Potential Standards

Courtesy ofDave Berry

(Computer RoomAir Conditioning)


EU Data Centre Code of Conduct• Covers best practices, required practices, metrics

and measurements for data centre energy usage• Draft available. Final document due September, 2008

• Paolo Bertoldi, EU Directorate General JRC• ".. inform and stimulate Data Centre operators to reduce

energy consumption in a cost effective manner without hampering the critical function ....”

• Grid and Virtualisation • “Processes should be put in place to require senior business

approval for any new service that requires dedicated hardware and will not run on a resource sharing grid or virtualised platform”

•• Growing motivation at the highest levels to pursue Growing motivation at the highest levels to pursue Green IT through grid concepts and standardsGreen IT through grid concepts and standards


Virtualization and Cloud Computing

•• ““ThereThere’’s Grid in them s Grid in them thar thar clouds!clouds!””• I. Foster’s blog, ANL & UC, Jan. 8, 2008

• Grids Encompass Clouds• Clouds have a simple user API effectively

hiding all the complexity of an ad hoc grid on the back-end, e.g., Amazon’s EC2 & S3, IBM’s Blue Cloud, Google’s App Engine, …

• If so, will this enable mass-market grids?• Users don’t have to be aware of using “a grid”

• If so, what does “cloud interoperability” require?• Is virtualization a means of achieving this?

• But there are different types of “clouds”!


A “Cloudy” Solution Space

OS Virtualization

OS Virtualization

Parallel Frameworks

Parallel Frameworks

Software as a Service

Software as a Service

MapReduce, GFS, BigTable

Amazon S3/EC2

Hadoop

Hadoop over EC2

Google AppEngine

Cohesive

Sun’s Caroline

MS Astoria

Mesh

RightScale, GigaSpaces,Elastra, 3Tera

Courtesy of Dennis Gannon, www.extreme.indiana.edu/~gannon/science-data-center.pdf

mapmap mapmap mapmap mapmap mapmap mapmap mapmap mapmap

reducereduce reducereduce

reducereduce

Data Collection Data Collection

…..

Vendors of highVendors of high--levellevelapplication configurationapplication configurationtools are potentialtools are potential““cloud framework providerscloud framework providers””


A Host of Issues• Data access and interoperability

• Will have to be approached at the application domain level, by the domain users involved

• Security• Can cloud providers provide sufficient security for sensitive applications?

• Reliability• Can cloud providers provide sufficient reliability for critical applications?

• Frameworks• How to manage sets of VMs and VOs?• These are essentially “cloud frameworks” and call for "cloud framework

providers"• Performance management

• Clouds tend to abstract away location• Managing the compute-data locality (affinity) can be important

• Costing models• How to compare your own infrastructure costs with a cloud computer?• How to compare two clouds?


OGF’s On-Going Work

• Identify/write use cases to feed DMTF virtualization work• Portals, aggregated virtualized resources, cloud computing• Leverage OGF’s Grid-Virt WG use case work as appropriate

• Packaging: OVF vs rPath• Evaluate, identify, and input changes to potentially merge OVF and

rPath so there is a single packaging format for the next version of the OVF spec

• EU Reservoir Project• EU FP7 project to provide “surge compute capacity”• Collaborating with Grid-Virt WG• OGF-Europe is participating member

•• This is just the beginning for cloud work in OGF!This is just the beginning for cloud work in OGF!


Run Your Own Cloud!

• Cloud client recently available• http://workspace.globus.org/clouds

• Deploy a cloud using your own resources• Any size cluster “on the back-end”

• Easy command line tools to deploy applications “in the cloud”

• Possible to interoperate with Amazon EC2


Sustainability• Moving Grids from “Research Infrastructure”

to “Operational Systems”• Application Domains

• Financial Services• Geospatial• Life Sciences• Digital Repositories & Digital Libraries

• European Efforts• The European Grid Initiative• OGF-Europe

• Focused Stakeholder Projects• A potential OGF “Business Model”


Grid Use Continues to Grow Exponentially• Firms Moving from 1,000s to 10,000+ and 100,000+ Nodes• Insatiable Demand – If Its Built, It Will Be Used• Demand is Being Driven Globally• Strategic and Market-Driven Events• Some Shared Ideas & Cross-Pollination• … Others more secret of perceived competitive advantageFinancial Services Drivers• Analytics• Risk Management• Pricing• Real Time & Batch Processing• New Regulations

Grids in Financial Services



Operations - Data Centers & Massive Infrastructures• Clusters of Automation = Disparate Management Systems• Additional Space / Power / Cooling Issues

• More Issues due to Latency Requirements and Proximity to Metro Areas• Operations & Management Tooling Can’t Keep Up!• Tension Between Stability (Stagnation) & Agility (Instability)

Evolving ArchitecturesGrids “Real Time Infrastructure”• Massive Scale, Continuous Availability• Rapid Change & Agility• Grids Don’t Solve all Problems• Real-Time Infrastructure & Utility Computing

• Applications & Services – Clouds, SOA et al• Platform – Virtualization, Multi-Core, Grid etc.

Financial Services Challenges


OGF engaging banks and vendors on existing OGF engaging banks and vendors on existing information models (CIM and GLUE)information models (CIM and GLUE) to to

coordinate vendor productscoordinate vendor products


Geospatial Data: Immense Applicability

Image: E. Gennai, C. Terbough, ESRI

Estimates vary, but ~80Estimates vary, but ~80--90% of all data collected or 90% of all data collected or produced by the human race is geospatially referencedproduced by the human race is geospatially referenced

• Natural exploration, e.g., oil & gas• Public administration• Civic planning and engineering• Weather and aviation• Satellite ground systems• Environmental planning• Disaster management


A Prime Example: GEO Grid

Global Earth Observation

GridS. Sekiguchi,

AIST


The OGCThe OGC--OGF CollaborationOGF Collaboration

• Promote best practices and international standardization for distributed geospatial data processing capabilities that is:•• TransparentTransparent -- users is not aware of the infrastructure•• InteroperableInteroperable -- the resources work together•• ScalableScalable -- small local, to massive distributed platforms

• Initial Technical Goals• Integrate OGC's Web Processing Service (WPS) with grid “back-end”,

programming and workflow tools• Integration of OGC Federated Catalogues and Data Repositories with grid

data movement tools• Jointly pursue technical goals in OWS-6

• OGC Web Services Testbed (OWS) is a (roughly) annual effort where sponsors define focused, near-term projects to achieve key tech. goals

• Invited paper in November IEEE Computer (just in time for SC08)


Possible Geospatial/Grid Topics• Integration, interoperability of geospatial and computing resource

• Information models (CIM/GLUE)• Distributed and federated catalogs (ebRIM, MDS, etc.)• Discovery services

• Back-ending of WPS into various grid systems• TeraGrid, NaReGI, NGS, EGEE

• Implementation of WPS on various grid tools• SAGA, GridRPC, HPC-Basic Profile

• Integration of WS-Eventing, WS-Notification, INFOD, etc., with geospatial systems, e.g., SWE

• Integration of configuration and lifecycle management tools (e.g., CDDLM) with geospatial systems

• Integration of WPS with workflow management tools• BPEL, Kepler, Taverna, Triana, Pegasus, etc.• Workflow design environments, execution engines, planning (data virtualization), data

provenance• Integration of WPS with grid security models

• SAML, XACML, VOMS, GSI• Support for virtual organizations

• Integration of geospatial data with rule-based, data management policy engines• iRODS

• Many other topics possible -- any choice depends on sponsor & user requirements!


Life Sciences

• Many established life science applications using grids• In silico docking for drug discovery, e.g., WISDOM

• Three projects discussed with Vincent Breton (CNRS, France; App. Coordinator for EGEE)• Avian Flu Surveillance Network• Tropical Disease Virtual Screening• Drug Distribution Tracking

• World-wide projects involving grid resources on potentially all continents• OGF would be the coordination venue• Life Sciences is an identified topic for OGF-

Europe• Searching for funding


Digital Repositories/Libraries• Digital Repositories/Libraries represent a growing part of

the “infomass”• Federated catalogues and federated storage

• OGF-Europe Workshop at OGF-23• 4 sessions, ~70 participants from industry, academia and

European Commission• Proposed OGF-Europe collaboration with Global Research

Library 2020• Proposed OGF-Europe involvement in GENESI-DR Digital

Repositories Seminar in Autumn 2008, Frascati• OGF-Europe PC membership for the DReSNet special DR

session at IEEE e-Science conference, Dec 2008• Eventual development of an OGF Working Group

dedicated to issues surrounding DRs


European Sustainability: EGI



30© 2008 Open Grid Forum 30

OGF-Europe Funded by EU

• The EU recognises interoperable software standardsinteroperable software standards are important for broader, more mainstream adoption

• OGF-EUROPE project crucial for realization of past 8 years of European investments in Grid computing into actionable and sustainable results, targeting the Small-to-Medium Enterprise sector

• OGF‐EUROPE: www.ogfeurope.eu• Surveys, Workshops, Community Outreach Seminars, International Events, Student Scholarship, Expert Fellowships

• 10 European Partners


A “Business Model” for OGF?

• What drives standardization?• Need for commonality and best practices across a significant

user community

• Technical Requirements• Feasible & appropriate to codify in the technical design

• Marketplace Drivers• User community must have critical mass• Major vendors will have no motivation to standardize otherwise

• Genuine standardization with wide-scale adoption will only occur when all of these conditions are met

•• How can we drive this process?How can we drive this process?•• How can we drive grid adoption and sustainability?How can we drive grid adoption and sustainability?


A Case Study: HPC Basic Profile• 2006

• Key stakeholders decided to demonstrate interoperabilitybetween their existing job submission infrastructures

• November 2006 (SC06)• Prototype implementations demonstrated

• 28-August-2007• HPC Basic Profile, Version 1.0, published• http://www.ogf.org/documents/GFD.114.pdf

• November 2007 (SC07)• Interoperability demonstrated by Altair, Microsoft, Platform,

OMII-UK, OMII-Europe, EGEE, UVa• 21-February-2008

• Interoperability Experiences with HPCBP, Version 1.0, published http://www.ogf.org/documents/GFD.124.pdf

• Commercial adoption plans by Altair, Microsoft & Platform


How to “Bottle” this Process??

•• Build Critical Mass of Key StakeholdersBuild Critical Mass of Key Stakeholders•• Continual polling and coordination across Continual polling and coordination across

the communitythe community•• They mustThey must agree on:agree on:

•• Clear GoalsClear Goals•• Clear Schedule (Clear Schedule (““timetime--boxbox”” the process)the process)•• Clear ResponsibilitiesClear Responsibilities•• Properly Provisioning the EffortProperly Provisioning the Effort

34© 2008 Open Grid Forum34

A General Process ModelA General Process Model

**RFQ/CFP = Request for Quotation/Call for ParticipationRFQ/CFP = Request for Quotation/Call for Participation

Develop & Develop & TestTest

Task DTask D

Selection & Selection & KickKick--offoff

Task CTask C

RFQ/CFP*RFQ/CFP*DevelopmentDevelopment

Task BTask B

ConceptConceptDevelopmentDevelopment

Task ATask A

Deploy & Deploy & PersistPersist

Task ETask E

Clear Schedule, Deliverables and Clear Schedule, Deliverables and Project ResponsibilitiesProject Responsibilities

OGF FacilitatesOGF Facilitates

Stakeholders ManageStakeholders Manage


Return on Investment

• What are the “carrots” to build the critical mass of stakeholders?• Get early influence in specification development,

early skills building, visibility, and opportunity for early market deployment of standards, but just as important…

•• Return on Investment (ROI)Return on Investment (ROI)• Investment

• Time, Money & People• Both Monetary and In-Kind (labor & materials)

• Timely Connection to Concrete Results• Stakeholders benefit from collaboration• Get more than they put in


““Business ModelBusiness Model”” SummarySummary

• This process could be used to drive:• Best practices• Standards• Interoperability testing• Compliance testing• … anything that requires collaboration among

stakeholders to drive progress• Key Process Issues:

• Building critical mass of key stakeholders• Clearly identified schedule, goals, and responsibilities• Properly provisioning the effort

• Possible “business model” for OGF!• Contact me: [email protected]

36


A Call to Action• Technology and Market Opportunities

• Technical• Data access and integration, Geospatial applications, Data

Centers, Virtualization, Cloud Computing, Green IT, …• Organizational

• OGF-Europe, DMTF, SNIA, OGC, IBM, eBay, MS, Platform, …• Nurture the grid research community

• Broad community events on innovative, hot topics• While targeting key efforts that produce high

impact results•• Efforts that connect the investment of time, money and Efforts that connect the investment of time, money and

people with concrete resultspeople with concrete results• Seek natural adjacencies for the development and

use of grid technology


And Finally…

• OGF-24 in Singapore• Biopolis, September 15-19, 2008

• OGF-25 with EGEE• Italy, Winter 2009

• OGF-26 in Chapel Hill• North Carolina, USA, Summer 2009

ConsiderJoining OGF!

ConsiderConsiderJoining OGF!Joining OGF!

Evolutionary Pressures on Grid Technology and MarketsEvolutionary Pressures on Grid Technology and...

Documents

Transcript of Evolutionary Pressures on Grid Technology and MarketsEvolutionary Pressures on Grid Technology and...