Migrating from Grid to Cloud: Case Study from GEO Grid · 2014-05-15 · Migrating from Grid to...

Migrating from Grid to Cloud:Migrating from Grid to Cloud:Migrating from Grid to Cloud:Migrating from Grid to Cloud:Case Study from GEO GridCase Study from GEO Grid

Kyoung-Sook Kim

Data Science Research GroupInformation Technology Research Institute

Interoperability and IntegrationInteroperability and Integration

Satellite imageGeology

Geo* Contents Applications

Individual interfaces & protocols for information exchangeSatellite image for information exchange

It is not easy to• Access and (re)use• Manage and control

GIS data Environment & Energy• Manage and control• Integrate geospatial contents and services

Cooperate with different

Sensor data

http://www.geofabrik.de/data/shapefiles.html

Disaster Response

• Cooperate with different organizations

High cost of application Sensor data High cost of application development and management

Geospatial World Forum 2014 @ Geneve, Switzerland2014/5/15 2


GEO Grid: OverviewGEO Grid: Overview*Global Earth Observation

ee--Science infrastructureScience infrastructure for global earth observation systems with heterogeneous geospatial datasets

GeologyGeo* Contents Applications

StandardStandard -based interoperability & IntegrationSatellite image

Geology

- Ease-of-use interfaces (search, access, process data)- Low integration efforts

StandardStandard-based interoperability & Integration

WMS WFSWeb Web

GIS dataEnvironment & EnergyWPS

WMS WFS

SOSWCS CS-W

Web Web Services Services

Sensor data


Disaster Response• Security managementSensor data

Disaster Response• Security management• Scalability of resources

(computing and storage)



GEO Grid: GEO Grid: OverviewOverviewee--Science infrastructureScience infrastructure for global earth observation systems

with heterogeneous geospatial datasets



Geology



WMS WFSWeb Web


WMS WFS

SOSWCS CS-W


Grid Computing Technologies &Sensor data


Disaster ResponseGrid Computing Technologies &

StandardsSensor data

Disaster Response

for sharing of geographically distributed resources


for sharing of geographically distributed resources and controlling resource sharing rules


GEO Grid: GEO Grid: OverviewOverviewee--Science infrastructureScience infrastructure for global earth observation systems

with heterogeneous geospatial datasets



Geology



WMS WFSWeb Web


WMS WFS

SOSWCS CS-W


Grid Computing Technologies &Sensor data


Disaster ResponseGrid Computing Technologies &Sensor data

Disaster Response

Distributed Distributed and Parallel Processing

Metadata Management(LOD/RDF)

Security and User Management

Storage Grids

Heterogeneous Heterogeneous and Distributed Database Federation


ProcessingProcessing(LOD/RDF)

Grid and Cloud InfrastructureGrid and Cloud Infrastructure

ManagementFederationFederation


GEO GEO Grid:Grid: System ArchitectureSystem ArchitectureOutcomesOutcomesOutcomesOutcomes

Decision MakingSupport


Sci-TechR&D

Sci-TechR&D

BusinessBusinessShare

Disaster Information

Share Disaster

Information SupportSupportInformationInformation

Application Application ServicesServices

ASTERDEM

QuiQuakeQuiQuake Flood Simulation

Flux Monitoring

HotspotDetection

Others

Science DCP

ServicesServices

OGC standard web services OGC standard web services

Heterogeneous

WMS WCS CS-W WPS SOS WFS

Grid and CloudGrid and CloudInfrastructureInfrastructure

GeoSpatialGeoSpatial ResourcesResources

Storage GridsSecurity and User Management

Heterogeneous and Distributed Database Federation

Computing Grid and Cloud

Our HPC clusterGeoSpatialGeoSpatial ResourcesResources Our HPC clusterData Archive Services

Satellite Data

Geological Data

40m

20m

10m

30m

-0.01-0.05-0.1

-0.2

-0.5

40m

20m

10m

30m

-0.01-0.05-0.1

-0.2

-0.5

40m

20m

10m

30m

-0.01-0.05-0.1

-0.2

-0.5

sensors

ASTER satellite archive~= 2million scenes~= 1PB

MODIS, JERS-1PALSAR,LANDSAT

Geological Map of Japan

CfreeVG10,SRTM, Other geospatial info

2014/5/15 6Geospatial World Forum 2014 @ Geneve, Switzerland

sensorsPALSAR,LANDSAT

geospatial info

resources

An An Example Example of GEO Grid of GEO Grid Public ServicesPublic Services

• QuiQuake (https:/ /gbank.gsj.jp/QuiQuake/ index.en.html)

• Quick estimation system for • Quick estimation system for earthquake maps triggered by observation records

• Provides wide-ranging and detailed (250m-grid) strong ground motion maps for quick disaster response.

• Produces the results soon after the occurrence of an earthquake.

• Archiving all data of past earthquake occurrences.


QuiQuakeQuiQuake Realization Realization in GEO Gridin GEO Grid

Share Disaster

Information

Share Disaster

Information

Sensor Data Decision MakingSupport


InformationInformation

High Performance

Provide the results as OGC services and formats

Geospatial Data

Data Integration

PerformanceComputing

OutcomeOutcomeApplication Application

ServicesServicesGrid and CloudGrid and CloudInfrastructureInfrastructure

GeoSpatialGeoSpatial ResourcesResources


GEO Grid SecurityGEO Grid Security• GSI (Grid Security Infrastructure)

– Secure communication (authenticated and confidential) between Grid elements– Security across organizational boundaries– Security across organizational boundaries– ”Single sign-on" for users of the Grid in multiple resources and/or sites

• VO (Virtual Organization) • VO (Virtual Organization) ManagementA dynamic group of individuals, groups, or organizations who define organizations who define the conditions and rules for sharing resources


Migration from Grid to CloudMigration from Grid to Cloud

2011 Tohoku Earthquake

GEO Grid at AIST (~200km away) was damaged by the earthquake.Cannot operate GEO Grid for 2 months by recovery works & subsequent power Cannot operate GEO Grid for 2 months by recovery works & subsequent power restriction request.

BUT satellite imageswere very important to understand situations and provide useful information for helping the disaster response activities.


provide useful information for helping the disaster response activities.

ASTER Comparison Before and After TsunamiASTER Comparison Before and After Tsunami

• ASTER: – Japanese sensor (optical, 15m) on NASA Terra

satellitesatellite– Can create 3D model by making stereo-matching – AIST can produce true-color images.

11

Mouth of Kitakami-river

GEO Grid Disaster GEO Grid Disaster Responses Responses

• Rapid Evacuation :– Moved our minimum environment to oversea volunteer servers

• Using VM/Cloud technology

– Changed our daily network (data transfer) workflow to use these servers– Changed our daily network (data transfer) workflow to use these servers• To get/receive latest images from the satellite(ASTER)ALOS/PALSAR

TDRSUntil March 11

- Data providing- Portal

70 GB/day (ASTER)

NASA

AIST

ERSDAC

JAXA360 GB/day (PALSAR)

- Archive (tape, B-ray) - Archive (on-Disk)- Processing- WMS

• ASTER data: NASA ERSDAC AIST


• ASTER data: NASA ERSDAC AIST• PALSAR data: JAXA ERSDAC AIST (AIST: processing, WMS, portal site, and data archive)

What We DIDWhat We DIDTerra/ASTERALOS/PALSAR

TDRSData Flow and Services

from March 11 to April 20

- Portal

NASA

(AIST)

ERSDAC

JAXA

Orkney

Google

- Processing- WMS- WMS

• ASTER data: NASA ERSDAC (AIST) (Orkney: processing and WMS, Google: portal site)


• ASTER data: NASA ERSDAC (AIST)• PALSAR data: JAXA ERSDAC (AIST)

(Orkney: processing and WMS, Google: portal site)

What We DIDWhat We DID

TDRS

Terra/ASTERALOS/PALSARData Flow and Services

From April 21

- Portal- Processing

NASA

(AIST)

ERSDAC

JAXA

Google

UCSD

OCC

UCSD

- Processing

NCHC

WMS Server- Processing- WMS

• ASTER data: NASA ERSDAC (AIST) NCHC@TW, SDSC@USA, and OCC@USA: processing

WMS Server

QuiQuakeHigh performance (long-running) computing using oversea servers for creating scientific information


• ASTER data: NASA ERSDAC (AIST)• PALSAR data: JAXA ERSDAC (AIST)

NCHC@TW, SDSC@USA, and OCC@USA: processingNCHC@TW: WMS Google: portal site

TimeTime--line of our Activityline of our Activity

Date Events and Activities DescriptionsMar 11 Occurrence of Earthq. GEO Grid services stop

Mar 14 GEO Grid disaster TF Establishment Support by private companies, research inst., universities worldwideworldwide

Mar 15 Start of data transport and migration of processing codes

ASTER and PALSAR from ERSDACComputer and WMS in OrkneyPortal site in Google site

Mar 19 HP open and WMS start ASTER and PALSAR, damage interpretation, QuiQuake(manual)Mar 19 HP open and WMS start ASTER and PALSAR, damage interpretation, QuiQuake(manual)

Mar 21 Start of data transport Formosat-2 by NSPO (via JAXA)

Mar 25 Geological maps in WMS Seamless geological map, active fault, and geochemical map

Mar 31 Value-added-ASTER Natural color, orthorectification, automation

Rough Numbers from 3.11 Rough Numbers from 3.11 1 week to evacuate and to start providing analyzed data to public1 month to recover the service using oversea sites2 months to recover/resume our environment

Rough Numbers from 3.11 Rough Numbers from 3.11 1 week to evacuate and to start providing analyzed data to public1 month to recover the service using oversea sites2 months to recover/resume our environment Mar 31 Value-added-ASTER Natural color, orthorectification, automation

Mar 31 CS-W deployment WMS list

Apr 1 QuiQuake open QuiQuake (automatic)

Apr 20 Value-added-PALSAR Crustal deformation by InSAR

2 months to recover/resume our environment Yet another 1to2 month(s) to get back the service from oversea sites2 months to recover/resume our environment Yet another 1to2 month(s) to get back the service from oversea sites

Apr 20 Value-added-PALSAR Crustal deformation by InSAR

Apr 21 Services move abroad NCHC (NARL-Taiwan), SDSC (UCSD), OCC (Univ.Chicago)

Apr 28 GEO Grid cluster resume

May 24 ASTER/AIST service resume Data publication to research users

May 27 QuiQuake/AIST resume


May 27 QuiQuake/AIST resume

June 30 PALSAR/AIST service resume

Towards Cloud ComputingTowards Cloud Computing

• Fortunately, we had VM images of satellite data processing– Ready to provisioning in a cloud environment– Connected to HPC international grid testbed (PRAGMA)– Connected to HPC international grid testbed (PRAGMA)

• Sharing our VM images in PRAGMA VM repository• We can boot our application VMs at any site by any PRAGMA colleagues

• However,• However,– Manual deployment at each site one by one – Lots of manual configuration due to heterogeneity and for security– Lots of manual configuration due to heterogeneity and for security– Tightly coupled data and computing servers– Not real-time– Heavy data formats and integration (OGC standards)– Heavy data formats and integration (OGC standards)– …


GEO Grid for Cloud ComputingGEO Grid for Cloud Computing• Virtualization as a Practical Mechanism

– Supporting Multiple VM Infrastructures (Xen, KVM, OpenNebula, CloudStack, Rocks, EC2)

• Data Intensity

• On-demand self-service• Broad network access• Resource poolingCloud

[NIST definition for using computing resources]

• Data Intensity– Data services to support data discovery, access,

processing, and delivery on demand with minimal transmission (ex., radiation monitoring service)

• Resource pooling• Rapid elasticity• Measured service

Cloud Characteristics

transmission (ex., radiation monitoring service)

• Complex Workflows– Machine learning platform based on Hadoop and

MapReduce– Lavatube (visual workflow engine)

• Software as a Service• Platform as a Service• Infrastructure as a

Service

Service Models

– Lavatube (visual workflow engine)– Real-time analytics

• High-performance Database for Linked Data– Distributed and parallel LOD processing

• Public• Private• Hybrid

Deployment Models – Distributed and parallel LOD processing

• Cloud Security– OpenID/OAuth for AuthN/AuthZ

• GeoSocial Media

• Hybrid• Community

Models

The NIST Definition of Cloud Computing, http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf • GeoSocial Media

– New data integration method


Global Earth Observation GridGlobal Earth Observation GridGlobal Earth Observation GridGlobal Earth Observation Gridhttp:/ /www.geogrid.org/http:/ /www.geogrid.org/

http://www.geogrid.org/

http://www.geogrid.org/

Migrating from Grid to Cloud: Case Study from GEO Grid · 2014-05-15 · Migrating from Grid to...

Documents

Transcript of Migrating from Grid to Cloud: Case Study from GEO Grid · 2014-05-15 · Migrating from Grid to...