Cloud-based Geospatial Data services and...

Cloud-based Geospatial Data services and

analysis Xuezhi Wang

Scientific Data Center

Computer Network Information Center Chinese Academy of Sciences

2014-08-25

Outlines

1 Introduction of Geospatial Data Cloud

2 Cloud-based Geospatial Data Analysis platform

3 Cloud-based Geospatial Data visualization platform

Geospatial Data Cloud

I. Platform overviews II. Geospatial data services III. Geospatial model services IV. Services achievements

Why do we do?

In 2008, USGS opened all geospatial datasets to the public for free.

• Landsat, MODIS is the most popular remote sensing dataset for scientists and researchers. Especially, the Landsat series.

• Due to the low international network speed, many Chinese scientists cannot take advantage of these resources easily.

• Based on the CNIC’s mass storage and high speed network( CSTNET: Chinese Science and Technology Network), we mirrored the geospatial datasets covered China and surrounding area, and provided online data services for

chinese scientists and researchers

Design a new cloud-based geospatial analysis and visualization platform

USGS opened its Landsat datasets. we contacted the USGS staffs through email.

We got an oral permit for Landsat redistribution from USGS. we download the Landsat datasets, and provide public

services for Chinese scientists

Cooperated with National Center for Atmospheric Research (NCAR)

Mirrored the NCAR datasets provide public services for Chinese

Scientists

We got a permit for SRTM dataset redistribution from International Centre for Tropical Agriculture(CIAT). We cache

SRTM global elevation data, and provide online data services.

Accroding to the user requirements, We make some remote sensing products from Landsat dataset and Modis dataset.

such as vegetation index, water cover dataset.

Dowload MODIS dataset covered CN Host the first user conference

Design a new cloud-based data platform

Geospatial data cloud opened for public services Host the second user conference with ESRI China

Platform History

Platform objectives Based on our cloud infrastructure, we hope to construct an 'one-stop'

geospatial data service platform for Chinese scientists and researchers.

An open cloud-based platform meeting the requirements of data search, data acquisition, online storage, scientific models, online calculation and map visualization.

Model

Online Calculation

Search Acquisition

Storage Visualization

http://www.gscloud.cn

General framework

Database cluster Online storage Near-line

storage

Data integration

Automatic Metadata crawling

Automatic data

entities cache

Mass Storage

Cloud environments

Distributed file system

Virtual Machines

Task scheduling

Data query

Data category

Integrated search

Complex search

One-key indexes

Data services

Smart data Order

Data application Favorite data Batch

download

Model services

Model package

Online execution

Web map services

Map publish Map visualization

Map style edit

Model management

User data spaces

Data management

Task management

Map management

Model flow

Map share

Service management

System managem

ent Users

Data

Models

Indexes

Tasks

Services

Resources monitor

Online data resources Open geospatial datasets: LANDSAT、MODIS、EO-1、DEM、NCAR、

NOAA and LUCC

More than 350TB data entities, and over 7 million data records.

Covered China and surrounding area

Online services for Chinese scientists and researchers with free of charge

Landsat1-3 MSS Landsat 4-5 MSS Landsat 4-5 TM Landsat 7 SLC-on Landsat7 SLC-off Landsat 8 OLI

TERRA AQUA

TERRA AQUA

GDEM 30m SRTM 90m Ocean elevation

HYP_L1R HYP_L1G

AL1

LANDSAT MODIS MODIS_L1B DEM EO-1

Generated geospatial products Water-cover of China from Landsat TM/ETM in Summer

Slope, Aspect, Topographic position indexes et. of China

Land Surface Temperature, Vegetation indexes (5 days, 15 days, and 30 days) of China, from 2000 – now

ASPECT Water Cover LST

Geospatial data query

Massive geospatial metadata

I need Landsat data in

summer, 2000

I need DEM data of Beijing

Input: geo-location or address

PostgreSQL+PostGIS Cluster

Based on geospatial

storage cluster and search

engine, we provide three

ways for users to access

their preferred data:

1: by address name

2: by geo-location

3: by administrative unit

Platform Screenshots 1. search by address name

Platform Screenshots 2. Search by geographic coordinates

Platform Screenshots 3. Search by polygon features

Platform Screenshots 4. Search by administrative unit

Platform Screenshots The result page that matched the query criteria

Extra attribute

filters

Geospatial model services Why do we need this services?

For example: If we need Beijing DEM data, there have two ways. Traditional way: search DEM data, download DEM data,

mosaic DEM data with ArcGIS, and then clip DEM data with Beijing shape mask .

Convenience way: select Beijing administrative unit in the web, submit a clip task to system, and then download Beijing DEM data.

Which is the best way? I prefer the second way.

That’s why we provide online geospatial model services.

We developed an elastic online task processing system with cloud techniques.

Users fill model parameters on the web page and submit tasks to the system;

The tasks were run under cloud virtual machines;

The results were stored in cloud storage.

Task scheduling

Task

Task

Task

Computing resource pool Storage pool

VM VM

VM

virtual machines in Cloud

Overall frameworks

Model page Model page

Online models and tools DEM tools:

DEM clip and mosaic; Reprojection; Aspect, Slope, Topographical index calculation of

specified region LANDSAT tools:

Online gap-fill model for Landsat ETM7 SLC-off dataset

MODIS tools: Online Land Surface Temperature calculation of user

specified time range Online vegetation indexes calculation of user specified

time range

Private cloud storage We provide user with private cloud storage to store their own model inputs and outputs data.

User manage their own data here, including upload, download, rename, delete, create folder and move

Platform Screenshots Online category page for Geospatial tools and models

Platform Screenshots Model parameters page of DEM clip and mosaic tool

User only need to fill task name, select a region in the web map, and choose output folder

Platform Screenshots Task tracking page list user’s submitted tasks

They can tracking task states, stop and start task, adjust task priority here.

Platform Screenshots Task result page

When the task was finished, the results can be download from here, or from user private cloud storage

Our platform was wildly used in many research fields in China: earth science, environmental science, ecology, hydrology, meteorology, oceanography, natural disasters, city planning ...

Service target

Based on statistics in November 2013

User characteristics

Currently, over 70,000 users

Doctor

Master Bachelor

Others

University

CAS

Government

Company

Others

Service statistics

Avg. downloaded over: 200 GB/day; online downloaded over: 250 TB;

offline copied over: 137 TB;

Outlines




Cloud-based Geospatial Data Analysis Platform

Why do we need this platform? For example: If we want to calculate Chinese monthly

temperature variation from 2000 to 2014, how can I achieve this?

Traditional way: download all data, and calculate with workstations. It may spent several months to download data. This is a very bad thing.

Shortcuts: using cloud computing technology. Apply virtual machines in the cloud; Write code in the virtual machines; read all data from cloud storage; Run program in parallel

Which way is the best? I prefer the second way. That’s why we want to construct this platform.

Overall frameworks

...... Storage resource pool

Cloud manager with OpenStack

Distributed file system

Data pool

Computing resource pool

Public geospatial data

User private geospatial data

virtual machine pool

VM VM VM VM

Currently progress of the platform We have finished the prototype system. The system is

under inner tests for access pressure and stability. In the end of this year, it will be provided for the public access.

Outlines




Cloud-based Geospatial Data Visualization Platform

Why do we need this platform? In many cases, users want to publish their geospatial data,

model results, vector map to the web site; They also want to customize the map layer styles through

the web pages; They also want to customize the components and styles of

the webgis pages; To achieve this goal, we construct this platform.

Optimized Web Map Service

Computing resource pool Storage pool

VM VM

VM

Dynamical render map in Cloud HTML5 map client ...

Tiles Storage

Dynamical cache

Map styles editor

Map request

Overall frameworks

Currently progress of the platform Most of system functions were already finished,

under inner tests and bug fixed.

Published services provided:

OGC WMS（web map service）, Google TMS , OGC WFS （web feature service）

GeoJson , WorldWind ...

System screenshots I. Render multi-bands raster layer with RGB false-color,

custom band composition

System screenshots II. Render thematic raster layer with unique color theme

System screenshots III. Render raster layer with classified color theme

System screenshots IV. Render raster layer with gradient color theme

System screenshots I. Render vector layer with Simple symbol

System screenshots II. Render vector layer with colorful symbol by unique

attributes

System screenshots III. Render vector layer with colorful symbol by classified

attributes

System screenshots IV. Render vector layer with sized symbols by classified

attributes

System screenshots Vector attributes query and edit

Thanks Any suggestions?

Email: [email protected]

Cloud-based Geospatial Data services and...

Documents

Transcript of Cloud-based Geospatial Data services and...