nci.org.au

GSKY

Geospatialrequest

FEATURES• Distributed• Scalable• Concurrent

INPU

T

OUTPUT

Request Response

User’sclient

GSKY:AScalable,DistributedGeospatialDataServer

© NCI Australia 2019

nci.org.au

GSKY: HighLevelArchitecture

Worker 1

Worker 2

Worker n

OWS

/g/dataGeospatial

Index

GeoCrawl

...

MAS

GSKY

OGCRequest

OGCResponse

© NCI Australia 2019

nci.org.au

GSKYrespondstoOpenGeospatialConsortium(OGC)APIoverhttpprotocol:• WebMapService(fordisplayingtheimagesonthemapserver)• WebCoverageService(fordeliveringtheactualdataas“coverages”- independentofthe

underlyingstorageformatorfiles)• WebProcessingservice

GSKYallows:• Performantaggregations,subsetting,subsampling,polygon/pencil/pixeldrills• Executionofon-the-flydatatransformations,re-projectionsandotheralgorithms

GSKYisimplementedusing• Richmetadataserverfordataquerye.g.,spatial,temporal,otherphysicalvariables• Clusteredbackendworkers– highperformanceI/Oandscale-outserver-sidecompute

© NCI Australia 2019

GSKYinanutshell:GeospatialBigDataService

nci.org.au

Ongoingimprovementstoseamlesslytransformingmoredataonthefly

GSKYfilehandlingofcommonlargeEarthObservationdatasetsAmountofdataisscalingupwithmoremodernsatelliteinstruments

MODIS(CSIRO&TERN)MODISdatafrom2001– present,with4timestampspermonthand1kfilespertimestamp.GSKYishandlingapprox.900kMODISfiles

Landsat-8dataset(DEA)Landsat8datasethasdatafrom2013– present,approximately3kfilespertimestamp.GSKYishandlingapprox.7millionfiles.

Sentinel2ARD(DEA)Approx 1400availabledays(2015-present)andabout12kfilesperdayforthisdataset.GSKYishandlingapprox.17millionfiles.

© NCI Australia 2019

nci.org.au

GSKY:• CurrentlyWMS,WPS,WCSforearthobs,

climateandgeophysicsdatasets• OpenSource:http://github.com/nci/gsky

Formoreinformation,see:http://gsky.nci.org.au/https://gsky.readthedocs.io/

Quicklook:http://nationalmap.gov.au/Adddata->MyData->AddWebDataURLs:http://gsky.nci.org.au/ows/dea

http://gsky.nci.org.au/ows/geoglam

NCI’sGSKY– AScalableGeospatialDataServer

© NCI Australia 2019