Status of the Open Data Cube -...
Transcript of Status of the Open Data Cube -...
Status of the Open Data Cube
Brian KilloughCEOS Systems Engineering Office (SEO)
WGISS-44 MeetingSeptember 27, 2017
What are Data Cubes?
• DataCube=Time-seriesmulti-dimensional(space,time,datatype)stackofspatiallyalignedpixelsreadyforanalysis
• ProvenconceptbyAustraliawithplansforglobalimplementation
• AnalysisReadyData(ARD)...Dependentonpre-processedproductstoreducetheburdenonusers
• Opensourcesoftwareapproachallowsfreeaccess,promotesexpandedcontributions,andincreasesdatausage.
• Uniquefeatures:exploitstimeseries,increasesdatainteroperability,andsupportsmanynew applications. TIME
DataCubesareapopulartopic…“TheDataCubeManifesto”(PeterBaumann,EU)and“TheSixFacesoftheDataCube”(PeterStrobl,EC)
Benefits of Data Cubes
§ ExpandeduseofCEOSsatellitedata…expandeduserbase§ Reducedprocessingburden..dependencyonARD§ Enhancedinteroperability...improvedbyMRI§ Efficienttimeseriesanalyses§ Freeandopenaccess§ Flexibledeployment(localorcloud)§ Useofacommonarchitecture§ Communitydevelopmentandsharing…viaGitHub
OurgoalisNOT tosellaproductordistributeatool.OurgoalistoprovideaSOLUTION thathasVALUE andincreasestheIMPACT ofsatellitedata.
Open vs. CEOS Data Cubes
§ TheODCinitiativeislargerthanCEOS.§ TheOpenDataCube(ODC)initiativewasestablished
byCEOS,withagoaltocreateandfosteranopen“community”ofcontributors.
§ TheODC usesacommonarchitectureamongthevariousimplementationssothatalluserscansharetoolsandapplications.
§ TheCEOSDataCube(CDC)isone“implementation”oftheODC.Similarly,DigitalEarthAustralia(DEA)andUSGSLandChangeMonitoring,Assessment,andProjection(LCMAP)areimplementations.
§ TheCDC goalistofocusonbuildingglobalcapacitytoutilise satellitedataandcontributetoglobalinitiatives(e.g.UN-SDG,GFOI,GEOGLAM)throughtheuseofDataCubes.
CEOS Data Cube Vision
AsolutionsupportingCEOSobjectives…§BuildcapabilityofuserstoapplyCEOSsatellitedata§ SupportingpriorityCEOS/GEOagendasandSDGs
CEOSAgencieswantingtoparticipate …§ ThroughprovisionofCEOSAnalysisReadyData(ARD)products§Contributingtodevelopmentanduptakeofsolutions
Customerfocused…§ Trainingmaterialsandeasyinstallation/maintenance§Abrandthatpeopleknowandtrust§Anactivecommunityofusers
Scalablesolution…§OperationalDataCubesin20countriesby2022§Keypartners(e.g.GEO,WorldBank)supportingdatacubeprojects
The “Road to 20” Operational Data Cubes by 2020
Operational
Under Development
Under Review or Expressed Interest
3 operational, 4 under development, 22 under review = 29 total
The “Road to 20” Highlights
§ Colombia hasanoperational DataCubesinceDec2016withover25,000historicLandsatimages.Theycontinuetoexpandtheuserbase,applicationsanddatasets.TheColombiaDataCubewontheNationalEnvironmentalAwardofColombianSocietyofEngineersinMay2017andhasbeenapprovedbytheColombiaGovernmentinto2018.
§ Switzerland hasanoperational DataCubesinceJuly2017withover4,000historicLandsatimages.TheyhavereceivedSwissgovernmentapprovalanddevelopedanewwebsite(swissdatacube.org).Theirfutureplansincludeexpandeddatasets(Sentinel)andincreasedapplicationswithbothgovernmentanduniversityinvolvement.
§ Vietnam isslowlymakingprogressbyestablishingpilotcubesinseveralregionsusinganewhighperformancecomputingsystem.Theirfocusisonforests,rice,andwaterapplications.VNSCishostinganinternalDataCubeWorkshoponSept17.
§ Taiwan ismakingprogressonalocalHPCinstallationthroughsupportfromCSIRO.Theirfocusisforestsandwaterapplications.
§ Uganda hasreceivedsupportfromtheU.K.toinstallademocubefortheKaramojaregiononacloud(AWS).TheyhavemaderapidprogresswithlittleCEOSsupport.
§ Seethe“Roadto20”documentontheODCwebsiteformore!
Other Highlights
§ Plansareinplacetoleverage theexperienceofseveraloperationalimplementationstoexpandthepresenceofDataCubes…§ Switzerland>>>Georgia,Moldova§ U.K.>>>SolomonIslands,Vanuatu,Nauru§ Taiwan>>>Hondurus
§ WearemakingprogresswithWorldBanktosupportthedeploymentofaDataCubeinUruguaytosupportanagricultureandwaterqualityprojectwithdirectlinkstoDINAMA(UN-SDGstatisticalagency).
§ 4DataCubesideeventsareplannedforGEO-17 onOct23-24.Each1.5hoursegmentwillhaveadifferentDataCubetopic.
§ FutureoutreachopportunitiesatPecora-20 (USGS)andIGARSS-2018(JulyinValencia,Spain).
ODC Progress§ Wehaveestablishedan“ODCPartners”groupwhichincludesrepresentatives
fromtheNASA-SEO,GA,CSIRO,USGS,andUK-Catapult.§ Wehaveestablishedan“ODCSteering”groupwhichincludetechnical
representativesfromNASA-SEO,GA,CSIRO,andUSGS.§ Wehaveestablishedanewwebsitehttps://opendatacube.org§ Wehavedeveloped“whitepapers”fortheODCandCDCthatdescribethegoalsof
eachinitiativeandanODCgovernancedocumentforcodemanagement.§ WeconductedthefirstODCWorkshopattherecentIGARSSconferenceinFort
Worth,Texas,USAinJuly.§ Weareplanningthe2nd Annual
ODCTechnicalMeetinginCanberra,AustraliaonFeb14-16,2018.
Technical Progress
§ TheCDChasestablisheddetailedcontenttosupportDataCubedeployments§ Installation – systemrequirements,installationguide§ DataPreparation – ARDguidance,dataacquisitionguidance§ DataCubeCreation– ingestors forallpopulardatasets§ Applications – AWSdemo,Pythonnotebooks,growinglistof
algorithms§ Forum – discussiongroupsforusersupport
§ DataCubeingestionhasdemonstratedsignificantreductionindatastoragerequirementswhencomparingtheingestedDataCubestotheoriginaldata.o Landsat =3xto7xreduction(varieswithdataparameterselections).
Forexample,a1-degx1-degx1-yearLandsatDataCubeis~900MB.o Sentinel-1GRD=6xreduction(basedon30mgrid,VVandVHonly)
DataCubes§ 16cubeswith10+yearseach.§ Kenya,Cameroon(LakeChad),Togo(coastalAfrica),Ghana,Colombia,Tonga(PacificIsland),Vietnam,Australia(MenindeeLakes),Bangladesh.
UserInterfaceFeatures§ 9applications:cloudcoveragemaps,customcloud-freemosaics,fractionalcover,NDVIanomaly,waterdetection,waterquality,landslides,coastalchangeandurbanization.
§ OutputsinGeoTIFFandGIFanimation.§ NewfeaturesaddedinSept2017:datavisualizationtools,ingestion“ondemand”fornewcubesorsubsetting,indices,mosaics(medoid,geometricmedian)
http://tinyurl.com/datacubeuiFree and Open!
Thisisthefirst“hands-on”globaldemooftheDataCubetoshowitspotentialforrapidtimeseriesanalysisanddiverseapplications
Amazon (AWS) Demo Portal
Near-term Developments
§ DevelopanewQGIStoolpluginwithaweb-based(WCS)connectiontoaDataCubehostedonAWS(cloudstorage).ThiswillbereadybyNov2017.
§ DevelopandtestsampleiPython NotebooksonAWStodemonstrateinteractiveDataCubeapplicationsandprogrammingsimplicity
§ TestthePyCCD landchangedetectionalgorithmwithradardatasets
§ DevelopandtestanewWaterQualityalgorithmfromTonyVodacek (LandsatScienceTeam)basedonaLook-Up-TableapproachtoinferChlorophyll-A,CDOMandTSSconcentrations.
§ TestSentinel-1GRDandSLC cubeswiththeRandomForestlandclassificationclusteringalgorithm
Lessons Learned
Throughourinitialcountryinteractions,wehavelearnedanumberoflessons …
§ CountryusersshouldhavesomePythonprogrammingskills§ Itisimportanttoclearlyunderstandcountryneedsandtoguidethemtoward
theneededsatellitedataandapplicationtools§ Itisimportanttomaintainconsistentcustomercommunication(bothface-to-
faceandremote)tosustaindeploymentprogressandbuildtrust§ Itisimportanttoutilise relationshipswithinvestmentbanks(e.g.WorldBank)
andGEOtoincreaseaccesstocountrycontactsandfacilitatedeployment§ TheODCcommunityneedstocontinuetogrowandexpandtobuildconfidence
towardsdesiredoutcomesandtobuildthesupplyofopensourcetoolsandapplications