Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data...

56
Spatial Data Science in ArcGIS: The Ecosystem Shaun Walbridge Kevin Butler

Transcript of Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data...

Page 1: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

SpatialDataScienceinArcGIS:TheEcosystem

ShaunWalbridge

KevinButler

Page 3: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

DataScience

Page 4: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Theapplicationofcomputationalmethodstoallaspectsoftheprocessofscientificinvestigation–dataacquisition,datamanagement,analysis,visualization,andsharingofmethodsandresults.

DataScience

Page 5: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ArcGISforspatialdatascience

ArcGISisasystemofrecord.Combinedataandanalysisfrommanyfieldsandintoacommonenvironment.Whyextend?Can’tdoitall,wesupportover1600GPtools—enablingintegrationwithotherenvironmentstoextendtheplatform.ArcGISisanecosystemthatlendsitselfverynicelytothewaythatspatialdatascientistsalreadywork.

Page 6: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

What’sintheEcosystem

Page 7: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

PythoninArcGISPythonAPIfordrivingArcGISDesktopandServerAfullyintegratedmodule:importarcpyInteractiveWindow,PythonAddins,PythonTooboxesArcGISAPIforPythonHostedNotebooksNotebooksinArcGISPro

Page 8: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:
Page 9: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:
Page 10: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Demo:NotebooksinPro

Page 11: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

CorePythonLibraries

Page 12: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

WhySciPy?Mostlanguagesdon’tsupportthingsusefulforscience,e.g.:

VectorprimitivesComplexnumbersStatistics

Objectorientedprogrammingisn’talwaystherightparadigmforanalysisapplications,butistheonlywaytogoinmanymodernlanguagesSciPybringsthepiecesthatmatterforscientificproblemstoPython.

Page 13: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

IncludedSciPyPackage KLOC Contributors Stars

52 229 4293

36 587 13408

85 214 7396

236 738 9868

183 1433 18431

387 699 5522

243 730 5617

Andover100additionalpackages.Checkthemout!

dask

IPython

JupyterLab

NumPy

Pandas

SciPy

SymPy

Page 14: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

PlottinglibraryandAPIforNumPydata

Proalsoincludesarcpy.chartforplottingviaProcharts

UC2020:EmbeddedProchartsinnotebooks

MatplotlibGallery

Page 15: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ArcGISwithNumPy

Page 16: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

1. Anarrayobjectofarbitraryhomogeneousitems2. Fastmathematicaloperationsoverarrays

,CC-BYSciPyLectures

Page 17: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ArcGISandNumPycaninteroperateonraster,table,andfeaturedata.SeeIn-memorydatamodel.Examplescriptto

ifworkingwithlargerdata.Usearcgis’SeDFifyouneedahigh-levelinterface

forfeaturedata

WorkingwithNumPyinArcGISprocessby

blocks

Page 18: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ArcGISwithNumPy

Page 20: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

UseCase:BenthicTerrainModeler

Page 21: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

LightweightSciPyIntegration

Usingscipy.ndimagetoperformbasicmultiscale

analysisUsingscipy.statstocomputecircularstatistics

Page 22: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

LightweightSciPyIntegration

Examplesource

importarcpyimportscipy.ndimageasndfrommatplotlibimportpyplotasplt

ras="data/input_raster.tif"r=arcpy.RasterToNumPyArray(ras,"",200,200,0)

fig=plt.figure(figsize=(10,10))

Page 23: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

LightweightSciPyIntegration

foriinxrange(25):size=(i+1)*3print"running{}".format(size)med=nd.median_filter(r,size)

a=fig.add_subplot(5,5,i+1)plt.imshow(med,interpolation='nearest')a.set_title('{}x{}'.format(size,size))plt.axis('off')plt.subplots_adjust(hspace=0.1)

Page 24: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:
Page 25: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Pandas

Page 26: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

PanelData—likeR“dataframes”BringarobustdataanalysisworkflowtoPythonDataframesarefundamental—treattabular(andmulti-dimensional)dataasalabeled,indexedseriesofobservations.

Page 27: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

SpatialDataFramesSamedataframemodel+geometriesArcPy+ArcGISAPIforPythonContinuestoexpandandimproveperformance

Page 28: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ArcPyImprovements

Page 29: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ArcPyImprovementsarcpy.metadatafortransformingyourmetadata

arcpy.naxforrichnetworkanalysis

Rastercelliteratorsforcustomper-cellrasteranalysiswithoutneedingtocopydatausingNumPy#DOCELLRISESarcpy.SetParameterSymbologyforrichanalyticalresultslikeChartsandpopups

Page 30: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ArcPyImprovementsRichrepresentationsfordatalikearcpygeometries,

rastersMorecomingUC2020

Page 31: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Integration

Page 32: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

IntegrationOK,sowe’vecoveredcorelibrariesthatexistwithintheProPythondistribution.Whataboutgoingbeyondthis?

Page 33: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

IntegrationWhatkindofcodeisbeingrun?

ThePrincipleofstackminimization

Page 34: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Demo:MetPy

Page 35: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

MassivedataparallelismthroughPythonComputesgraphsofthecomputationalstructure

Page 36: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Demo:Dask&TyingItTogether

Page 37: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:
Page 38: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

RRStatisticalProgrammingLanguagePowerfulcoredatastructuresforanalysisUnparalleledbreathofstatisticalroutines

Page 39: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

R-ArcGISBridgeAccesstolocalandremotedata

TransformtonativeRspatialtypes(sf,sp,raster)

CallArcPythroughreticulate

UseinRStudio

MakeGPtoolswhichcallR

JupyterNotebookswithR:condainstallr-arcgis-essentials

Page 40: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Demo:R

Page 41: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

fromfutureimport*

Page 42: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

RoadAheadContinuedimprovementsinDeepLearninginPro—makethisexperienceasseamlessandassimpleaspossibleRichrepresentations(__repr__)formanyobjectsin

ArcPyandProArcPyinExternalCondaenvironments(detectsPro)

Page 43: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ProExternalEnvironments

Page 44: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Resources

Page 45: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

NewtoPythonCourses:

Books:

ProgrammingforEverybodyCodecademy:PythonTrack

LearnPythontheHardWayHowtoThinkLikeaComputerScientist

Page 47: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ScientificCourses:

PythonScientificLectureNotesHighPerformanceScientificComputingCodingtheMatrix:LinearAlgebrathroughComputerScienceApplicationsTheDataScientist’sToolbox

Page 48: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ScientificBooks:

Free:

verycompellingbookonBayesianmethodsinPython,usesSciPy+PyMC.

ProbabilisticProgramming&BayesianMethodsforHackers

KalmanandBayesianFiltersinPython

Page 49: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ScientificPaid:

HowtouselinearalgebraandPythontosolveamazingproblems.

ThecannonicalbookonPandasandanalysis.

CodingtheMatrix

PythonforDataAnalysis:DataWranglingwithPandas,NumPy,andIPython

Page 50: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

PackagesOnlyrequireSciPyStack:

Scikit-learn:

IncludesSVMs,canusethoseforimageprocessingamongotherthings…

FilterPy,Kalmanfilteringandoptimalestimation:

Lecturematerial

FilterPyonGitHubAnextensivelistofmachinelearningpackages

Page 51: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Code

AnopensourcecollectionoffunctionchainstoshowhowtodocomplexthingsusingNumPy+scipyontheflyforvisualizationpurposes

withahandfulofdescriptivestatisticsincludedinPython3.4+.TIP:WantacodebasethatrunsinPython2and3?

,whichhelpsmaintainasinglecodebasethatsupportsboth.Includesthefuturizescripttoinitiallyaprojectwrittenforoneversion.

ArcPy+SciPyonGithubraster-functions

statisticslibrary

Checkoutfuture

Page 52: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ScientificArcGISExtensions

CombinesPython,R,andMATLABtosolveawidevarietyofproblems

speciesdistribution&maximumentropymodels

PySALArcGISToolboxMovementEcologyToolsforArcGIS(ArcMET)MarineGeospatialEcologyTools(MGET)

SDMToolbox

BenthicTerrainModelerGeospatialModelingEnvironmentCircuitScape

Page 53: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Conferences

ThelargestgatheringofPythonistasintheworld

AmeetingofScientificPythonusersfromallwalks

ThePythoneventforPythonandGeoenthusiasts

TalksfromPythonconferencesaroundtheworldavailablefreelyonline.

PyCon

SciPy

GeoPython

PyVideo

PyVideoGIStalks

Page 54: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

Closing

Page 55: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn:

ThanksGeoprocessingTeamArcGISAPIforPythonTeamThemanyamazingcontributorstotheprojectsdemonstratedhere.

Getinvolved!AllareonGitHubandhappilyacceptcontributions.

Page 56: Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. Packages Only require SciPy Stack: Scikit-learn: