Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data...
Transcript of Spatial Data Science in ArcGIS: Making the Most of the Ecosytem · Python for Data Analysis: Data...
SpatialDataScienceinArcGIS:TheEcosystem
ShaunWalbridge
KevinButler
https://github.com/scw/ds-scipy-devsummit-2020-talk
HighQualityPDF(5MB)
ResourcesSection
DataScience
Theapplicationofcomputationalmethodstoallaspectsoftheprocessofscientificinvestigation–dataacquisition,datamanagement,analysis,visualization,andsharingofmethodsandresults.
DataScience
ArcGISforspatialdatascience
ArcGISisasystemofrecord.Combinedataandanalysisfrommanyfieldsandintoacommonenvironment.Whyextend?Can’tdoitall,wesupportover1600GPtools—enablingintegrationwithotherenvironmentstoextendtheplatform.ArcGISisanecosystemthatlendsitselfverynicelytothewaythatspatialdatascientistsalreadywork.
What’sintheEcosystem
PythoninArcGISPythonAPIfordrivingArcGISDesktopandServerAfullyintegratedmodule:importarcpyInteractiveWindow,PythonAddins,PythonTooboxesArcGISAPIforPythonHostedNotebooksNotebooksinArcGISPro
Demo:NotebooksinPro
CorePythonLibraries
WhySciPy?Mostlanguagesdon’tsupportthingsusefulforscience,e.g.:
VectorprimitivesComplexnumbersStatistics
Objectorientedprogrammingisn’talwaystherightparadigmforanalysisapplications,butistheonlywaytogoinmanymodernlanguagesSciPybringsthepiecesthatmatterforscientificproblemstoPython.
IncludedSciPyPackage KLOC Contributors Stars
52 229 4293
36 587 13408
85 214 7396
236 738 9868
183 1433 18431
387 699 5522
243 730 5617
Andover100additionalpackages.Checkthemout!
dask
IPython
JupyterLab
NumPy
Pandas
SciPy
SymPy
PlottinglibraryandAPIforNumPydata
Proalsoincludesarcpy.chartforplottingviaProcharts
UC2020:EmbeddedProchartsinnotebooks
MatplotlibGallery
ArcGISwithNumPy
1. Anarrayobjectofarbitraryhomogeneousitems2. Fastmathematicaloperationsoverarrays
,CC-BYSciPyLectures
ArcGISandNumPycaninteroperateonraster,table,andfeaturedata.SeeIn-memorydatamodel.Examplescriptto
ifworkingwithlargerdata.Usearcgis’SeDFifyouneedahigh-levelinterface
forfeaturedata
WorkingwithNumPyinArcGISprocessby
blocks
ArcGISwithNumPy
Computationalmethodsfor:
Integration( )Optimization( )Interpolation( )FourierTransforms( )SignalProcessing( )LinearAlgebra( )Spatial( )Statistics( )Multidimensionalimageprocessing( )
scipy.integratescipy.optimizescipy.interpolate
scipy.fftscipy.signal
scipy.linalgscipy.spatial
scipy.statsscipy.ndimage
UseCase:BenthicTerrainModeler
LightweightSciPyIntegration
Usingscipy.ndimagetoperformbasicmultiscale
analysisUsingscipy.statstocomputecircularstatistics
LightweightSciPyIntegration
Examplesource
importarcpyimportscipy.ndimageasndfrommatplotlibimportpyplotasplt
ras="data/input_raster.tif"r=arcpy.RasterToNumPyArray(ras,"",200,200,0)
fig=plt.figure(figsize=(10,10))
LightweightSciPyIntegration
foriinxrange(25):size=(i+1)*3print"running{}".format(size)med=nd.median_filter(r,size)
a=fig.add_subplot(5,5,i+1)plt.imshow(med,interpolation='nearest')a.set_title('{}x{}'.format(size,size))plt.axis('off')plt.subplots_adjust(hspace=0.1)
Pandas
PanelData—likeR“dataframes”BringarobustdataanalysisworkflowtoPythonDataframesarefundamental—treattabular(andmulti-dimensional)dataasalabeled,indexedseriesofobservations.
SpatialDataFramesSamedataframemodel+geometriesArcPy+ArcGISAPIforPythonContinuestoexpandandimproveperformance
ArcPyImprovements
ArcPyImprovementsarcpy.metadatafortransformingyourmetadata
arcpy.naxforrichnetworkanalysis
Rastercelliteratorsforcustomper-cellrasteranalysiswithoutneedingtocopydatausingNumPy#DOCELLRISESarcpy.SetParameterSymbologyforrichanalyticalresultslikeChartsandpopups
ArcPyImprovementsRichrepresentationsfordatalikearcpygeometries,
rastersMorecomingUC2020
Integration
IntegrationOK,sowe’vecoveredcorelibrariesthatexistwithintheProPythondistribution.Whataboutgoingbeyondthis?
IntegrationWhatkindofcodeisbeingrun?
ThePrincipleofstackminimization
Demo:MetPy
MassivedataparallelismthroughPythonComputesgraphsofthecomputationalstructure
Demo:Dask&TyingItTogether
RRStatisticalProgrammingLanguagePowerfulcoredatastructuresforanalysisUnparalleledbreathofstatisticalroutines
R-ArcGISBridgeAccesstolocalandremotedata
TransformtonativeRspatialtypes(sf,sp,raster)
CallArcPythroughreticulate
UseinRStudio
MakeGPtoolswhichcallR
JupyterNotebookswithR:condainstallr-arcgis-essentials
Demo:R
fromfutureimport*
RoadAheadContinuedimprovementsinDeepLearninginPro—makethisexperienceasseamlessandassimpleaspossibleRichrepresentations(__repr__)formanyobjectsin
ArcPyandProArcPyinExternalCondaenvironments(detectsPro)
ProExternalEnvironments
Resources
NewtoPythonCourses:
Books:
ProgrammingforEverybodyCodecademy:PythonTrack
LearnPythontheHardWayHowtoThinkLikeaComputerScientist
GISFocusedPythonScriptingforArcGISArcPyandArcGIS-GeospatialAnalysiswithPythonPythonDevelopersGeoNetCommunityGISStackexchange
ScientificCourses:
PythonScientificLectureNotesHighPerformanceScientificComputingCodingtheMatrix:LinearAlgebrathroughComputerScienceApplicationsTheDataScientist’sToolbox
ScientificBooks:
Free:
verycompellingbookonBayesianmethodsinPython,usesSciPy+PyMC.
ProbabilisticProgramming&BayesianMethodsforHackers
KalmanandBayesianFiltersinPython
ScientificPaid:
HowtouselinearalgebraandPythontosolveamazingproblems.
ThecannonicalbookonPandasandanalysis.
CodingtheMatrix
PythonforDataAnalysis:DataWranglingwithPandas,NumPy,andIPython
PackagesOnlyrequireSciPyStack:
Scikit-learn:
IncludesSVMs,canusethoseforimageprocessingamongotherthings…
FilterPy,Kalmanfilteringandoptimalestimation:
Lecturematerial
FilterPyonGitHubAnextensivelistofmachinelearningpackages
Code
AnopensourcecollectionoffunctionchainstoshowhowtodocomplexthingsusingNumPy+scipyontheflyforvisualizationpurposes
withahandfulofdescriptivestatisticsincludedinPython3.4+.TIP:WantacodebasethatrunsinPython2and3?
,whichhelpsmaintainasinglecodebasethatsupportsboth.Includesthefuturizescripttoinitiallyaprojectwrittenforoneversion.
ArcPy+SciPyonGithubraster-functions
statisticslibrary
Checkoutfuture
ScientificArcGISExtensions
CombinesPython,R,andMATLABtosolveawidevarietyofproblems
speciesdistribution&maximumentropymodels
PySALArcGISToolboxMovementEcologyToolsforArcGIS(ArcMET)MarineGeospatialEcologyTools(MGET)
SDMToolbox
BenthicTerrainModelerGeospatialModelingEnvironmentCircuitScape
Conferences
ThelargestgatheringofPythonistasintheworld
AmeetingofScientificPythonusersfromallwalks
ThePythoneventforPythonandGeoenthusiasts
TalksfromPythonconferencesaroundtheworldavailablefreelyonline.
PyCon
SciPy
GeoPython
PyVideo
PyVideoGIStalks
Closing
ThanksGeoprocessingTeamArcGISAPIforPythonTeamThemanyamazingcontributorstotheprojectsdemonstratedhere.
Getinvolved!AllareonGitHubandhappilyacceptcontributions.