Connecting ArcGIS with R and Conda - 4326.us · from interactive environments like Spyder to...
49
Connecting ArcGIS with R and Conda Shaun Walbridge
Transcript of Connecting ArcGIS with R and Conda - 4326.us · from interactive environments like Spyder to...
ConnectingArcGISwithRandCondaShaunWalbridge
https://github.com/scw/nyc-r-ws
HighQualityPDF
ArcGIS
Today:RandCondaConda
IntroductionOptionaldemo
RandtheR-ArcGISBridgeIntroductionDemoRoadAhead
SoftwareEcosystem
ConnectingwithArcGIS
Don’thaveanequivalentAPIforaccessingourservicesviaRliketheArcGISAPIforPython(yet)ArcGISisasystemofrecord.Combinedataandanalysisfrommanyfieldsandintoacommonenvironemnt.Whyconnect?Can’tdoitall,wesupportover1000GPtools—enablingintegrationacrossenvironmentsforrealisticworkflows
TheSciPyStack
DatasciencePython(SciPystack,Jupyter,scikit-learn,…)C++(Tensorflow,Shark,MLC++)Java(SparkMLlib,Weka)R( )Manyworkflowsrequirecombiningcomponentsfrommultipleenvironments.
MLtaskview
PackageswithConda
Whypackagemanagement?
Softwareiscomposedofmanysmallercomponents,oftencalledpackagesorlibraries.It’softenbettertoreusecodethatsolvesaproblemwellratherthanrecreatingitBut,sharingcodeisahardproblem.Doyouhavethesamepackagesofthesameversionsasthedeveloperdid?
WhyConda?
ScientificPythoncommunityidentifiedthattherewasagapnotbeingaddressedbythecorePythoninfrastructure,limitingtheirabilitytogetpackagesintothehandsofusers
Industrystandardbuiltbypeoplewhocareaboutthisspace—ContinuumAnalytics(Esripartner)
WhyConda?
Itsolvesthehardproblem:
Handlesdependenciesformanylanguages(C,C++,RandofcoursePython)BuiltforPythonfirst,butitreallysolvesamuchbroaderinfrastructuralissue.
Conda
Cross-platform:SimplydeveloprecipesforbuildingandinstallingsoftwareonLinux,OSXandWindows.Opensource:Esriisusingit,youcanuseitinyourownprojectsforothercontexts
Whatcanitinstall?Notjustscientificpackages,caninstalleverythingfrominteractiveenvironmentslike to .Spyder JupyterNotebooks
Environments—Canisolateanenvironment,flexiblymakechangeswithotaffectinginstalledsoftware.Requirements—includeexplicitstateinformation,notjustthepackagename.Packages—anenvironmentisbuiltupfromoneormorepackages,canbefrommanylanguages–fromRtoC++toFortran.AlsohandlesplatformsandJupyternotebooks
Wheredopackagescomefrom?
Condapackagescancomefromavarietyoflocations:
manythousandsofpackagesRepositories(e.g.AnacondaCloud,self-hosted)Ondisk
anaconda.org
CondaBasics
Condainfoisthestartingpoint—ittellsyouthestateoftheenvironment.
conda --help
conda info
CondaBasicsconda list
# packages in environment at C:\ArcGIS\bin\Python\envs\arcgispro-py3:#colorama 0.3.7 py35_0 defaultscycler 0.10.0 py35_0 defaultsfuture 0.15.2 py35_0 defaultsmatplotlib 1.5.3 np111py35_0e [arcgispro] esrimpmath 0.19 py35_1 defaultsnetcdf4 1.2.4 py35_0e [arcgispro] esrinose 1.3.7 py35_1 defaultsnumexpr 2.6.1 np111py35_0e [arcgispro] esrinumpy 1.11.2 py35_0e [arcgispro] esripandas 0.19.0 np111py35_0 defaultspip 8.1.2 py35_0 defaultspy 1.4.31 py35_0 defaultspyparsing 2.1.4 py35_0 defaultspypdf2 1.26.0 py_0 esripytest 2.9.2 py35_0 defaultspython 3.5.2 0 defaultspython-dateutil 2.5.3 py35_0 defaultspytz 2016.6.1 py35_0 defaultsrequests 2.11.1 py35_0 defaultsscipy 0.18.1 np111py35_0e [arcgispro] esrisetuptools 27.2.0 py35_1 defaultssympy 1.0 py35_0 defaultswheel 0.29.0 py35_0 defaults
CondaBasicsActivatingenvironments,acoupleways:
UsetheshortcutsManuallyactivatetheenvironment:
cd C:\ArcGIS\bin\Python\Scripts activate arcgispro-py3
CondaBasicsAcollectionofpackagesandPythoninstalliscalledanenvironmentorenv,thebuildingblockformanagingPythonwithCondaCanhavemultipleenvironmentsandseamlesslyswitchbetweenthem
Condavs…Name Means Included?
Conda Thecommanditself ✓Miniconda AminimumsetofPythonpackagesto
buildandrunConda.✓
Anaconda Adistribution200+packagesbuiltwithConda
AnacondaServer
Hostthefullinfrastructureinternally
DeeperDive
HowcanIusethis?ArcGIS(Desktop&Server)shipstheSciPystack—powerfulandoutoftheboxinallproductsCondacommandandaCondarootPythoninstallNewmodules(e.g.requests),environmentwithPro
HowcanIusethis?ArcGIS(Desktop&Server)shipstheSciPystack—powerfulandoutoftheboxinallproductsCondacommandandaCondarootPythoninstallNewmodules(e.g.requests),environmentwithProGetpackages,expandyourpossibilityspacePackageyourwork:thisisanopportunitytodistributeit,possiblyincludingcommercialsideaswell.
HowcanIusethis?
R
Esriand ?IntegrationviaArcGIS–RbridgeJoined andMoretocome—GIShashistoricallybeenmorecoupledwithPython
RConsortium RFoundation
Why ?Powerfulcoredatastructuresandoperations
Dataframes,functionalprogrammingUnparalleledbreadthofstatisticalroutines
ThedefactolanguageofStatisticiansCRAN:6400packagesforsolvingproblemsVersatileandpowerfulplotting
RDataTypesyou’reusedtoseeing…
Numeric-Integer-Character-Logical-timestampDatatypes
RDataTypesyou’reusedtoseeing…
Numeric-Integer-Character-Logical-timestamp…butothersyouprobablyaren’t:
vector-matrix-data.frame-factor
Datatypes
DataFramesTreatstabular(andmulti-dimensional)dataasalabeled,indexedseriesofobservations.Soundssimple,butisagamechangerovertypicalsoftwarewhichisjustdoing2Dlayout(e.g.Excel)
DataTypes# Create a data frame out of an existing sourcedf.from.csv <- read.csv( "data/growth.csv", header=TRUE)
DataTypes# Create a data frame from scratchquarter <- c(2, 3, 1)person <- c("Goodchild", "Tobler", "Krige")
met.quota <- c(TRUE, FALSE, TRUE)df <- data.frame(person, met.quota, quarter)
DataTypesR> df person met.quota quarter1 Goodchild TRUE 22 Tobler FALSE 33 Krige TRUE 1
0D:SpatialPoints1D:SpatialLines2D:SpatialPolygons3D:Solid4D:Space-time
spTypesEntity+Attributemodel
StatisticalFormulas
DomainspecificlanguageforstatisticsSimilarpropertiesinotherpartsofthelanguage
formodelspecificationconsistency
fit.results <- lm(pollution ~ elevation + rainfall +
caret
R—ArcGISBridge
R—ArcGISBridge
R—ArcGISBridge
ArcGISdeveloperscancreatetoolsandtoolboxesthatintegrateArcGISandRArcGISuserscanaccessRcodethroughgeoprocessingscriptsRuserscanaccessorganizationsGIS’data,managedintraditionalGISways
https://r-arcgis.github.io
R—ArcGISBridgeStoreyourdatainArcGIS,accessitquicklyinR,returnRobjectsback
toArcGISnativedatatypes(e.g.geodatabasefeatureclasses).
Knowshowtoconvertspatialdatatospobjects.PackageDocumentation
AccessArcGISfromRStartbyloadingthelibrary,andinitializingconnectiontoArcGIS:
# load the ArcGIS-R bridge librarylibrary(arcgisbinding)# initialize the connection to ArcGIS. Only needed when running directly from R.arc.check_product()
AccessArcGISfromRFirst,selectadatasource(canbeafeatureclass,alayer,oratable):
Then,filterthedatatothesetyouwanttoworkwith(createsin-memorydataframe):
ThiscreatesanArcGISdataframe–lookslikeadataframe,butretainsreferencesbacktothegeometrydata.
input.fc <- arc.open('data.gdb/features')
filtered.df <- arc.select(input.fc, fields=c('fid', 'mean'), where_clause="mean < 100")
AccessArcGISfromRNow,ifwewanttodoanalysisinRwiththisspatialdata,weneedittoberepresentedasspobjects.arc.data2spdoestheconversion
forus:
arc.sp2datainvertsthisprocess,takingspobjectsandgeneratingArcGIScompatibledataframes.
df.as.sp <- arc.data2sp(filtered.df)
AccessArcGISfromRFinishedwithourworkinR,wanttogetthedatabacktoArcGIS.Writeourresultsbacktoanewfeatureclass,witharc.write:
arc.write('data.gdb/new_features', results.df)
BuildingRScriptTools
RArcGISBridgeDemo
DetailsofmodelbasedclusteringanalysisintheRSampleTools
HowToInstallInstallwiththeRbridgeinstallDetailedinstallationinstructions
WhereCanIRunThis?
