Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS...
Transcript of Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS...
![Page 1: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/1.jpg)
Unstructured Data Analysis in ArcGIS
James Jones - Esri
Julia Bell - Esri
Scott Graff - Microsoft
![Page 2: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/2.jpg)
What is Unstructured Data?
• Does not have a recognizable structure or is loosely structured
• Can be in a variety of formats and storage mechanisms
• Word Documents
• Social Media Posts
• PowerPoint
• Share drive
![Page 3: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/3.jpg)
“Every two days we create as muchinformation as we did up to 2003”
Eric Schmidt, 2010
![Page 4: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/4.jpg)
What does that look like?Every minute…
Twitter sees new 350,000 tweets Facebook has 510,000 comments posted, 293,000 statuses updated
15.2 million Text Messages are sent
954,000 new Microsoft Office documents are created
144 million e-mails are sent
![Page 5: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/5.jpg)
How much spatial information are we missing out on?
![Page 6: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/6.jpg)
How much spatial information are we missing out on?
How can we capture this information in ArcGIS?
![Page 7: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/7.jpg)
CoordinatesCustom Locations
User defined keywords
Native Esri
Capability
What are you looking for?
What is the best tool?
How is it best used?
• Data is at least somewhat understood• Data benefits from identifiable and
repeating patterns• Little to no programming experience
available/needed
How to Integrate Unstructured Data into ArcGIS
ArcGIS Pro w/ LocateXT
ArcGIS Enterprise w/ LocateXT
ArcGIS Pro for Intelligence
![Page 8: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/8.jpg)
Extract Locations from Unstructured Data
ArcGIS LocateXT
![Page 9: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/9.jpg)
Extracting Locations with ArcGIS
• LocateXT Extension for ArcGIS Desktop and Enterprise
• Available for ArcMap 9.1 and later
• Available in ArcGIS Pro at 2.3
• 100% Feature function as ArcGIS Pro 2.4
• Uses pattern matching regular expressions (REGEX) to search for coordinates in a variety of formats
• Uses custom location list to match/extract other patterns (place names, codes, other terms)
![Page 10: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/10.jpg)
Extracting Locations in ArcGIS Pro
• New option added to the “Add Data” button
• Allows for a user to drag and drop documents or copied text into a window
• Can create a new feature class or append it to an existing one
• Included with ArcGIS Pro for Intelligence
![Page 11: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/11.jpg)
Extracting Locations in ArcGIS Pro
• Two Geoprocessing Tools added
• Located in the Conversion Tools –> To Geodatabase
• Extract Locations from Document
• Extract Locations from Text
![Page 12: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/12.jpg)
Extracting Custom Attributes
• Ability to create custom attributes based on content within document or near a location
• Triggered by location extraction
• Based on keywords
• Tag locations based on keywords
• Scrape/harvest portions of document based on keywords
• Ability to extract based off of:
• Number of characters/words
• Number of lines/blank line
• Stop string
• Previously built in separate LocateXT desktop application (until Pro 2.4)
![Page 13: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/13.jpg)
Extracting Addresses
• Ability to extract addresses from documents based on combination of:
• State
• Zipcode
• Ex. VA 22182
• The combination of extracted text and pre-text is geocoded
![Page 14: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/14.jpg)
Explore Unstructured Data through LocateXT and Custom Attributes
![Page 15: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/15.jpg)
How to Integrate Unstructured Data into ArcGIS
CoordinatesCustom Locations
User defined keywords
LocationsPeople/Organizations
EventsDates
Relationships
Native Esri
Capability
What are you looking for?
What is the best tool?
How is it best used?
• Data is at least somewhat understood• Data benefits from identifiable and
repeating patterns• Little to no programming experience
available/needed
• Data is not well understood• Data does not contain identifiable
and/or repeating patterns• Integration needed
Third Party
Integration
Natural Language Processing
ArcGIS Pro w/ LocateXT
ArcGIS Enterprise w/ LocateXT
ArcGIS Pro for Intelligence
![Page 16: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/16.jpg)
with ArcGIS
Integrating NLP
![Page 17: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/17.jpg)
Integrating NLP Capabilities with ArcGIS
• Many NLP offerings have Python APIs/SDKs or communicate over REST
• Integrates near seamlessly with ArcPy
• Create Python Toolboxes/Script Tools
• Allows to extract relevant data based on data local to their machine or as part of Enterprise Pipeline
• ArcGIS.Learn has incorporated support for Entity Recognition
APIs
Apps
Desktop
ArcGIS
NLTK
NLP Tools
![Page 18: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/18.jpg)
Processing Unstructured Data Using ArcGIS and Microsoft Azure
Source Processing Storage Analysis Apps/Visualization
Key Take-aways:
1. Leverages modern, serverless processes and integration apps
2. Allows for a variety of NLP processes to be ran
3. Deep analytics with ArcGIS and Azure Cognitive Services/Machine Learning
![Page 19: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/19.jpg)
Building an Unstructured Pipeline to Understand World Events
Source Processing Storage Analysis Apps/Visualization
Logic Apps / Power Automate watch RSS feeds and Website.Microsoft Cognitive Services extracts entities and analyze
sentiment.
Data is passed to ArcGIS GeoEvent Server for ingestion
into ArcGIS Platform. JSON files are stored in Azure Storage. Tabular data is
stored in a Azure SQL Data Warehouse.
ArcGIS Pro for Intelligence allows non-GIS Intel
professionals access to a custom experience of ArcGIS Pro to provide deep analysis. Microsoft Cognitive Services
further enriches data by running computer vision
against embedded images.
Operations Dashboard and ArcGIS Insights allows for very tailored views of the data to be
quickly analyzed and viewed by decision makers and non-
GIS Professionals.
![Page 20: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/20.jpg)
ArcGIS Pro for Intelligence
Multi-int workstation for the intelligence professional
• Create and manage intelligence information
• Visualize and display your data in maps, charts, and timelines
• Perform spatial, temporal, relational, and predictive analysis
• Produce and disseminate intelligence products
![Page 21: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/21.jpg)
NLP Integration with ArcGIS
![Page 22: Unstructured Data Analysis in ArcGIS · 2020. 7. 2. · Processing Unstructured Data Using ArcGIS and Microsoft Azure Source Processing Storage Analysis Apps/Visualization Key Take-aways:](https://reader030.fdocuments.in/reader030/viewer/2022013023/5fddb93a5754e13788463fc3/html5/thumbnails/22.jpg)
Download the Esri
Events app and find your event
Select the session
you attended
Scroll down to
“Survey”
Log in to access the
survey
Complete the survey
and select “Submit”
Please Share Your Feedback in the App