Big Data Symposium: Analytics and Applications for Federal Big Data - FEMA Dr. Brand Niemann...
-
Upload
sharleen-coral-harrell -
Category
Documents
-
view
229 -
download
6
Transcript of Big Data Symposium: Analytics and Applications for Federal Big Data - FEMA Dr. Brand Niemann...
1
Big Data Symposium:Analytics and Applications for Federal Big
Data - FEMADr. Brand Niemann
Director and Senior Enterprise Architect – Data ScientistSemantic Community
http://semanticommunity.info/AOL Government Blogger
http://gov.aol.com/bloggers/brand-niemann/March 5-6, 2013
http://semanticommunity.info/Big_Data_Symposia
2
Begin With the End in Mind• Ms. Jo Strang, Associate Administrator, Safety, Federal Railroad
Administration, Department of Transportation “Open Gov 2.0 and Safety.Data.Gov”– New safety data sources and challenge from the National Institute of
Justice and new data from Open FEMA• Open FEMA
– Could not find with Google search– Start with Fema.gov
• http://www.fema.gov/
• Safety.Data.Gov– Could not find at that Web site– Start with Bureau of Justice Statistics
• http://www.bjs.gov/
3
My Process• Found Data Feeds at Fema.gov• Documented the Web Site, Metadata, and Data Sources for:
– USGS Earthquakes– World Map– FEMA Disaster Declarations Summary– FEMA Hazard Mitigation Program Summary– Public Assistance Subgrantee Funded Projects Detail– Public Assistance Funded Projects Summary
• Used Google Earth for the FEMA and USGS KML Files• Imported the Above Data Sets Into Spotfire• Created a Separate Visualization Tab for Each Data Set in Spotfire• Started a Unified Data Architecture• Submitted a Question About the Data Definitions to FEMA• Documented My Data Science Work in PowerPoint Slides• Provided Conclusions and Recommendations
6
Fema.gov Data Feeds Examples• RSS:
– Disaster Declarations• Query Form and List of Links
• KMZ:– Disaster Declared Counties - National View
• Designated Counties
• META and XLS:– Disaster Declarations Summary
• Same as RSS Disaster Declarations
– Hazard Mitigation Program Summary• Web Page
– Public Assistance Funded Projects Detail• No Web Page
– Public Assistance Funded Projects Summary• Web Page
• KML and CSV:– Worldwide M1+ Earthquakes, Past 7 Days (USGS)
• KMZ and TXT (No Web Page) My Note: But I found a Way Cool One!– http://earthquake.usgs.gov/earthquakes/map /
7
RSS: Disaster Declarations
http://www.fema.gov/disasters
8
META: Disaster Declarations Summary
http://www.fema.gov/library/viewRecord.do?id=6292
XLS
9
XLS: Disaster Declarations Summary
View / Download / Print (xls 9004K)
InstructionsMetadata
10
Hazard Mitigation Program Summary
http://www.fema.gov/hazard-mitigation-grant-program
11
META: Hazard MitigationProgram Summary
http://www.fema.gov/library/viewRecord.do?id=6293
XLS
12
XLS: Hazard MitigationProgram Summary
View / Download / Print (xls 16563K)
13
META: Public AssistanceFunded Projects Detail
http://www.fema.gov/library/viewRecord.do?id=6295
XLS
14
XLS: Public AssistanceFunded Projects Detail
View / Download / Print (xls 129914K)
15
Public Assistance FundedProjects Summary
http://www.fema.gov/public-assistance-local-state-tribal-and-non-profit
16
META: Public Assistance FundedProjects Summary
http://www.fema.gov/library/viewRecord.do?id=6299
17
XLS: Public Assistance FundedProjects Summary
View / Download / Print (xls 18935K)
InstructionsMetadataProjects 1-9Applicants
18
Worldwide M1+ Earthquakes,Past 7 Days (USGS)
http://earthquake.usgs.gov/earthquakes/catalogs/eqs7day-M1.txt
TXT not CSVSpotfire handles that!
20
Default View in Spotfire
Illustrates Data Visualization Design Using Shneiderman’s Mantra: Overview First, Zoom and Filter, Then Details-on-Demand
21
KMZ: FEMA Designated Counties and USGS Worldwide M1+ Earthquakes, Past 7 Days (USGS)
https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire
22
USGS Earthquakes: Spotfire
My Note: It is hard to see at thisscale, but the data points areall related in each visualization.
My Note: Could Spotfire do this? Can USGS do what Spotfire does?
https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire
23
World Map and Data Tables: Spotfire
My Note: I needed a world map and to see how to integrate it with the Earthquake and other data tables. This Spotfire functionality helps me do this and is the beginning of a “unified data architecture”(Scott Gnau, President, Terra Data Labs).
https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire
24
FEMA Disaster Declarations Program Summary: Spotfire
My Note: Thevisualizationsare connected!
https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire
25
FEMA Hazards Mitigation Program Summary: Spotfire
My Note: Thevisualizationsare connected!
https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire
26
FEMA Public AssistanceFunded Projects Detail: Spotfire
https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire
27
FEMA Public Assistance FundedProjects Summary: Spotfire
https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire
28
Conclusions and Recommendations
• USGS Earthquakes:– Could Spotfire do this? Can USGS do what Spotfire does?
• Google Earth of FEMA and USGS:– Need a way to access Google Earth within Spotfire.
• World Map:– One can easily find and import geospatial data files (Shape) into Spotfire.
• Unified Data Architecture:– Spotfire provides a way to produce this by displaying all the data tables and their columns.
• FEMA Disaster Declarations Summary:– I submitted a question about the data definitions to FEMA and am waiting for an answer.
• FEMA Hazard Mitigation Program Summary:– There is county data, but the names do not match those used in the standard Shape file for county
maps so I had to use the State designation.• Public Assistance Subgrantee Funded Projects Detail:
– Parts 1 and 2 of the Data Set need to be merged.• Public Assistance Funded Projects Summary:
– The nine additional tabs of spreadsheet data should be used in a separate Spotfire file.