Where Do We Put It All? Lessons Learned Housing Large Geospatial Data Collections In OCUL's Scholars...
22
Where Do We Put It All? Lessons Learned Housing Large Geospatial Data Collections In OCUL’s Scholars GeoPortal Jo Ashley GIS Analyst, Scholars Portal
-
Upload
nacisslides -
Category
Design
-
view
45 -
download
0
Transcript of Where Do We Put It All? Lessons Learned Housing Large Geospatial Data Collections In OCUL's Scholars...
Where Do We Put It All? Lessons Learned Housing Large Geospatial Data Collections In OCUL’s Scholars GeoPortal
Jo AshleyGIS Analyst, Scholars Portal
Scholars GeoPortal
Context
Ontario is a large Province with diverse data needs and interests represented by OCUL
Timeline of Development
2002 2002-2007 2008 2009 2010-
2012 2012 2016
Founded
Establishedservices
Draft proposalfor GeoPortalGeoVisioningWorkshopOCUL MapGroup drivingforce
Project grantawarded forGeoPortal
GeoPortaldevelopment
Official launch
Going strong1,500 datasets120 TB’s andcounting…
Overview• How do hardware & software requirements guide
our data loading?• How do we prioritize, manage and load large
amounts of geospatial data?• How can we improve our overall processing
workflow?• How do we make all the data loaded discoverable
to the user?
Hardware & software
Hardware & software
Current• Encountering
performance issues related to size and shear amount of data 10.0.
• Updating HD & software system aims to reduce these issues.
• Currently producing clusters in architecture
Moving forward• Will look into ArcGIS
Online & Portal for ArcGIS if improvements in performance don’t improve… early days yet.
Datasets
Vector (map services)
DMTI Local
DLI OGDE
Vector (map services)
Vector loading challenges
• Balancing different needs across different schools.
• Size of data and length of time to process.
• Popularity or value to research community.
• In future we will need to consider loading researcher data (to comply with funder mandates and archiving policies).
Raster (image services)SCOOP 2013 TIFFTiles: 35,923Data: 3.4TB (102MB each tile) Overviews: 1.66TB
FRI 2007-2010 Block I & JTiles: 2,306Data: 2.7TB (1GB each tile) Overviews: 887GB
Raster (image services)
TIFF MrSID JPEG JPEG2000
Compression
Lossless (raw) Lossy (JPEG)
Lossless or Lossy
Lossy (Lossless*)
Lossless or Lossy
File size Usually largest Small to Moderate Small Small to
Moderate
Maps (image services) 030M11
19091921Waterdamage
1931Not gridded
1931 Gridded
Raster loading challenges
• Be mindful of imagery size, type (i.e. orthos vs. DSM derivative), and storage capacity (jpegs vs. tiffs)
• Consider loading larger data into the Cloud (Ontario Library Research Cloud) to reduce redundancies and facilitate preservation.
Overall process issues
Original vector data
Original raster data
ArcMapmxd
Mosaic
*
Map service
Caching
*SDE GDB
File GDBImage service
Dataset layer available on
the GeoPortal
* * *
Imagery available on
the GeoPortal
* *
Overall process solutions
• Migration to 10.4 will reduce redundancy by half.
• Automation of process will make service production more efficient.
Data discovery issues
Print vs. ExportAnnotationsShare vs. permalinkData table
Data discovery solutions
• Work with the OCUL community to determine preferred functionality, portal objective(s) and overall functionality
• Review usage statistics and let them assist in future development
Lessons learned
• Imperative to upgrade to ArcGIS 10.4 to support continued growth of GeoPortal
• Must automate data loading process in order to meet ongoing demands
• Work with OCUL community and analyze usage stats to prioritize loading
• Continue to review and upgrade our interface to improve data discovery
