1 November 17, 2009 Jim Russell - EOPM Review GATS AIM Data Accessibility John McNabb, AIM Project...
-
Upload
camron-martin-reeves -
Category
Documents
-
view
213 -
download
0
Transcript of 1 November 17, 2009 Jim Russell - EOPM Review GATS AIM Data Accessibility John McNabb, AIM Project...
1November 17, 2009Jim Russell - EOPM Review
GATS
AIM Data Accessibility
John McNabb,
AIM Project Data Center Manager
Hampton University
2November 17, 2009Jim Russell - EOPM Review
GATS
AIM Data Accessibility
• AIM Science Data System (SDS)
• AIM Project Data Center (PDC)
• Data Availability and Access Statistics
• Lessons Learned
3November 17, 2009Jim Russell - EOPM Review
GATS
AIM Science Data System
• Distributed Data Processing and Delivery– Instrument Data Processing Centers (DPC):
• Process, store and disseminate data products for their instrument
• Centralized Meta-data, Search, and Archiving– Project Data Center (PDC):
• Provide search capability for instrument data products
• Interfaces to:– Affiliated Systems(Co-Investigators, NSSDC, E/PO K-14)– Non-affiliated Systems (Science community, Public)
4November 17, 2009Jim Russell - EOPM Review
GATS
Command Flow
Telemetry Flow
Post MissionMission Operations Elements
Science Processing Elements
Affiliated Systems
Non-affiliated Systems
MOC Mission Operations Center DPC Data Processing Center PDC Project Data CenterPOC Payload Operation Center
GroundStation
MOCLASP / CU
Co-
Investigators
AIM PDCHU
K-14Community
NSSDC
SOFIE
DPCGATS
CIPS/CDE
DPCLASP / CU
E/POHU
Public &Scientific
Community
POC
POC
5November 17, 2009Jim Russell - EOPM Review
GATS
MOCLASP
DPCCIPS/CDE
DPCSOFIE
Data Users
PDCHU
Raw Data Raw Data
GATS FTP
Meta-Data Meta-Data
Data Distribution
Search Queries
LASP FTP
Data Products Data Products
6November 17, 2009Jim Russell - EOPM Review
GATS
MOCLASP
DPCCIPS/CDE
DPCSOFIE
PDCHU
Raw Data Raw Data
Data Archiving
Data Products Data Products
NSSDC
Final Archive
LASP GATS
7November 17, 2009Jim Russell - EOPM Review
GATS
AIM Project Data Center
• Primary source of information to outside groups– Online Data Search tools– Preliminary Archive– Orbital Predictions
aim.hamptonu.edu
8November 17, 2009Jim Russell - EOPM Review
GATS
Online data search tool
9November 17, 2009Jim Russell - EOPM Review
GATS
Preliminary Archive
• Provides ongoing development and testing platform for final archive procedures.
• Collect all publicly available data as soon as it is published.
• Collect all data products and documentation in preparation for final archive.
10November 17, 2009Jim Russell - EOPM Review
GATS
Orbital Predictions
• Have provided regular orbital predictions for ground based sensors, sounding rocket launches, and Space Shuttle missions
• Online on-demand predictions available
11November 17, 2009Jim Russell - EOPM Review
GATS
Data Availability: CDE• CDE has released two levels of science
products:– Level 1: Charge
• Amount of charge generated by PVDF upon particles impact
– Level 2: Mass• Calculated mass of impacting particle
– All data available on Hampton Univ. AIM website• Currently at final version, V5
– Data available from June 1, 2007 to June 1, 2009
12November 17, 2009Jim Russell - EOPM Review
GATS
Data Availability: CIPS
• Preliminary data is available within 24 hours to Science Team and others by special request.
• After 5 days definitive revision of current version of data is created and published.
13November 17, 2009Jim Russell - EOPM Review
GATS
Data Availability: SOFIE• Preliminary data is available within 24
hours to Science Team
• After review (typically 2-3 days) definitive revision of current version of data is published.
• Online tools provide powerful plotting options
14November 17, 2009Jim Russell - EOPM Review
GATS
SOFIE Online Plotting Tools
http://sofie.gats-inc.com/sofie
15November 17, 2009Jim Russell - EOPM Review
GATS
Data Access Statistics: PDC
Average unique IP addresses per month based on Fall 2009 (excluding bots and local users)
• ~1500 to AIM web site
• ~700 to AIM homepage
• ~260 to outreach web pages
• ~75 to Data Search Tool
16November 17, 2009Jim Russell - EOPM Review
GATS
Data Access Statistics: CIPS/CDE
• Based on Google Analytics
• 175 Unique visitors over 3 months
• 41% new visitors
• Referrers:– 50% direct access– 25% google.com– 12% aim.hamptonu.edu
17November 17, 2009Jim Russell - EOPM Review
GATS
Data Access Statistics: SOFIE
• Based on Google Analytics and Webalizer data
• 25% new visitors
• Referrers:– 86% direct access. – 11% referred by aim.hamptonu.edu
• ~50 daily average visits
18November 17, 2009Jim Russell - EOPM Review
GATS
Data Access Statistics: SOFIE
19November 17, 2009Jim Russell - EOPM Review
GATS
Lessons Learned
• Product Availability Notices (PANs) sent by email can fall prey to changing SPAM filters. Realistic test cases for sending PANs need to include large numbers of files in a short period of time representing reprocessing of the whole data set.
• In file naming, version numbering system should include at least two integer places, one decimal, and a two character revision: "v02.1_rev03"
• NetCDF-3 is not well suited for storing CIPS data. Official file types should be chosen that works best for the type of data needing to be stored, and that might be different for different data sets on a project.
• A distributed data dissemination system that takes advantage of shared data servers can work, however decentralized anonymous FTP makes tracking data downloads difficult. People will not generally register for update notifications unless required to in order to get access to the data.
• Overlapping transition period is very helpful when personnel change