NEES Grid Data Overview
-
Upload
lafayette-isai -
Category
Documents
-
view
32 -
download
0
description
Transcript of NEES Grid Data Overview
![Page 2: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/2.jpg)
Introduction
• The data approach has evolved significantly in the past year– Second version of the Data Repository (security, access
control, performance improvements)– Data Turbine as unified real-time storage– NTCP has become increasingly capable– Model activity is bearing fruit (multiple) - Protégé / RDF /
XML Schema– Data Curation summit has provided vision– We now have a notebook which captures metadata
• As we see more detail in these areas, we find new areas that need exploration
![Page 3: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/3.jpg)
Boxology
NEES GridData Approach
Data Models
ExperimentManagement
DataAcquisition
ExperimentMonitoring
DataAnalysis
CentralRepository
LocalRepository
Notebook
![Page 4: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/4.jpg)
Data LifecycleData Models
Experiment Prep
Data Analysis
Data Publishing
Data Curation
Data Discovery and Reuse
Experiment Management
Data Monitoring
![Page 5: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/5.jpg)
Data/MetadataCapture
Throughout
Data Models
Experiment Prep
Data Analysis
Data Publishing
Data Curation
Data Discovery and Reuse
Experiment Management
Data Monitoring
![Page 6: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/6.jpg)
Data Models
• Data models are developed in RDF• Local repository supports multiple
simultaneous data models with cross-model linkages
• Metadata browser (aka Project browser) becomes the Project Browser, Notebook Browser, Site Specification Database Browser
• Metadata browser can federate multiple sources of Metadata
![Page 7: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/7.jpg)
InstrumentationSetup
SensorGroup
Sensor
Specimen
EquipmentSetup
CalibrationSet
Equipment
CameraDataEquipment
![Page 8: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/8.jpg)
Project Model Proj
Exp
Trial
Sensor
Site
Person Facility
Equipment
Specimen
Element Element
Site Model
Multiple Models
Notebook
Chapter Entry
![Page 9: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/9.jpg)
Overall Data Modeling EffortsOverall Data Modeling Efforts
NEES
Site A Site CSite B
Equipment People
Experiments Trials
Equipment People
Experiments Trials
Data Data Data
TsnumaiSpecimen
Shake TableSpecimen
GeotechSpecimen
CentrifugeSpecimen
Units Sensors Descriptions
SiteSpecificationsDatabase
ProjectDescription
Domain Specificmodels
Common Elements
Data / Observations
Ref. Source: Chuck Severance
![Page 10: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/10.jpg)
Models + Data Model
Repo
Models
Configure
Data
Load
Con
figur
e
RDF/OWL
RDF<owl:ObjectProperty rdf:ID="hasPublications"> <rdfs:domain> <owl:Class> <owl:unionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Project"/> <owl:Class rdf:about="#Task"/> </owl:unionOf> </owl:Class> </rdfs:domain> <rdfs:range rdf:resource="#Publications"/> </owl:ObjectProperty>
![Page 11: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/11.jpg)
Protégé - 2K
Models + Data Model
Repo
Models
Configure
Data
Load
Con
figur
e
RDF/OWL
RDF
![Page 12: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/12.jpg)
Experiment Preparation
• Notebook– Allows the creation of material without needing a model– The model is pages, chapters, and “stuff”– It is all captured with data and metadata– A notebook can be attached to any object in the model
structure (i.e. a project can have a notebook, a trial can have a notebook, etc…)
• Resources• Discussions• Project Browser
– Setup basic structured metadata for the experiment - Trials, descriptions, sensors, etc… This material is captured in accordance to and with the data model
![Page 13: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/13.jpg)
DOE ELN / Example
![Page 14: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/14.jpg)
Setting up and Experiment
• Prior to running an experiment, the project browser will be used to create a trial, and experiment configuration, set up sensors, etc.
• In some cases, setup information may be done on the DAQ itself and the configuration information may be pulled from the DAQ
![Page 15: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/15.jpg)
NEESgrid Experiment Data FlowNEESgrid Experiment Data Flow
NEESGridData
Repository
ProjectBrowser
DataTurbine
DataIngestion
ExperimentControl
StreamingViewer
DAQC
D
SiteSpecific
ProjectRelated
ExperimentalSetup
ExperimentalElement
DataElement
Data Model
DAQDisk
StoredViewer
![Page 16: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/16.jpg)
Experiment Management
• Simple reference implementations for– Experiment configuration (pull / push)– Experiment Start– Experiment Stop
• Some combination of LabView and CHEF code
![Page 17: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/17.jpg)
DT Main System
PTZ/USB
StillCapture
DT Client
BT848Video
Frames
DT Client
Capturing Video and Data
Camera ControlGateway
DAQData
CaptureDT Client
SimulationCoordinator
Site A Site B
![Page 18: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/18.jpg)
DT Main System
Data Monitoring Tools
Still Image / Camera Control
~
< >^
^
< >
Camera ControlGateway
Creareviewers
Still imagecameracontrol
Thumb-nail
![Page 19: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/19.jpg)
Working with Creare
• We want to leverage Creare’s live capture and viewers– Integrated Live Video and Data Viewer– Audio capability in addition to Video– JMF DataSource Capability - Use JMStudio
• SI will focus on the extraction, repository, data model, and stored viewer aspects
![Page 20: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/20.jpg)
Data Stored in Data Turbine
Video
Stills
Data
4 5 6 7 8TimeStep*
* Time Step is only present for Pseudo-dynamic
Wall Clock Time
![Page 21: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/21.jpg)
Data Extraction / Ingestion
• A tool will be developed to extract data from Data Turbine and place it in the NEES repository in the appropriate format– Video Channels– Image Channels– Data Channels
• The information will be stored in a format suitable for viewing using the stored viewer and appropriate metadata will be placed in the repository so that the information can be viewed
• This process is the primary new work in this plan
![Page 22: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/22.jpg)
DT Main System
Data Extraction For Analysis
Data ExtractionPseudo-Dynamic
Continuous
Time Step Channel xyz
Start Time Step 1
End Time Step 9999
Export Auto Export
NEESData
Repository
![Page 23: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/23.jpg)
Pseudo-Dynamic Extraction
Video
Stills
Data
4 5 6 7 8TimeStep*
Wall Clock Time
![Page 24: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/24.jpg)
Continuous Extraction
Video
Stills
Data
Wall Clock Time
![Page 25: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/25.jpg)
![Page 26: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/26.jpg)
Stored Data Viewer Improvements
• Interactive Mode allowing reconfiguration of views within the Applet (insta-view)
• Linear combinations of data values• Ability to launch from the Project
Browser• Looking at integration with notebook
(i.e. launch from the notebook)
![Page 27: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/27.jpg)
Central Repository / Curation
• Curation and the Central Repository are different than the local repository and the running / management of experiments
• Data must be packaged, kept, indexed, and maintained for the long term
![Page 28: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/28.jpg)
Curation Flow
Project Model Proj
Exp
Trial
Sensor
Site
Person Facility
Equipment
Specimen
Element Element
Site Model
Multiple Models
Notebook
Chapter Entry
NEESgrid NEESgrid Experiment Data FlowExperiment Data Flow
NEESGridData
Repository
ProjectBrowser
DataTurbine
DataIngestion
ExperimentControl
StreamingViewer
DAQC
D
SiteSpecific
ProjectRelated
ExperimentalSetup
ExperimentalElement
DataElement
DataModel
DAQDisk
StoredViewer
DT Main System
PTZ/USB
StillCapture
DT Client
BT848Video
Frames
DT Client
Capturing Video and Data
Camera ControlGateway
DAQData
CaptureDT Client
SimulationCoordinator
Site A Site B
CurationBundle
• At some point, a project, experiment, etc is ready for curation. We must save all the information (models, notebooks, sensor data, etc) for transfer to the central repository
![Page 29: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/29.jpg)
Data/MetadataCapture
Throughout
Data Models
Experiment Prep
Data Analysis
Data Publishing
Data Curation
Data Discovery and Reuse
Experiment Management
Data Monitoring
![Page 30: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/30.jpg)
Workflow in Central Repository
• The workflow of the central repository will be defined over time - here are some sample concepts– Incoming materials collect in an inbox– The curator processed the materials - adds required metadata,
checks incoming data models, distinguishes information, and makes the bundle ready for publication
– Some data is published immediately, other data is held for a period of time (perhaps to allow for publication)
– Published data can be searched and viewed used and downloaded
• There are people in the curation loop• The software for this is non trivial and will evolve over
time with requirements• Sometimes it will be necessary to alter/convert data to
insure its value over time.
![Page 31: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/31.jpg)
Workflow in Central Repository
CurationBundle
CurationBundle
InBoxProcessed
Published
Search
Hold forTime
NeedConversion
![Page 32: NEES Grid Data Overview](https://reader033.fdocuments.in/reader033/viewer/2022052217/56812fe8550346895d955f7a/html5/thumbnails/32.jpg)
Conclusion
• This is a significant adjustment in priority• But not a significant shift in approach or
architecture• All of the elements which have been
discussed can still be delivered - the elements described herein are just higher priority.