Science Data Plan including SOC Plan: - Stanford...

Changed on 12 November 2009 1:28 AM

HMI & AIA Joint Science Operations Center

Science Data Processing

Plan Overviewa.k.a.

JSOC-SDP OverviewHMI-S019

12 November 2009 1:28 AM

version 0.8

/tt/file_convert/5b3b13057f8b9a986e8bd781/document.doc i

JSOC GDS Science Data Processing Plan version 0.8

ii


Contents

Word did not find any entries for your table of contents.In your document, select the words to include in the table of contents, and then in the Formatting Palette under Styles, click a heading style. Repeat for each heading that you want to include, and then insert the table of contents in your document. You can also create a table of contents by clicking the Create with Manual Formatting option and then type the entries manually.

THE table of contents got messed up

iii

Changed on 12 November 2009 1:28 AM

1. Overview

The SDO HMI Contract Performance Specification states that:

“9.3 Science Operations Control Center (SOC)

The Contractor shall design, develop and operate a Science Operations Center (SOC). The SOC will interface with the SDO Mission Operations Center (MOC) to provide operational interfaces for the on-orbit operation and monitoring of the instrument and the transfer of instrument housekeeping and science data collected by the spacecraft and transmitted to the MOC ground stations. The SOC shall subsequently convert this raw science data into valid research quality data and data products for archival and public access and distribution. Some key functions to be performed in the SOC include:

a. Instrument Flight Operations. The conduct of instrument on-orbit flight operations in concert with the SDO MOC and its associated Flight Operations Team (FOT). The Contractor shall be responsible for the command planning and the health and safety monitoring of the flight instrument. The specific extent of command authority will be resolved during mission operations development dialogues with the SDO Project, but as a minimum, any command activity that constitutes a hazardous command activity or requires coordination with the FOT for reconfiguring of spacecraft resources or reconfiguring another instrument shall be planned, integrated and authorized by the FOT. The SOC shall provide all the required operator interfaces to display, monitor and analyze the instrument operating state, operating condition and trended behavior; support the instrument operation and observation planning and the build of associated commands and command loads; and provide effective communication means for contact and coordination with the MOC and among the instrument operations team members.

b. Science Data Processing. The receipt, sorting, quality checking and process of the instrument science data forwarded to the SOC by the SDO ground system. The Contractor shall provide the required software and computational algorithms to process this data into the required science data products on a regular, routine basis. This effort includes the need to monitor the calibration of the flight instrument and adjust the processing software accordingly. The Contractor shall store and archive the science data and perishable data products, and shall provide public access and distribution to the data and data products. The Contractor shall prepare and submit the plan for the architecture, data flow, processing, archival and distribution of the science data through the SOC (CDRL SD326).”

This document is the first of the three parts of CDRL SD326.

/tt/file_convert/5b3b13057f8b9a986e8bd781/document.doc 1


1.1. Scope of this document

The HMI Science Operations Center (SOC) has been merged with the AIA SOC to become the HMI/AIA Joint Science Operations Center (JSOC). The JSOC consists of two parts. The Ground Data System (GDS) at Stanford University provides the HMI and AIA data capture from the SDO DDS at White Sands, the processing through level-0 and level-1 data products, and permanent local and offsite archiving of the telemetry data streams. The JSOC-GDS also provides higher level science data processing for HMI through to the level-2 data products identified as HMI standard data products. The JSOC-OPS center at Lockheed Solar and Astrophysics Laboratory provides the instrument commanding and health and safety monitoring.

This document contains the top level description of the JSOC-GDS.

1.2. History of JSOC

The HMI investigation was selected for SDO in August 2002. The SDO plan for operations, data capture and data analysis calls for each instrument investigation team to provide a “Science Operations Center” or SOC where the instrument will be operated and monitored and where the science data will be archived and, processed into high level data products. The science analysis of these data products is also a responsibility of the selected investigations but is beyond the scope of the SOCs. The original plan for the HMI SOC was to build upon the successful SOHO/MDI SOI Science Support Center at Stanford University for the science data handling functions of the HMI SOC and to operate the HMI instrument jointly with our Co-I team at Lockheed Martin Solar and Astrophysics Laboratory (LMSAL).

In the fall of 2003 NASA decided to add the LMSAL proposed AIA instrument to the SDO mission. At that time it became clear that the most cost effective and science-efficient plan was to merge the AIA SOC with the HMI SOC to form the Joint Science Operations Center (JSOC) for the HMI and AIA investigations. For several reasons including ease of access by visitors at Stanford, in place secure access areas at LMSAL, lower cost of procurement and staffing at Stanford, and the operations experience at LMSAL we jointly concluded that the commanding and health monitoring for both instruments would best occur at LMSAL while the data archiving, low level processing, and data export functions would best be handled at Stanford. The higher level science analysis for each instrument will be centered at the lead institution for each instrument. Thus we have the present JSOC split into two parts, the JSOC-OPS at LMSAL and the JSOC-GDS at Stanford.

For many years the solar physics groups at Stanford University and LMSAL have cooperated on a number of missions under the umbrella of the Stanford-Lockheed Institute for Space Research. This agreement of cooperation between the two institutions helps to enable a seamless joint effort.

2


1.3. JSOC role in SDO

SDO, in essence, is a spacecraft that orients itself toward the sun, powers up, and observes. Its standard observation program is to run in this mode, autonomously and continuously, for a complete solar cycle. Although other modes of operation exist (such as HMI’s calibration modes and AIA’s additional standard modes), this standard observation program will be adhered to as much as possible. Compared to SOHO, the JSOC operations will be simple. The plan is to have four dedicated commanding workstation for each of the HMI and AIA instruments – two at the MOC and two at LMSAL, one of each of these pairs serving as a backup. These systems will generate command loads, monitor instrument health and status, and capture the instrument and observatory housekeeping data received from the MOC. Communication with the MOC is restricted to these workstations, which are located in rooms that meet NASA data-system IT-2 security requirements. Personnel will monitor the workstations during regular business hours.

The bulk of the JSOC activities involve science data generation and distribution. HMI and AIA will generate data at a rate of 55Mbps and 67Mbps, respectively. Although this is a significant dataflow, about 15Mbytes/sec or 1.4TB/day, the rate of image generation and the complexity of image types are comparable to that of the SOHO/MDI program. HMI and AIA will produce 32 and 48 images per minute, respectively. Every 90 seconds, HMI will produce 36 types of images (6 wavelengths at 6 polarizations), and every 10 seconds AIA will produce 8 types of images (4 cameras each capturing two different wavelength bands). MDI captures 5-6 images per minute, which contributes to dozens of image types under normal operations. Thus, while the HMI/AIA data volume is roughly 800 times that of MDI, the complexity is commensurate. More importantly, the HMI/AIA data volume rate relative to computer processing power is approximately equivalent to that of MDI. The amount of hardware needed to support this dataflow is larger than that found in a typical solar physics analysis facility but significantly less than that at a supercomputer center. The “floor space” required to accommodate the computer hardware for the HMI/AIA JSOC project, scheduled to launch in early 2010, and the floor space needed for the MDI SOHO project, which launched in 1995, are similar.

In line with the LWS Data Management Plan the term “SOC” as used here, refers to instrument operations, data capture, data processing of telemetry to science data-products, distribution of data to the science teams (including the PI’s science teams), data archive, and data export. Science investigation per se is not under the purview of the SOC, but PI teams, Co-Investigators, and other LWS investigators will use JSOC facilities to execute such investigations. Figure 1 shows the “big picture” role of the JSOC in the SDO mission while Figure 2 shows the connectivity between the components of the JSOC, the SDO MOC, the SDO DDS, and the user community.

The SDO SOC provides several services that fall under the auspices of three centers established by the HMI and AIA teams. These are the JSOC-IOC, which is located at the LMSAL facility in Palo Alto, the JSOC-SDP, which is located on the Stanford University campus, and the AIA Visualization Center (AVC) located at LMSAL. The JSOC-IOC tasks and the lower-level JSOC-SDP for HMI are essentially identical to those for AIA. The processing of higher-level

3


HMI data differs from that of AIA data. Accordingly, the former is executed at the JSOC-SDP while the latter is performed at the AVC, with cooperation in place between the two centers to achieve common goals.

The HMI goals require the processing, in sequential steps, of the HMI data into higher-level helioseismology and magnetic-field standard data products. This processing expedites the data analysis needed to advance the investigations. These higher-level processing tasks are very data intensive and require facile access to the whole volume of HMI data. As the computational system used for the low-level data processing is similarly suitable for these tasks, the same infrastructure is used for this higher-level processing.

The AIA science goals encompass interactive processing of AIA data into movie-like sequences of data images and derived products. The definition of these processing steps by the AIA team was, in large part, based upon techniques gleaned from the TRACE and Yohkoh missions. In order to provide the high data rate required needed to support this data-handling, the Stanford team maintains a high-bandwidth networking connection between the Stanford JSOC-SDP and the AVC.

In summary, for both HMI and AIA the JSOC-IOC: provides a secure workstation area for instrument commanding and status monitoring. serves as the access point for the dedicated network SDO Mission Operations Center

(MOC) connections. provide an isolated facility for reviewing “quicklook” science data provided by the

JSOC-SDP. implements the science-observing plan developed by each instruments’ Science Planning

Team. provides to the JSOC-SDP the command history logs used for data analysis.

For both HMI and AIA, the JSOC-SDP: houses the JSOC-SDP Data Capture System, which provides:

o an isolated area for data received from the SDO Data Distribution System (DDS). o an access point for the dedicated DDS network connections.o the data capture machines, which collect and archive data received from the DDS,

and copy it offsite.o software that transforms raw data into level-0 images and associated meta-data.o software that compiles JSOC-IOC logs and SDO FDS and planning data.o software that transfers the level-0 data and logs, FDS data, and planning data to

the JSOC-SDP pipeline-processing system.

houses the JSOC-SDP Pipeline-Processing System, which:o archives and distributes (upon request) level-0 data.o produces quicklook images and delivers them to JSOC-IOC. o produces level-1 calibrated HMI-science-observables.o produces level-1 calibrated AIA-science-observables.o processes, archives, and distributes JSOC-IOC logs and SDO FDS and planning

data.

4


o produces level-2 HMI data-products.o distributes, upon request, level-1 and level-2 data in standard data-storage

formats.o produces and distributes selected HMI and AIA “Space Weather” data products.o archives selected level-1 and level-2 products.o produces, archives, and distributes selected HMI higher level products.o receives, archives, and distributes selected AIA higher-level science-analysis

products.o provides a data catalog to be accessed by the broad user community.o supports LMSAL-led development of AIA calibration and quicklook processing

software.o supports HMI and AIA Co-Investigator-led development of analysis software that

runs in the JSOC processing pipeline.o supports E/PO related access to selected data products.

5


6

Telemetry & Command(T&C)

SystemASIST / FEDS

Telemetry MonitoringCommand Management

HK Data Archival (DHDS)HK Level-0 Processing

Automated OperationsAnomaly detection

Flight DynamicsSystem

Orbit DeterminationManeuver PlanningProduct Generation

R/T Attitude DeterminationSensor/Actuator Calibration

SDO Mission Operations Center

EVE SOC(LASP /

Boulder Co.)

Acquisition Data

Observatory Commands

Observatory Housekeeping Telemetry

Tracking Data

Trending

Mission Planning& Scheduling

plan daily/periodic eventscreate engineering planGenerate Daily Loads

HMIScience Data

55Mbps

Ka-Band:150 Mbps

Science Data

S-Band: TRK,Cmd & HK Tlm

Instrument Commands / Loads

Data DistributionSystem(Incl. 30-Day

Science Data Storage)

SDO Ground Site #1(White Sands)

S-Bandground system

Ka-Bandground system

Ka Science Data

AIAScience Data

67Mbps

EVEScience Data

7 Mbps

R/T Housekeeping Telemetry/ Science Planning and FDS Products

Science Planning and FDS Products

ExternalTracking Station

S-Band HK Tlm, TRK Data

AcquisitionData

Station/DDS Control

Station/DDS Status

Ground StationControl System

DDSControl System


Ka-Band:150 Mbps

Science Data

R/T Housekeeping Telemetry

S-Band: TRK, Cmd & HK Tlm

Cmd & HK TlmS-Band: TRK,

Cmd(Includes 72-hr storage)

S-Bandground system


Alert NotificationSystem (ANS)

Flight SoftwareMaintenance Lab

(FLATSAT)

Flight software loadsSimulated housekeeping telemetry

Memory dumpsSimulated commands

(Includes 48-hr storage)

Same Interfacesas Prime Ground Site



Status and Control

HMI AIA JSOC(Palo Alto Ca.)

StanfordScience Data Processing

LMSALInstrument Monitoring

& Control Instrument Commands / Loads

Telemetry & Command(T&C)

SystemASIST / FEDS

Telemetry MonitoringCommand Management

HK Data Archival (DHDS)HK Level-0 Processing

Automated OperationsAnomaly detection

Flight DynamicsSystem

Orbit DeterminationManeuver PlanningProduct Generation

R/T Attitude DeterminationSensor/Actuator Calibration

SDO Mission Operations Center

EVE SOC(LASP /

Boulder Co.)

Acquisition Data

Observatory Commands

Observatory Housekeeping Telemetry

Tracking Data

Trending

Mission Planning& Scheduling

plan daily/periodic eventscreate engineering planGenerate Daily Loads

HMIScience Data

55Mbps

Ka-Band:150 Mbps

Science Data

S-Band: TRK,Cmd & HK Tlm

Instrument Commands / Loads

Data DistributionSystem(Incl. 30-Day

Science Data Storage)


S-Bandground system


Ka Science Data

AIAScience Data

67Mbps

EVEScience Data

7 Mbps

R/T Housekeeping Telemetry/ Science Planning and FDS Products

Science Planning and FDS Products

ExternalTracking Station

S-Band HK Tlm, TRK Data

AcquisitionData

Station/DDS Control

Station/DDS Status

Ground StationControl System

DDSControl System


Ka-Band:150 Mbps

Science Data

R/T Housekeeping Telemetry

S-Band: TRK, Cmd & HK Tlm

Cmd & HK TlmS-Band: TRK,

Cmd(Includes 72-hr storage)

S-Bandground system


Alert NotificationSystem (ANS)

Flight SoftwareMaintenance Lab

(FLATSAT)

Flight software loadsSimulated housekeeping telemetry

Memory dumpsSimulated commands


Same Interfacesas Prime Ground Site



Status and Control




& Control





& Control Instrument Commands / Loads

Figure1. Role of JSOC in SDO mission.

Figure 2. Basic Connectivity between SDO Mission Operations, Data Distribution, and instrument Science Operations Centers.

Science TeamForecast Centers

EPOPublic

Catalog

Primary Archive

HMI & AIAOperations

House-keeping

Database

MOCDDS

Redundant Data

Capture System

30-DayArchive

OffsiteArchive

OfflineArchive

HMI JSOC Pipeline Processing System

DataExport& WebService

Stanford

LMSAL

High-LevelData Import

AIA AnalysisSystem

Local Archive

QuicklookViewing

GSFCWhite Sands

WorldScience Team

Forecast CentersEPO

Public

Catalog

Primary Archive

HMI & AIAOperations

House-keeping

Database

MOCDDS

Redundant Data

Capture System

30-DayArchive

OffsiteArchive

OfflineArchive



Stanford

LMSAL


AIA AnalysisSystem

Local Archive

QuicklookViewing

GSFCWhite Sands

World

Catalog

Primary Archive

Catalog

Primary Archive

HMI & AIAOperations

House-keeping

Database

House-keeping

Database

MOCDDS

Redundant Data

Capture System

30-DayArchive

OffsiteArchiveOffsiteArchive

OfflineArchive



Stanford

LMSAL


AIA AnalysisSystem

Local Archive

QuicklookViewing

GSFCWhite Sands

World


1.4. Heritage from MDI

As a result of the similarity of the task to our experience with MDI and TRACE we have adopted similar approaches for HMI and AIA, respectively. The basic system can be described as a “pipeline” of processing steps managed by automatically generated scripts with the aid of a relational database to keep track of the locations and status of each dataset.

The current MDI components of data validation, pipeline execution, standard data product generation, parallel virtual machine, data storage management, database server, data catalog, Oracle DBMS, media archive server, quality reporter, job management, SOAP data query URL, and extensive data export methods will be retained with modifications for the HMI demands. As a result, large-scale system prototyping can begin immediately following the initial system engineering studies. Changes do need to be made to data formats, and to remove hardware specific dependencies from some components.

Particular changes will be made to handle the HMI/AIA telemetry stream and required data product generation. A small version of the MDI system has been adapted to serve the high speed data capture for the EGSE during instrument development. A new simulation subsystem was built to generate the telemetry formats and rates expected from HMI and AIA. The final data capture subsystem will be a limited version of the pipeline processing system with only sufficient processors, disk, and tape units to deal with the capture, level-0 processing, and archiving role.

There will be expanded provisions for inserting known data so that data validation at each processing level can be performed.

The keyword data associated with each dataseries will now be kept in database tables independently of the image data. So information can be extracted, or keywords reprocessed without accessing the files containing the image data.

The binary format of the data “inside the pipe” will be designed for efficient storage while the format of exported data will continue to be in community standard data formats.

Some terminology will be changed to ease the conversion to a modified handling of the data collection in a more flexible manner more suited to end user requests while still allowing efficient pipeline processing.

1.5. Data Access Policy

HMI and AIA are committed to an open data policy. As with MDI and TRACE, JSOC data will be available to the science community in each level of reduction as soon as it is available at the JSOC. The current best calibration parameters and software will also be freely available. The HMI and AIA team will coordinate with EVE investigators to determine appropriate formats for data exchange, catalog exchange, and archive locations. We suggest that the JSOC be the NASA designated mission archive for HMI and AIA data for the duration of the mission. At the conclusion of the mission the raw, reduced, and calibrated data will be deposited in an appropriate NASA specified data archive. A number of HMI and AIA data products will be of immediate value for space-weather analysis. These products will be computed from the best available data set in near real time for rapid delivery to users. The particular set will be

7


determined in Phase-D but will certainly include full-disk magnetograms, continuum images, and farside activity images from HMI and selected lower resolution images and image sequences (movies) from AIA.

1.6. Related Documents

The basic requirements for the JSOC-GDS are derived from the HMI and AIA Contracts between NASA and Stanford University and Lockheed-Martin. These requirements originated in the SDO Definition Study published in 2002 and are reflected in the SDO “Level 1” mission requirements. These documents and others listed below are all available via the HMI web site at http://hmi.stanford.edu/doc/index.html

The JSOC-GDS and related topics are described in more detail in:

CDRL SD301: Instrument Science Requirements which is a section of the HMI Science Plan (HMI-S014)

The AIA Instrument Science Requirements

CDRL SD326b: Science Data Analysis, Processing, Archive and Distribution Plan;

CDRL SD326c: Functional Descriptions of Data Products

JSOC IT Security Plan

8

http://hmi.stanford.edu/doc/index.html


2. JSOC-SDP Center Functions

The JSOC-SDP Center consists of several distinct components. Figure 2 shows a top level schematic of the planned JSOC system. The Stanford components are on the left and the LMSAL components are on the right. The connections to the SDO-MOC and SDO-DDS are shown at the top with connections to the science teams and the world on the lower right. The entire JSOC operation must meet high standards of data security. In addition, specific components must meet more stringent data security criteria (delineated by dashed-line enclosures): the LMSAL components must meet NASA data-security controls level IT-2, and the Stanford components must meet level IT-3.

2.1. Basic JSOC Data Handling and Processing Concepts – Common Infrastructure

The data handling and processing environment for the data capture system and the pipeline processing system share a common infrastructure, derived in large part from the MDI processing system (which resides in the SOI Science Support Center (SSSC) at Stanford.)

HMI and AIA data are, in essence, chronological sequences of images, which can be analyzed with techniques appropriate for time series. Over the duration of the baseline mission, these series will grow rapidly, given the large size and cadence of these images. Given the cost of online storage (disk drives), it is not practical to keep entire maturing series online. This situation is not uncommon: this was also the case for SOHO/MDI data in 1995 and, in fact, for WSO data in 1975. We formulated a solution to this problem with the aid of 15 years of hindsight gained from working on WSO. The answer was to assign to each data series a name independent of its physical location (e.g., a directory). This name is descriptive – it reflects the type of data contained in the series. This allowed the data files that compose the series to reside in any arbitrary location, a tape system even. As long as the system keeps track of the location of the data files of a named series, accessing the series is possible. To request data, a user can provide the system a data series name, and the system can return the location of the data files. In fact, the data can be split into files that that reside in multiple locations – some files can reside in one directory, some in another, and some on tape. For JSOC, we have made incremental modifications to the SOI system to increase convenience when storing and retrieving data, to cope with a much larger data volume, to take advantage of the relatively lower cost of online storage, and to support AIA’s needs for “near-line” storage.

This JSOC infrastructure includes the Data Record Management System (DRMS), the Storage Unit Management System (SUMS), the Application Programming Interface (API) that provides access to the HMI/AIA data, and the Pipeline Manager, which controls the interaction between user programs and data within the pipeline environment.

2.1.1. Data Organization and Naming

Each collection of related data items is called a “dataseries”. The origin of this name reflects the fact that most “dataseries” are sequences of data in time. In spite of this slight misnomer, a dataseries can be a collection of any arbitrary data items. The dataseries is an abstract entity that can be visualized as a table with rows and columns of metadata (such as observation time,

9


wavelength, camera number, etc.). Each column contains a series of values for one type of metadata (e.g. observation time). Each row is a data record that contains one metadata value from each of the columns’ series. In the context of a time series, each data record contains the values for all metadata types for a single time step. In addition, there exists a subset of the metadata types, a subset of N columns in this table, such that at most one record has the arbitrary values V1, V2, …, VN for these columns. In other words, if this special subset contains two medata types – the “observation time” metadata and the “camera” metadata, and one were to ask which records have the values “2009.12.05_UT” in the “observation time” column and “1” in the column containing the “camera” metadata, at most one record would satisfy this query. In this way, a requestor can specify any desired set of records by specifying an appropriate query. This special subset of columns is knows as the primary key. In the time-series type of dataseries, this will be some kind of time metadata.

It is important to understand that the dataseries and its columns, rows, and primary key, is an abstract entity. It is implemented as a set of PostgreSQL database objects – a relation, a sequence, several rows in several other relations, etc. It is tempting to view the dataseries as a single table in a database since it has a table structure, but this is not the case. One cannot look inside a database and see a single table that has all the information that composes a dataseries. In particular, database tables contain primary keys, but the primary key of the dataseries is not the primary key of any database table that is part of the implementation of the dataseries.

The types of metadata contained in each data series record fall three classes: keywords (akin to FITS keywords), data-segment references (“pointers” to data files), and links (“pointers” to other data records). Keyword values typically provide information about a data file (e.g., an image file), including attributes like observation time, image quality, etc. Segment values link the data record with one or more data files (typically images). Each data record can be associated with data that are not stored inside any database object. Instead these data are stored in one or more files that may reside online (on hard disk), offline (on a tape in a cabinet), or nearline (on a tape in a drive or tape robot). Since data-segments are “pointers”, the data files for a dataseries can reside anywhere, yet they can be easily retrieved because the pointer is reset every time the data files are relocated. Link values link the data record to one other data record, which makes the second data record’s metadata available in queries involving the first data record.

The off-database data files reside in one or more storage units, which are managed by the Storage Unit Management System (SUMS). Like DRMS, SUMS uses a PostgreSQL database to organize and store the metadata needed for it to perform its functions. Each storage unit contains one or more files from a single dataseries, and it is this storage unit as a whole that is moved from online to offline to nearline storage. Physically, a storage unit is a directory of files and/or subdirectories. SUMS maps each data record to a single storage unit, but more than one data record can map to the same storage unit. Allowing storage units to contain many data records enables efficient tape storage, as large transfers to tape are more efficient than small transfers. The optimal number of data records will depend on the size of record’s data files. Accordingly, a blocking parameter, which controls the number of data records per storage unit, can be specified by the user when the dataseries is created.

10


SUMS manages the allocation of disk space to new storage units and the movement of storage units from online storage to offline storage and back, and it satisfies DRMS requests for the location of storage units. When a data series is created, parameters must be specified which tell SUMS whether or not a dataseries data files is to be archived permanently to tape, how long the files should remain online (after this retention period, they are subject to clean up by SUMS and reused by other storage units), and how to group data records’ data files on tape (data files belonging to disparate data series may be grouped together on tape). Once a user program has written a data file, it becomes the property of SUMS – SUMS will be able to delete and move the file, but the user can never modify or delete that file again. The user hands over ownership of the file by calling a DRMS API function just before the user’s program is about to terminate.

Users obtain access to these storage units by making calls to DRMS API functions. The DRMS library will then call SUMS API functions to request storage units. Users almost always gain access to data files by using the DRMS API, not the SUMS API. To illustrate, a user provides a query to a DRMS API that returns one or more data records. Each of these records can then be used in an additional DRMS API call that returns the directory containing the appropriate storage unit, if it is online. If the storage unit is offline or nearline, then SUMS will allocate online storage space, name a directory, and copy the storage unit into that directory.

The naming system is the query language used when deriving a text query that identifies a subset of data records from various dataseries. This system is a method for naming any arbitrary set of data records, and when this name is provided as an argument to certain DRMS API functions, it is considered a query, and each of these functions returns a record-set. As described previously, this record-set contains pointers to data files accessible via other DRMS API functions. The name can contain any number of comma-separated sub-queries, each of which identifies a subset of records from a single dataseries. The naming system is documented in detail at http://jsoc.stanford.edu/jsocwiki/DrmsNames. In general, a name consists of one or more text strings, each of which is composed of a dataseries name followed by a filter. The filter is either a list of records (data records are numbered), or more commonly, a logic statement that includes the names of the prime-key keywords, or an SQL where clause. An example is

hmi.vw_V_lev18[T_REC=2010.05.12_00:00_TAI/1d]

The series name hmi.vw_V_lev18 has two parts separated by a “.”. The first part, hmi in this example, is the namespace. It is used to organize series into groups that share a number of features, including permissions. The second part, vw_V_lev18, refers to the specific series within the namespace. The parts of the MDI multi-part name scheme are now combined into a single series name. The filter specifies logic involving prime-key keyword values. In this example, the logic translates into “find the range of records whose first record has the T_REC value of May 12, 2010 and include all records that follow this record for one day”.

11

http://jsoc.stanford.edu/jsocwiki/DrmsNames


2.1.2. Metadata

As described above, HMI and AIA images are accompanied by a number of ancillary data quantities, or metadata. The keyword, data-segment, and link metadata are stored in columns in data series. The keyword metadata contain not only values typically found in solar-data FITS-file headers, but they also include information related to the processing of data, such as the processing time, the version of the telemetry-unpacking program, program command-line options. The data-segment metadata include file and data attributes specific to the segment protocol (e.g., FITS, generic, TAS – tiled array storage), such as the data-file protocol, the data-file data type, the dimensionality of the image, and the compression parameters. The link metadata include attributes needed to locate the target record of the target series, such as the name of the target series, the type of link (static vs. dynamic), and the record number or prime-key values of the target record. In addition, certain data-file metadata are stored in SUMS tables, including file timestamps, file sizes, and file online paths. And certain optional metadata having to do with program execution are stored additional DRMS database tables, and in log files.

This scheme is an improvement over that used in MDI where the metadata were bundled with the data files in the dataset (analogous to JSOC’s storage unit). In MDI the metadata were not accessible when the dataset was offline, but in JSOC, the metadata never go offline and they are always available via a data-series query.

2.1.3. Data Catalog

Two primary database catalogs compose the JSOC system: the Storage Unit Catalog and the Data Record Catalog. The former is a list of all storage units and their current working directories, total sizes, parent dataseries, tape locations, storage group numbers, online retention times, etc. The latter comprises several lists, including: 1. a list of all keywords and their data types, print formats, record scopes (constant or variable across data records), parent series, default values, and data-record-specific values; 2. a list of all record links and their parent series, target series, and link types; 3. a list of all data-segments and their parent series, record scopes, data types, and data dimensionalities; 4. a list of all dataseries and their authors, owners, unit sizes (number of records per storage unit), archive flags, default retention times, tapegroup numbers, and primary indices; and 5. a list of all DRMS sessions (a session is a single transaction between a user program and the DRMS database) and the host/port database connections used during the session, their corresponding session log-file paths, and session statuses.

2.1.4. Data formats

The format of the telemetry data varies as the data are processed from telemetry packets into higher- and higher-level data-products. HMI and AIA data arrives from the DDS as a stream of data packets containing compressed image data and produced by onboard processing. Level-0 processing uncompresses the onboard-compressed data to reconstruct the CCD images, then uses Rice compression, a lossless scheme, to store these images in DRMS dataseries. The data of higher-level products are similarly compressed and stored in separate DRMS dataseries. Upon

12


export to local disk, to Lockheed or other closely coupled users, or to general users via our web interface, the data are converted into the requested data format. Currently, data can be exported as FITS files, or “as-is” in the JSOC internal format (does not contain keyword metadata). But we are likely to support VOTables in the near future. We may support other formats, such as CDF or HDF, if a need is identified. External GRID users will probably prefer to use the internal format since they can seamlessly use the pipeline libraries.

2.1.5. Processing Control

The JSOC-SDP contains a Grid Engine queueing system that manages processing-cluster computing resources that are available to user programs. Users wishing to use the pipeline processing cluster write programs and scripts that run these programs. They then submit these scripts to the queueing system. DRMS database access is managed by the database server, and storage-unit allocation and access are managed by SUMS. DRMS modules (programs linking against the DRMS libraries) contain code to manage the DRMS session and database transactions.

A pipeline manager controls the logical flow among the products of a pipeline. For each pipeline a scientist maintains a list of dependencies among these products. The pipeline manager then builds the pipeline products by first building product dependencies, if necessary (e.g., if they do not exist, or if they are obsolete). Then the manager builds the products that require the now built dependencies.

2.1.6. Five Kinds of Users

The JSOC needs to support five kinds of “end users” defined by their location and ease of access to the JSOC computer system. These users are pipeline internal users, local users not using the pipeline, remote co-operating users such as the AIA team at LMSAL, remote investigators using remote systems, and GRID users such as the UK Co-I group. To support these user communities we will implement the user level API in two versions, one for the internal pipeline use where the data is in compressed images in datarecords in dataunits with metadata in the JSOC catalog and outside users where the data is in, e.g., FITS files with the metadata embedded into the data files. The goal is to make the “C” and the “IDL” library interface seen by the user identical in the two environments. This will allow remote users (as well as local users) to develop and test analysis code in their own environment and easily port the programs into the pipeline environment. Figure 3 shows the five user types and service layers provided by the JSOC for each.

NEED TO TALK ABOUT SOLARSOFT HERE

NEED PHIL’S INPUT

NetDRMS users.

13


2.2. Data Capture

The data capture system is a set of four machines, two of them redundant, responsible for acquiring HMI and AIA data from the SDO DDS. One machine, connected to DDS with an OC3 line, is dedicated to HMI data. A separate machine and OC3 line are dedicated to AIA data. Each machine is equipped with a tape drive, on which it generates back-up LTO-4 tapes that are hand delivered, twice a week, to an offsite, secure repository at LMSAL. Each machine has 13 TB of disk storage, which is large enough to cache 19 days of telemetry files. The third machine serves as a warm-standby backup for both of the first two machines. Scripts are in place to enact warm-stanbdy failover to the warm-standby should there be a failure on either of the primary machines. The data capture system also serves as a pass-through for SDO Flight Dynamics data, mission planning data, other data originating at the SDO MOC, and HMI and AIA housekeeping data and operations logs originating at the JSOC-OPS facility at LMSAL.

Processes running on a dedicated cluster node ingest the raw telemetry files into dataseries, hmi.tlm for HMI data and aia.tlm for AIA data. At the same time, the processes reconstruct the original CCD images from the telemetry, saving those data into hmi.lev0 and aia.lev0 dataseries. Upon successful completion of both the back-up of telemetry to tape archive and verification of the tapes, and ingestion of the telemetry into the hmi.tlm and aia.tlm dataseries, the DCS machines send an acknowledgement back to DDS, allowing the latter to free its cache.

Every minute the DDS copies, via scp, four “.tlm” files, one per virtual channel per instrument, to a SOC directory. Each file consists of a stream of compressed VCDUs (after the R/S Check

14

Figure 3. The five kinds of JSOC users. The pink shaded services will be provided as part of the JSOC.


Symbols have been stripped). After each “.tlm” file is written, an associated “.qac” file (quality and accounting) is written containing the name, size and md5 fingerprint of the “.tlm” file. When the JSOC detects the “.qac” file it ingests the “.tlm” file and marks this file as received. Once an hour the DDS sends a “.dsf” file (data status file) containing all the “.tlm” file names it believes it has transferred, but have not yet been acknowledged, in the last hour. In response, the DCS creates an acknowledgement status file (“.asf”) acknowledging all those “.tlm” files it has received and verified. Any file listed in the “.dsf” file that has not been received and verified is marked for retransmission in the “.asf” file. At the end of each day, the DCS sends an “.arc” file listing all the “.tlm” files it has successfully and permanently archived. Receipt by DDS of this file containing the names of “.tlm” files is a prerequisite for deletion of the named “.tlm” files at the DDS.

2.3. Level-0 Data Processing

Level-0 processing converts the raw telemetry data into images that look like they did on the CCD. The internal file format for level-0 data is a Rice-compressed FITS file, with one file per image. The FSN is the prime key of the dataseries that contain the level-0 images. Level-0 storage units contain 17 images, and by default they will remain in online storage for 60 days for HMI and 30 days for AIA. They will also be permanently archived. HMI level-0 images are not used as-is for science analysis. Instead, they must first be processed into level-1 observable data. However, AIA level-0 data will be used for science analysis. Consequently, AIA level-0 data will remain in near-line storage (in a tape robot, never on a shelf) throughout the duration of the mission.

The housekeeping data that is bundled into the high-rate channel will also is stored as level-0 data types. – WHAT IS THIS?

2.4. Level-1 Data Processing

Needs Phil/Rock’s review

Level-1 processing converts the level-0 images into calibrated images of observable physical quantities. Whereas the units of level-0 data are CCD DNs (“digital numbers”), the level-1 data values have units of m/s, gauss, etc. (for HMI). The level-1 data serves as the primary data used in various science analyses, and it is the input for higher-level data-product pipelines. Level-1 data will remain online for 60 days.

I think we’re confusing level-1 with level-1.5.

The goal is to maintain all the level-1 data online for at least three months. Some key HMI level-1 products will be maintained online for the duration of the mission. For AIA the level-1 data will have passed through cosmic ray cleaning and flat fielding, offset removal, and brightness calibrations to standard units. AIA level-1a snapshots of extracted regions, lower resolution movies, etc will be generated at Stanford or LMSAL depending on the particular product. These will be archived at Stanford (thus the 1a designation). Other AIA level-1 products will not be archived but will be recomputed at LMSAL or Stanford as requested. The system will be designed to hold the active data online at the beginning of the mission with the expectation that for the first few years seldom accessed data will be recalled when needed. Some level-1

15


products will be archived offline but some will be recomputed from level-0 if the online version is lost or removed for storage space needs. The decision of offline or nearline archiving of higher level products will be made based on the compute resources needed to regenerate them as compared to the resources needed to store and retrieve them. This tuning need not be done until near launch.

2.5. HMI Higher-Level Processing

HMI and AIA data will undergo standard and automatic higher-level processing to produce higher-level science data-products. These will be the main input for science analysis. The pipeline processing system at the JSOC-SDP center that produces level-0 and level-1 data also produces higher-level HMI products. Higher-level AIA processing, on the other hand, is performed at the AVC at LMSAL. This facility provides interactive processing by means of advanced visualization tools.

Not sure of the usage of level-0, level-1, level-2 here. Normally, level-0 isn’t produced onboard, and the observables are level-1.5. Also, it isn’t entirely clear who is identifying the active regions.

The HMI higher-level processing workflow is shown in Figure 4. The filtergrams are produced onboard, as are the level-0 data products. The second column in Figure 4 contains the physical observables: velocity, magnetograms, and brightness. These are the level-1 products. The “processing” columns represent intermediate “level-2” data products. The column labeled “Data Product” contains the higher-level, “level-3” data-products. The level-1 and level-3 data-products are archived and form the primary data sources for the HMI science community.

Level-2 intermediate data-products are created by sampling, filtering, projections/mapping, transposing, and applying spatial and temporal transformations of level-1 data.

The level-2 data-products include:

a) Global helioseismology Heliographic maps of Doppler velocity Spherical Harmonic Time series Mode frequencies and splitting

b) Local-area helioseismology Tracked tiles of Doppler velocity Local wave frequency shifts derived by ring diagram analysis Wave travel times derived by time-distance analysis Wave phase shift maps derived by helioseismic holography

c) Line-of-sight and vector magnetography Full-disk averaged maps of Stokes parameters Tracked tiles of Stokes parameters and/or vector magnetograms Vector magnetograms

d) Continuum intensity data products Tracked full-disk 1-hour averaged continuum maps Solar limb parameters

16


Brightness feature maps

Level 2 data are not normally archived, but generated on-demand as needed. Documentation of the algorithms, of the actual code, and of the calibration data used to create the level-2 products from lower-level data will accompany the higher-level data products as ancillary information. This will typically include version and configuration information for the relevant pipeline analysis modules, in addition to references to the lower-level data-products needed to recreate the higher-level data-product in question.

Level-3 data represent the results of scientific model analysis, such as application of helioseismic mode fits, mode inversions, magnetic and velocity field reconstruction, and feature identification.

The level-3 data products include:

a) Global helioseismology Internal rotation and large scale flows Internal sound speed

b) Local-area helioseismology Full-disk maps of velocity and sound speed at depths of 0-30Mm High resolution maps of velocity and sound speed near active regions at depths of

0-30Mm Deep focus maps of velocity and sound speed at depths of 0-200Mm Far-side activity index

c) Line-of-sight and vector magnetography Line of sight magnetic flux maps Full-disk vector magnetic field, filling factor, and other thermodynamic

parameters Coronal magnetic field extrapolations Coronal and Solar wind models

d) Continuum intensity data products Brightness images

17


18

Level-0

Level-1

HM

I Dat

a A

naly

sis

Pip

elin

e

Dop

pler

Vel

ocity

Hel

iogr

aphi

cD

oppl

er v

eloc

itym

aps

Trac

ked

Tile

sO

f Dop

pler

gram

s

Sto

kes

I,V

Filte

rgra

ms Con

tinuu

mB

right

ness

Trac

ked

full-

disk

1-ho

ur a

vera

ged

Con

tinuu

m m

aps

Brig

htne

ss fe

atur

em

aps

Sol

ar li

mb

para

met

ers

Sto

kes

I,Q,U

,VFu

ll-di

sk 1

0-m

inA

vera

ged

map

s

Trac

ked

Tile

s

Line

-of-s

ight

Mag

neto

gram

s

Vec

tor M

agne

togr

ams

Fast

alg

orith

m

Vec

tor M

agne

togr

ams

Inve

rsio

n al

gorit

hm

Egr

essi

on a

ndIn

gres

sion

map

s

Tim

e-di

stan

ceC

ross

-cov

aria

nce

func

tion

Rin

g di

agra

ms

Wav

e ph

ase

shift

map

s

Wav

e tra

vel t

imes

Loca

l wav

efre

quen

cy s

hifts

Sph

eric

alH

arm

onic

Tim

e se

ries

To l=

1000

Mod

e fre

quen

cies

And

spl

ittin

g

Brig

htne

ss Im

ages

Line

-of-S

ight

Mag

netic

Fie

ld M

aps

Cor

onal

mag

netic

Fiel

d E

xtra

pola

tions

Cor

onal

and

Sol

ar w

ind

mod

els

Far-s

ide

activ

ity in

dex

Dee

p-fo

cus

v an

d c s

map

s (0

-200

Mm

)

Hig

h-re

solu

tion

v an

d c s

map

s (0

-30M

m)

Car

ringt

on s

ynop

tic v

and

c sm

aps

(0-3

0Mm

)

Full-

disk

vel

ocity

, v(r

,Θ,Φ

),A

nd s

ound

spe

ed, c

s(r,Θ

,Φ),

Map

s (0

-30M

m)

Inte

rnal

sou

nd s

peed

,c s

(r,Θ

) (0<

r<R

)

Inte

rnal

rota

tion

Ω(r

,Θ)

(0<r

<R)

Vec

tor M

agne

ticFi

eld

Map

s

HM

I Dat

aD

ata

Prod

uct

Proc

essi

ng

Level-0

Level-1

Level-0

Level-1

HM

I Dat

a A

naly

sis

Pip

elin

e

Dop

pler

Vel

ocity

Hel

iogr

aphi

cD

oppl

er v

eloc

itym

aps

Trac

ked

Tile

sO

f Dop

pler

gram

s

Sto

kes

I,V

Filte

rgra

ms Con

tinuu

mB

right

ness

Trac

ked

full-

disk

1-ho

ur a

vera

ged

Con

tinuu

m m

aps

Brig

htne

ss fe

atur

em

aps

Sol

ar li

mb

para

met

ers

Sto

kes

I,Q,U

,VFu

ll-di

sk 1

0-m

inA

vera

ged

map

s

Trac

ked

Tile

s

Line

-of-s

ight

Mag

neto

gram

s

Vec

tor M

agne

togr

ams

Fast

alg

orith

m

Vec

tor M

agne

togr

ams

Inve

rsio

n al

gorit

hm

Egr

essi

on a

ndIn

gres

sion

map

s

Tim

e-di

stan

ceC

ross

-cov

aria

nce

func

tion

Rin

g di

agra

ms

Wav

e ph

ase

shift

map

s

Wav

e tra

vel t

imes

Loca

l wav

efre

quen

cy s

hifts

Sph

eric

alH

arm

onic

Tim

e se

ries

To l=

1000

Mod

e fre

quen

cies

And

spl

ittin

g

Brig

htne

ss Im

ages

Line

-of-S

ight

Mag

netic

Fie

ld M

aps

Cor

onal

mag

netic

Fiel

d E

xtra

pola

tions

Cor

onal

and

Sol

ar w

ind

mod

els

Far-s

ide

activ

ity in

dex

Dee

p-fo

cus

v an

d c s

map

s (0

-200

Mm

)

Hig

h-re

solu

tion

v an

d c s

map

s (0

-30M

m)

Car

ringt

on s

ynop

tic v

and

c sm

aps

(0-3

0Mm

)

Full-

disk

vel

ocity

, v(r

,Θ,Φ

),A

nd s

ound

spe

ed, c

s(r,Θ

,Φ),

Map

s (0

-30M

m)

Inte

rnal

sou

nd s

peed

,c s

(r,Θ

) (0<

r<R

)

Inte

rnal

rota

tion

Ω(r

,Θ)

(0<r

<R)

Vec

tor M

agne

ticFi

eld

Map

s

HM

I Dat

aD

ata

Prod

uct

Proc

essi

ng

HM

I Dat

a A

naly

sis

Pip

elin

e

Dop

pler

Vel

ocity

Hel

iogr

aphi

cD

oppl

er v

eloc

itym

aps

Trac

ked

Tile

sO

f Dop

pler

gram

s

Sto

kes

I,V

Filte

rgra

ms Con

tinuu

mB

right

ness

Trac

ked

full-

disk

1-ho

ur a

vera

ged

Con

tinuu

m m

aps

Brig

htne

ss fe

atur

em

aps

Sol

ar li

mb

para

met

ers

Sto

kes

I,Q,U

,VFu

ll-di

sk 1

0-m

inA

vera

ged

map

s

Trac

ked

Tile

s

Line

-of-s

ight

Mag

neto

gram

s

Vec

tor M

agne

togr

ams

Fast

alg

orith

m

Vec

tor M

agne

togr

ams

Inve

rsio

n al

gorit

hm

Egr

essi

on a

ndIn

gres

sion

map

s

Tim

e-di

stan

ceC

ross

-cov

aria

nce

func

tion

Rin

g di

agra

ms

Wav

e ph

ase

shift

map

s

Wav

e tra

vel t

imes

Loca

l wav

efre

quen

cy s

hifts

Sph

eric

alH

arm

onic

Tim

e se

ries

To l=

1000

Mod

e fre

quen

cies

And

spl

ittin

g

Brig

htne

ss Im

ages

Line

-of-S

ight

Mag

netic

Fie

ld M

aps

Cor

onal

mag

netic

Fiel

d E

xtra

pola

tions

Cor

onal

and

Sol

ar w

ind

mod

els

Far-s

ide

activ

ity in

dex

Dee

p-fo

cus

v an

d c s

map

s (0

-200

Mm

)

Hig

h-re

solu

tion

v an

d c s

map

s (0

-30M

m)

Car

ringt

on s

ynop

tic v

and

c sm

aps

(0-3

0Mm

)

Full-

disk

vel

ocity

, v(r

,Θ,Φ

),A

nd s

ound

spe

ed, c

s(r,Θ

,Φ),

Map

s (0

-30M

m)

Inte

rnal

sou

nd s

peed

,c s

(r,Θ

) (0<

r<R

)

Inte

rnal

rota

tion

Ω(r

,Θ)

(0<r

<R)

Vec

tor M

agne

ticFi

eld

Map

s

Brig

htne

ss Im

ages

Line

-of-S

ight

Mag

netic

Fie

ld M

aps

Cor

onal

mag

netic

Fiel

d E

xtra

pola

tions

Cor

onal

and

Sol

ar w

ind

mod

els

Far-s

ide

activ

ity in

dex

Dee

p-fo

cus

v an

d c s

map

s (0

-200

Mm

)

Hig

h-re

solu

tion

v an

d c s

map

s (0

-30M

m)

Car

ringt

on s

ynop

tic v

and

c sm

aps

(0-3

0Mm

)

Full-

disk

vel

ocity

, v(r

,Θ,Φ

),A

nd s

ound

spe

ed, c

s(r,Θ

,Φ),

Map

s (0

-30M

m)

Inte

rnal

sou

nd s

peed

,c s

(r,Θ

) (0<

r<R

)

Inte

rnal

rota

tion

Ω(r

,Θ)

(0<r

<R)

Vec

tor M

agne

ticFi

eld

Map

s

HM

I Dat

aD

ata

Prod

uct

Proc

essi

ng

Figure 4. HMI pipeline processing


19

Figure 5. AIA higher level processing.


2.6. AIA Higher Level Processing

Need Rock’s/Phil input. Also Figure 5 needs to be replaced by a newer figure.

The AIA higher level processing is described in the AIA science plan. An overview of that plan is shown here in Figure 5 to show the links to the JSOC-SDP-Center part of the JSOC.

LEVEL Description Examples RATE[GB/day]

RATE[TB/year]

Cache[day]

%Archive

Raw Telemetry 723 264 30 1000 Primary

DataLoss-less 1,100 400 30 100

1a Organized Sets, Current Calibration

Movies,Extracted Regions

55 20 100 (mission)

100

1 Best calibrations

On demand 1,100 400 100 10

2 Higher Level Products

DEM, Movies

28 10 1900 (mission)

100

An important set of AIA science goals rely upon special processing capabilities present at the LMSAL AIA Visualization Center (AVC). This processing requires receipt from the JSOC-SDP Center all level-0 and level-1 AIA data, and all HMI magnetogram observables. At the AVC, processing creates low-resolution images suitable for users who wish to examine the evolution of solar features and their correlation with magnetic-field regions. The images are combined into movie sequences. These data are of prime interest to modelers of space-weather forecasts. The AVC processing also extracts data regions from these images to create “data cubes” that track the passage of each NOAA active region across the sun’s disc. To perform correlational studies, scientists will examine the equivalent regions from magnetogram images. The AVC manages an “event database” that catalogs solar feature and events of interest, including the coordinates of NOAA active regions identified at the JSOC-SDP Center. This catalog is available from both the AVC and JSOC-SDP Center for export to the online science community at large.

2.7. Archives

The phrase “JSOC Archive” refers to storage on magnetic tapes. This includes nearline storage (tapes in a tape robot) and offline storage (tapes in a cabinet, but not in a robot). Access of offline data requires manual movement of a tape from a cabinet to the tape robot. The result is the movement of other data from nearline storage to offline storage so that the tape robot can accommodate the previously offline tape. Online data (on magnetic hard disks) have a retention time, specified as a number of days. After this retention time has elapsed, the storage holding the data is subject to reclamation by SUMS. The timing of reclamation following the expiration of a

20


retention time is not known – SUMS will reclaim the space only if space is needed for other writes. Data being written to disk may optionally be written to tape. If archiving is specified for data, then as they are written to disk, these to-be-archived data are also written to tape (or they are written shortly after). These tapes remain nearline, unless they are supplanted by the placement offline-storage tapes into the tape robot. Certain tapes are not subject to movement from nearline to offline storage, as specified by the data’s tape group number.

Level-0 HMI and AIA data and level-1 HMI data will remain online (on magnetic disk) for 60 days, and they will be archived. After the 60-day retention period, they are subject to deletion from online storage. HMI data will migrate from nearline to offline storage in response to low usage patterns, but AIA will remain nearline for the duration of the mission.

In addition to the JSOC archive, there is an offsite archive as discussed in section XX. It is the repository for tapes containing the raw telemetry files. Twice a week, these tapes will be hand-delivered from Stanford to LMSAL. If the JSOC-SDP telemetry series need to be restored from these offsite tapes, copies of the offsite tapes, not the original tapes, will be transferred back to Stanford, and ingested into the appropriate data series.

2.8. Export System

CoSEC?

There are four ways that remote users may export data for their use: 1. Users with direct access to a NetDRMS site can use HMI and AIA data replicated at that site; 2. The JSOC-SDP hosts two websites that allow users to query and receive data; 3. Users with IDL and Solarsoft installed can install a JSOC-SDP interface that directly accesses DRMS data and SUMS data files. There is a similar interface available for Matlab users; 4. The Virtual Solar Observtory (VSO) hosts a website that provides a common interface allowing users to select and retrieve data, not just for HMI and AIA, but for many types of solar data.

With the first method, users would access data in much the same way as JSOC-SDP staff would access the data, except that they access the data from a site that is not the JSOC-SDP Center. Institutions can become “NetDRMS” sites, which means that an institution installs DRMS and SUMS code and PostgreSQL client and server code, and it instantiates DRMS and SUMS databases, with the appropriate set of database objects (like tables and functions). It creates various user accounts, both on its Linux machines, and in its database instances. Then it creates private-public encryption keys (used for data transfer), and it sets up and runs a SUMS server. Once the institution has completed this list of tasks, it can then initiate transfer of DRMS data from Stanford (or any other DRMS site acting as a NetDRMS exporter). It must subscribe to the dataseries of interest by running a program that communicates with Stanford, and it must run another program that fetches this data, and ingests it into its DRMS database.

After these steps have been completed, the NetDRMS site then has access to all the metadata of the subscribed-to dataseries. The site can then decide if it wants to pre-fetch chunks of storage units (to which the file metadata “point”) from Stanford. Or it may decide to simply fetch the needed data files on-demand, as the data files are requested by users. The latter happens automatically, provided all the previous steps have been successfully completed. Once all the

21


metadata (stored in the DRMS database) and the data files have been transferred to the site, then the site is autonomous, with respect to the dataseries to which the site is subscribed. Users at the NetDRMS site can access those data from their NetDRMS site in a way that is completely identical to the way Stanford users access data from the JSOC-SDP Center.

The second method (the “webexport” method) entails two web sites for obtaining HMI and AIA data. The first site lists a set of roughly a dozen commonly desired data-products, and provides UI elements that allow for the download of one or more data files. The UI is intentionally very basic, allowing access by users who are not familiar or interested in the details of understanding the DRMS dataseries naming scheme. The second site has a more intricate UI. It allows the user much finer control over the selection of data, and it allows access to all public DRMS dataseries, not just the dozen data products available in the first site. Both these sites make use of cgi programs hosted at the JSOC-SDP Center. Requested data that is online will often be available immediately, although it may take up to 24 hours to satisfy requests for nearline and, particularly, offline data. Since most level-1 HMI data will remain online during the period of its most intense use (shortly after it is published), and all AIA level-0 data will be online or nearline, completion of requests for these data will timely.

The third method is geared towards users of Solarsoft and Matlab. Interface libraries have been created for these users who want to access, directly from the JSOC-SDP Center, HMI and AIA data. These interfaces make use of the same cgi programs that the second method employs. They also present the data in ways to which Solarsoft and Matlab users are accustomed. For example, Solarsoft users specify data in much the same way as a JSOC-SDP user would – they would provide a query involving a dataseries name and a filter. However, the Solarsoft interace would return one or more file URLS in response to this query, and the Solarsoft user would then read that data by providing those URLS to Solarsoft and other IDL routines. These interfaces also provide the ability for users to store data back into JSOC-SDP dataseries.

The fourth method, download from the VSO website, is for users who take advantage of the unified searching capabilities of VSO. The JSOC-SDP Center is a “VSO provider”, and VSO communicates with the JSOC-SDP server via cgi programs similar to the ones used for the second and third method above. The JSOC-SDP Center provides a more specialized and streamlined interface to VSO than it does for the other two methods, as performance is more important than than the flexibility and generality needed by the other two methods.

2.9. Integration & Test

Rock/Phil please review this

With the provision of injecting known telemetry data into the JSOC front end, a complete test of data flow and integrity can be performed for each stage of the processing. Standard regression test suites and validation procedures will be developed to verify processing at each stage through all the system additions and revisions. Throughput and load balancing tuning will be performed. All development source code is under a Configuration Management (CM) system. The CM is based on CVS. CVS provides configuration management functionality as well as the means for multiple users to work concurrently on a common source tree with minimal conflict.

22


3. JSOC Data Volumes

Need to update Figure 6 and Table 1

The data-flow estimates through the JSOC, shown in Figure 6, are based on the assumptions listed in Table 1. The raw telemetry data are highly compressed onboard SDO and are stored as-is in the hmi.tlm and aia.tlm dataseries. The level-0 and level-1 data are also stored in compressed form (with Rice compression, where the compression factor is roughly two), but the level of compression of the higher-level data products varies from component to component.

4. JSOC Hardware Configuration

4.1. JSOC-SDP Data Center

The data center is located in the climate-controlled basement of the Physics and Astrophysics building at Stanford University. Important systems and components are configured with Uninterruptable Power Supply (UPS) devices, and some devices will auto-switch over to building back-up power in the event of a loss of regular power. Critical components are monitored for failure in two ways. Many systems are outfitted with monitoring software that generates email notifications if certain events occur. Also, a webpage exists that lists the status of each component.

23

Figure 6. HMI-AIA SOC data volume. Estimates of data flow in Gigabytes/day. See Table 1 for assumptions.

Dataflow (GB/day)Joint Ops

ScienceArchive440TB/yr(Offiste)

Data Capture2 processors each

1230

1610

HMI &AIA Science

Hk

0.04

30d cache40TB each

Quick Look

LMSAL secure host

Level 0(HMI & AIA)2 processors

75

Level 1(HMI)

16 processors

Online Data325TB+50TB/yr

HMI High LevelProcessingc. 200 processors

HMI Science Analysis Archive 650TB/yr

Redundant data capture system

1210

1210

Data Exports

1200

LMSAL Link(AIA Level 0, HMI Magnetograms)240

1610

1820

1230

rarelyneeded

1230

2 processorsSDO Scientist &User Interface

Dataflow (GB/day)Joint Ops

ScienceArchive440TB/yr(Offiste)

Data Capture2 processors each

1230

1610

HMI &AIA Science

Hk

0.04

30d cache40TB each

Quick Look

LMSAL secure host

Level 0(HMI & AIA)2 processors

75

Level 1(HMI)

16 processors

Online Data325TB+50TB/yr

HMI High LevelProcessingc. 200 processors

HMI Science Analysis Archive 650TB/yr

Redundant data capture system

1210

1210

Data Exports

1200

LMSAL Link(AIA Level 0, HMI Magnetograms)240

1610

1820

1230

rarelyneeded

1230

2 processorsSDO Scientist &User Interface


4.2. Capture system

The data capture system is a set of four machines, two of them redundant, responsible for acquiring HMI and AIA data from the SDO DDS. One machine, connected to DDS with an OC3 line, is dedicated to HMI data. A separate machine and OC3 line are dedicated to AIA data. Ethernet (gigabit) connections exist between the DCS and the rest of the JSOC-SDP Center. Each machine is equipped with a tape robot, with which it generates back-up LTO-4 tapes. Each tape robot, which has 30 slots and one drive, is capable of storing 24 TB of data (a little over 30 days of data). Each machine has 13 TB of disk storage, which is large enough to cache 19 days of telemetry files. The third machine serves as a warm-standby backup for both of the first two machines, and the fourth machine, another complete backup, resides at the MOC where it will be used only in the catastrophic event that the other three machines stop functioning for at least 20 continuous days. Each system has a UPS device to allow for clean shut down, or switching to building backup power. The hard disks have been arranged into a RAID configuration, and they have redundant controllers.

HMI: 55,000,000 bps 553 30 16AIA: 67,000,000 bps 674 30 20HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 530 30 16AIA: 4k*4k*2 bytes * 8 imgs per 10 1080 30 32HMI: V,M,Ic @ 45s & B, ld, ff @ 130 46AIA: same as level-0 1080 100 105HMI: See HMI-S015 and Figure 5 20 0 7AIA (lev1a): movies & extracted regions. 54 0 19HMI: Magnetograms (M, B) 59 100 6AIA: Full Level-0 data+lev1_extract 1134 100 111HMI: 2 * Higher Level products + 5*10 49 60 3AIA: 3* higher Level products (TRACE 162 60 9HMI: tlm 553 197AIA: tlm 674 30 20 240HMI: Lev0, Lev-1, All Higher 680 100 66 242AIA: Lev0, Lev1a 1134 365 404 404HMI Disk Totals 40 53AIA Disk Totals 277 19HMI Tape Totals 439AIA Tape Totals 644Combined Disk (TB) 317 73Combined Tape per year (TB) 1084

Totals

Offsite tape

1227

Offline tape

1814

LMSAL Link

1193

Export 211

Level-1 1210

Higher level

74

In from DDS

1227

Level-0 1610

Table 1. Data Flow AssumptionsData Path

Assumptions Volume (GB/day)

Combined

Online cache

Cache size (TB)

Perm TB/yr

24


4.3. Pipeline Processing system

The majority of HMI and AIA data processing will be performed on a 512-CPU computing cluster (64 x86-64 Intel nodes, each with dual processors, each with four CPU cores). We deemed this cluster size to be appropriate by analyzing model calculations performed during Phase-A of the mission. The model considers the CPU utilization by MDI and the expected 50-fold increase in CPU utilization required for HMI. Given this model, we expect to require the following numbers of high-end 64-bit contemporaneous processors: 2 processors for data capture, 2 processors for level-0 processing, 16 processors for level-1 HMI processing, and approximately 190 processors for higher-level HMI processing.

We have taken a number of steps to ensure reliability of the cluster and its components. All machines have been tested fully for performance and reliability with simulated data at realistic data rates and volumes (the cluster predates the existence of true data). And all components are under vendor service warranties. Not only is redundancy is a byproduct of having a large number of nodes, but onsite spare drives and components exist. Finally, all disk systems are configured as RAID-6 disk-arrays, protected with checksums.

4.4. Archive (online, offline, nearline, shelf, offsite)

Are these correct? Phil/Rock please fix.

Disks: For HMI we plan to keep a 30-day cache of raw telemetry (20 TB), a 90-day cache of filtergrams (90 TB), and observables for the life of the mission (50 TB/yr) on RAID arrays. For a 5-year mission the required total disk capacity for these components is 400 TB. In addition,

I’m not understanding this - why does the above say we need 400TB for 5 years, but below it says we need to start with 450TB and grow to 550TB (also doesn’t say when we’ll need 550 TB either).

Online data storage will need to contain 60 days of level-0, 60 days of level-1, and all higher level products. If this model does not change we must start with 450TB of disk and grow to 550TB. We desire to maintain the Level-1 data online for the full mission. This would increase the initial configuration to 900TB and require an additional 450 TB per year. It is unlikely that this volume of disk can be obtained within our cost limits at the start of the mission but may well be possible after a few years if disk capacities and prices continue present trends.

Given these numbers, we estimate that the cost of the hard disks will be $320K for the first 5 years.

The file server currently has 450TB of hard-disk (online) storage. The plan is to augment this amount by 150 TB per year by adding new hard disks.

Tapes: The data center is currently using high-density, 800 GB capacity, LTO-4 cartridge tapes for nearline and offline storage. As the mission progresses, we plan to upgrade to larger-capacity tapes as needed.

25


The next section doesn’t make sense to me – it makes it sound like we are discarding any telemetry older than 90-days, and any filtergrams older than 270 days. Phil/Rock please fix.

Tape Libraries: For HMI we plan to keep a 90-day cache of raw telemetry (60 TB), a 270-day cache of filtergrams (270 TB), and enough blank tapes for 30 days' archiving needs (60 TB) near-line. This makes the nearline storage requirement about the same as the online disk capacity. For AIA the tape library requirement is much larger. The need will start at 300TB for a 9-month capacity and grow by XXX TB per year.

The data center has a 2200-slot, 12-drive, LTO-4 tape library. Full tapes will be placed in a cabinet as needed to make room for the placement of new tapes into the library. Access of data from a cabinet-bound tape (offline storage) requires manual replacement of a tape currently in the library with the desired tape from the cabinet. If deemed desirable, expansion plans include the purchase of one for more additional 1300-slot extensions that can be attached to the existing library. Each extension increases the nearline storage by 1300 tapes.

4.5. Connectivity (web, grid, DDS, LMSAL, MOC, etc)

Need a new diagram

26

Stanford

DDS

Firewall

Firewall

Router

Sci W/S

Firewall

World

CMD

MOC

Display

LMSAL

JSOC Disk array

Router

LMSAL W/S

Firewall

Front End Pipeline

1 GbPrivate line

Router

JSOC Disk array

Router

Firewall

Router

NASAAMES

“White”Net

Pipeline

Stanford

DDS

Firewall

Firewall

Router

Sci W/S

Firewall

World

CMD

MOC

Display

LMSAL

JSOC Disk array

Router

LMSAL W/S

Firewall

Front End Pipeline

1 GbPrivate line

Router

JSOC Disk arrayJSOC Disk array

Router

Firewall

Router

NASAAMES

“White”Net

Pipeline

Figure 7. Network connectivity of the JSOC. The connection to the DDS is 2 OC-3 lines, one for HMI and one for AIA. The LMSAL-MOC connections are dedicated secure lines.


2 OC-3 to DDS, gigabit to LMSAL, XXX to GRID, …

27


5. Development Plan

As of late 2009, much of the JSOC-SDP Center’s JSOC system is complete. The hardware is in place and is fully functional – only upgrades of older equipment are occurring as needed. Software development of infrastructure features is also mostly complete. Development activity centers on identifying and fixing minor defects, although the data-replication to NetDRMS sites is in the final stages of implementation (but a final plan exists). Integration with VSO is also ongoing, with completion expected by the end of the year. The software that processes telemetry into level-0 and level-1 observables is also virtually complete – final verification is awaiting spacecraft commissioning. Data export and distribution code is functional, albeit rudimentary.

Development of higher-level processing software is the current focus of much of the JSOC-SDP effort. A few data-processing pipelines are now complete (pending final verification during commissioning), most will be complete by the end of the year, and a couple will not be complete until sometime during the first half of 2010.

DO WE NEED DETAILS ABOUT THE development efforts for each of the pipelines.

Throughout Phase-E, we plan to upgrade aging hardware components (as described above). And we plan to add new disks, RAM, tapes, tape-cabinets, etc. as needed (and as described above). We also plan to enhance distribution (export) tools to meet the growing needs of the scientific community at large.

During Phase-E First two years of mission continue Co-I pipeline testing support FIGURE 8 below is not referenced anywhere, and it needs to be updated or jettisoned.

28


5.1. Data EGSE

Phil/Rock – not sure what an EGSE is. – do we need this here

The needs of the system for the high-speed bus component of the EGSE used for instrument development and testing is nearly the same as that needed for the JSOC capture system. The data EGSE system has been completed in its first simple version and now functions with data delivered from the SDO spacecraft simulator. The first version has 2 processors and XXX GB on disk. The second version, HMI-2 and AIA-2, will have the same software system but will have additional disk and a tape subsystem so that test data may be imported into the JSOC Pipeline system prototype for analysis.

29

Figure 8. JSOC development schedule.

Science Data Plan including SOC Plan: - Stanford...

Documents

Transcript of Science Data Plan including SOC Plan: - Stanford...