SOFIA DCS History and Overview Ian Gatley. SOFIA March 7-8 2000DCS Preliminary Design Review2 The...

SOFIA

DCS History and Overview

Ian Gatley

March 7-8 2000 DCS Preliminary Design Review

2

SOFIA The South Pole CARA Project:

A DCS demonstration• A data cycle system was produced by the RIT consortium

for the CARA project (Harper, PI) at the South Pole. • The South Pole DCS has the following history:

– The South Pole IR telescope operated for 5 years as a PI project.

– NSF indicated that the project needed to make telescope time with their camera publicly available.

– NOAO agreed to run the proposal selection.– NOAO selected 21 proposals, 13 of which were actually

implemented.– Gatley, a member of the CARA consortium, and his

colleagues at RIT developed and implemented the data cycle pipeline for this experiment.


3

SOFIA

South Pole DCS: Implementation

• Both raw and calibrated data were provided to the PIs.• Data were archived so they could be used for further

global processing– For example, Kastner kept track of L-band photometry in

order to assess the site's performance at this band.

• Some 6 - 8 months of observations (IR imaging) took 1 month of processing at RIT.

• The process was not optimized, and it was written only for the CARA mode of observing.

• Rhody and Gatley spearheaded the project.• Kastner (who planned to use the data, and who came to RIT as a

faculty member in July 1999) improved and refined the process by interacting with Rhody, and by actually using the pipeline.


4

SOFIA

South Pole DCS: Philosophy

• Commercial software tools were used whenever possible.• Software written by the consortium defines the

interaction between tools and/or packages and the data.• In addition, standard packages employed in other

astronomy pipelines were used.• Examples of commercial software include:

– XML (extensible markup language); – CORBA (Common Object Resource Broker Architecture); – IDL Lib_Astro; – JAVA; – SQL (a database query language)


5

SOFIA

South Pole DCS: Portability and Scalability

• Because this was a straightforward application, Harvey Rhody had all of the DCS for CARA on his computer at RIT.

• The system is portable, and could be configured to run on several resident computers with balanced loads or on separated computers over the internet.

• This experiment represents a proof of concept:– the RIT consortium has designed and executed a DCS.


6

SOFIA

SOFIA DCS History

• HAWC presented their pipeline designs to SOFIA because:– Gatley and Harper (the South Pole DCS principals) were

also principals in the HAWC experiment,– Harper (PI HAWC) proposed an end-to-end design, – Gatley and Harper had already teamed to successfully

implement the data pipeline for the CARA project.

• Becklin was interested in a common pipeline approach for all the SOFIA instruments

• Since RIT already had demonstrated competence, they were asked to describe plans for a DCS to SOFIA


7

SOFIA

Slides from Becklin presentation Nov 98


13

SOFIA

Conceptual Design Review(June 1999)


14

SOFIA

The CoDR material is on the web

• http://sofia-usra.arc.nasa.gov/internal/science/DCS/dcs_cdr.html

• Username = science

• Password = sofia


15

SOFIA

The DCS MissionThe DCS shall provide the infrastructure to support the operation of facility instruments on SOFIA using best current knowledge and tools, and supporting continuous improvement in an efficient, extensible, and modular architecture.

http://sofia-usra.arc.nasa.gov/internal/science/DCS/dcs_cdr.html


16

SOFIA

Results of Conceptual Design Review


17

SOFIA

Need for DCSThe CoDR committee finds that a DCS is necessary for making the facility instruments on SOFIA accessible to the largest possible astronomy community.

Therefore, the Director should consider the development of the DCS of equivalent importance to the development of the FSIs.


18

SOFIA

Recommendations of CoDR Committee

1. The USRA should commission the development of a DCS, and the first priority for the DCS is the recording and archiving of all data necessary for complete analysis, calibration and future access to observational data obtained with the FSIs.

2. In order for the FSIs to be readily accessible to the community of US astronomers, the second priority for the DCS is to establish a select number of observing modes supported by pipeline reduction, including quicklook analysis, for each of the FSIs.


19

SOFIA

Recommendations of CoDR Committee

3. The Director should enable a close working relationship between the DCS team(s) and the FSI teams.

4. The DCS should become the vehicle for providing on-line documentation and information necessary for the GI to propose and carry out observations with the FSIs.

5. The DCS should be implemented as a system of distributed modules, integrated by an extensible hardware and software infrastructure, as presented at the DCS Conceptual Review.


20

SOFIA

Top Level Requirements

• The recommendations of the CoDR committee ARE the top level requirements for the DCS


21

SOFIA

Top level requirements (1)• The DCS shall record and archive

all data necessary for complete analysis, calibration and future access to observational data obtained with the FSIs.

• The DCS shall support a select number of observing modes with pipeline reduction, including quicklook analysis, for each of the FSIs.


22

SOFIA

Top level requirements (2)

• The DCS shall provide on-line documentation necessary for the GI to propose and carry out observations with the FSIs.

• The DCS shall be implemented as a system of distributed modules, integrated by an extensible hardware and software infrastructure.


23

SOFIA

The DCS in plain English

• The program architecture planned consists of "dumb" modules (e.g. the parser) that need descriptions of what to do, and where and how to find the appropriate tools.

• Scripts are written which are particular to the data-taking mode for the instrument, and which utilize these tools on the data.

• All of these components are also modular. • The scripts can be written by RIT (or the PI).


24

SOFIA

The CARA example• Experiments “scripted” for execution • Observations made by the operator (Charlie Kaminsky).

– Standard modes of operation, common standards

• Data sent by satellite to White Sands and then by Internet.• Dark-frame subtraction, linearization, flat-fielding, co-addition,

background-subtraction etc., performed as required, using IDL. – Saved the method used, the raw data, and the final product

• intermediate products not generally retained;• intermediate products from some point in the path can be retained if

desirable

• Images calibrated.• Data written in the desired format*.

* There would be a stage to put the product into IPAC archival form for SOFIA

• Data archived.• Data products disseminated.

Overview of the PDR• The "Core Development" DCS has been provided priorities by the

June '99 Conceptual Design Review.• This core development must have the capability to capture the

data, reduce it, and provide documentation, all within a modular infrastructure utilizing extensible software/hardware tools.

• The DCS consortium will present their view of the required large components, and the software tools and programs needed to get the components to work together efficiently in the pipeline.

• The DCS consortium will also present the collected requirements for support of extensibility, including (presently unfunded, but envisaged):

• expanded scope for facility instrument support;• additional participation (universities, ARC, GSFC, and IPAC);• tools for USRA FSI flight planning; • tools for USRA to manage the request for general investigator telescope

time;• tools for visualizing the use of SOFIA facility instrumentation; • unified approach to USRA control of the facility instruments;• Support of PI instruments.


26

SOFIA Interface Management and Continuous Improvement

• With Facility Science Instruments• With MCCS • With Archives (IPAC, etc)• With General Investigators

• Supporting change and continuous improvement– The strength of SOFIA and the fundamental

difference between this and the other Great Observatories.


27

SOFIA

Summary

• Proof of concept exists and this team has track record. • But the scope of the CARA DCS was very much less than that of

the SOFIA DCS.

• The initial design of the SOFIA DCS is focussed on one mode per FSI instrument, and deals solely with data collection, reduction, and archiving.

• The core DCS will be delivered to the SSMOC, and the next stages of the USRA program will build upon that deliverable.

• An augmentation proposal (Becklin PI) is being submitted so that the SSMOC can support DCS expanded scope.

SOFIA DCS History and Overview Ian Gatley. SOFIA March 7-8 2000DCS Preliminary Design Review2 The...

Documents

Transcript of SOFIA DCS History and Overview Ian Gatley. SOFIA March 7-8 2000DCS Preliminary Design Review2 The...