1

STScI PipelineThermal-vacuum data handling

I. Neill Reid for WFC3 Instrument CALibration team

ICAL

2

Introduction

The WFC3 thermal-vacuum campaign will incorporate several innovations, including:

• Fully-planned science calibration campaign

• All images obtained during T/V will be archived in the HDA

- science: calibration & performance verification

- technical: optical alignment

• Data characterising the configuration of the illuminating optical stimulus (OS) will be incorporated automatically in image headers

3

Background

• Thermal-vacuum tests scheduled for Feb-April, 2003 at GSFC

• Science calibration mid-April 2003 (17 days)

• Expect up to 10,000 images; ~2,000 science

• Data-handling system design has been considered by the WFC3 ICAL

Full system concept has been developed and impact on pipeline assessed

Details of other sub-systems still under consideration

4

Functional Requirements for T/V data system

• Data acquisition - Acquire SDI and SHP files from SITS and OS snapshots from the Stimulus; merge and convert into a uniquely-named FITS file.

• Data logging - Automatically log all calibration activities -records what SMSs are being run and what pictures are being generated

• OS monitoring - Store all OS configuration and telemetry data in ancillary files.• Quick-look and real-time access to local database - Images taken during T/V shall be

available for immediate display and real-time science assessment on site. • Data archiving – Data will be transferred to STScI for permanent storage and retrieval

in the HDA.• Exposure traceability - FITS files will be traceable back to the SMS scripts. • Archive retrieval - DADS will allow use of keywords to retrieve images from HDA• Automation - Provide automation to minimise data-handling errors and optimise use of

manpower.• Reuse - Legacy software will be used to minimize development costs• Compatibility – For efficiency, the same software should be compatible with both

UVIS and IR channels

5

System Concepts

Two main architectures considered, both utilising heritage software:

• ACS-like system– Local operating system and database designed specifically for T/V – Flexible, rapid access to data via custom-built cataloguing facilities– Limited capabilities for easily-accessible long-term archive

• OPUS-like system– STScI-based system linked to Hubble Data Archive– Designed for pre-specified programmatic data acquisition– Well-defined, easily-accessible long-term archive

Proposed system combines strengths of alternatives in a hybrid system

6

Recommended Data Handling System

CalibrationPlan TestProcedure

Quick LookReal TimeImage Displayand Analysis

Data LoggingAcquisitionSoftware

OpticalStimulus

SITS ExcelSpreadsheetSMS Tool

Real TimeManual CCLCommands

SMS Scripts

WFC3

Retrieval /Traceability

STSDASIRAF Tools

Real TimeLocal Database

Local ShortTermArchive

Science IPT /SOC RemoteImage DisplayAnalysis WorkStation

HST RemoteLong TermArchive

StimulusComputer

DataArchivingDistributionSystem(DADS)

OSConfigurationScripts

Science IPTWorkstation

OPUSPipelineWorkstation

SHP Files

SDI Files

OS

Snap

sho

t C

onfig

uration F

iles

Local Real Time Quick Look System - GSFC

OS Ancillary Files

Data Ingest / Retrieval

TelemetryCommands

Proposed WFC3 Thermal Vacuum Data Flow

OPUS System - STScI

OPUSStagingDisk

Time StampFiles

LocalStagingDisk

SMS

Unique Merged FITSFiles with OS Key Words

7

System characteristics

• Utilise existing capabilities and resources

• Modular design

- Science Calibration plan

- SITS

- Optical stimulus

- File transfer & time stamp

- Quick-look ACS-style Analysis System

- OPUS/pipeline system

This allows concurrent development and a phased integration schedule

• Minimise development cost and schedule impacts

• All calibration files will be generated from HDA images

8

Schedule

Detailed schedule for full system currently under review - outline:

• Science Calibration Plan – final version, Nov 30 2001

• Prototype systems for SITS/OS/Quick-look/Time-stamp modules will be developed for testing with Optical Stimulus, late-May to July 2002

• Remaining issues can be addressed post-prototype testing, but before December 2002

• Full system test in December 2002

9

Pipeline deliverables

1. File names 15 Oct 2001

unique identifier to associate SDI, OSC & log files and permit cross-referencing with quick-look data:

ipppssoot_csiiyyddhhmmss.extension2. Definition of unique T/V-specific keywords 31 Oct 20013. Incorporation of auxiliary information 31 May 2001 - OS configuration files - comment/log files4. Identify instrument set-up parameters for incorporation in science

header 31 May 2001 - supplement/replace parameters normally in PMDB5. T/V specific archive retrieval tools 1 April 2001