docushare.lsst.org€¦ · Web viewThe LSST Science Requirements Document specifies a set of...

Large Synoptic Survey Telescope Data Management Subsystem Requirements LSE-61

Document # Date Effective Status

LSE-61 8/3/2011Version 1.1

Author(s)

Gregory Dubois-Felsmann

Data Management Requirements

Subsystem/Office

Data ManagementDocument Title

Data Management Subsystem Requirements

The Large Synoptic Survey Telescope(LSST)

Data Management Subsystem Requirements

The contents of this document are subject to configuration control and may not be changed, altered, or their provisions waived without prior approval of the LSST Change Control Board.

2023-05-15 of 46


Change History Log

Revision Effective Date Description of Changes

1.1 8/3/2011 Requirements numbering applied. Original numbering preserved for a transition period.

1 6/28/2011 Captured DM Functional Requirements Specification as version 1 of new document LSE-61

0.5 February 2009 Major revisions from meeting with Science Collaborations0.1 2007 Initial major release


2023-05-15 of 46


Table of Contents

Data Management Requirements..............................................................................................61 Introduction.......................................................................................................................62 Application Layer..............................................................................................................7

2.1 Data Products Overview...........................................................................................................72.1.1 Level 1, 2, and 3 Data Products............................................................................................7

2.1.1.1 General Data Products Requirements............................................................................7Public Access to Science Data...................................................................................................7Timely Access To Transient Data..............................................................................................7Create and Maintain Science Data Archive ...............................................................................8

Nightly Data Accessible Within 24 hrs....................................................................................8Produce Data Releases..........................................................................................................8

Data Releases Produced Yearly..........................................................................................82.1.2 Overview of Pipeline Processing..........................................................................................8

2.1.2.1 General Production and Pipeline Requirements.............................................................8Pipeline Infrastructure................................................................................................................9

drProductionInfrastructure......................................................................................................9Pipeline Availability.................................................................................................................... 9

productionAvailability..............................................................................................................9Simulated, pre-cursor, and real data..........................................................................................9

2.1.2.2 Alert Production............................................................................................................10Alert Production Requirements................................................................................................10

Exposure Data Acquisition and Archiving.............................................................................10Raw Data Acquisition.........................................................................................................10Raw Data Archiving...........................................................................................................11Wavefront Sensor Data Acquisition...................................................................................11Wavefront Sensor Data Archiving......................................................................................11Cross-talk Corrected Data Acquisition...............................................................................11

Instrument Signature Removal Pipeline (ISR) Requirements...............................................12Provide ISR Pipeline..........................................................................................................12Provide Image Assembly Software....................................................................................12Provide Linearization Software..........................................................................................12Provide Artifact Masking Software.....................................................................................12Provide PSF Determination Software................................................................................12Provide Image Data Quality Assessment Software...........................................................13

WCS Pipeline Requirements.................................................................................................13Generate Photometric Zeropoint for Exposure..................................................................13Generate WCS.................................................................................................................. 13

astrometricQuality...........................................................................................................13Provide Astrometric Calibration Software..........................................................................13

Image Difference Pipeline Requirements..............................................................................13Perform Image Differencing...............................................................................................14

Detection Pipeline (DP) Requirements.................................................................................14Provide Detection Pipeline.................................................................................................14

Association Pipeline (AP) Requirements..............................................................................14Provide Association Pipeline..............................................................................................14

Moving Object Pipeline (MOP) Requirements.......................................................................14Provide Moving Object Pipeline Software..........................................................................15Moving Object Pipeline Accuracy......................................................................................15

2.1.2.3 Data Release Production..............................................................................................15Data Release Production Requirements..................................................................................15

General Data Release Requirements...................................................................................16Calibrated Photometry of Difference Sources....................................................................16Detect Faint Objects..........................................................................................................16


2023-05-15 of 46


Detect Time Varying Point Source Superimposed on Extended Object............................16Enable BAO Analysis.........................................................................................................16Measure Intrinsic Ellipticities of Small Galaxies.................................................................16Provide Astrometric Model.................................................................................................16Provide Calibrated Photometry..........................................................................................17

Provide Calibrated Photometry corrected for Supernova SED.......................................17Provide PSF For an Exposure...........................................................................................17Provide Photometric Redshift of Supernovae....................................................................17Provide Photometric Redshifts of Galaxies........................................................................17

CoAdd Images Pipeline Requirements.................................................................................17General CoAdd Images Pipeline Requirements................................................................17

Deep Detection Pipeline (DDP) Requirements.....................................................................17Provide Image Addition Software.......................................................................................18Provide Deep Detection Software......................................................................................18Provide Shape Measurement Software.............................................................................18

Enable a Range of Shape Measurement Approaches....................................................18Astrometric Calibration Pipeline............................................................................................18

General Astrometric Calibration Pipeline Requirements....................................................18Photometric Calibration Pipeline...........................................................................................19

Provide Photometric Calibration Software.........................................................................192.1.2.4 Calibration Products Production...................................................................................19

Calibration Products Requirements..........................................................................................19Correct for Camera Bias Structure........................................................................................19Correct for Camera Crosstalk...............................................................................................19Correct for Detector Fringing.................................................................................................19Correct for Instrument Sensitivity Variation...........................................................................19Produce Bad Pixel Map........................................................................................................20Produce Bias Frame.............................................................................................................20Produce Crosstalk Correction Matrix....................................................................................20Produce Illumination Correction Frame.................................................................................20Produce Monochromatic Flatfield Data Cube........................................................................20Astrometric Quality................................................................................................................20Photometric Quality...............................................................................................................20

2.1.3 Contents of Principal Database Tables...............................................................................202.1.3.1 Object Table.................................................................................................................212.1.3.2 MovingObject Table......................................................................................................212.1.3.3 Source Table................................................................................................................212.1.3.4 DIASource Table..........................................................................................................212.1.3.5 FaintSource Table........................................................................................................212.1.3.6 Exposure Table............................................................................................................21

2.1.4 Common Data Objects.......................................................................................................212.1.4.1 MaskedImages.............................................................................................................212.1.4.2 Exposures.................................................................................................................... 222.1.4.3 Co-Added Exposures...................................................................................................222.1.4.4 Alerts............................................................................................................................ 22

2.2 Level 1 Data Products............................................................................................................222.2.1 Exposures........................................................................................................................... 22

2.2.1.1 Exposures Requirements.............................................................................................22General Exposure Requirements.............................................................................................23

Exposure Archive Publicly Accessible..................................................................................23Exposure Formats................................................................................................................. 23Keep Exposure Archive........................................................................................................23

Raw Science Exposure............................................................................................................23Raw Science Images Available within 24 hours....................................................................23Raw Science Image Attributes..............................................................................................23

Processed Science Exposure..................................................................................................24The contents of this document are subject to configuration control and may not be changed, altered, or

their provisions waived without prior approval of the LSST Change Control Board.2023-05-15 of 46


Produce Processed Science Exposures...............................................................................24Determine Accurate PSF for an Exposure.........................................................................24Calibrated Science Exposures Available witihin 24 hours.................................................24Processed Science Exposure Attributes............................................................................24

Difference Exposure................................................................................................................. 24Difference Exposures Available within 24 hours...................................................................24Difference Exposure Attributes.............................................................................................25

2.2.2 Catalogs............................................................................................................................. 252.2.2.1 Catalogs Requirements................................................................................................25

General Catalog Requirements................................................................................................25Queryable............................................................................................................................. 25Keep Science Data Archive..................................................................................................25Maintain Archive Publicly Accessible....................................................................................25External Formats................................................................................................................... 25

Difference Source Catalog.......................................................................................................26Difference Source Attributes.................................................................................................26Difference Sources Available within 24 hours.......................................................................26

Object Catalog.........................................................................................................................26Produce Object Catalog........................................................................................................26Source List Updates..............................................................................................................26Level 1 Object Attributes.......................................................................................................27Summary Metadata Attributes...............................................................................................27Summary Metadata Updates................................................................................................27

Orbit Catalog............................................................................................................................ 27Produce Orbit Catalog..........................................................................................................27Source Lists Update Frequency............................................................................................27Moving Object Attributes.......................................................................................................27Moving Objects Available within 24 hours.............................................................................28

2.2.3 Alerts.................................................................................................................................. 282.2.3.1 Alerts Requirements.....................................................................................................28

Generate Alerts........................................................................................................................ 28Detect in Two Exposures in Visit...........................................................................................28Alert Attributes......................................................................................................................28

Public Distribution of Alerts......................................................................................................28Keep Historical Alert Archive....................................................................................................29Archive Alerts within 24 hours..................................................................................................29

2.2.4 Nightly Summary Products.................................................................................................292.2.4.1 Nightly Summary Products Requirements....................................................................29

Generate Data Quality Report within 4 hours...........................................................................29Data Quality Report Contents...............................................................................................29

Generate DMS Performance Report within 4 hours.................................................................29Performance Report Contents..............................................................................................29

Generate Calibration Report within 4 hours.............................................................................30Calibration Report Contents..................................................................................................30

2.2.5 Engineering and Facility Database Archive........................................................................302.2.5.1 E&F Database Requirements.......................................................................................30

Provide E&F Database Archive................................................................................................302.3 Level 2 Data Products............................................................................................................30

2.3.1 Co-Added Exposures..........................................................................................................312.3.1.1 Co-Added Exposures Requirements............................................................................31

Produce Images for EPO.........................................................................................................31Produce Co-Added Exposures.................................................................................................31Co-Added Images Updated at Least Every 6 months..............................................................31Co-Added Image Atrributes......................................................................................................31

2.3.2 Catalogs............................................................................................................................. 322.3.2.1 Object Catalog – Level 2 Data......................................................................................32


2023-05-15 of 46


Astrometric Model....................................................................................................................32Calibrated Light Curve.............................................................................................................32Calibrated Point Source Photometry........................................................................................32Calibrated Position Curve.........................................................................................................32Extended Object Photometry...................................................................................................32Level 2 Object Attributes..........................................................................................................33Object Classification................................................................................................................. 33Photomeric Redshift PDF.........................................................................................................33Photometric Redshifts..............................................................................................................33Extended Object Shape Parameters........................................................................................33

2.3.2.2 Source Catalogs - Level 2 Data....................................................................................33Low SNR Source Attributes......................................................................................................33Source Attributes...................................................................................................................... 34

2.4 Level 3 Data Products............................................................................................................342.4.1 General Level 3 Data Product Requirements.....................................................................35

2.4.1.1 DAC resource allocation for Level 3 processing...........................................................352.4.1.2 Level 3 Data Product Self Consistency........................................................................352.4.1.3 Provenance for Level 3 processing at DACs................................................................35

2.4.2 Level 3 Catalog Processing Requirements.........................................................................352.4.2.1 Access to catalogs for external Level 3 processing......................................................352.4.2.2 Access to input catalogs for DAC-based Level 3 processing.......................................362.4.2.3 Federation with external catalogs.................................................................................362.4.2.4 Software framework for Level 3 catalog processing.....................................................36

2.4.3 Level 3 Image Processing Requirements...........................................................................362.4.3.1 Access to images for external Level 3 processing........................................................362.4.3.2 Access to input images for DAC-based Level 3 processing.........................................362.4.3.3 Software framework for Level 3 image processing.......................................................36

2.5 Calibration Data Products......................................................................................................372.5.1 Calibration Products Requirements....................................................................................37

2.5.1.1 Crosstalk Correction Matrix Attributes..........................................................................372.5.1.2 Calibration Data Products.............................................................................................372.5.1.3 Calibration Images Available 1 Hour Before Observing................................................372.5.1.4 Calibration Image Attributes.........................................................................................38

2.6 Detection and Measurement of Objects................................................................................382.6.1 Deep Detection Processing................................................................................................382.6.2 Difference Exposure Processing.........................................................................................382.6.3 Measuring the properties of Objects...................................................................................38

2.6.3.1 Slowly Moving Point Source Model..............................................................................392.6.3.2 Small Object Model......................................................................................................392.6.3.3 Large Object Model......................................................................................................402.6.3.4 Solar System Model.....................................................................................................40

2.6.4 The Multifit Algorithm..........................................................................................................402.6.5 Model Residuals.................................................................................................................402.6.6 Limitations and Possible Extensions...................................................................................42

2.6.6.1 Faint Transients............................................................................................................422.6.6.2 Faint Solar System Objects..........................................................................................432.6.6.3 Possible Multifit Improvements.....................................................................................43

2.7 Inserting Synthetic Objects...................................................................................................432.8 Data Access............................................................................................................................ 43

2.8.1 Computing Resources........................................................................................................442.8.2 Data Access Tools..............................................................................................................44

2.9 Incorporation of User Analysis Modules..............................................................................453 Middleware Layer............................................................................................................45

3.1 Middleware Layer Requirements...........................................................................................453.1.1 Provide Data Access Services............................................................................................453.1.2 Provide Pipeline Execution Services..................................................................................46


2023-05-15 of 46


3.1.3 Provide Management and Control Services.......................................................................463.1.4 Provide Pipeline Construction Services..............................................................................463.1.5 Provide Data/Catalog Construction Services......................................................................463.1.6 Provide User Interface Services.........................................................................................47

4 Infrastructure Layer........................................................................................................474.1 Infrastructure Layer Requirements.......................................................................................47

4.1.1 General Infrastructure Requirements..................................................................................474.1.1.1 Optimization of Cost, Reliability and Availability in Order.............................................474.1.1.2 Pipeline throughput.......................................................................................................474.1.1.3 Re-processing Capacity...............................................................................................474.1.1.4 Temporary Storage for Communications Links.............................................................484.1.1.5 Infrastructure Sizing for "catching up"...........................................................................484.1.1.6 Incorporate Fault-Tolerance.........................................................................................484.1.1.7 Incorporate Autonomics................................................................................................48

4.1.2 Mountaintop Site Requirements..........................................................................................484.1.2.1 Mountaintop Site Data Communications......................................................................484.1.2.2 Mountaintop Site Temporary Storage...........................................................................484.1.2.3 Prefer Computing and Storage Down...........................................................................48

4.1.3 Mountain to Base Network Requirements..........................................................................494.1.3.1 Mountain to Base Network...........................................................................................494.1.3.2 Mountain to Base Network Availability..........................................................................494.1.3.3 Mountain to Base Network Reliability...........................................................................494.1.3.4 Mountain to Base Network Secondary Link..................................................................494.1.3.5 Mountain to Base Network Ownership and Operation..................................................49

4.1.4 Base Facility Requirements................................................................................................494.1.4.1 Base Facility Infrastructure...........................................................................................494.1.4.2 Base Facility Temporary Storage.................................................................................504.1.4.3 Base Facility Co-Location with Existing Facility............................................................504.1.4.4 Base Facility Data Quality Assessment........................................................................50

4.1.5 Base to Archive Network Requirements.............................................................................504.1.5.1 Base to Archive Network..............................................................................................504.1.5.2 Base to Archive Network Availability............................................................................504.1.5.3 Base to Archive Network Reliability..............................................................................504.1.5.4 Base to Archive Network Secondary Link.....................................................................514.1.5.5 Base to Archive Network Lease....................................................................................51

4.1.6 Archive Center Requirements.............................................................................................514.1.6.1 Archive Center..............................................................................................................514.1.6.2 Archive Center Disaster Recovery................................................................................514.1.6.3 Archive Center Co-Location with Existing Facility.........................................................51

4.1.7 Archive to Data Access Center Network Requirements......................................................514.1.7.1 Archive to Data Access Center Network.......................................................................514.1.7.2 Archive to Data Access Center Network Availability.....................................................524.1.7.3 Archive to Data Access Center Network Reliability......................................................524.1.7.4 Archive to Data Access Center Network Secondary Link.............................................524.1.7.5 Archive to Data Access Center Network Lease............................................................52

4.1.8 Data Access Center Requirements.....................................................................................524.1.8.1 Data Access Centers....................................................................................................524.1.8.2 Data Access Center Simultaneous Connections..........................................................524.1.8.3 Data Access Center VO Standards..............................................................................524.1.8.4 Data Access Center Geographical Distribution.............................................................534.1.8.5 No Limit on Data Access Centers.................................................................................53

Data Management Requirements


2023-05-15 of 46


This package contains the subsystem level requirements for the LSST Data Management System (DMS).

1 IntroductionThe LSST Science Requirements Document specifies a set of science goals to be achieved from the LSST observing program. To enable the achievement of these goals, the LSST Data Management System (DMS) is required to generate, or enable the generation of, a set of data products, and to make them available to scientists and the public. To carry out this mission the DMS performs the following major functions:∑ Processes the incoming stream of images generated by the camera system during observing to

produce transient alerts and to archive the raw images.∑ Roughly once per year, creates and archives a Data Release (DR), which is a static self-consistent

collection of data products generated from all survey data taken from the date of survey initiation to the cutoff date for the Data Release. The data products include optimal measurements of the properties (shapes, positions, fluxes, motions) of all objects, including those below the single visit sensitivity limit, astrometric and photometric calibration of the full survey object catalog, and limited classification of objects based on both their static properties and time-dependent behavior. Deep coadded images of the full survey area are produced as well.

∑ Periodically creates new calibration data products, such as bias frames and flat fields, that will be used by the other processing functions.

∑ Makes all LSST data available through an interface that utilizes, to the maximum possible extent, community-based standards such as those being developed by the Virtual Observatory (VO), and facilitates user data analysis and the production of user-defined data products at Data Access Centers (DAC) and at external sites.

The data management system begins at the data acquisition interface between the camera and telescope subsystems and flows through to the data products accessed by end users. On the way, it moves through three types of managed facilities supporting data management, as well as end user sites that may conduct science using LSST data or pipeline resources on their own computing infrastructure.The Data Management facilities are a Mountain Summit/Base Facility, a central Archive Center, multiple Data Access Centers, and a System Operations Center. The data will be transported over existing high-speed optical fiber links from the mountain summit/base facility in South America to the archive center in the U.S. Data will also flow from the mountain summit/base facility and the archive center to the data access centers over existing fiber optic links.∑ The Mountain Summit/Base Facility is composed of the mountaintop telescope site, where data

acquisition must interface to the other LSST subsystems, and the Base Facility, where rapid-turnaround processing will occur for data quality assessment and near real-time alerts.

∑ The Archive Center is a super-computing class data center with high reliability and availability. This is where the data will undergo complete processing and re-processing and permanent storage. It is also the main repository feeding the distribution of LSST data to the community.

∑ Data Access Centers for broad user access are envisioned, according to a tiered access model where the tiers define the capacity and response available. There are two project funded Data Access Centers co-located with the Base Facility and the Archive Center. These centers provide replication of all of the LSST data to ensure that disaster recovery is possible. They provide Virtual Observatory interfaces to the LSST data products. LSST is encouraging non-US/non-Chilean funding and partners to host additional Data Access Centers to increase end user access bandwidth and help amortize observatory operations costs.

∑ The System Operations Center (SOC) provides a control room and large-screen display for supervisory monitoring and control of the DM System. Network and facility status are available as well as the capability to "drill down" to individual facilities. DM support to observatory science operations and an end user help desk are also at the SOC.


2023-05-15 of 46


2 Application Layer

The Data Products and the Productions and Pipelines that produce those products comprise the DMS Application Layer.

2.1Data Products Overview

2.1.1 Level 1, 2, and 3 Data Products∑ Level 1 products are generated by pipeline processing the stream of data from the camera system

during normal observing. Level 1 data products are therefore continuously generated and / or updated every observing night. This process is of necessity highly automated, and must proceed with absolutely minimal human interaction. In addition to science data products, a number of Level 1 SDQA data products are generated to assess quality and to provide feedback to the Observatory Control System.

∑ Level 2 products are generated as part of a Data Release, which is required to be performed at least yearly, and will be performed more frequently during the first year of the survey. Level 2 products use Level 1 products as input, and include data products for which extensive computation is required, often because they combine information from many exposures. Although the steps that generate Level 2 products may be automated, significant human interaction will be required at key points to ensure the quality of the data.

∑ Level 3 data products are derived from Level 1 and / or Level 2 data products to support particular science goals, often requiring the combination of LSST data across significant areas on the sky. The DMS is required to facilitate the creation of Level 3 data products, for example by providing suitable APIs and computing infrastructure, but is not itself required to create any Level 3 data product. Instead these data products are created externally to the DMS, using software written by, e.g., science collaborations. Once created, Level 3 data products may be associated with Level 1 and Level 2 data products through database federation (Wikipedia 09). In rare cases, the LSST Project, with the agreement of the Level 3 creators, may decide to incorporate Level 3 data products into the DMS production flow, thereby promoting them to Level 2 data products.

Level 1 and Level 2 data products that have passed quality control tests are required to be accessible to the public without restriction. Additionally, the source code used to generate them will be made available, and LSST will provide support for builds on selected platforms. The access policies for Level 3 data products will be product- and source-specific, and in some cases will be proprietary.

2.1.1.1 General Data Products RequirementsThese requirements apply to all Data Products.

Public Access to Science DataID: DM-APP-DP-GEN-5ID: DMS-REQ-0001

Last Modified: 6/16/2011

Timely Access To Transient DataID: DM-APP-DP-GEN-6ID: DMS-REQ-0002


Transient alerts shall be published to community alert distribution networks using community-standard


2023-05-15 of 46


protocols, to be determined during the LSST construction phase as community standards evolve.

Create and Maintain Science Data Archive ID: DM-APP-DP-GEN-1ID: DMS-REQ-0003


Nightly Data Accessible Within 24 hrsID: DM-APP-DP-GEN-3ID: DMS-REQ-0004


Produce Data ReleasesID: DM-APP-DP-GEN-4ID: DMS-REQ-0005


Data Releases Produced YearlyID: DM-APP-DP-GEN-2ID: DMS-REQ-0006


2.1.2 Overview of Pipeline Processing

2.1.2.1 General Production and Pipeline Requirements∑ The Alert Production runs at the Base Facility, and processes the images streaming from the LSST

Camera to produce real time transient Alerts∑ The Data Release Production runs at the Archive Facility, and is responsible for creating a new Data

Release, starting from Raw Science Exposures.∑ The Calibration Products Production runs at the Archive Facility, and services both the Alert and Data

Release Productions. It is responsible for creating the calibration data products, such as flats and biases, that are required to remove the instrumental signature from the Raw Science Exposures.

Pipeline InfrastructureID: DM-APP-PL-1ID: DMS-REQ-0007


Discussion: For now, for compatibility with earlier releases of the FRS, this time remains set at one year, but this can only be correct if it includes time for development and release validation between production cycles. Most likely the actual production period will need to be significantly shorter than one year in order to meet the higher-level requirement of one year for the interval between releases.

drProductionInfrastructureLast Modified: 9/19/2010

Description Value Unit Name1.0 Years drProcessingP

eriod


2023-05-15 of 46


Pipeline AvailabilityID: DM-APP-PL-2ID: DMS-REQ-0008


Discussion: This applies to active pipelines only. It is allowed for pipelines to be down for longer periods during observatory scheduled maintenance, for Alert Production, and during development/validation periods between productions, for Data Release Production.

productionAvailabilityLast Modified: 9/19/2010

Description Value Unit NameMaximum allowable outage of active DM production.

24 Hour productionMaxDowntime

Simulated, pre-cursor, and real dataID: DM-APP-PL-3ID: DMS-REQ-0009


2.1.2.2 Alert Production∑ Acquire the raw science images from the Camera, and move them to the Archive Center for

permanent storage.∑ Acquire the raw wavefront sensor images from the Camera, and move them to the Archive Center for

permanent storage and access by the Telescope and Site group.∑ Process the crosstalk-corrected images from the Camera to detect transient events within 60 seconds

of shutter closure for the second exposure in a Visit.∑ Package information about detected transients as Alerts, and distribute them to the community as

VOEvents.∑ Continuously assess the data quality of the data stream.The major steps in the processing flow are:∑ Image processing of the Raw Exposures to remove the instrumental signature.∑ Determination of the WCS, PSF, and rough photometric zeropoint. This produces Processed

Exposures. WCS solutions are also reported to the Observatory control system for use by the TCS.∑ Subtraction of a registered Template Exposure from the Processed Exposure, producing a Difference

Exposure. The Template Exposure is a coadd created by the Data Release Production.∑ Detection of sources of either polarity in the Difference Exposure, producing DIASources.∑ Visit processing logic, which compares the DIASources from the two Exposures in the Visit to

discriminate against cosmic rays, and to flag very rapidly moving Solar System objects.∑ FaintSources, abbreviated measurements of low SNR detections, are produced for Objects of

particular interest, e.g. predicted positions for Solar System objects, or objects that have previously produced Alerts.

∑ Comparison of positive flux DIASources with predictions from the Moving Object System for already known Solar System objects, as contained in the MovingObject table.

∑ The Association Pipeline is run to match DIASources to already known astronomical objects, as contained in the Object table.

∑ Generation of Alerts. DIASources that are detected in both Exposures of a Visit, and are not matched to a known Solar System object, will produce an Alert.

∑ SDQA is performed at every pipeline stage, stored in database tables, and fed to the OCS as required.

∑ The Moving Object Pipeline (MOPS) is run during the day to interpret each new detection of a moving object as a new measurement of a Solar System object already in the MovingObject table, or as a previously unknown object, which will be added to the MovingObject table. All orbits are refined


2023-05-15 of 46


based on the new measurements from the night.As the raw images arrive at the Archive Center, the same processing flow is performed there, with the consistency of the databases at the Base and Archive Centers being periodically checked. The duplication of processing is carried out to reduce the data bandwidth required between the Base and Archive Centers.

Alert Production Requirements

Exposure Data Acquisition and Archiving

Raw Data AcquisitionID: DMS-REQ-0018 Last Modified: 6/16/2011

Discussion: The manner of data acquisition is a matter for the DM-Camera ICD in this area.

Raw Data ArchivingID: DMS-REQ-0019 Last Modified: 6/16/2011

Discussion: This will involve assembly of the acquired data at the Base Center and its transfer to the Archive Center.

Wavefront Sensor Data AcquisitionID: DMS-REQ-0020 Last Modified: 6/16/2011

Discussion: The manner of data acquisition is a matter for the DM-Camera ICD in this area. It should be identical, except for the selection of the specific data sources, to that for the raw science sensor data.

There is no currently established requirement for the acquisition or archiving of any raw guider sensor data.

Wavefront Sensor Data ArchivingID: DMS-REQ-0021 Last Modified: 6/16/2011

Discussion: This will involve assembly of the acquired data at the Base Center and its transfer to the Archive Center.

Cross-talk Corrected Data AcquisitionID: DMS-REQ-0022 Last Modified: 6/16/2011

Discussion: The manner of data acquisition is a matter for the DM-Camera ICD in this area. It may be different for that for the raw science sensor data, as it is subject to quite different latency and reliability requirements.

This data is the input to the main Alert Production pipelines. It is not planned to be archived.


2023-05-15 of 46


Instrument Signature Removal Pipeline (ISR) Requirements∑ Level 1 Science Reduction Mode∑ Level 2 Science Reduction Mode∑ Calibration Mode.These modes produce the associated Image Data Products. The modes are implemented by sequencing the execution of a mode-specific subset of a common set of image processing tasks.

Provide ISR PipelineID: DMS-REQ-0023 Last Modified: 6/16/2011

Provide Image Assembly SoftwareID: DM-APP-PL-IP-1ID: DMS-REQ-0024


Provide Linearization SoftwareID: DM-APP-PL-IP-2ID: DMS-REQ-0025


[ Want to be limited by systematics, rather than random noise. Req will be different for each product

]

[Is "linearization" really the right community-standard name for this?]

Provide Artifact Masking SoftwareID: DM-APP-PL-IP-3ID: DMS-REQ-0026


Provide PSF Determination SoftwareID: DM-APP-PL-IP-5ID: DMS-REQ-0027


TBD metric/accuracy requirement?

[ The PSF determined here is mostly used for image DQA, possibly for increasing the astrometric and/or photometric precision. It will not be good enough to use for WL analysis. It's very unclear how to set a requirement on the PSF accuracy for the latter purposes]

Provide Image Data Quality Assessment SoftwareID: DM-APP-PL-IP-7ID: DMS-REQ-0028


[ What remains to be provided here is a list of metrics.]


2023-05-15 of 46


WCS Pipeline Requirements

Generate Photometric Zeropoint for ExposureID: DMS-REQ-0029 Last Modified: 6/16/2011

Generate WCSID: DMS-REQ-0030 Last Modified: 6/16/2011

astrometricQualityLast Modified: 6/16/2011

Description Value Unit NameThis value corresponds to about one-quarter of a pixel.

50 mili-Arcsec

astrometricAccuracy

5 int astrometricMinStandards

Provide Astrometric Calibration SoftwareID: DM-APP-PL-IP-4ID: DMS-REQ-0031


Image Difference Pipeline Requirements

Perform Image DifferencingID: DMS-REQ-0032 Last Modified: 6/16/2011

Detection Pipeline (DP) Requirements∑ Calibrated Science Image∑ Subtracted Science Image∑ Co-added Science ImageThese modes produce the associated Source Catalog Data Product.

[ Not quite right - needs work ]

Provide Detection PipelineID: DM-APP-PL-DP-1ID: DMS-REQ-0033


Association Pipeline (AP) Requirements∑ Proper motions and parallaxes of stars∑ Classification of variable stars∑ Phased light curves for periodic variable stars∑ Orbital elements for solar system objects


2023-05-15 of 46


∑ Recognition that an alertable transient event has occurred.

Provide Association PipelineID: DM-APP-PL-AP-1ID: DMS-REQ-0034


TBD metric/accuracy requirement?

[ possibly require, for each source, the probability that the source is correctly associated with the object]

Moving Object Pipeline (MOP) Requirements

Provide Moving Object Pipeline SoftwareID: DM-APP-PL-MO-1ID: DMS-REQ-0035


Discussion: The magnitude and angular velocity limits for this identification are TBD. These limits may be driven more by computational resource constraints than by the raw reach of the collected data. The software may well be capable of exceeding the required limits, but at an unacceptable cost.

Moving Object Pipeline AccuracyID: DM-APP-PL-MO-2ID: DMS-REQ-0036


2.1.2.3 Data Release Production∑ As in the Alert Production, all Raw Exposures from the camera are processed to remove the

instrumental signature, and to determine the WCS and PSF, producing Processed Exposures. This is done with the best available calibration products, which in general will not be those available when the processing was initially done.

∑ The survey region is tessellated into a set of sky patches, and several Coadded Exposures are produced for each patch from the Processed Exposures. These are a per-band Template Coadd used for image subtraction; a Detection Coadd used in the Deep Detection Pipeline, possibly per-band; and a RGB Coadd used for visualization.

∑ The Deep Detection Pipeline is run, populating the Object, Source, and FaintSource tables. Deep Detection uses the Multifit algorithm to fit object models simultaneously to the entire stack of Exposures which contain the Object. This results in a set of optimal measurements of the Object attributes over the full time span of the survey, including astrometric parameters such as proper motion and parallax.

∑ The Image Subtraction Pipeline is run, as in the Alert Production, yielding DIASources and FaintSources, for transient objects

∑ The Moving Object Pipeline is run on DIASources, to yield a complete set of orbits for Solar System Objects in the MovingObject table.

∑ The Photometric Calibration Pipeline is run on the full set of measurements in the Source, DIASource, and FaintSource catalogs, incorporating measurements from the Auxiliary Telescope and other sources of data about the atmosphere to perform a global photometric calibration of the survey. In addition to accurate photometry for every measurement, this yields an atmosphere model for every Exposure.


2023-05-15 of 46


Data Release Production Requirements

General Data Release Requirements

Calibrated Photometry of Difference SourcesID: DM-APP-PL-DRP-2ID: DMS-REQ-0037


Discussion: SRD: 1 - 2% accuracy required for Solar System Object taxonomy based on photometry

Detect Faint ObjectsID: DM-APP-PL-DRP-3ID: DMS-REQ-0038


Detect Time Varying Point Source Superimposed on Extended ObjectID: DM-APP-PL-DRP-4ID: DMS-REQ-0039


Discussion: Supernovae detected by LSST will typically appear superimposed on a resolved galaxy.

Enable BAO AnalysisID: DM-APP-PL-DRP-5ID: DMS-REQ-0040


Measure Intrinsic Ellipticities of Small GalaxiesID: DM-APP-PL-DRP-1ID: DM-APP-PL-DRP-1ID: DMS-REQ-0041


The measurement error goal for this is TBD.

Provide Astrometric ModelID: DM-APP-PL-DRP-6ID: DMS-REQ-0042


Provide Calibrated PhotometryID: DM-APP-PL-DRP-7ID: DMS-REQ-0043


Provide Calibrated Photometry corrected for Supernova SEDID: DMS-REQ-0044 Last Modified: 8/3/2011


2023-05-15 of 46


Discussion: The Spectral Energy Distribution (SED) of supernovae is very different from that of stars.

Provide PSF For an ExposureID: DMS-REQ-0047 Last Modified: 8/3/2011

Provide Photometric Redshift of SupernovaeID: DM-APP-PL-DRP-10ID: DMS-REQ-0045


Provide Photometric Redshifts of GalaxiesID: DM-APP-PL-DRP-11ID: DMS-REQ-0046


CoAdd Images Pipeline Requirements

General CoAdd Images Pipeline RequirementsID: DMS-REQ-0048 Last Modified: 8/3/2011

The software shall support the generation of monochromatic, panchromatic, and polychromatic coadds.

Each Exposure contributing to a Co-Added Exposure shall be projected onto a common reference frame using a projection selected from a set of options by a Policy.

Deep Detection Pipeline (DDP) Requirements

Provide Image Addition SoftwareID: DM-APP-PL-DD-1ID: DMS-REQ-0049


Provide Deep Detection SoftwareID: DM-APP-PL-DD-2ID: DMS-REQ-0050


Provide Shape Measurement SoftwareID: DM-APP-PL-DD-3ID: DMS-REQ-0051


Discussion: This does not specify the algorithms or architecture to be used, and in particular does not specify which of the approaches described in the narrative section "Detection and Measurement of Objects" below may be used.

Enable a Range of Shape Measurement ApproachesID: DMS-REQ-0052 Last Modified: 6/16/2011


2023-05-15 of 46


Astrometric Calibration Pipeline

General Astrometric Calibration Pipeline RequirementsID: DMS-REQ-0053 Last Modified: 8/3/2011

Photometric Calibration Pipeline

Provide Photometric Calibration SoftwareID: DM-APP-PL-IP-6ID: DMS-REQ-0054


Discussion: Need to resolve why the word "Preliminary" appeared in the original text here, and what the relationship with the global photometric calibration is. gpdf: I think the latter may not be adequately framed in the existing requirements.

2.1.2.4 Calibration Products Production

Calibration Products Requirements(Note: the requirements below remain to be fleshed out pending the current review of the Calibration Plan by the new Calibration Scientist.)

Correct for Camera Bias StructureID: DM-APP-PL-CAP-1ID: DMS-REQ-0055


TBD

Correct for Camera CrosstalkID: DM-APP-PL-CAP-2ID: DMS-REQ-0056


TBD

Correct for Detector FringingID: DM-APP-PL-CAP-4ID: DMS-REQ-0057


TBD

Correct for Instrument Sensitivity VariationID: DM-APP-PL-CAP-3ID: DMS-REQ-0058


TBD


2023-05-15 of 46


Produce Bad Pixel MapID: DM-APP-PL-CAP-7ID: DMS-REQ-0059


TBD

Produce Bias FrameID: DM-APP-PL-CAP-5ID: DMS-REQ-0060


TBD

Produce Crosstalk Correction MatrixID: DM-APP-PL-CAP-6ID: DMS-REQ-0061


TBD

Produce Illumination Correction FrameID: DM-APP-PL-CAP-8ID: DMS-REQ-0062


TBD

Produce Monochromatic Flatfield Data CubeID: DM-APP-PL-CAP-9ID: DMS-REQ-0063


TBD

Astrometric QualityLast Modified: 6/22/2009

Photometric QualityLast Modified: 6/22/2009

2.1.3 Contents of Principal Database TablesTwo important definitions that underly the design of the database are those for "Object" and "Source". An Object (or, when confusion is possible, an "AstroObject") is a representation of an astrophysical object: a star; a galaxy; a Solar System object. A "Source" is the representation of a measurement of an Object's properties from a single image that contains its footprint on the sky. As we discuss in the remainder of this section, both Objects and Sources come in a few different flavors that are specialized for particular situations that frequently arise.The highest level tables are Object and MovingObject, which together organize on a per-astrophysical-object basis all information from the survey. An entry in either of these tables links to all the individual measurements of the Object, contained in the DIASource, FaintSource, and Source tables. Each of these measurements, in turn, links to the metadata for each telescope exposure in the Exposure table.

2.1.3.1 Object Table

The Object Table has a row for every non-Solar-System astrophysical object found in the LSST images.


2023-05-15 of 46


Each Object Table row has a set of columns which together summarize what has been measured for the object over the history of the survey.

2.1.3.2 MovingObject TableNote that there will be occasions during nightly alert processing in which an Object with only a single associated measurement is subsequently found to be a measurement of a MovingObject. In these relatively rare cases, the original Object will be deleted.

2.1.3.3 Source TableAn entry in the Source Table is made in conjunction with Multifit measurement of an Object, and represents a measurement at high SNR from a single Exposure.

2.1.3.4 DIASource TableAn entry in the DIASource Table is made as a result of a high SNR measurement of an Object in a difference Exposure.

2.1.3.5 FaintSource TableAn entry in the FaintSource Table is made in conjunction with a low SNR measurement of an Object either in Deep Detection and Measurement (e.g., with Multifit) or in a difference Exposure.

2.1.3.6 Exposure Table

2.1.4 Common Data Objects

2.1.4.1 MaskedImages∑ The Science image, with the pixel data type being either 16-bit integers or 32-bit floats.∑ The Mask image, typically 16 bits deep, with each plane representing a particular logical statement

about the corresponding science image pixel, e.g. "this pixel is saturated".∑ The Variance image, which represents the expected variance in the corresponding science pixel

values.

2.1.4.2 Exposures

2.1.4.3 Co-Added Exposures

2.1.4.4 AlertsThe data bundled with the Alert includes:∑ The rows from the Exposure table for the two Exposures in the Visit.∑ The rows from the DIASource table for the two relevant DIASources.∑ The row from the Object table for the associated Object.


2023-05-15 of 46


∑ A postage stamp image centered on the DIASource from each of the two Difference Exposures, and from the Template Exposure.

Note that no explicit classification of an Alert is provided, but users can readily construct classifiers and filters based on information in the Science Database. This includes past time dependence, colors, and shape information for the associated Object. Additionally, database queries can readily be formulated which will identify Exposures that have generated anomalously large numbers of Alerts, presumably due to image artifacts or processing problems.

2.2Level 1 Data Products

Level 1 data products are divided into Exposures, Catalogs, Alerts, Nightly Summary Products, and the Engineering and Facility Database Archive.

2.2.1 Exposures∑ Raw Exposures∑ Processed Exposures (trimmed, debiased, flattened, etc.)∑ Difference Exposures.Note that the Raw Exposures are sent to the Archive Center, where they are immediately made available for retrieval. Processed and Difference Exposures are recreated on-the-fly at the Archive Center as needed.

2.2.1.1 Exposures Requirements

General Exposure RequirementsThe requirements in this section apply to all Exposures.

Exposure Archive Publicly AccessibleID: DM-APP-DP-IM-2ID: DMS-REQ-0064


All releases of the DMS exposure archive shall be maintained and preserved in a publicly accessible statefor the entire operational life of the LSST observatory.

Exposure FormatsID: DM-APP-DP-IM-3ID: DMS-REQ-0065


Additionally, full exposures shall be made available in LSST Pipeline Input Format, for processing by LSST pipelines or by external pipelines.

Keep Exposure ArchiveID: DM-APP-DP-IM-1ID: DMS-REQ-0066


The Archive shall be designed to be capable of managing the data for the full duration of the survey and


2023-05-15 of 46


associated data processing.

Raw Science Exposure

Raw Science Images Available within 24 hoursID: DM-APP-DP-IM-4ID: DMS-REQ-0067


The raw science images and associated metadata shall be available for public access in the DMS image archive within 24 hours of their read-out from the Camera.

Raw Science Image AttributesID: DM-APP-DP-IM-5ID: DMS-REQ-0068


* Time of exposure start and end, referenced to TAI* Site metadata (site seeing, transparency, weather)* Telescope metadata (telescope pointing, active optics state, environmental state)* Camera metadata (shutter trajectory, wavefront sensors, environmental state)

Processed Science Exposure

Produce Processed Science ExposuresID: DMS-REQ-0069 Last Modified: 1/27/2009

Determine Accurate PSF for an ExposureID: DMS-REQ-0070 Last Modified: 6/16/2011

Intrinsic Object shapes can be measured only if the PSF is accurately known at all focal plane positions within the Exposure

Calibrated Science Exposures Available witihin 24 hoursID: DM-APP-DP-IM-6ID: DMS-REQ-0071


Calibrated science exposures shall be available for public access in the DMS Exposure archive within 24 hours of their generation by the DMS.

Processed Science Exposure AttributesID: DM-APP-DP-IM-7ID: DMS-REQ-0072


* List of readouts of a single ccd that have been combined into a single image structure* Associated pixel mask, identifying anomalous sensor pixels, saturated pixels, and pixels affected by artifacts such as satellite trails, cosmic rays, or stray light from nearby bright sources* Astrometric calibration in the form of a WCS, ie. a mapping from pixel coordinates to ICRS sky coordinates* Photometric calibration, i.e a mapping from a source's intensity in data units to its AB magnitude at the top of the atmosphere for the filter used.


2023-05-15 of 46


* A representation of the spatially varying point spread function (PSF)* Data quality assessment

Difference Exposure

Difference Exposures Available within 24 hoursID: DM-APP-DP-IM-8ID: DMS-REQ-0073


Difference Exposures shall be available for public access in the DMS exposure archive within 24 hours of their generation by the DMS.

Difference Exposure AttributesID: DM-APP-DP-IM-9ID: DMS-REQ-0074


* The ratio of the RMS level of the image noise in areas far from bright sources to the sky noise in the input images, and the spatial variation of that ratio.* The ratio of the RMS level of the image noise in areas centered on bright sources compared the source level, and the spatial variation of that ratio.* Representation of the PSF matching kernel used in the differencing.

2.2.2 Catalogs∑ Exposure∑ Object∑ MovingObject∑ DIASource∑ FaintSource.

2.2.2.1 Catalogs Requirements

General Catalog RequirementsThe requirements in this section apply to all Catalogs.

QueryableID: DMS-REQ-0075 Last Modified: 1/23/2009

The catalogs shall be queryable in SQL

Keep Science Data ArchiveID: DM-APP-DP-CA-1ID: DMS-REQ-0076


The DMS shall provide a science data archive containing catalogs of all detected sources and objects andassociated metadata that meet data quality thresholds for retention.

Maintain Archive Publicly Accessible


2023-05-15 of 46


ID: DM-APP-DP-CA-2ID: DMS-REQ-0077


All releases of the DMS catalog archive shall be maintained and preserved in a publicly accessible state for the entire operational life of the LSST observatory.

External FormatsID: DM-APP-DP-CA-3ID: DMS-REQ-0078


* Comma-separated ASCII text* eXtensible Markup Language (XML) format, including VOTable (http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOTable)

Difference Source Catalog

Difference Source AttributesID: DMS-REQ-0079 Last Modified: 2/6/2009

* Focal plane position centroid (pixels)* Expected focal plane position centroid error (pixels)* Instrumental flux (ADU)* Expected instrumental flux error (ADU)* Sky background level (ADU)* Expected error in sky background (ADU)* Source extraction flags, which identify any problems encountered by the DP in processing the Difference Source* Shape characterization, including at least an extendedness parameter* ICRS sky position derived from the focal plane position (radec)* Expected error in ICRS sky position (radec)

Difference Sources Available within 24 hoursID: DM-APP-DP-CA-4ID: DMS-REQ-0080


Detected sources and associated metadata shall be available for public access in the DMS science data archive within 24 hours of their generation by the DMS.

Object Catalog

Produce Object CatalogID: DM-APP-DP-CA-6ID: DMS-REQ-0081


The LSST DMS shall produce a catalog of all objects detected by the Detection Pipeline in at least two Difference Exposures from a single Visit.

Source List UpdatesID: DM-APP-DP-CA-7ID: DMS-REQ-0082



2023-05-15 of 46


The Difference Source lists shall be updated following each exposure with a latency of no more than 24 hrs.

Level 1 Object AttributesID: DM-APP-DP-CA-8ID: DMS-REQ-0083


* A unique Object ID* All Difference Sources, Sources, and Low SNR Sources associated with the Object* A set of Object properties. A restricted set of properties is generated by the Alert Production, while the full set is generated by the Data Release Production.

Summary Metadata AttributesID: DM-APP-DP-CA-9ID: DMS-REQ-0084


* Mean ICRS sky position* Best estimate (eg median) for AB top-of-atmosphere magnitude in each filter band* Variability measures (eg Welch-Stetson) for each filter band* Proper motion (None if not measured)* Parallax (None if not measured)* Object classification

Summary Metadata UpdatesID: DM-APP-DP-CA-11ID: DMS-REQ-0085


The object summary metadata attributes shall be updated no less frequently than every 6 months.

Orbit Catalog

Produce Orbit CatalogID: DM-APP-DP-CA-12ID: DMS-REQ-0086


The LSST DMS shall produce a catalog of all moving objects detected by the Moving Object Pipeline (MOP)

Source Lists Update FrequencyID: DM-APP-DP-CA-13ID: DMS-REQ-0087


The source lists shall be updated following each exposure with a latency of no more than 24 hrs.

Moving Object AttributesID: DM-APP-DP-CA-14ID: DMS-REQ-0088


* Orbital elements* Probable errors in orbital elements


2023-05-15 of 46


* Summary photometric information, as for other objects in Object Catalog* The Sources associated with this Moving Object, as for other Objects in Object Catalog

Moving Objects Available within 24 hoursID: DM-APP-DP-CA-15ID: DMS-REQ-0089


Detected moving objects and associated metadata shall be available for public access in the DMS science data archive within 24 hours of their generation by the DMS.

2.2.3 AlertsAlerts are distributed as VOEvents, and archived at the Archive Center.

2.2.3.1 Alerts Requirements

Generate AlertsID: DMS-REQ-0090 Last Modified: 6/22/2009

Detect in Two Exposures in VisitID: DM-APP-DP-AL-1ID: DMS-REQ-0091


Alert AttributesID: DM-APP-DP-AL-3ID: DMS-REQ-0092


* Class of event detected* Confidence of detection* All associated information from the Object Catalog (includes Sources associated with this Object)* Postage stamp images centered on the Object from both difference exposures of the visit which triggered the alert, and from the template image used in image substraction.

Public Distribution of AlertsID: DM-APP-DP-AL-2ID: DMS-REQ-0093


Keep Historical Alert ArchiveID: DM-APP-DP-AL-4ID: DMS-REQ-0094


The DMS shall preserve and keep in an accessible state an alert archive with all issued alerts for a historical record and for false alert analysis.

Archive Alerts within 24 hoursID: DM-APP-DP-AL-5ID: DMS-REQ-0095



2023-05-15 of 46


Alerts shall be available for public access in the DMS alert archive within 24 hours of their initial issue.

2.2.4 Nightly Summary ProductsA variety of reports will be generated every night to summarize the performance of the DMS and the SDQA metrics on the Level 1 data. The contents of these is TBD.

2.2.4.1 Nightly Summary Products Requirements

Generate Data Quality Report within 4 hoursID: DM-APP-DP-SR-1ID: DMS-REQ-0096


The DMS shall generate a Data Quality Report in both human-readable and machine-readable forms within 4 hours of the completion of each observing night.

Data Quality Report ContentsID: DM-APP-DP-SR-2ID: DMS-REQ-0097


* Photometric zero point vs. time for each utilized filter* Sky brightness vs. time for each utilized filter* Seeing vs. time for each utilized filter - this is a broad brush measure of image quality* PSF parameters vs. time for each utilized filter - this is a lot more information, since it includes asymmetries and field dependence* Detection efficiency for point sources vs. mag for each utilized filter.

Generate DMS Performance Report within 4 hoursID: DM-APP-DP-SR-3ID: DMS-REQ-0098


The DMS shall generate a DMS Performance Report in both human-readable and machine-readable forms within 4 hours of the completion of each observing night.

Performance Report ContentsID: DM-APP-DP-SR-4ID: DMS-REQ-0099


* Number of observations successfully processed through each pipeline* Number of observations for each pipeline that had recoverable failures - group by failure type (e.g. hardware or cpu / software failure cleared by automated intervention)* Number of observations for each pipeline that had unrecoverable failures - similar grouping* Number of observations archived at base camp* Number of observations archived at archive center* Number of observations satisfying the science criteria for each active science program

Generate Calibration Report within 4 hoursID: DM-APP-DP-SR-5ID: DMS-REQ-0100


The DMS shall generate a Calibration Report in both human-readable and machine-The contents of this document are subject to configuration control and may not be changed, altered, or



Calibration Report ContentsID: DM-APP-DP-SR-6ID: DMS-REQ-0101


* A super-flat from all observations on the sky, with filters applied to avoid using cloudy exposures and implementing quality thresholds * A combined bias from all biases.

2.2.5 Engineering and Facility Database ArchiveEvery 24 hrs, the DMS synchronizes the Engineering and Facility Database, which is generated by the Observatory Control System (OCS), with the copy at the Archive Center. This Level 1 product contains comprehensive metadata for the entire LSST observatory, and can be queried together with the Science Database.

2.2.5.1 E&F Database Requirements

Provide E&F Database ArchiveID: DM-APP-DP-EF-1ID: DMS-REQ-0102


2.3Level 2 Data Products

Level 2 data products include Coadded Science Exposures, Calibration Products, and Catalogs.

2.3.1 Co-Added Exposures∑ Template Co-Add for creating Difference Exposures. This Co-Add is optimized for a narrow PSF, so

that during image differencing the Template is never narrowed by the matching kernel, even in the best seeing of the survey. A Template is specific to a filter band.

∑ Detection Co-Add for object detection. This Co-Add is optimized so that the peaks from faint objects have the highest signal-to-noise. The Detection Co-Add may combine information from multiple filter bands.

∑ RGB Co-Add for visualization. This Co-Add will be formed by combining the Template Co-Adds for the individual filter bands to give a visually pleasing and useful color image.

2.3.1.1 Co-Added Exposures Requirements

Produce Images for EPOID: DMS-REQ-0103 Last Modified: 6/16/2011

Discussion: This is expected to include polychromatic (e.g., RGB JPEG) images for casual users. The contents of this document are subject to configuration control and may not be changed, altered, or



Produce Co-Added ExposuresID: DM-APP-DP-CO-1ID: DMS-REQ-0104


Each Exposure contributing to a Co-Added Exposure shall be projected onto a common reference frame using a projection selected from a set of options by a Policy.

[ACTION: harmonize with co-add pipeline requirement, enumerate the types of coadds required]

Co-Added Images Updated at Least Every 6 monthsID: DM-APP-DP-CO-2ID: DMS-REQ-0105


Co-Added Images shall be created from the latest image archive no less frequently than every 6 months and for every Deep Detection processing run.

Co-Added Image AtrributesID: DM-APP-DP-CO-3ID: DMS-REQ-0106


* registration information* PSF convolution information

2.3.2 Catalogs∑ Exposure∑ Object∑ MovingObject∑ Source∑ DIASource∑ FaintSource.Exposure is updated with the latest derived metadata, such as WCS and PSF.

2.3.2.1 Object Catalog – Level 2 Data

Astrometric ModelID: DMS-REQ-0107 Last Modified: 1/25/2009

Calibrated Light CurveID: DMS-REQ-0108 Last Modified: 1/25/2009

Calibrated Point Source PhotometryID: DMS-REQ-0109 Last Modified: 1/25/2009

Calibrated Position CurveID: DMS-REQ-0110 Last Modified: 1/25/2009


2023-05-15 of 46


Extended Object PhotometryID: DMS-REQ-0111 Last Modified: 6/16/2011

TBD

Level 2 Object AttributesID: DM-APP-DP-DO-1ID: DMS-REQ-0112


* extended object shape parameters* extended object photometry* photometric redshifts* calibrated point source photometry* fully sampled photometric time history* astrometric model

Object ClassificationID: DMS-REQ-0113 Last Modified: 6/16/2011

TBD

Photomeric Redshift PDFID: DMS-REQ-0114 Last Modified: 1/30/2009

Photometric RedshiftsID: DMS-REQ-0115 Last Modified: 1/30/2009

Photometric redshifts shall be calculated for all extended objects which have calibrated magnitudes in all filters.

Extended Object Shape ParametersID: DM-APP-DP-CA-10ID: DMS-REQ-0116


Weak lensing studies require the image moments (or equivalent) through N=2.

For weak lensing extended objects of size greater than TBD (10 arcsec?) do not require shape parameters to be measured

2.3.2.2 Source Catalogs - Level 2 Data

Low SNR Source AttributesID: DMS-REQ-0117 Last Modified: 6/16/2011

* Instrumental flux (ADU)* Expected error in instrumental flux (ADU)* Sky background level (ADU)


2023-05-15 of 46


* Expected error in sky background (ADU)* Source extraction flags, which identify any problems encountered by the DP in processing the Source

Source AttributesID: DM-APP-DP-CA-5ID: DMS-REQ-0118


* Focal plane position centroid (pixels)* Expected focal plane position centroid error (pixels)* Instrumental flux (ADU)* Expected instrumental flux error (ADU)* Sky background level (ADU)* Expected error in sky background (ADU)* Source extraction flags, which identify any problems encountered by the DP in processing the Source* Shape characterization, including at least an extendedness parameter* ICRS sky position derived from the focal plane position (radec)* Expected error in ICRS sky position (radec)

2.4Level 3 Data Products∑ Phase-folded light curves for periodic variables \item catalogs of specific subsets of objects∑ Maps of derived properties such as lensing shear∑ Catalogs of clusters∑ Catalogs of morphologically analyzed galaxies.An essential feature of Level 3 data products is that their creation is not a responsibility of the DMS. They will arise from user analysis projects. The generation of Level 3 data products may thus involve the use of code and query definitions from outside the LSST project, and that is not part of the project's open-source code base. When Level 3 data products are created as the output of analysis of LSST data, whether with the basic interactive user tools or via custom pipelines, tools will be provided so that they may be federated with the Level 1 and Level 2 datasets, so that joins may easily be made between data-release and user-provided tables.It is anticipated that certain of the Level 3 data products will be archived by the DMS, using project resources provided for this purpose. The allocation of these resources to archiving, and the duration of storage, will be determined by the LSST Project based on the value of the data to the community and the cost of recomputing the data versus persisting it. The DMS is further required to facilitate the archiving of Level 3 data products using external resources, by providing data import and export tools, and tools to assist external users in maintaining the consistency of large multi-file datasets.It is anticipated that some, but not all, Level 3 data products will be generated using the physical resources of the Data Access Centers (DACs) provided as part of the LSST project; others will be generated using external resources. These may be in the form of additional DACs externally funded, but configured and managed much like the internal ones, or may be truly external computing facilities brought to bear on specific LSST data analysis challenges. The DMS is thus required to facilitate the production of Level 3 data products by clients external to the DMS, both at LSST-controlled DACs and at external sites. This will be done by providing stable and well-documented APIs and libraries based on open-source software, freely downloadable to external sites, and by providing a modest level of user support from the project.

2.4.1 General Level 3 Data Product Requirements

2.4.1.1 DAC resource allocation for Level 3 processingID: DM-APP-DP-LTG-1ID: DMS-REQ-0119



2023-05-15 of 46


2.4.1.2 Level 3 Data Product Self ConsistencyID: DM-APP-DP-LTG-2ID: DMS-REQ-0120


The DMS shall provide a means for ensuring that users' Level 3 processing tasks are carried out on self-consistent inputs - i.e., catalogs, images, metadata, calibrations, camera configuration data, etc., that match each other and all arise from consistent Level 1 and Level 2 processings.

2.4.1.3 Provenance for Level 3 processing at DACsID: DM-APP-DP-LTG-3ID: DMS-REQ-0121


The DMS should also provide an optional means for Level 3 processing users at DACs to maintain basic provenance information on their own inputs to a processing task, such as code or additional calibration data.

Rationale: the DMS should facilitate Level 3 processing users in being able to carry out their work in a reproducible way.

2.4.2 Level 3 Catalog Processing Requirements

2.4.2.1 Access to catalogs for external Level 3 processingID: DM-APP-DP-LTC-2ID: DMS-REQ-0122


2.4.2.2 Access to input catalogs for DAC-based Level 3 processingID: DM-APP-DP-LTC-1ID: DMS-REQ-0123


The DMS shall provide access to all Level 1 and Level 2 catalog products through the LSST project's Data Access Centers, and any others that have been established and funded, for Level 3 processing that takes place at the DACs.

2.4.2.3 Federation with external catalogsID: DM-APP-DP-LTC-4ID: DMS-REQ-0124


2.4.2.4 Software framework for Level 3 catalog processingID: DM-APP-DP-LTC-3ID: DMS-REQ-0125


2.4.3 Level 3 Image Processing RequirementsThis section contains requirements on the Level 3 processing of images, separated from those for catalogs because they may in time diverge.


2023-05-15 of 46


2.4.3.1 Access to images for external Level 3 processingID: DM-APP-DP-LTI-2ID: DMS-REQ-0126


2.4.3.2 Access to input images for DAC-based Level 3 processingID: DM-APP-DP-LTI-1ID: DMS-REQ-0127


The DMS shall provide access to all Level 1 and Level 2 image products through the LSST project's Data Access Centers, and any others that have been established and funded, for Level 3 processing that takesplace at the DACs.

2.4.3.3 Software framework for Level 3 image processingID: DM-APP-DP-LTI-3ID: DMS-REQ-0128


2.5Calibration Data Products∑ Bias Exposures∑ Monochromatic Dome Flats∑ Broadband Dome Flats∑ Pupil Ghost Image ∑ Crosstalk Correction Matrix∑ Fringe Images∑ Illumination Correction∑ A variety of products required for the Auxiliary Telescope (TBD).

2.5.1 Calibration Products Requirements

2.5.1.1Crosstalk Correction Matrix AttributesID: DMS-REQ-0129 Last Modified: 6/22/2009

The crosstalk correction matrix will supply the coupling coefficient due to electronic crosstalk between the pixel value read out in each channel of the camera and the value read out in every other channel.

2.5.1.2 Calibration Data ProductsID: DM-APP-DP-CI-1ID: DMS-REQ-0130


* Monochromatic dome flats over a representative wavelength range* Sky flat* Bias frame* Dark frame* Fringe frame* Illumination correction frame* Crosstalk correction matrix


2023-05-15 of 46


2.5.1.3 Calibration Images Available 1 Hour Before ObservingID: DM-APP-DP-CI-2ID: DMS-REQ-0131


Calibration Images shall be available for nightly observing in the DMS image archive at least 1 hour prior to the start of observing.

2.5.1.4 Calibration Image AttributesID: DM-APP-DP-CI-3ID: DMS-REQ-0132


* The calibration image type (eg dome flat, superflat, bias, etc)* The filter to which it applies (not relevant for bias)* The range of raw image exposure dates for which the calibration image is valid

2.6Detection and Measurement of Objects

2.6.1 Deep Detection Processing

2.6.2 Difference Exposure ProcessingNote that this process cannot be perfect, since measuring the extendedness of objects near the PSF size will always be uncertain. Consequently, there will be cases where flux from a supernova or AGN point source will be incorrectly added to the underlying galaxy rather than to a new point source. Between successive Data Releases, however, these errors will decrease in number: as the survey goes deeper, and accumulates images in better seeing, extendedness will be better measured by the Multifit procedure.

2.6.3 Measuring the properties of ObjectsOur choice of models for Multifit is driven by astrophysics, by characteristics of the LSST system, and by computing practicalities, as follows:∑ Within the context of the LSST survey, clearly extended objects do not have measurable proper

motion or parallax. Comets are a clear exception, but they are processed only within difference exposures, not through Multifit. Supernova light echoes might also be an exception, and they deserve further thought.

∑ While co-added exposures largely erase the effects of the gaps between the individual CCDs, individual exposures are strongly affected by them. Processing of objects in individual exposures that extend across CCD gaps creates a significant overhead of programming complexity and computational cost, and will not initially be implemented in the DMS. This capability can be incorporated in later versions of the DMS if it is warranted by the science gain.

∑ Given the above constraint, Multifit is only useful if the object being fit is wholly contained within a single CCD field for the majority of the exposures in which it appears. Note that this does not mean that an object needs to appear in the same CCD in different exposures! This will rarely occur, given the survey's dithering pattern. If we want the object containment probability to be at least 0.8, the object size, d can be no larger than 0.1 D, where D is the angular size of the CCD on the sky, 13 arcmin in the case of LSST. This sets a natural upper limit on object size to be processed by Multifit of approximately 1 arcmin. We note that this size comfortably encompasses the objects required for the SRD's science drivers.


2023-05-15 of 46


2.6.3.1 Slowly Moving Point Source ModelThe SMPS Model will be fit only to objects which are leaf nodes in the segmentation tree.

2.6.3.2 Small Object ModelThe SO Model will be fit only to objects which are leaf nodes in the segmentation tree.

2.6.3.3 Large Object ModelA typical large object will be segmented into smaller component objects. The leaf objects in the resulting segmentation tree will be measured as described above, providing they qualify as ``small''. The large object itself, the root of the segmentation tree, will be represented only by its ``Footprint'', with the following attributes:∑ Ellipse equivalent to Object footprint (same moments through second order)∑ Footprint bounding box∑ Flux within footprint.Parameters resulting from model fitting, and from analysis of time dependent properties, will be absent. Fits of morphological models (eg bulge/disk) to large objects must be created as Level 3 data products.Objects larger than "large" will not be represented in the DMS.

2.6.3.4 Solar System ModelThe details of the FaintSource measurement process are not yet well defined. In particular, it is unclear if the measurements should be at a position entirely fixed by the orbit prediction, or should be allowed to compensate for prediction error by "peaking up" within some error bound around the prediction.

2.6.4 The Multifit AlgorithmThe library of model types is described in the preceding subsections. An initial model will be fit to the co-add, to provide a good starting point for the fit to the full dataset. Multifit will then read in all the pixels from the $n$ exposures and perform a maximum likelihood fit for the model which, when convolved with the $n$ PSFs, best matches the n observations. This naturally incorporates the effects of varying seeing, as the contribution of the better-seeing images to the likelihood will be sharper. This approach also facilitates proper accounting for masked areas, cosmic rays, etc. The best-fit model parameters and their uncertainties will be recorded in an Object table row.Slightly simplified versions of this fitting approach have already been used in major surveys. 2MASS used it to fit point sources to their multiple-epoch observations which had nontrivial seeing variations. SDSS uses it to fit a model simultaneously to the ugriz images. The extension here is to fit simultaneously across wavelengths and across a large number of multiple-epoch observations with seeing variations.

2.6.5 Model Residuals1. An Example: Supernova in a Visible GalaxySuppose that a supernova explodes in a ``small'' galaxy that is clearly resolved in LSST imagery, and is already listed in the Object table from a previous Data Release. Suppose further that the supernova is bright enough that it will be above the detection threshold in at least one difference exposure. The following sequence of events will occur during the nightly Alert Production processing:∑ The first time the supernova is detected above threshold in both difference exposures from a Visit, the

Association Pipeline (AP) will attempt to match the resulting DIASources to an Object in the Object table. In this case, it will find that each DIASource is contained within the footprint of its host galaxy, but based on the fact that the galaxy is extended, and the DIASources are not, the AP will create a new Object (SN) at the position of the supernova.

∑ An Alert will be issued for the supernova.


2023-05-15 of 46


∑ On subsequent Visits, as long as the supernova remains above the detection threshold, new DIASources will be created from each Exposure, and the AP will associate them to the SN object. A query to the science database will readily retrieve all difference image photometry for SN from the DIASources linked to the SN Object entry.

∑ A likely policy for the survey to follow will be to add every Object that results in an Alert to a list of Objects to be force photometered in Difference Exposures. If this is done, the SN will result in FaintSource entries even after it has dropped below the detection threshold above which it would generate DIASources. This will allow a query to the science database to retrieve this photometry as well.

When it is time to create a new Data Release, a new, empty, science database is created. It is populated with the Raw Exposures from the survey extending from the inception of the survey up until the cutoff date for the DR, but nothing else. Note in particular that none of the other Level 1 data products created by the Alert Production are imported into the DR. Processing always begins from scratch. The following sequence of events involving the supernova will then occur during Data Release processing:∑ As with all sky patches, a subtraction template co-add for each filter is created for the patch of sky

containing the supernova by combining all the survey exposures that cover that patch. The combination algorithm strongly discriminates against transient flux, eg with a median. The template co-add will therefore contain an image of the galaxy uncontaminated by the supernova.

∑ A detection co-add is created for the patch of sky containing the supernova by combining all the survey exposures that cover that patch. The combination algorithm will discriminate against single visit transients, eg from solar system objects, but otherwise optimally combines the flux from all exposures, eg with the Kaiser algorithm. The detection co-add therefore contains the galaxy with the superimposed supernova. The brightness of the supernova in the co-add depends on the fraction of the total co-add duration where the supernova has significant flux (so it becomes progressively fainter in subsequent DRs).

∑ Object detection and deblending is run on the detection co-add. There are two cases to consider:∑ The supernova is bright enough in the co-add to trigger deblending into a galaxy plus a point source

(Case A). Three entries are made in the Object table: Gal+SN, Gal, and SN. The Gal+SN object is the root of the segmentation tree, while Gal and SN are the deblended leaves.

∑ The supernova is faint enough in the co-add that deblending is not triggered. A single object is detected, with the galaxy flux slightly distorted by the supernova flux (Case B). A single entry is made in the Object table: Gal*. Gal* is not deblended, so it is a leaf node in the segmentation tree.

∑ Multifit is run on the exposure stack for the Objects entered into the table that are leaf nodes in the segmentation tree, two objects for Case A, one for Case B. Given that Gal is ``small'', both the SO and SMPS models will be fit for these objects. Let us consider Case A and B separately:

∑ For Case A, Multifit is run separately on Gal and SN, fitting the SO and SMPS models for each. The model residuals should clearly favor the SO model for Gal and the SMPS for SN, but note that the model parameters for both Gal and SN will be distorted by the presence of the other. For example, the position of the SN/SMPS will appear to vary with time, since at low flux levels its profile will be significantly affected by the galaxy, but less so at high flux levels. The Gal/SO position is not allowed to vary with time, by definition of the SO model, but its flux will vary due to the presence of the SN. [This argues that it is desirable to run Multifit simultaneously on all models that overlap, or at least iteratively, subtracting models from the images. See Section 6.7] As a result of running Multifit, Source entries will be generated for Gal at each epoch in the stack. Source entries will be generated for SN at each epoch in the stack where its SNR exceeds some detection threshold. FaintSource entries will be created for Gal and for SN at each epoch in the stack from forced photometry with the model shape and position.

∑ For Case B, Multifit fits both the SO and SMPS models to Gal*. The model residuals for the individual exposures will likely vary greatly with time. If the Gal* was not deblended simply because the supernova occupied only a small fraction of the co-add time range, both SO and SMPS models will show large residuals, and in the case of SMPS, spurious motion as well. Both Source and FaintSource entries will be generated for Gal* at every epoch.

∑ With the Multifit procedure complete, the difference exposures are formed by psf-matching and


2023-05-15 of 46


subtracting the template exposure from each exposure in the exposure stack. By hypothesis, the supernova exceeds the detection threshold in at least some of these exposures. For each exceedance, a DIASource is created, and an attempt is then made to match it with an object already in the Object table just created by Multifit. Again, the outcome is different for Case A and Case B:

∑ For Case A, the DIASources will match the SN object, and will be given that Object Id.∑ For Case B, the first DIASource will not match Gal* because the supernova is a point source, while

Gal* (by hypothesis) is measurably extended (it is also unlikely to match in position, but it may). Therefore, a new Object is created for that DIASource, call it SN*. All subsequent DIASources will then be associated to SN*.

For both Case A and B, FaintSource entries will be created for every epoch in the stack by forced photometry on the difference exposures.When the processing of the Data Release is complete, the following information will be available for the supernova and its underlying galaxy:∑ Entries in the Object table for both the supernova and the galaxy. For Case A, the model parameters

reported for the supernova and the galaxy may each be significantly biased by the presence of the other.

∑ A full lightcurve for the supernova measured in difference exposures and reported in the FaintSource entries associated with the supernova object.

∑ A full set of Sources for the galaxy, each of which quantifies the residual between the model and the exposure from which the Source was measured.

∑ For Case A, a full set of Sources for the supernova, each of which quantifies the residual between the model and the exposure from which the Source was measured.

∑ A partial set of DIASources for the supernova, limited to those epochs where it is brighter than the detection limit.

Note that the case of a time variable AGN embedded in a small galaxy is broadly parallel to the supernova case considered above. Unlike the supernova case, however, where the FaintSources from difference exposures give a nearly optimal extraction of the full light curve, in the AGN case, only the "AC" part of the AGN light curve will be so measured. Some "DC" part of the AGN lightcurve will be part of the flux associated with the galaxy.

2.6.6 Limitations and Possible Extensions

2.6.6.1 Faint Transients1. The object is above the detection threshold in the co-add used for object detections and/or2. The peak brightness is above the detection limit for difference images.We can imagine a case where condition 2 is not satisfied, but the luminosity peak is covered by enough LSST exposures that a co-add concentrated near the peak satisfies condition 1. If the peak occurs near the beginning of the survey, the transient will be measured in DR1, but conceivably missed in subsequent DR's, because the significance of the peak in the co-add will become progressively less as more exposures from the quiescent phase are added. This behavior is certainly undesirable, and will have an undetermined negative impact on transient science. It could be overcome by a strategy which created a variety of co-adds spanning different time intervals. This would clearly create significant additional computing costs, and is not currently planned. This decision can be revisited if there is a compelling science case.

2.6.6.2 Faint Solar System Objects

2.6.6.3 Possible Multifit Improvements


2023-05-15 of 46


2.7Inserting Synthetic ObjectsThe strategy for this is still under discussion. An outline of the current strawman design:∑ LSST DM will provide a general facility to add synthetic objects to existing LSST exposures, fully

taking into account the observed PSF and sky background for each exposure.∑ Science teams that wish to ascertain their detection efficiency for a particular class of objects will

supply a model for the object class.∑ One or more ``mini-DR''s will be created using the normal Data Release procedures, with appropriate

synthetic objects added to the input exposures. Separate tables describing the synthetic objects will be maintained within the Science Database. A mini-DR will span a limited area of sky and/or time so that the computational requirements are tractable.

∑ The Science Database for each mini-DR will be maintained as a completely separate database from those of any other DR.

2.8Data Access

2.8.1 Computing Resources

2.8.2 Data Access ToolsQueries will be analyzed before their dispatch to DAC computing resources. Simple queries are expected to be able to be carried out with sub-minute latency. More complex or I/O intensive ones, up to and including ones requiring full table scans, will be bundled together for efficiency and scheduled for execution on a regular basis---hourly, in the present plan. The complexity and resource requirements of such queries have in the past been found to follow a power law. Based on the present analysis of expected queries, we expect to be able to support all but the most CPU-intensive queries on the project-provided DAC resources.The Data Management Functional Requirements Specification already anticipates the existence of a class of analyses and data products (Level 3) that will be carried out and obtained beyond the scope of these tools and of the required public data products and associated pipelines that are included in the project baseline (Level 1 and Level 2). Generally these will be analyses requiring bulk access to the data. Data Management is required to facilitate such custom pipeline analyses, including ones involving proprietary science algorithms. This will be done by providing a set of interfaces and tools that can be used at the DACs, with a modest level of user support provided by the project. These are intended to be the same tools that are used for the project-required Science Data Quality Analysis activity. These tools include both the APIs required to access image and catalog data within the LSST pipeline framework, and the ``steering'' tools required to ensure that such analyses may be carried out across the full dataset and to record the provenance information required to make such analyses reproducible.As noted, it is possible that the volume of data analysis work will exceed the baselined capacity of the DACs, so that external facilities will be required to carry out the full science program. To facilitate this, the custom pipeline interfaces and tools will be open source and will also be made available for download to computing sites beyond the project-provided DACs. These tools will be designed according to industry standards, so as to be portable across a reasonable range of flavors of Unix, and are planned to have dependencies only on freely available, open-source packages.Data Management obviously cannot provide quality assurance over the scientific content of user analyses, but the DM-provided tools will allow users to maintain technical quality control of their analyses, e.g., ensuring that they have run on complete and self-consistent inputs and associated calibrations. Users will be able to provide quality control metrics from their analyses and will be able to aggregate them over all their processing of the full dataset to monitor the progress of their pipelines.


2023-05-15 of 46


2.9Incorporation of User Analysis ModulesIt would be beneficial to LSST and all its users to streamline the process of code contribution, with special attention given to low-level basic tools that might be used by many groups. Data Management will work to facilitate this by providing well-documented open-source interfaces, programming standards, and software quality metrics that external users can rely on in constructing their own pipelines and tools. Contributions which meet these standards will be eligible for inclusion in the project's code base. Data Management will provide a modest level of user support to assist users in meeting these requirements.There are some constraints, however: for instance, if contributed code brings in new external libraries or otherwise expands the dependencies of the code base, it may not be possible to include it. In addition, LSST may not be able to provide staff to support non-trivial external code contributions in the long term. We will encourage user groups to provide such support for their code, and will generally be willing to assist in this by making available the same support tools (such as for documentation and problem tracking) that we use in-house. User groups will be expected to produce documentation of the use and science data quality of contributed data products and code, and to take responsibility for its robustness when applied at LSST scales.A possible final stage of the incorporation of contributed code would be for it to become part of the standard LSST data production and used to generate public data products centrally. We expect that this will turn out to be desirable at some point in the life of the project for some subset of contributed code. There are important caveats: this will not be possible if it significantly increases the CPU, memory, or archival storage requirements of the system beyond its baseline, unless those costs are covered in some way at the time of the decision. It will also require an explicit agreement on the part of the contributors to support their code for the long term or find funding to increase the central support resources. Finally, any contributed code incorporated into production will have to be demonstrated to be highly reliable and must be integrated in the standard SDQA system, with the contributor responsible for defining and implementing the necessary metrics. For non-trivial contributions a peer review of the quality assurance plan would be advisable.

3 Middleware Layer

3.1Middleware Layer Requirements

3.1.1 Provide Data Access ServicesID: DM-MW-1ID: DMS-REQ-0155


* Transparent access to pipeline input data regardless of execution and storage location* Transparent access to all output data products across LSST sites* Automated ingestion of data products * Data staging and transfer * Services for data access into files, structured objects, and database query results* Secure access control to all data* Services to download any of the released LSST data products, from raw to stacked, as desired, subject to the users’ computing capabilities (e.g., network bandwidth, storage)* Services to submit queries against all LSST catalogs and receive results in a research-ready form* Services to track the provenance of a data object, such as a transient detection in a catalog, back through the processing used and to the original raw data from which it was generated* Services to manage the replication of data across LSST facilities as needed.

3.1.2 Provide Pipeline Execution ServicesID: DM-MW-2 Last Modified: 8/3/2011


2023-05-15 of 46


ID: DMS-REQ-0156

* Services for parallel processing on high-performance platforms, including distributed-memory and shared-memory architectures* Real-time monitoring of execution of pipelines* Automated detection of and recovery from faults.

3.1.3 Provide Management and Control ServicesID: DM-MW-3ID: DMS-REQ-0157


* Services for re-configuring and redeploying pipelines, in particular, change where the particular processing takes place without manually rebuilding the pipeline.* Services to allow a pipeline to be paused in order to change module parameters.* Services to allow a pipeline to be restarted without extensive repetition of previously executed processing steps, as a new instance of processing with associated provenance.* Services to allow portions of pipelines to be executed on selected sets of data without major reconfiguration.* Component interfaces that allow both local and secure, remote control of the data management system* Adaptive and prioritized process management/control/scheduling services* Workflow management* Monitoring, error handling, and failure recovery services* Secure access control to pipelines and data

3.1.4 Provide Pipeline Construction ServicesID: DM-MW-4ID: DMS-REQ-0158


* Services to statically (i.e. not during execution) insert a new module in an existing process flow without recompilation of modules whose interface has not changed.* Services to perform both composition-time and run-time type checking of modules and data structures for compatibility.* Services to estimate the processing and storage workload represented by a given pipeline configuration and dataset to be processed.* Services to configure a data-driven strategy for use by pipeline execution management.* Services to integrate user-generated query or analysis modules into the LSST processing infrastructure at LSST facilities, subject to security restrictions and available LSST resources.* Services to perform TBD analysis operations remotely

3.1.5 Provide Data/Catalog Construction ServicesID: DM-MW-5ID: DMS-REQ-0159


* Services to define new data types, including new science-related entities such as object classifications, new meta-data entities, and new data relationships.* Services to extend catalogs to include new data types and concepts* Services to integrate new types of storage hardware* Services to create queries and searches for new data access patterns* Services to define new data access control policies

3.1.6 Provide User Interface ServicesID: DM-MW-6ID: DMS-REQ-0160



2023-05-15 of 46


* Services and tools to browse LSST data products through TBD astronomical views or visualizations* Services to create and serve "best" images of selectable regions of the sky in a TBD form* Services to resample and re-project images to TBD* Services to co-add LSST with external images [ I don't think this is useful - TSA]

4 Infrastructure Layer

4.1Infrastructure Layer Requirements

4.1.1 General Infrastructure RequirementsThe requirements in this section apply to all Infrastructure elements.

4.1.1.1 Optimization of Cost, Reliability and Availability in OrderID: DM-INF-1ID: DMS-REQ-0161


The allocation of processing and storage to facilities will be done to optimize (in priority order) cost, reliability, and availability of the total data management subsystem.

4.1.1.2 Pipeline throughputID: DM-INF-2ID: DMS-REQ-0162


4.1.1.3 Re-processing CapacityID: DM-INF-3ID: DMS-REQ-0163


The infrastructure will be sized such that complete re-processing of the entire raw image dataset may occur once per year without interrupting observatory operations.

4.1.1.4 Temporary Storage for Communications LinksID: DM-INF-4ID: DMS-REQ-0164


The infrastructure will provide for temporary storage for a minimum of 200% of the mean time to repair of any communications network link at the source end of that link.

4.1.1.5 Infrastructure Sizing for "catching up"ID: DM-INF-5ID: DMS-REQ-0165


4.1.1.6 Incorporate Fault-ToleranceID: DM-INF-6ID: DMS-REQ-0166


The infrastructure will incorporate as fault-tolerance features to prevent loss of data in the event of


2023-05-15 of 46


hardware or software failure.

4.1.1.7 Incorporate AutonomicsID: DM-INF-7ID: DMS-REQ-0167


4.1.2 Mountaintop Site Requirements

4.1.2.1 Mountaintop Site Data CommunicationsID: DM-INF-MS-1ID: DMS-REQ-0168


The DMS shall provide data communications infrastructure to accept science and associated metadata read-outs and transfer to the nightly processing pipelines.

4.1.2.2 Mountaintop Site Temporary StorageID: DM-INF-MS-2ID: DMS-REQ-0169


The DMS shall provide four nights (two nights, redundantly) of raw data storage on the Mountaintop Site in the event of Mountain to Base network outages.

4.1.2.3 Prefer Computing and Storage DownID: DM-INF-MS-3ID: DMS-REQ-0170


The DMS computing and storage equipment will be preferentially located at Base Facility versus the Mountaintop Site due to lower support costs and fewer reliability issues (lower altitude), therefore any processing that can be done in either location will be allocated to the Base Facility.

4.1.3 Mountain to Base Network Requirements

4.1.3.1 Mountain to Base NetworkID: DM-INF-MB-1ID: DMS-REQ-0171


The DMS shall provide communications infrastructure between the Mountaintop Site and the Base Facilitysufficient to carry scientific data and associated metadata for each image in <= 6 seconds.

4.1.3.2 Mountain to Base Network AvailabilityID: DM-INF-MB-2ID: DMS-REQ-0172


The Mountain to Base communications shall be highly available, with Mean Time Between Failures (MTBF) > 90 days, measured over a one-year period.

4.1.3.3 Mountain to Base Network ReliabilityID: DM-INF-MB-3ID: DMS-REQ-0173



2023-05-15 of 46


The Mountain to Base communications shall be highly reliable, with Mean Time to Repair (MTTR) < 24 hours, measured over a one-year period.

4.1.3.4 Mountain to Base Network Secondary LinkID: DM-INF-MB-4ID: DMS-REQ-0174


4.1.3.5 Mountain to Base Network Ownership and OperationID: DM-INF-MB-5ID: DMS-REQ-0175


The Mountain to Base communications link shall be owned and operated by LSST and/or the operations entity to ensure responsiveness of support.

4.1.4 Base Facility Requirements

4.1.4.1 Base Facility InfrastructureID: DM-INF-BF-1ID: DMS-REQ-0176


The Base Facility shall provide sufficient computing, storage, and network infrastructure to support nightlyprocessing, including image processing, detection, association, and moving object pipelines, the generation of all time-critical data products, i.e. alerts, and the assessment of nightly data quality.

4.1.4.2 Base Facility Temporary StorageID: DM-INF-BF-2ID: DMS-REQ-0177


The Base Facility shall provide at least four nights (two nights, redundantly) of raw data storage in the event of Base to Archive Center network outage.

4.1.4.3 Base Facility Co-Location with Existing FacilityID: DM-INF-BF-3ID: DMS-REQ-0178


The Base Facility shall be co-located at an existing facility to leverage existing support and facility resources

4.1.4.4 Base Facility Data Quality AssessmentID: DM-INF-BF-4ID: DMS-REQ-0179


The Base Facility shall provide computing infrastructure sufficient to perform nightly data quality assessment on at least 10% of the nightly data, to ensure optimal observatory operations.

4.1.5 Base to Archive Network Requirements

4.1.5.1 Base to Archive Network


2023-05-15 of 46


ID: DM-INF-BA-1ID: DMS-REQ-0180


The DMS shall provide communications infrastructure between the Base Facility and the Archive Center sufficient to carry scientific data and associated metadata for a given night before the start of observing preparation for the next night.

4.1.5.2 Base to Archive Network AvailabilityID: DM-INF-BA-2ID: DMS-REQ-0181


The Base to Archive communications shall be highly available, with MTBF > 180 days, measured over a one-year period.

4.1.5.3 Base to Archive Network ReliabilityID: DM-INF-BA-3ID: DMS-REQ-0182


The Base to Archive communications shall be highly reliable, with MTTR < 48 hours, measured over a one-year period.

4.1.5.4 Base to Archive Network Secondary LinkID: DM-INF-BA-4ID: DMS-REQ-0183


4.1.5.5 Base to Archive Network LeaseID: DM-INF-BA-5ID: DMS-REQ-0184


The Base to Archive communications shall be leased from a public or commercial entity, due to high cost of deployment and support for long-haul links.

4.1.6 Archive Center Requirements

4.1.6.1 Archive CenterID: DM-INF-AC-1ID: DMS-REQ-0185


The Archive Center shall provide computing, storage, and network infrastructure to support pipeline processing and reprocessing, permanent storage for all data products (with provenance), and serve data for replication to data centers and end user sites.

4.1.6.2 Archive Center Disaster RecoveryID: DM-INF-AC-2ID: DMS-REQ-0186


4.1.6.3 Archive Center Co-Location with Existing FacilityID: DM-INF-AC-3ID: DMS-REQ-0187



2023-05-15 of 46


The Archive Center shall be hosted at an existing NSF/DOE-funded supercomputing center.

4.1.7 Archive to Data Access Center Network Requirements

4.1.7.1 Archive to Data Access Center NetworkID: DM-INF-AD-1ID: DMS-REQ-0188


The DMS shall provide communications infrastructure between the Archive Center and Data Access Centers sufficient to carry scientific data and associated metadata in support of community and EPO access. Aggregate bandwidth for data transfers from the Archive Center to Data Centers shall be at least 10 Gbps.

4.1.7.2 Archive to Data Access Center Network AvailabilityID: DM-INF-AD-2ID: DMS-REQ-0189


The Archive to Data Access Center communications shall be highly available, with MTBF > 180 days, measured over a one-year period.

4.1.7.3 Archive to Data Access Center Network ReliabilityID: DM-INF-AD-3ID: DMS-REQ-0190


The Archive to Data Access Center communications shall be highly reliable, with MTTR < 48 hours, measured over a one-year period.

4.1.7.4 Archive to Data Access Center Network Secondary LinkID: DM-INF-AD-4ID: DMS-REQ-0191


4.1.7.5 Archive to Data Access Center Network LeaseID: DM-INF-AD-5ID: DMS-REQ-0192


The Archive to Data Access Center communications shall be leased from a publicly funded research network entity, due to high cost of deployment and support for long-haul links.

4.1.8 Data Access Center Requirements

4.1.8.1 Data Access CentersID: DM-INF-DC-1ID: DMS-REQ-0193


The Data Access Centers shall provide computing, storage, and network infrastructure to support open access to LSST data products (with provenance) by end users.

4.1.8.2 Data Access Center Simultaneous Connections


2023-05-15 of 46


ID: DM-INF-DC-2ID: DMS-REQ-0194


At least 300 simultaneous connections shall be supported at each Data Access Center.

4.1.8.3 Data Access Center VO StandardsID: DM-INF-DC-3ID: DMS-REQ-0195


All Data Access Centers shall provide access to LSST data products via public interfaces and services in accordance with IVOA/NVO standards.

4.1.8.4 Data Access Center Geographical DistributionID: DM-INF-DC-4ID: DMS-REQ-0196


The Data Access Centers will be hosted at facilities selected in at least the U.S. and the observatory host country to permit widest possible access to LSST data with the fewest possible network hops.

4.1.8.5 No Limit on Data Access CentersID: DM-INF-DC-5ID: DMS-REQ-0197



2023-05-15 of 46

docushare.lsst.org€¦ · Web viewThe LSST Science Requirements Document specifies a set of...

Documents

Transcript of docushare.lsst.org€¦ · Web viewThe LSST Science Requirements Document specifies a set of...