Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April...

Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009

Improved and automated QC process

Reinhard Hanuschik and the DPQC group

• fast data transfer: July 2008• full data stream VLT and VLTI (ex burst-mode data)• > 50% within 10 minutes

• new ways in data processing @ DPQC• impact on shared QC

• 2 main goals:• calibration completeness • calibration quality


Mission #1: calibration completeness (“save last night”)

• calibrations incomplete: science OBs to be repeated• focus on calibration plan:

– known, predictable, configured

– can be checked automatically

calChecker

1. Setting the stage


Mission #2: calibration product quality (“save next night”)

combining Health Check monitor and scoresscores:

quick-look HC monitor


data acq (Paranal)

data in Garching

opslog

• any problem not discovered on site: up to 10 days affected• strong need for 2 QC processes:

daytime astronomer Paranal (preliminary),QC scientist Garching (final)

• basic real-time QC with opslog data

(after ~ 10 days)

until 2008:


data acq (Paranal)

data in Garching

now (2009):• fast data delivery

• fast QC feedback

• improved QC feedback– measure quality (QC1 parameters) – compare quality (trending)– rate quality (scoringscoring) new!


• www.eso.org/qc/ALL/daily_cal.html• evaluation for 7 last days (columns) and all science setups (rows)• driven by science data• checks:

– all science setups for a given instrument

– checks calibration plan

– knows about validity and required number of files

life examples

2. calChecker


• evaluates each box in colors:– green (OK): all calibrations complete and within validity

(e.g.: 5 BIAS & 3 FLATs & 1 ARC & 1 STD within validity)

– yellow (NOK): calibrations complete but some outdated(e.g.: as above but 3 FLATs are 2 days old)

– red (MISS): calibrations incomplete, at least one missing(e.g.: 5 BIAS & 3 FLATs ok, 1 ARC & 1 STD missing)

• missing or outdated calibrations are indicated


• links to details– for each box: complete list of science files

– prog_id, obs_id etc. for easy reference

– OB grade and comment: useful for analysis


• ANALYSIS– QC scientist to provide analysis/comments,

but this requires interaction ( not always available!)

– may override tool NOK/MISS results

• BOCAL, EOCAL– signal about daytime calib queue

– calChecker displays “pending”, “ongoing”, “finished” for current date


• called: once per hour by cronjob– has update information, knows about its age

don’t hesitate to contact QC!


• documentation• association rules• contact

• planned: – include ticket mechanism (PSO scheme)

– memory function for long-term calibrations (once a month etc.) and HC calibrations


• general rule: calChecker follows calOBbuilder

• not planned:– short-term night calibrations (like standard stars within 2 hrs)

– complex dependencies (multiple standard stars a night; alternative calibrations; calibrations for calibrations)

• to be done on Paranal:– optimize calOBbuilder queue (input provided by QC)


• goes beyond completeness, includes quality• links to the QC processing page for that date (AB product monitorAB product monitor)

3. Links to product quality

calChecker HC monitor

products/AB monitor


AB product monitor


Scores

automatic evaluation scheme:

• 2 thresholds defined

• QC1 parameter evaluated

• total AB score: 0 / 1 / more than 1

• counts outliers


• links to other reports– data report

– nightlog (password protected, sciops)


• www.eso.org/HC

• final QC Garching data now display as quick as/quicker than opslog!(forget 10 day gap…)

4. Quick-look HC monitor


products/AB monitor


full version

score version

(Quick-look)


• Information on demand:

click on “more” if you want graphs, on “less” if you want scores only


• HC monitor: selected scores • selection: “relevant for instrument health”

• AB product monitor: all scores of a date


score propagation:– data scores for last week only

– plot: last data scores summed upplot score green if last data score is green (previous red scores ignored)

• report score: plot scores are aggregated in report score• report scores in group scores, and finally in instrument score

• all higher-level scores turn red if at least one lower-level score is red


• big advantage of score hierarchies:– instrument score green you know within a secondwithin a second: all is OK!

– red: the score overview can tell you more

• scheme intended to guide you to issues


• Health Check monitor– update pattern: refresh most important ones upon new opslog data

– less important ones at least once per 24 hour

• process health– monitored: opslog transfer, parser (1x per hour)

– coming soon: data transfer, ngas access


• planned: – defocus the OPSLOG data

(to become a fallback solution except for VIRCAM/OCAM)

– improve update pattern for HC reports (immediate after new data)

– product quality links (same as for calChecker)

– agree on mail alert mechanism



products/AB monitor


• the end …


Other QC services within this context

• Agora: FAQ page for PSO-QC related issues• refs: reference frames web page• QC: tutorial pages, documentation


• avoid being under-sensitive:– overlook that raw file is saturated or empty

– miss important properties = QC1 parameters

• avoid being too sensitive:– have too narrow thresholds

– have unstable thresholds (3SIG)

– use not fully understood parameters

– select data sets (setups) rarely acquired ( trending impossible or uncertain)

– use same QC1 parameter in various HC plots ( multiple alarms)

• rule of thumb: – every red score should have an explanation

– likely trigger an action

[Excursion: The art of scoring … ]

[E. Munch, Cry]


5. Operational impact

daytime astronomer: check completeness– take action for ylw/red boxes:

e.g. trigger calibrations

check Health Check monitor– check scores– pick red ones– analyze problem– trigger actions

check product quality (AB monitor)

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April...

Documents

Transcript of Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April...