TRAP (transient detection pipeline) status update

23
TRAP STATUS UPDATE TRAnsients Pipeline Gijs Molenaar [email protected] @gijzelaerr Thursday, July 11, 13

description

These are the slides from the talk I gave at the 'Radio Transients with SKA Pathfinders and Precursors' conference at Kruger Park, South Africa. 9-12 July 2013

Transcript of TRAP (transient detection pipeline) status update

Page 1: TRAP (transient detection pipeline) status update

TRAP STATUS UPDATETRAnsients Pipeline

Gijs Molenaar

[email protected]

@gijzelaerr

Thursday, July 11, 13

Page 2: TRAP (transient detection pipeline) status update

ABOUT TRAP

•TRAnsients Pipeline

•Detect and classify transients in multi-frequency radio sky image time series

• Emit VOevents

• 99% Python

Thursday, July 11, 13

Page 3: TRAP (transient detection pipeline) status update

STEPS

Thursday, July 11, 13

Page 4: TRAP (transient detection pipeline) status update

A LOT HAPPENED

• Version 1.0 imminent

• Focused on code quality and performance

•No big new science features

Thursday, July 11, 13

Page 5: TRAP (transient detection pipeline) status update

PERFORMANCE

• A lot faster

• Really a lot faster

• 0.85 image per second per core

• Scales well

minutes

Thursday, July 11, 13

Page 6: TRAP (transient detection pipeline) status update

RSM CYCLE0 RUN0

• 3402 images

• processing record - 5:21 min

• 2 machines, 36 cores

• 5645 unique sources

• 667 detected transients

• previous version: 400 min on 40 cores

Thursday, July 11, 13

Page 7: TRAP (transient detection pipeline) status update

TRAP & AARTFAAC

• AARTFAAC

• 48 images/s

• 57 (real) cores required

• 1 or 2 big fat systems will do!

Thursday, July 11, 13

Page 8: TRAP (transient detection pipeline) status update

INSTALLABILITY

•Merged TKP into TRAP

• Almost open source

• Easy database setup

• Remove many dependencies

• Like Lofar System Software (closed source)

Thursday, July 11, 13

Page 9: TRAP (transient detection pipeline) status update

QUALITY CONTROL

• Automated rejection of bad images

• Known bright source in FOV

• RMS x times higher than theoretical noise

• oversampled / undersampled / highly elliptical

Thursday, July 11, 13

Page 10: TRAP (transient detection pipeline) status update

STORAGE

• Added support for PostgreSQL

• fast with small datasets

•Many off-the-shelf tools available

Thursday, July 11, 13

Page 11: TRAP (transient detection pipeline) status update

UNDER THE HOOD

• Switched to celery

• asynchronous job queue

• based on distributed message passing

•No more cuisine

Thursday, July 11, 13

Page 12: TRAP (transient detection pipeline) status update

WHY CELERY

• Easier to use / install / debug

• Faster - hot processes

•Many off-the-shelf tools

•CEP1 compatible

• Easy to add compute nodes

Thursday, July 11, 13

Page 13: TRAP (transient detection pipeline) status update

Thursday, July 11, 13

Page 14: TRAP (transient detection pipeline) status update

DISCO?

•Maybe add support for Disco in the future

• Similar

• Map - Reduce

•Hadoop for Python

•Distributed file system

Thursday, July 11, 13

Page 15: TRAP (transient detection pipeline) status update

USABILITY

• tkp-manage.py

• Pipeline management tool

• Inspired by Django manage.py command

• Easy to

• setup pipeline

• add and run jobs

• run celery workers

• Add new commands

Thursday, July 11, 13

Page 16: TRAP (transient detection pipeline) status update

DEMO?

Thursday, July 11, 13

Page 17: TRAP (transient detection pipeline) status update

SUPPORTED TELESCOPES

• Support for FITS and CASA tables

• field parsers for LOFAR

• Possible to add telescope specific field parsing and quality checks

•ThunderKAT next week

Thursday, July 11, 13

Page 18: TRAP (transient detection pipeline) status update

PROJECT CLEANUP

• removed 40% of code

• 80% unit tested

• Added jenkins build server

• Performance regression tests

• Pull request/review work flow

• hipchat for central communication

Thursday, July 11, 13

Page 19: TRAP (transient detection pipeline) status update

WEB INTERFACE BANANA

•New web interface

•Rewrite of TKP-web

• Future ready

• Scientist friendly

Thursday, July 11, 13

Page 20: TRAP (transient detection pipeline) status update

Thursday, July 11, 13

Page 21: TRAP (transient detection pipeline) status update

DEMO?

Thursday, July 11, 13

Page 22: TRAP (transient detection pipeline) status update

FUTURE WORK

• More stable releases

• Add support for non-LOFAR data

• More quality checks

• Source storage and association performance

• Distributed file system

• Automated classification

• Web based data exploration

Thursday, July 11, 13

Page 23: TRAP (transient detection pipeline) status update

QUESTIONS

[email protected]

@gijzelaerr

Thursday, July 11, 13