1. LOFAR & PythonGijs Molenaar gijs@pythonic.nl

2. Agenda LOFAR AARTFAAC / TKP Python MonetDB 3. Non Python stuff 4. What is LOFAR 5. What is LOFAR Low-Frequency Array Distributed sensor network No dishes Simple and cheap antennas Omni-directional Concurrent observations Software telescope Flexible Requires supercomputer 6. What is LOFARBiggest Radio telescopearray in the world 7000 antennas 10 to 80 MHz (LBA) 120 to 240 MHz (HBA) Multiple small stations NL, DE, UK, FR, SE New (unique) design 7. New design Many unsolved problems Multi-terabit/s data scale How to Process Store Distribute Make results useful Small scale example for SKA 8. Usage Epoch of re-ionization Cosmic rays Extragalactic surveys Transients Pulsars 9. Correlation 10. Correlation IBM blue Gene/P Groningen Alotta computer power Will be replaced withFPGAs (for ourpurpose) No Python 11. Example output Cygnus A 12. Actual data 13. Transient Key Project Static sources are boringTransient something that changes Detect sourcesStore extracted informationAssociate with existing observationsDid something change?This is where the Python fun starts! 14. The glue Distributing computation Image processing Source extraction Source association Database interactions 15. Why Python A lot of existing code Good (scientific) libraries Good astronomical libraries Easy to use Get it working first, then fast NumPy is quite fast when used correctly 16. What do we use? Python NumPy / SciPy Django MonetDB Pyrap (casacore wrapper) Does all astro shizzle you can imagine 17. Distributed Computation Home made (ASTRON)SSH basedCalling remote scripts Shared code base (NFS) Protocol for passing arguments and resultsUses logging module a lotDifficult to debug Difficult to profileHadoop probably better 18. The Lofar Cluster 100 nodes 24 cores each 20 TB raid5 storage 40 Gbps InfiniBand But for AARTFAAC we build our own Eventually 19. Main loop Extract source properties Store properties Do association Detect transients 20. The data 1 datacube per second 10 frequency bands In the future 10 images per second In the future 4 different polarization Non stop 21. Source extraction 22. Source extraction & NumPy Calculate average noise Subtract Find islands Fit gauss 23. Store the data 24. Database Move calculation to the data Independent data Naturally separable by sky coordinates ~100 TB/year 10.000 insert/second 25. MonetDB Dont believe the hype NoNoSQL Relational Column store DB Fast Auto tuning! Developers next door (CWI) Cross research funding 26. INSERT INTO tempbasesources(xtrsrc_id,datapoints,I_peak_sum,I_peak_sq_sum,weight_peak_sum,weight_I_peak_sum,weight_I_peak_sq_sum)SELECT b0.xtrsrc_id,b0.datapoints+ 1 AS datapoints,b0.I_peak_sum+ x0.I_peak AS i_peak_sum,b0.I_peak_sq_sum+ x0.I_peak * x0.I_peak AS i_peak_sq_sum,b0.weight_peak_sum+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum,b0.weight_I_peak_sum+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sum,b0.weight_I_peak_sq_sum+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sq_sumFROM basesources b0,extractedsources x0WHERE x0.image_id = @imageidAND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight) AS INTEGER)AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight) AS INTEGER)AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)+(x0.y - b0.y)*(x0.y - b0.y)+(x0.z - b0.z)*(x0.z - b0.z))/2)/SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)< @assoc_r; 27. MonetDB vs MySQL 28. MonetDB and Python We maintain the MonetDB Python API http://pypi.python.org/pypi/python-monetdb/ Problems? Ask me :) 29. Challenges Debugging queries MonetDB still in active development 30. Djonet MonetDB backend for Django https://github.com/gijzelaerr/djonet brew install monetdb pip install python-monetdb djonet Contributions are welcome! 31. VO events Standardized language Report observations of astronomical events Hey world, check this supernova out overthere http://comet.transientskp.org 32. VOevents 33. Visualisation Web interface Django! Not public (yet) 34. Source code? No sorry. Maybe some day Legal issues, bureaucracy UvA, Astron 35. More http://www.transientskp.org/ http://www.lofar.org/ http://www.aartfaac.org/ 36. Questions?