Python Crash CoursePython Crash CoursePyFits, AstropyPyFits, Astropy

3rd year Bachelors

V1.0

dd 05-09-2013

Hour 7

Handling FITS files - PyFITSHandling FITS files - PyFITS

• http://www.stsci.edu/resources/software_hardware/pyfits

• Read, write and manipulate all aspects of FITS files

– extensions

– headers

– images

– tables

• Low-level interface for details

• High-level functions for quick and easy use

PyFITS - reading PyFITS - reading

>>> import pyfits

>>> imgname = “testimage.fits”

>>> img = pyfits.getdata(imgname)

>>> img

array([[2408, 2408, 1863, ..., 3660, 3660, 4749],

[2952, 2408, 1863, ..., 3660, 3115, 4204],

[2748, 2748, 2204, ..., 4000, 3455, 4000],

...,

[2629, 2901, 2357, ..., 2261, 2806, 2261],

[2629, 2901, 3446, ..., 1717, 2261, 1717],

[2425, 2697, 3242, ..., 2942, 2125, 1581]], dtype=int16)

>>> img.mean()

4958.4371977768678

>>> img[img > 2099].mean()

4975.1730909593043

>> import numpy

>>> numpy.median(img)

4244.0

PyFITS – reading FITS imagesPyFITS – reading FITS images

• row = y = first index• column = x = second index• numbering runs as normal (e.g. in ds9) BUT zero indexed!

>>> x = 348; y = 97

>>> delta = 5

>>> print img[y-delta:y+delta+1,

... x-delta:x+delta+1].astype(numpy.int)

[[5473 5473 3567 3023 3295 3295 3839 4384 4282 4282 3737]

[3295 4384 3567 3023 3295 3295 3295 3839 3737 3737 4282]

[2478 3567 4112 3023 3295 3295 3295 3295 3397 4486 4486]

[3023 3023 3023 3023 2750 2750 3839 3839 3397 4486 3941]

[3295 3295 3295 3295 3295 3295 3839 3839 3397 3941 3397]

[3295 3295 2750 2750 3295 3295 2750 2750 2852 3397 4486]

[2887 2887 2887 2887 3976 3431 3159 2614 3125 3669 4758]

[2887 2887 3431 3431 3976 3431 3159 2614 3669 4214 4214]

[3159 3703 3159 3703 3431 2887 3703 3159 3941 4486 3669]

[3703 3159 2614 3159 3431 2887 3703 3159 3397 3941 3669]

[3431 3431 2887 2887 3159 3703 3431 2887 3125 3669 3669]]

PyFITS – reading FITS tablesPyFITS – reading FITS tables

>>> tblname = ‘data/N891PNdata.fits’

>>> d = pyfits.getdata(tblname)

>>> d.names

('x0', 'y0', 'rah', 'ram', 'ras', 'decd', 'decm', 'decs', 'wvl', 'vel',

'vhel', 'dvel', 'dvel2', 'xL', 'yL', 'xR', 'yR', 'ID', 'radeg', 'decdeg',

'x', 'y')

>>> d.x0

array([ 928.7199707 , 532.61999512, 968.14001465, 519.38000488,…

1838.18994141, 1888.26000977, 1516.2199707 ], dtype=float32)

>>> d.field(‘x0’) # case-insensitive

array([ 928.7199707 , 532.61999512, 968.14001465, 519.38000488,…

1838.18994141, 1888.26000977, 1516.2199707 ], dtype=float32)

>>> select = d.x0 < 200

>>> dsel = d[select] # can select rows all together

>>> print dsel.x0

[ 183.05000305 165.55000305 138.47999573 158.02999878 140.96000671

192.58000183 157.02999878 160.1499939 161.1000061 136.58999634

175.19000244]

PyFITS – reading FITS headersPyFITS – reading FITS headers

>>> h = pyfits.getheader(imgname)

>>> print h SIMPLE = T /FITS header BITPIX = 16 /No.Bits per pixel NAXIS = 2 /No.dimensions NAXIS1 = 1059 /Length X axis NAXIS2 = 1059 /Length Y axis EXTEND = T / DATE = '05/01/11 ' /Date of FITS file creation ORIGIN = 'CASB -- STScI ' /Origin of FITS image PLTLABEL= 'E30 ' /Observatory plate label PLATEID = '06UL ' /GSSS Plate ID REGION = 'XE295 ' /GSSS Region Name DATE-OBS= '22/12/49 ' /UT date of Observation UT = '03:09:00.00 ' /UT time of observation EPOCH = 2.0499729003906E+03 /Epoch of plate PLTRAH = 1 /Plate center RA PLTRAM = 26 / PLTRAS = 5.4441800000000E+00 / PLTDECSN= '+ ' /Plate center Dec PLTDECD = 30 / PLTDECM = 45 / >>> h[‘KMAGZP’]

>>> h['REGION']

'XE295‘

# Use h.items() to iterate through all header entries

PyFITS – writing FITS imagesPyFITS – writing FITS images

>>> newimg = sqrt((sky+img)/gain + rd_noise**2) * gain

>>> newimg[(sky+img) < 0.0] = 1e10

>>> hdr = h.copy() # copy header from original image

>>> hdr.add_comment(‘Calculated noise image’)

>>> filename = ‘sigma.fits’

>>> pyfits.writeto(filename, newimg, hdr) # create new file

>>> pyfits.append(imgname, newimg, hdr) # add a new FITS extension

>>> pyfits.update(filename, newimg, hdr, ext) # update a file

# specifying a header is optional,

# if omitted automatically adds minimum header

PyFITS – writing FITS tablesPyFITS – writing FITS tables

>>> import pyfits

>>> import numpy as np

>>> # create data

>>> a1 = numpy.array(['NGC1001', 'NGC1002', 'NGC1003'])

>>> a2 = numpy.array([11.1, 12.3, 15.2])

>>> # make list of pyfits Columns

>>> cols = []

>>> cols.append(pyfits.Column(name='target', format='20A',

array=a1))

>>> cols.append(pyfits.Column(name='V_mag', format='E', array=a2))

>>> # create HDU and write to file

>>> tbhdu=pyfits.new_table(cols)

>>> tbhdu.writeto(’table.fits’)

# these examples are for a simple FITS file containing just one

# table or image but with a couple more steps can create a file

# with any combination of extensions (see the PyFITS manual online)

PyFITS - advancedPyFITS - advanced

>>> f = pyfits.open(tblname)

>>> f.info()

Filename: data/N891PNdata.fits

No. Name Type Cards Dimensions Format

0 PRIMARY PrimaryHDU 4 () uint8

1 BinTableHDU 52 223R x 22C [E, E, E, E, E,

E, E, E, E, E, E, E, E, E, E, E, E, E, E, E, E, E]

>>> table = f[1] # data extension number 1 (can also use names)

>>> d = f[1].data # data, same as returned by pyfits.getdata()

>>> h = f[1].header # header, same as returned by pyfits.getheader

()

>>> # make any changes

>>> f.writeto(othertblname) # writes (with changes) to a new file

>>> f = pyfits.open(tblname, mode=‘update’) # to change same file

>>> # make any changes

>>> f.flush() # writes changes back to file

>>> f.close() # writes changes and closes file

AstropyAstropy

>>> import astropy

>>> from astropy.io import fits

The astropy package (alternatively known as the “core” package) contains various classes, utilities, and a packaging framework intended to provide commonly-used astronomy tools. It is divided into a variety of sub-packages (http://docs.astropy.org)

In [1]: import astropy.

astropy._compiler astropy.io astropy.tests

astropy.builtins astropy.logger astropy.time

astropy.config astropy.logging astropy.units

astropy.conftest astropy.nddata astropy.utils

astropy.constants astropy.setup_helpers astropy.version

astropy.coordinates astropy.sphinx astropy.version_helpers

astropy.cosmology astropy.stats astropy.wcs

astropy.cython_version astropy.sys

astropy.extern astropy.table

Astropy ConstantsAstropy Constants

astropy.constants contains a number of physical constants useful in Astronomy. Constants are Quantity objects with additional meta-data describing their provenance and uncertainties.

>>> from astropy.constants import G

>>> from astropy import constants as const

>>> print const.c

Name = Speed of light in vacuum

Value = 299792458.0

Error = 0.0

Units = m / (s)

Reference = CODATA 2010

>>> print const.c.to('pc/yr')

0.306594845466 pc / (yr)

>>> F = (const.G * 3. * const.M_sun * 100 * u.kg) / (2.2 * u.au) ** 2

>>> print F.to(u.N)

0.367669392028 N

Astropy UnitsAstropy Units

astropy.units handles defining and converting between physical units, and performing arithmetic with physical quantities (numbers with associated units).>>> from astropy import units as u

>>> u.pc.to(u.m)

3.0856776e+16

>>> cms = u.cm / u.s

>>> mph = u.mile / u.hour

>>> cms.to(mph, 1)

0.02236936292054402

>>> cms.to(mph, [1., 1000., 5000.])

array([ 2.23693629e-02, 2.23693629e+01, 1.11846815e+02])

>>> u.nm.to(u.Hz, [1000, 2000])

UnitsException: 'nm' (length) and 'Hz' (frequency) are not convertible

>>> u.nm.to(u.Hz, [1000, 2000], equivalencies=u.spectral())

array([ 2.99792458e+14, 1.49896229e+14])

>>> F = (const.G * 3. * const.M_sun * 100 * u.kg) / (2.2 * u.au) ** 2

>>> print F.to(u.N)

0.367669392028 N

Astropy nddataAstropy nddata

astropy.nddata provides the NDData class and related tools to manage n-dimensional array-based data (e.g. CCD images, IFU data, grid-based simulation data, ...). This is more than just numpy.ndarray objects, because it provides metadata that cannot be easily provided by a single array

>>> from astropy.nddata import NDData

>>> array = np.random.random((12, 12, 12)) # a random 3-dimensional array

>>> ndd = NDData(array)

>>> ndd.ndim

3

>>> ndd.shape

(12, 12, 12)

>>> ndd.dtype

dtype('float64')

>>> ndd.data # the underlying numpy array

array([[[ 0.97717024, 0.76279957, 0.88721772, ..., 0.28542013, 0.61843553, 0.23758867],

[ 0.32538728, 0.12459703, 0.40686256, ..., 0.22809324,

Astropy nddataAstropy nddata

Nndata handles Masks, Flags, Uncertainties and Meta data

>>> ndd.mask = ndd.data > 0.9

>>> ndd.flags = np.zeros(ndd.shape)

>>> ndd.flags[ndd.data < 0.1] = 1

# multiple flag layers

>>> from astropy.nddata import FlagCollection

>>> ndd.flags = FlagCollection(shape=(12, 12, 12))

>>> ndd.flags['photometry'] = np.zeros(ndd.shape, dtype=str)

>>> ndd.flags['photometry'][ndd.data > 0.9] = 's'

>>> ndd.flags['cosmic_rays'] = np.zeros(ndd.shape, dtype=int)

>>> ndd.flags['cosmic_rays'][ndd.data > 0.99] = 99

>>> from astropy.nddata import StdDevUncertainty

>>> ndd.uncertainty = StdDevUncertainty(np.ones((12, 12, 12)) * 0.1)

>>> ndd.meta['exposure_time'] = 340.

>>> ndd.meta['filter'] = 'J‘

# back to numpy array

>>> arr = np.array(ndd)

Astropy nddataAstropy nddata

astropy.nddata includes a convolution function that offers improvements compared to the scipy astropy.ndimage convolution routines, including:

•Proper treatment of NaN values

•A single function for 1-D, 2-D, and 3-D convolution

•Improved options for the treatment of edges

•Both direct and Fast Fourier Transform (FFT) versions

from astropy.nddata import convolve, convolve_fft

result = convolve(image, kernel)

result = convolve_fft(image, kernel)

>>> convolve([1, 4, 5, 6, np.nan, 7, 8],

[0.2, 0.6, 0.2],

boundary='extend')

array([ 1.6, 3.6, 5. , 5.9, 6.5, 7.1, 7.8])

Astropy tablesAstropy tables

Astropy.table provides functionality for storing and manipulating heterogenous tables of data in a way that is familiar to numpy users. A few notable features of this package are:

– Initialize a table from a wide variety of input data structures and types.– Modify a table by adding or removing columns, changing column names, or

adding new rows of data.– Handle tables containing missing values.– Include table and column metadata as flexible data structures.– Specify a description, units and output formatting for columns.– Interactively scroll through long tables similar to using more.– Create a new table by selecting rows or columns from a table.– Full support for multidimensional columns.– Methods for Reading and writing Table objects to files

Astropy tablesAstropy tables

>>> from astropy.table import Table, Column

>>> a = [1, 4, 5]

>>> b = [2.0, 5.0, 8.2]

>>> c = ['x', 'y', 'z']

>>> t = Table([a, b, c], names=('a', 'b', 'c'), meta={'name': 'first table'})

>>> print t

a b c

--- --- ---

1 2.0 x

4 5.0 y

5 8.2 z

>>> t['a'] # Column 'a'

<Column name='a' units=None format=None description=None>

array([1, 4, 5])

>>> t['a'][1] # Row 1 of column 'a'

4

>>> t.add_column(Column(data=[1, 2, 3], name='d')))

>>> t.remove_column('c')

Astropy tablesAstropy tables

>>> t = Table()

>>> t.add_column(Column(data=[1, 4], name='a'))

>>> t.add_row((1, 2.0, ‘b'))

>>> Table([t['a']**2, t['b'] + 10])

<Table rows=2 names=('a','b')>

array([(1, 16.0), (11, 12.0)],

dtype=[('a', '<i8'), ('b', '<f8')])

>>> arr = {'a': [1, 4],

... 'b': [2.0, 5.0],

... 'c': ['x', 'y']}

>>>

>>> Table(arr)

<Table rows=2 names=('a','c','b')>

array([(1, 'x', 2.0), (4, 'y', 5.0)],

dtype=[('a', '<i8'), ('c', '|S1'), ('b', '<f8')])

>>> t['b'].mask = [True, False] # Modify column mask (boolean array)

>>> print(t)

a b

--- ---

1 --

-- 4

Astropy timeAstropy time

The astropy.time package provides functionality for manipulating times and dates. Specific emphasis is placed on supporting time scales (e.g. UTC, TAI, UT1) and time representations (e.g. JD, MJD, ISO 8601) that are used in astronomy. It uses Cython to wrap the C language SOFA (standards of Fundamental astronomy, IAU) time and calendar routines.

>>> from astropy.time import Time

>>> times = ['1999-01-01 00:00:00.123456789', '2010-01-01 00:00:00']

>>> t = Time(times, format='iso', scale='utc')

>>> t

<Time object: scale='utc' format='iso' vals=['1999-01-01 00:00:00.123' '2010-01-01 00:00:00.000']>

>>> t.jd

array([ 2451179.50000143, 2455197.5 ])

>>> t.mjd

array([ 51179.00000143, 55197 ]

>>> t1 = Time('2010-01-01 00:00:00', scale='utc')

>>> t2 = Time('2010-02-01 00:00:00', scale='utc')

>>> dt = t2 - t1 # Difference between two Times

>>> dt

<TimeDelta object: scale='tai' format='jd' vals=31.0>

>>> dt.sec

2678400.0.

Scale Description

tai International Atomic Time (TAI)

tcb Barycentric Coordinate Time (TCB)

tcg Geocentric Coordinate Time (TCG)

tdb Barycentric Dynamical Time (TDB)

tt Terrestrial Time (TT)

ut1 Universal Time (UT1)

utc Coordinated Universal Time (UTC)

Astropy ioAstropy io

Astropy provides a unified interface for reading and writing data in different formats. For many common cases this will simplify the process of file I/O and reduce the need to master the separate details of all the I/O packages within Astropy.

>>> from astropy.table import Table

>>> t = Table.read('2mass.tbl', format='ipac')

>>> t = Table.read('aj285677t3.txt', format='cds')

>>> t = Table.read('photometry.dat', format='daophot')

>>> t = Table.read('paper_table.tex')

>>> t = Table.read('photometry.dat', format='ascii', data_start=2, delimiter='|')

>>> t = Table.read('observations.hdf5', path='group/data')

>>> t = Table.read('catalog.xml', table_id='twomass', format='votable')

Astropy ioAstropy io

The astropy.io.fits package provides access to FITS files. FITS (Flexible Image Transport System) is a portable file standard widely used in the astronomy community to store images and tables.

>>> from astropy.io import fits

>>> hdulist = fits.open('input.fits')

>>> hdulist.close()

>>> hdulist.info()

Filename: test1.fits

No. Name Type Cards Dimensions Format

0 PRIMARY PrimaryHDU 220 () int16

1 SCI ImageHDU 61 (800, 800) float32

2 SCI ImageHDU 61 (800, 800) float32

>>> prihdr = hdulist[0].header

>>> prihdr['targname'] = ('NGC121-a', 'the observation target')

>>> prihdr['targname']

'NGC121-a'

>>> prihdr.comments['targname']

'the observation target'

Astropy ioAstropy io

Image data

>>> scidata = hdulist[1].data # pointer to the data

>>> scidata.shape

(800, 800)

>>> scidata.dtype.name

'float32‘

>>> scidata[30:40, 10:20]

# array manipulation to convert the image data from counts to flux:

>>> photflam = hdulist[1].header['photflam']

>>> exptime = prihdr['exptime']

>>> scidata *= photflam / exptime

>>> hdulist.writeto('newimage.fits')

>>> f = fits.open('original.fits', mode='update')

... # making changes in data and/or header

>>> f.flush() # changes are written back to original.fits

Astropy WCSAstropy WCS

framework to represent celestial coordinates and transform between them>>> from astropy import coordinates as coord

>>> from astropy import units as u

>>> coord.ICRSCoordinates(ra=10.68458, dec=41.26917, unit=(u.degree, u.degree))

<ICRSCoordinates RA=10.68458 deg, Dec=41.26917 deg>

>>> coord.ICRSCoordinates('00h42m44.3s +41d16m9s')

<ICRSCoordinates RA=10.68458 deg, Dec=41.26917 deg>

>>> c = coord.ICRSCoordinates(ra=10.68458, dec=41.26917, unit=(u.degree, u.degree))

>>> c.ra

<RA 10.68458 deg>

>>> c.ra.hours

0.7123053333333333

>>> c.ra.hms

(0.0, 42, 44.2992000000001)

>>> c.dec

<Dec 41.26917 deg>

>>> c.dec.radians

0.7202828960652683

Astropy WCSAstropy WCS

Coordinate transformations

>>> c.galactic

<GalacticCoordinates l=121.17422 deg, b=-21.57283 deg>

>>> c.transform_to(coord.GalacticCoordinates)

<GalacticCoordinates l=121.17422 deg, b=-21.57283 deg>

# Lookup object names at CDS, requires internet connection

>>> c_eq = coord.ICRSCoordinates.from_name("M33")

>>> c_eq

<ICRSCoordinates RA=23.46210 deg, Dec=30.65994 deg>

Introduction to languageIntroduction to language

End