1

An Introduction to IDL (The Interactive Data Language)

and IDL in Astronomy

Astronomy Seminar Series 11/12/2005

1. IDL Features 2. Basics of IDL

3. 1D & 2D Display 4. FITS I/O in IDL

5. Image Processingcqras@126.com

2

I.IDL Features

http://www.astro.virginia.edu/class/oconnell/astr511/IDLguide.html

3

IDL vs. Mathematica, Matlab, Maple

http://amath.colorado.edu/computing/mmm/brief.html

4

IDL vs. Traditional Astronomical Software

http://www.astro.virginia.edu/class/oconnell/astr511/IDLguide.html

5

IDL in Astronomy

http://idlastro.gsfc.nasa.gov/other_url.html

6

Very Good Places and Packages

IDL Astronomy User's Libraryhttp://idlastro.gsfc.nasa.gov/homepage.html

http://idlastro.gsfc.nasa.gov/ftp/astron.tar.gz

FITS http://idlastro.gsfc.nasa.gov/fitsio.htmlSolar Softwarehttp://lmsal.com/solarsoft/sswdoc/index_menu.htmlftp://sohoftp.nascom.nasa.gov/solarsoft/offline/swmaint/tar/ ssw_ssw_gen.tar.Z

Coyote’s Guide to IDL Programminghttp://www.dfanning.comftp://ftp.dfanning.com/pub/dfanning/outgoing/coyote2nd/

IDL Newsgroup (comp.lang.idl-pvwave)http://groups.google.com/group/comp.lang.idl-pvwave

Markwardt IDL Library (Fitting)http://cow.physics.wisc.edu/~craigm/idl/idl.html

JHUAPL IDL Libraryhttp://fermi.jhuapl.edu/s1r/idl/s1rlib/local_idl.html

IDL + EMACS http://www.idlwave.org/http://idlwave.org/download/idlwave-help.tar.bz2

Important: Practice, Take notes,

Google idl + keywords

IDL Astro Package astron.dir.tar.gzCoyote Program Library coyote2ndfiles.zipSSW General Package ssw_ssw_gen.tar.Z

tar xvfz ssw_ssw_gen.tar.Zmv gen /usr/local/rsi/idl/lib/sswgenOthers in a similar way; or see Page 8

7

Working with IDL

MenuToolbars

Edit Window

Output Log

Variable Watch

Command Input

Status Bar

compile run

My favoriteLinux/Unix Console

Demo

8

Personal SETUP for IDL (recommended)

Personal IDL_STARTUP1.edit ~/.cshrc : setenv IDL_STARTUP ~/.idl_startup.pro2.edit ~/.idl_startup.pro (or other file names), Device, retain = 2, decomposed = 0 !path = !path + ‘:~/idlpros1:~/idlpros2’

#Windows: File Preferences Startup... you can select any file that is similar to that under Linux.

Flashing Colors in Linux/Unix

#edit ~/.Xresources idl.gr_visual: TrueColor   idl.gr_depth: 24   idl.retain: 2   idl.colors: -1

Memory Limits in Hubble (Alpha)

# edit ~/.cshrc limit stacksize unlimited limit datasize unlimited

Working DirectoryIDL> cd, ‘myworkdir’Windows: File Preferences … Startup Startup

9

II. Basics of IDLIDL> PRINT, 3 * 5, [30,5,50] 15 30 5 50

IDL> x = 'Hello! IDL World' & HELP, x X STRING = 'Hello! IDL World'IDL> x = indgen(15) & y = sin(2*!dpi*x/15) X INT = Array[10] Y DOUBLE = Array[10]

IDL> FOR i = 0, 15-1 DO PRINT, i, x[i], y[i]

IDL> DEVICE, decomposed = 0IDL> plot, loaddata(1), psym=-4, $ title='plot', xtitle='Month', ytit='Sth‘ ;,$ ;/ylog, yrange=[5e-1,40], ystyle=1

Dynamic Datatype

Array Zero-Ordered

Array Operation

Direct GraphicsParameters

10

Some Symbols, Definitions and Others

? – online help.run, .compile, .r; & $ ! ↑↓

http://fermi.jhuapl.edu/s1r/idl/idl_syntx.html

11

IDL Variables: Dynamic

Scalar, Array (1—8D) Structure: collection of scalars, arrays, or other structures

System Variables (!)– !dpi (3.1415926)

– !p: Display. e.g., !p.font, !p.color

– !d: Device. e.g. !d.name

HELP, variable or HELP, variable, /struct Data type: dynamic

12

Type Len Creation Array ConversionByte 1 A=5B Bytarr ByteInteger 2 B=0;b=0S Intarr FixUint 2 C=0U Uintarr unitLong 4 D=0L Lonarr LongUlong 4 E=0UL Ulonarr UlongLong64 8 F=0LL Long64arr Long64Ulong64 8 G=0ULL Ulon64arr Ulong64Float 4 H=0.0 Fltarr FloatDouble 8 I=0.0D Dblarr DoubleComplex 8 J=complex(1.0,0.0) Complexarr ComplexDcomplex 16 K=dcomplex(1.0,0.0) Dcomplexarr DcomplexString ? L=’hello’ Strarr StringPointer 4 M=ptr_new() Ptrarr ---Object 4 N=obj_new() Objarr ---

IDL Variables: Basic Datatypes

Indexed Array Creator: e.g., findgen() float index generatorTable c.f. intenet

13

Control Statements IF: conditionalif exp then statemif exp then statem1 else statem2 For Loops: for i = init, limit, step do statem While Loops: while exp do statem Repeat Loops: repeat statem until exp Case: GOTO: goto, label Blocks:

Begin statem1 …… statemxEndxxx if x lt 0 then begin print,x & a=2 & endif

e.g.

http://fermi.jhuapl.edu/s1r/idl/idl_syntx.html

if x lt 0 then begin print,x a=2 endif

for i=0, 10 do begin readf, lun, txt print,txtendfor

14

Programs: Procedures & Functions

Batch files: one or more IDL statements or commands. $ @batchfile: interpreted one by one, exactly as if it was from th

e keyboard. Main-level Programs: a series of program statements that are compiled

and executed once an END statement is encountered.

Programs: pro name, param1, param2, ... paramx ended with END Functions: function name, param1, param2, ... Paramx ended with END

Variable Access: Batch and Main (globe), Program & function (local)

IDL: an interactive tool, also a powerful programming language.

15

Parameters Passing

Actual (caller) and Formal (called) Parameters Correspondence by position or keyword Keyword Inheritance (_Extra)

Passing Mechanism• Expressions, constants, system variables, and subscripted va

riable references are passed by value.• Variables are passed by reference.

Parameters and Keywords Checking• n_params() ： number of parameters in calling an procedure/ function• n_elements() ： returns zero for undefined variable.• keyword_set() ： check a Boolean keyword parameter.• arg_present() ： defined and reference passing?

16

Working with Arrays! (Essential for IDL)

A(n,m): Array with n columns and m rowsA[n,m]: Element

pos = n*i+jj = pos MOD ni = (pos - j)/n

A[i1, j:*]

17

Array-Oriented Operation

Arrays work the same as Scalars. e.g. 2*A, A + B, A-B, A/B, SQURT(A), …… Try to avoid use of loops (slow!) Array Creation: xxxArr, xIndGen, Replicate, …

Array Manipulation:[], ARRAY_INDICES, CONGRID, HISTOGRAM, INVERT,MAX, MIN, MEDIAN, N_ELEMENTS, REBIN, REFORM,REVERSE, ROT, ROTATE, SHIFT, SIZE, SORT, TOTAL,MEAN, TRANSPOSE, UNIQ, WHERE, etc.

18

WHERE much faster than IF

Set all values between 5 to 8 equal = 15.

Loop Way:For j=0,2 Do Begin

For k=0,3 Do Begin

IF (array(j,k) GE 5) AND (array(j,k) LE 8) THEN array(j,k) = 15

EndFor

EndFor

IDL Way:index = Where((array GE 5) AND (array LE 8), count)

IF count GT 0 THEN array[index] = 15

http://www.dfanning.com/powerpoint/index.html

19

Where(): where are they?

file= FILEPATH('galaxy.dat', subdir = ['examples', 'data'])imagesize = [256, 256]image = READ_BINARY(file, data_dims =imagesize)

DEVICE, decomposed = 0 & LOADCT, 4WINDOW, 0, xsize = imagesize[0], ys= 2 * imagesize[1]

indices = Where((image GE 200) AND $ (image LE 230), count)

IF count GT 0 THEN BEGIN result = Array_Indices(image, indices) col = Reform(result [0,*]) row = Reform(result [1,*]) TV, image, 0 & TV, image, 1 PlotS, col, row, /Device, $ Color=FSC_Color(‘white'), psym=1ENDIF

20

Mask using Where()

file = FILEPATH('worldelv.dat', $subdir = ['examples', 'data'])file = FILEPATH('worldtmp.png', $subdir = ['examples', 'demo', 'demodata'])

TV, elvImage, 0 & TV, tmpImage, 1ocean = WHERE(elvImage LT 125) image = tmpImageimage[ocean] = elvImage[ocean] TV, image, 2

land = WHERE(elvImage GE 125) image = tmpImageimage[land] = elvImage[land] TV, image, 3

Temperature Distribution in Land and Ocean

21

img1 = BytScl(Loaddata(4), Top=99)

img2 = BytScl(Loaddata(5), Top=99)+100B

Window, XSize=256, YSize=256*3

LoadCT, 13, NColors=100 & TV, img1, 2

LoadCT, 3, NColors=100, Bottom=100 & TV, img2, 1

LoadCT, 13, NColors=100

LoadCT, 3, NColors=100, Bottom=100

index = $

Where((Indgen(256L*256L) MOD 2) EQ 0)

img1[index] = img2[index]

TV, img1, 0

Simultaneous look at two images.

Index Manipulation

http://www.dfanning.com/powerpoint/index.html

22

Examples: the IDL Way

n = 5 i = REBIN(LINDGEN(n), n, n) j = REBIN(TRANSPOSE(LINDGEN(n)), n, n) Print, i Print, j 0 1 2 3 4 0 0 0 0 0 0 1 2 3 4 1 1 1 1 1 0 1 2 3 4 2 2 2 2 2 0 1 2 3 4 3 3 3 3 3 0 1 2 3 4 4 4 4 4 4

mask = (i GE j)Print, mask 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 0 0 0 0 1

e.g., Upper Triangular Matrix

http://www.dfanning.com/idl_way/

23

Direct Graphics

• Device oriented; display in a specific device

• Set_plot:

X/WIN/MAC, PS, PRINTER, METAFILE, Z, CGM, PCL, NULL

• Device: retain, decomposed,

set_character_size, pseudo_color, index_color, true_color

• Window Coordinates: Data, Device, Nomal

24

PS files CreationcurName=!D.name ; ‘X’ or ‘Win’ set_plot,'ps' device, file=‘test.ps‘, posi=[1,1,9.5,9]/10, bits_per_pixel=8;device, file=‘test.eps‘, posi=[1,1,9.5,9]/10 ,/encapsulated;device, file=‘test.ps‘, posi=[1,1,9.5,9]/10,/colorplot, loaddata(1) device,/close & set_plot,curName

For X/MS windows devices, write_jpeg, ‘test.jpg', tvrd() write_jpeg, ‘test.jpg', tvrd(true=1),true=1

PS Layout configurations:http://www.dfanning/documents/programs.htmlhttp://cow.physics.wisc.edu/~craigm/idl/printing.html

25

Working with Colors (R,G,B) triple: any Color decomposed to Red, Green, and Blue components, each with value 0~255

Indexed Color Model and RGB Color Model • Indexed Color Model (also 8 bit color): Color Lookup Table Color Palette 2^8 = 256 colors Device, decomposed = 0• RGB Color Model (also 24 bit color) : specify color values explicitly, using an RGB triple 2^8 * 2^8 * 2^8 = 16777216 colors Device, decomposed = 1

Indexed?Dynamic

HELP, /DEVICELOADCT, TVLCT, XLOADCT, XPALETTE

About COLORBAR, FSC_COLOR, etc.http://www.dfanning/documents/programs.html

26http://www.dfanning.com/powerpoint/index.html

Comparison: 24-Bit and 8-Bit Color

27

Color Decomposition

Data=loaddata(1)

Plot, data, Color=255, posi=[0,0,1,1]Help, /Device

Device, Decomposed=1 ; ONPlot, data, Color=11829830L,$Background='00ff00'xL, posi=[0,0,1,1]

TVLCT, 70, 130, 180, 240Device, Decomposed=0; OFFPlot, data, Color=240,$Background=255,posi=[0,0,1,1]

28

RGB Image: Ant Nebula read_jpeg,'Ant.jpg',ant

r=reform(ant[0,*,*]) g=reform(ant[1,*,*]) b=reform(ant[2,*,*])Help, ant

window, 0, xsize=2*info[2], ysize=2*info[3]tv, ant, true=1,0tvscl,r,channel=1 & tvscl,g,channel=2 & tvscl,b,channel=3tvscl,r,channel=1,1 & tvscl,g,channel=2,2 & tvscl,b,channel=3,3

Google Images: Ant Nebula

device,decom=1

29

Image Types

• Byte: 0~255; otherwise, BYTSCL()• Binary Image: values only with 0 or 1• Gray-scale Image: B-W LUT• Indexed Images: 0~255, LUT• RGB Images: (R,G,B), each 0~255

COLOR_QUAN() : RGB Indexed Image

FILEPATH, QUERY_IMAGE, READ_BINARY, READ_IMAGEREAD_JPEG/WRITE_JPEG, WRITE_IMAGE,……

30

III. 1D and 2D Data Display plot/oplot, plots, axis, xyouts, ploterr/oploterr/errplot/, ploterr

or(!!!), vel/velovect/plot_field/flow3

BTLSCL, REBIN, REFORM tv/tvscl/bytscl, imdisp(!!!), plot_image, tvimage, contour, im

age_cont, surface, surf_shade, show3, median, smooth/convol, reberts/sobel, defroi, profiles, etc.

Histogram, plot_hist

box_cursor, plot_box, rdpix, curval

live_tools(live_plot,...), itools(iplot,icontour,...) iTools (iplot, ……)

!p.multi, position, xrange, xstyle, psym, ……

31

User PSYMdevice, decomposed=0!p.multi=[0,1,2]data=loaddata(1)Plot, data, PSym=-2

;filled circlephi = Findgen(32) * (!PI * 2 / 32.)phi = [ phi, phi[0] ]UserSym, Cos(phi), Sin(phi), /Fill

Plot, data, /NoData;tvlct, 178,34,34,10 & OPlot, data, Color=10OPlot, data, Color=FSC_Color('firebrick')Oplot, data, color=FSC_Color('forest green'), $ PSym=8, Symsize=1.5

32

Plotting Error Bars

xtime = indgen(101) & data = loaddata(1)xerr = randomN(seed,101)*2 yerr = randomN(seed,101)*4device,decomposed=0 & loadct, 0 tvlct, r, g, b, /get & tvlct, 255-[[r],[g],[b]] & tvlct, 0,255,0,100

ploterror, xtime, data, xerr, yerr, psym = -2, xstyle=1, $ xtitle='!7b(!6cm!U-2!N !6s!D-1!N)', ytitle='!6H!7a' oplot,xtime,data,color=100,thick=2 ERRPLOT, X, Y-yerr1, Y+yerr2

33

TV and TVSCL

file = FILEPATH('hurric.dat', subdir = ['examples', 'data'])hurric = READ_BINARY(file, DATA_DIMS = [440, 340])

34

Pseudo-Color Images in PSfile = FILEPATH('worldelv.dat', subdir = ['examples', 'data'])image = READ_BINARY(file, DATA_DIMS = [360, 360])

IMDISP better than TV/TVSCLAlso try PLOT_IMAGE

curName=!d.name & set_plot,'PS'device,/color,file='elev.eps',bits_per_pixel=8,/encaploadct,13 & imdisp, image ; plot_image,imagedevice,/close & set_plot,curName

35

True-Color Images in PS

file=filepath('rose.jpg',subdir='examples/data')read_jpeg,file,rose & tvlct,r,g,b,/get

curName=!d.name & set_plot,'PS‘device,/color,file='rose.eps',bits_per=8,/encaploadct,0 & imdisp,rosedevice,/close & set_plot,curNametvlct,r,g,b

Color table always activein 24 bits mode, ps device

Google Images: Rosette Nebula

36

VI. FITS I/O in IDL FITS (Flexible Image Transport System) is a standardized data format wh

ich is widely used in astronomy.

Briefly, a FITS file consists of a sequence of one or more Header and Data Units (HDUs). A header is composed of ASCII card images that in IDL is usually read into a string array variable. The header describes the content of the associated data unit, which might be a spectrum (IDL vector), an image (IDL array), or tabular data in ASCII or binary format (often read as an IDL structure). Image and vector data can be present in any HDU, but tabular data cannot appear in the first HDU. The HDUs following the first (or primary) HDU are also known as extensions, and thus a FITS file containing tabular data must contain at least one extension.

The FITS Support Office/NASA http://fits.gsfc.nasa.gov/

Four Classes of Procedures:MRDFITS()/MWRFITS: READFITS()/WRITEFITS FX* Procedures

FITS_* and FTAB_* Procedures

FITS I/O in IDLAstrohttp://idlastro.gsfc.nasa.gov/fitsio.html

37

FITS I/O: File Information

IDL> file = ‘WFPC2u5780205r_c0fx.fits’IDL> Fits_Info, ‘WFPC2u5780205r_c0fx.fit

s’WFPC2u5780205r_c0fx.fits has 1 extensionsPrimary header: 263 recordsImage -- Real*4 array ( 200 200 4 )Extension 1 -- u5780205r_cvt.c0h.tab Header : 354 records ASCII Table ( 796 4 )

IDL> file=‘EUVEngc4151imgx.fits’IDL> fits_help, file XTENSION EXTNAME EXTVER EXTLEVEL BITPIX GCOUNT PCOUNT NAXIS NAXIS*

0 8 0 0 0 1 IMAGE ds 16 1 0 2 512 x 512 2 IMAGE sw_night 16 1 0 2 2048 x 300 3 IMAGE mw 16 1 0 2 2048 x 300 4 IMAGE lw 16 1 0 2 2048 x 300 5 BINTABLE ds_limits 8 1 0 2 16 x 3 6 BINTABLE sw_night_limits 8 1 0 2 20 x 2 7 BINTABLE mw_limits 8 1 0 2 20 x 2 8 BINTABLE lw_limits 8 1 0 2 20 x 2

IDL> images=mrdfits(file,0,head0)MRDFITS: Image array (200,200,4) Type=Real*4IDL> help,images,head0IMAGES FLOAT = Array[200, 200, 4]HEAD0 STRING = Array[263]IDL> table=mrdfits(file,0,head1)MRDFITS: Image array (200,200,4) Type=Real*4IDL> help,table,head1TABLE FLOAT = Array[200, 200, 4]HEAD1 STRING = Array[263]

http://fits.gsfc.nasa.gov/fits_samples.html

38

Fits Example: Orion Nebuladata=readfits('masterf673.fits.gz',head)loadct,10 & tvlct,r,g,b,/getr[0]=34 & g[0]=139 & b[0]=34 & tvlct,r,g,bplot_image,alog10(data>5e-1)loadct,10plot_image,alog10(data[1000:2000,1200:2200]>0.5)

http://casa.colorado.edu/~bally/HST/HST/master/

39

Solar Map Software An IDL map is a structure that contains two-dimensional (2-d) image data with accompanying pixel coordinate and spatial scale information. The latter parameters are defined as properties of the map and are unique for each image source. Defined in this manner, an arbitrary image can be manipulated or transformed in a manner that is independent of the image source.

http://hesperia.gsfc.nasa.gov/~ptg/trace-align/

http://orpheus.nascom.nasa.gov/~zarro/idl/maps.html

40

V. Image Processing

A Short Introduction to Digital Image Processinghttp://web.uct.ac.za/depts/physics/laser/hanbury/intro_ip.html

41

BYTSCL (Byte Scale)

READ_DICOM(FILEPATH('mr_brain.dcm', subdir = ['examples', 'data']))

BYTSCLmr_brain

42

Dragon in EIT 304

01/23/2001 19:19:43

http://umbra.nascom.nasa.gov/eit/eit-catalog.html

43

Color Table Highlights, Contrast

file = FILEPATH('mineral.png', subdir = ['examples', 'data'])image = READ_PNG(file, r, g, b)

colorLevel = [[0, 0, 0], [255, 0, 0], [255, 255, 0], [0, 255, 0], $ [0, 255, 255], [0, 0, 255], [255, 0, 255], [255, 255, 255]] numberLevel = CEIL(!D.TABLE_SIZE/8.) level = INDGEN(!D.TABLE_SIZE)/numberLevel newR = colorLevel(0, level) & newR[!D.TABLE_SIZE - 1] = 255 newG = colorLevel(1, level) & newG[!D.TABLE_SIZE - 1] = 255 newB = colorLevel(2, level) & newB[!D.TABLE_SIZE - 1] = 255

TVLCT, newR, newG, newBTVLCT, 13TVLCT, r,g,b

44

Histogram EqualizationLeft: BYTSCL(HISTOGRAM(image))Right: Mineral

also, H_EQ_CT, H_EQ_INT

Left:BYTSCL(HISTOGRAM(hq_image))Right:hq_image = HIST_EQUAL(image)

ADAPT_HIST_EQUALLeft:HISTOGRAM(equalizedImage)Right:ADAPT_HIST_EQUAL(image)

45

Adjust Histogram

http://web.uct.ac.za/depts/physics/laser/hanbury/intro_ip.html

46

Fast Fourier Transform de-noise

file= FILEPATH('abnorm.dat', subdir = ['examples', 'data'])powerSpec = ALOG10(SHIFT(FFT(image), zz[0]/2, zz[0]/2))CONGRID(powerSpec, zz2[0], zz2[1])

mask = FLOAT(scaledSpec) GT 2.6 maskedSpec = scaledSpec * mask + MIN(powerSpec)

inverseTran = ABS(FFT(SHIFT(10.^(maskedSpec), $ zz[0]/2,zz[1]/2), /inverse))

scaledSpec = powerSpec - Min(powerSpec)

47

FFT: removing corrugated effect

http://web.uct.ac.za/depts/physics/laser/hanbury/intro_ip.html

48

Inverse Laplace Trans

mask = HANNING(imagesize[0], imagesize[1]) maskedSpec = (scaledSpec * mask) + MIN(powerSpec)

inverseTrans = ABS(FFT(SHIFT(10.^(maskedSpec), $ imagesize[0]/2, imagesize[1]/2), /inverse))

maskedSpecinverseTrans

49

Median Smoothing

file= FILEPATH('rbcells.jpg', subdir = ['examples', 'data'])

MEDIAN(rbcells, 5)rbcells

50

Rim Enhancing

croppedSize = [96, 96]file= FILEPATH('nyny.dat', subdir = ['examples', 'data'])croppedImage = image[200 : croppedSize[0] - 1 + 200, 180 : croppedSize[0] - 1 + 180]

ROBERTS SOBELcroppedImage