Montage: Experiences in Astronomical Image Mosaicking on the TeraGrid ESTO.

Montage: Experiences in Astronomical Image Mosaicking on

the TeraGrid

http://montage.ipac.caltech.edu/

ESTO

Contributors

• Nathaniel Anagnostou IPAC

• Bruce Berriman IPAC

• Ewa Deelman ISI

• John Good IPAC

• Joseph C. Jacob JPL

• Daniel S. Katz JPL

• Carl Kesselman ISI

• Anastasia Laity IPAC

• Thomas Prince JPL

• Gurmeet Singh ISI

• Mei-Hui Su ISI

• Roy Williams CACR

IPAC Infrared Processing and Analysis Center, JPL/CaltechISI Information Science Institute, USCJPL Jet Propulsion LaboratoryCACR Center for Advanced Computing Research, Caltech

Large Astronomical Image ArchivesIRAS 1 GB 1 arcmin All Sky 4 Infrared Bands

DPOSS 4 TB 1 arcsec All Northern Sky 1 Near-IR, 2 Visible Bands

2MASS 10 TB 1 arcsec All Sky 3 Near-Infrared Bands

SDSS-DR2 5 TB 0.4 arcsec3,324 square degrees(16% of Northern Sky)

1 Near-IR, 4 Visible Bands

IRAS2MASSDPOSS

SDSS

IRAS

SDSS-DR2

DPOSS2MASS

Total Image Size Comparison

Wavelength Comparison

Science Drivers for Mosaics

• Many important astrophysics questions involve studying regions that are at least a few degrees across

• Need high, uniform spatial resolution

• BUT… cameras give high resolution or wide area but not both => need mosaics

• Mosaics can reveal new structures & open new lines of research

• Star formation regions, clusters of galaxies must be studied on much larger scales to reveal structure and dynamics

• Mosaicking multiple surveys to the same grid – image federation – required to effectively search for faint, unusual objects, transients, or unknown objects with unusual spectrum

Montage at a High Level• Delivers custom, science grade image mosaics

• User specifies projection, coordinates, spatial sampling, mosaic size, image rotation

• Preserves astrometry & flux

• Designed with modular “toolbox” design • Loosely-coupled engines for Image Reprojection, Background

Rectification, Co-addition

• Control testing and maintenance costs

• Flexibility; e.g., custom background algorithm; use as a reprojection and co-registration engine

• Implemented in ANSI C for portability

• Public service will be deployed on the TeraGrid• Mosaics will be ordered through web portal

Example Mosaics

2MASS 3-color mosaic of galactic plane • 44 x 8 degrees; 36.5 GB per band; CAR projection• 158,400 x 28,800 pixels; covers 0.8% of the sky• 4 hours wall clock time on cluster of 4 x 1.4-GHz Linux boxes

100 µm sky; aggregation of COBE and IRAS maps (Schlegel, Finkbeiner and Davis, 1998)

• 360 x 180 degrees; CAR projection

Montage UsersMontage Adopted by Spitzer Legacy Teams

SWIRE uses Montage for:

• Building sky simulations for use in mission planning

• Fast background rectification and co-addition of in-flight images

GLIMPSE uses Montage for:

• Large scale infrared mosaics of the Galactic Plane

• Quality Assurance of GLIMPSE Spitzer data by co-registration of 2MASS J, H, K and MSX 8 m images

Color composite of co-registered 2MASS and MSX. Each square

is 0.5 x 0.5 degrees

Right: Spitzer IRAC 3 channel mosaic (3.6m in green, 4.5m in red, and i-band optical in blue); high redshift non-stellar objects are visible in the full resolution view (yellow box).

Montage Users (cont.)

• Spitzer Space Telescope Outreach

• IRSA (NASA’s InfraRed Science Archive)

• COSMOS (a Hubble Treasury Program)

• Michelson Science Center Navigator Program ground-based archives

• NSF National Virtual Observatory (NVO) Atlasmaker project

• UK Astrogrid project

Public Release of Montage• Source code and user’s guide available for download

at http://montage.ipac.caltech.edu/• First release: Montage version 1.7

• Emphasized accuracy in photometry and astrometry• Testing on synthetic images showed flux errors of < 0.1% and

location errors of < 0.1 pixel

• Images processed serially

• Extensively tested (2,595 test cases) and validated on 2MASS 2IDR images for 10 WCS projections with mosaics smaller than 2 x 2 degrees and coordinate transformations Equ J2000 to Galactic and Ecliptic

• Second release: Montage version 2.2• More efficient reprojection algorithm: up to 30x speedup

• Improved memory efficiency: capable of building larger mosaics

• Enabled for parallel computation with MPI

• Enabled for processing on TeraGrid using standard grid tools

Montage v1.7 Reprojection: mProject module

Arbitrary Input Image

Central to the algorithm is accurate calculation of the area of spherical polygon intersection between two pixels (assumes great circle segments are adequate between pixel vertices)

Input pixels projected on celestial sphere

Output pixels projected on celestial sphere

SIMPLE = T / BITPIX= -64 / NAXIS = 2 / NAXIS1= 3000 / NAXIS2= 3000 / CDELT1= - 3.333333E-4 / CDELT2= - 3.333333E-4 / CRPIX1= 1500.5 / CRPIX2= 1500.5 / CTYPE1=‘RA---TAN’ CTYPE2=‘DEC--TAN’ CRVAL1= 265.91334 / CRVAL2= -29.35778 / CROTA2= 0. /END

FITS header defines output projection

Reprojected Image

• Transform directly from input pixels to output pixels

• Approach developed by Spitzer for tangent plane projections

• Performance improvement in reprojection by x 30

Montage v2.2 Reprojection:mProjectPP module

• Montage version 2.2 includes a module, mTANHdr, to compute “distorted” gnomonic projections to make this approach more general

• Allows the Spitzer algorithm to be used for other projections (in certain cases)

• For “typical” size images, pixel locations distorted by small distance relative to image projection plane

• Not applicable to wide area regions

1 2 3

mProject 1 mProject 2 mProject 3

1 2 3

mDiff 1 2 mDiff 2 3

mFitplane D12 mFitplane D23

ax + by + c = 0 dx + ey + f = 0

a1x + b1y + c1 = 0

a2x + b2y + c2 = 0

a3x + b3y + c3 = 0

mBackground 1 mBackground 2 mBackground 3

1 2 3

mAdd

Final Mosaic

D12D23

Montage Workflow

mConcatFit

mBgModel

ax + by + c = 0

dx + ey + f = 0

Montage Parallelization with MPI1 2 3


1 2 3

mDiff 1 2 mDiff 2 3


ax + by + c = 0 dx + ey + f = 0

a1x + b1y + c1 = 0

a2x + b2y + c2 = 0

a3x + b3y + c3 = 0


1 2 3

mAdd

Final Mosaic

D12D23

mConcatFit

mBgModel

ax + by + c = 0

dx + ey + f = 0

mProject and mBackground jobs parallelized with MPI

• Round-robinby input image



1 2 3

mDiff 1 2 mDiff 2 3


ax + by + c = 0 dx + ey + f = 0

a1x + b1y + c1 = 0

a2x + b2y + c2 = 0

a3x + b3y + c3 = 0


1 2 3

mAdd

Final Mosaic

D12D23

mConcatFit

mBgModel

ax + by + c = 0

dx + ey + f = 0

mDiff and mFitplane jobs parallelized with MPI

• Round-robinby input image pair



1 2 3

mDiff 1 2 mDiff 2 3


ax + by + c = 0 dx + ey + f = 0

a1x + b1y + c1 = 0

a2x + b2y + c2 = 0

a3x + b3y + c3 = 0


1 2 3

mAdd

Final Mosaic

D12D23

mConcatFit

mBgModel

ax + by + c = 0

dx + ey + f = 0

mAdd job parallelized with MPI

• By block of rows in output mosaic

Montage Parallel Performance6 x 6 degrees 2MASS mosaic of M16 at 1” sampling

Montage_v2.1 Execution Times on NCSA TeraGrid Cluster(using mProject algorithm from Montage_v1.7)

2546.6

1310.3

679

350.9

188.8

108.373.4

52

1

10

100

1000

10000

1 2 4 8 16 32 64 128

Number of Nodes (1 Processor Per Node)

Wall Clock Time (minutes)

Total (mProject)

mProjExec (mProject)

mDiffExec

mFitExec

mBgExec

mAdd

Montage_v2.1 Execution Times on NCSA TeraGrid Cluster

280.8

150.2

93.165.7

4437.3 36.85 32.4

1

10

100

1000

10000

1 2 4 8 16 32 64 128

Number of Nodes (1 Processor Per Node)

Wall Clock Time (minutes)

Total (mProjectPP)

mProjExec (mProjectPP)

mDiffExec

mFitExec

mBgExec

mAdd

mProjExec (mProject)

Total (mProject)

Time (Days) Number of Nodes

Algorithm 1 2 4 8 16 32 64 128

mProject 2026.9 1042.9 540.4 279.3 150.3 86.2 58.4 41.4

mProjectPP 223.5 119.5 74.1 52.3 35.0 29.7 29.3 25.8

So, how long would it take to mosaic the entire sky?

Montage on the Grid• “Grid” is an abstraction

• Array of processors, grid of clusters, …

• Use a methodology for running on any “grid environment”• Exploit Montage’s modular design in an approach applicable to any grid

environment • Describe flow of data and processing (in a Directed Acyclic Graph - DAG), including:

• Which data are needed by which part of the job

• What is to be run and when

• Use standard grid tools to exploit the parallelization inherent in the Montage design

• Build an architecture for ordering a mosaic through a web portal • Request can be processed on a grid

• Our prototype uses the Distributed Terascale Facility (TeraGrid)

• This is just one example of how Montage could run on a grid

Montage on the Grid UsingPegasus (Planning for Execution on Grids)

Example DAG for 10 input files

mAdd

mBackground

mBgModel

mProject

mDiff

mFitPlane

mConcatFit

Data Stage-in nodes

Montage compute nodes

Data stage-out nodes

Registration nodes

Pegasus

Grid Information Systems

Information about available resources, data location

Grid

Condor DAGMan

Maps an abstract workflow to an executable form

Executes the workflow

MyProxy

User’s grid credentials

http://pegasus.isi.edu/

Montage TeraGrid Portal

AbstractWorkflow

mGridExec

Pegasus

Concrete Workflow

Condor DAGman

Grid Scheduling

and Execution ServiceISI

AbstractWorkflow

Image List

DAGMan

TeraGrid Clusters

SDSC

NCSA

ISI Condor Pool

Computational Grid

Location, Size, Band

User Portal

JPL

mDAGFiles

Abstract Workflow Service

JPL

m2MASSList

2MASS Image List

ServiceIPAC

mNotify

User Notification

ServiceIPAC

Montage w/ Pegasus Performance

6 x 6 degree 2MASS Mosaic, computed on NCSA TeraGrid using Pegasus Portal

and Montage version 3.0 beta 6

Montage on the Grid Performance

• MPI version:• Best performance

• Requires a set of processors with a shared file system

• Pegasus/DAGman version• Very good performance

• Built-in fault tolerance

• Can use multiple sets of processors

Summary

• Montage is a custom astronomical image mosaicking service that emphasizes astrometric and photometric accuracy

• Second public release, Montage version 2.2, available for download at the Montage website

• A prototype Montage service has been deployed on the TeraGrid• It ties together distributed services at JPL, Caltech IPAC, and ISI

• Montage at SC2004:• See ISI at ANL booth for demo of Pegasus portal running Montage on

the TeraGrid• See us at NASA and Caltech CACR booths to hear this presentation

again or for general questions about Montage

• Montage website: http://montage.ipac.caltech.edu/

Montage: Experiences in Astronomical Image Mosaicking on the TeraGrid ESTO.

Documents

Transcript of Montage: Experiences in Astronomical Image Mosaicking on the TeraGrid ESTO.