Optimal Compression of Floating-point Astronomical ImagesWithout Significant Loss of Information

W. D. PenceNASA Goddard Space Flight Center, Greenbelt, MD 20771

William.Pence@nasa.gov

R. L. WhiteSpace Telescope Science Institute, Baltimore, MD 21218

and

R. SeamanNational Optical Astronomy Observatories, Tucson, AZ 85719

ABSTRACTWe describe a compression method for floating-point astronomical images that gives compres-

sion ratios of 6 — 10 while still preserving the scientifically important information in the image.The pixel values are first preprocessed by quantizing them into scaled integer intensity levels,which removes some of the uncompressible noise in the image. The integers are then losslesslycompressed using the fast and efficient Rice algorithm and stored in a portable FITS formatfile. Quantizing an image more coarsely gives greater image compression, but it also increasesthe noise and degrades the precision of the photometric and astrometric measurements in thequantized image. Dithering the pixel values during the quantization process greatly improvesthe precision of measurements in the more coarsely quantized images. We perform a series ofexperiments on both synthetic and real astronomical CCD images to quantitatively demonstratethat the magnitudes and positions of stars in the quantized images can be measured with thepredicted amount of precision. In order to encourage wider use of these image compression meth-ods, we have made available a pair of general-purpose image compression programs, called fpackand funpack, which can be used to compress any FITS format image.

Subject headings: image compression, FITS format

1. Introduction

The exponential growth in the volume of as-tronornical data being generated from ever largerformat imaging detectors continues to drive upthe data handling costs of both large and smalltelescope projects. The cost of archiving the im-ages is only one part of the problem. A typicalobservational work flow involves a long cascadeof temporary and permanent copies of the datareplicated from observatory to data processing fa-cility to deep storage site to science archive cen-

ter to virtual observatory portal to multiple endusers. Each science image provided as input to• pipeline will produce several output images as• result of processing operations such as resam-pling onto a standard grid, co-adding, mosaicking,and any analysis steps specific to the science pro-gram. The problem is one of throughput — not juststorage — of the total system data flow. :Networktransmission costs also rival or exceed the cost ofstorage media and can be breathtakingly large forspacecraft or remote mountaintops. Often there is

https://ntrs.nasa.gov/search.jsp?R=20100024392 2020-07-30T06:11:40+00:00Z

an upper limit to the network bandwidth at anyprice. Making effective use of the available imagecompression technologies is an important compo-nent in dealing with these data handling costs.

Lossy data compression techniques, which donot exactly preserve each image pixel value butdo still preserve the required scientific informationcontent of the image, are already used by manyprojects. To cite just a few examples, the Keplerspace telescope project (Caldwell et al. 2010) usesa lossy image quantization method to reduce thedata volume to fit within the limited bandwidthfrom its heliocentric orbit, the GONG helioseis-mology project (Harvey et al. 1996; Goodrich etal. 2004) relies on lossy compression to transmitimages from its telescopes deployed at six locationsworldwide, and the NOAO High-PerformancePipeline (Valdes & Swaters 2007) includes a finalquantization step from an internal floating-pointrepresentation to the final integer archive dataproducts to achieve greater image compression.

In a previous contribution (Pence, Seaman, &White 2009, hereafter, Paper I), we demonstratedthat the maximum possible lossless compressionratio for integer format astronomical images is de-termined by the amount of noise in the backgroundpixels (i.e., the "sky") in the image. The noise inastronomical images often has 2 main components:Poissonian-distributed photon noise and Gaussiandistributed "read-out" noise. If each pixel is rep-resented by BITPIX bits (usually 16 or 32 bits)and if, on average, Nbits of those bits are filledwith uncompressible, randomly fluctuating noise,then the maximum theoretical compression ratiofor that image is given by BITPIX / Nbits . Inpractice, no compression algorithm is 100% effi-cient, so the actual maximum compression ratio,R, is given by

R = BITPIX/(IVbits + K) (1)

where K is an empirical measure of the efficiency(or overhead) of the algorithm in units of bitsper pixel. We compared several lossless compres-sion algorithms and found that the Rice algorithm(Rice, Yeh, & Miller 1993; White & Becker 1998),which has a small K value of about 1.2 bits perpixel, provided the best combination of speed andcompression efficiency.

The relationship between noise and entropy inimages is discussed more fully in the appendix of

Paper I, based on the seminal work by Shannon(1948), where we showed that the "equivalent"number of noise bits per pixel in an image canbe calculated from the noise (0') of the pixels inbackground regions of the image such that

Nbits = 1092(0' 12) = 1092(0') + 1.792 (2)

which, when combined with equation 1, gives

R = BITPIX/(109 2 (0') + 1.792 + K) (3)

In this current article we extend the previousanalysis of integer images to study compressiontechniques for astronomical images in floating-point format. In the cases we are mainly con-cerned with here, these images originally had in-teger pixel values, but were converted into 32-bitIEEE floating-point format during the calibrationprocessing (e.g., bias subtraction, flat-fielding, ab-solute flux calibration, etc.). Equation 3 appliesto floating point images as well as integer im-ages, however typical floating-point astronomicalimages contain so much noise that it is impossibleto achieve significant amounts of compression withlossless algorithms (often less than a factor of 2).One explanation for this excess noise is that a 32-bit IEEE floating-point number can express 7 deci-mal places of precision but this usually far exceedsthe inherent precision of individual image pixelvalues. As a result many of the least significantbits in the mantissa of the floating-point pixel val-ues are effectively filled with quasi-random noisewhich is inherently uncompressible. While thereare counter-examples of floating-point FITS arraysthat have very little noise and can be losslesslycompressed effectively (e.g., generated by theoret-ical simulations), these are not typical of the typesof floating-point images commonly found in largeastronomical data archives and are not the subjectof this article.

There are many published articles on loss-less floating-point data compression schemes (seeLindstrom & Isenburg 2006, and reference thereinas recent examples) but it is beyond the scope hereto summarize them in detail. The main point isthat ultimately all of these lossless compressiontechniques face the same Shannon entropy limit,and the only way to achieve greater compressionof noisy floating-point images is to use techniquesthat discard some of the noise. These methods

are technically "lossy" because they do not ex-actly preserve the pixel values, however, if onlynoise is discarded, then the compression can stillbe considered lossless from a scientific standpointbecause all the useful information is retained.

In the remainder of this article we describe alossy compression technique for floating-point as-tronomical images that provides an optimal com-bination of speed, compression ratio, and preser-vation of information content. It is faster andachieves much higher compression than genericlossless file compression algorithms like GZIP(Gailly & Adler 1992), yet it produces no sig-nificant loss of information in the image whenused appropriately. In §2 we describe in detailthe quantization and compression techniques thathave been implemented in our publicly availablefpack and funpack image compression utility pro-grams. Then in §3 we describe the results ofseveral experiments that demonstrate that astro-nomical floating-point images can be compressedby up to a factor of 10 without significant lossof astrometric or photometric precision. Finally,^4 summarizes the results and gives recommenda-tions for achieving the best compression of astro-nomical images.

2. Quantization and Compression Meth-ods

The most common lossy compression techniquefor floating-point images is to preprocess the pixelvalues by quantizing them into a smaller set of dis-crete values prior to applying a lossless compres-sion algorithm. In the simplest case, the valuesare rounded into a grid of equally spaced floating-point levels. This reduces the number of differentbit patterns in the image pixels (i.e., reduces theentropy in the image) and improves the efficiencyof file compression programs like GZIP which ac-cumulate a dictionary of the most common bit pat-terns in the file and represent them using a shortercode in the compressed file. Watson (2002) ap-plied an analogous technique to integer images.In order to accommodate compression algorithms,like Rice, that only operate on integer arrays, thequantized floating-point values are usually repre-sented by scaled integers so that the image pixel

values are approximated by

FloatValue = ScaleFactorx IntegerValue+ZeroPoint(4)

Note that this integer scaling technique was theonly way to represent floating-point images in theFITS data format (Hanisch et al. 2001) beforesupport for the IEEE floating-point format wasofficially added in 1990.

As an aside, there are many articles in the liter-ature (e.g., Nieto-Santisteban et al. 1999; Gowen& Smith 2003; Nicula, Berghmans, & Hochedez2005; Seaman, White, & Pence 2009; Bernstein etal. 2010) that advocate using a square root scal-ing function, in part because the Poissonian shotnoise scales by this same factor. What is usuallynot stated in these studies is that practically thesame increase in compression that is obtained af-ter applying a square root scaling function can beobtained by linearly scaling all the pixels in theimage by the same factor as is applied to the back-ground pixels during the square root scaling. Sincethe compression ratio is determined mainly by thenoise in the background areas of the image (fromequation 3), the amount of scaling that is appliedto the relatively infrequent bright pixels usuallymakes little difference to the overall compressionratio of the image. In experiments on simulatedastronomical images, Bernstein et al. (2010) foundthat using square -root scaling only produced sig-nificantly better compression than linear scalingwhen more than 10% of the image pixels are af-fected by isolated bright objects or cosmic rays.

2.1. FITS Tiled Image Compression Con-vention

White & Greenfield (1999) developed the tech-nique that forms the basis of the FITS tiled-imagecompression format that is used here with a fewnew refinements. Each row of the image (or inprinciple, any other rectangular `tile' in the image)is compressed separately to provide fast randomaccess to individual sections of an image withouthaving to uncompress the entire image. In thecase of floating-point images, the pixel values areconverted to integers with a scale factor that, bydefault, is proportional to the measured amountof noise in that tile.

The noise in each the of the image is calcu-lated using a robust algorithm that was originally

developed to measure the signal-to-noise in spec-troscopic data (Stoehr et al. 2007). In particular,we use their third order "Median Absolute Dif-ference" (MAD) formula to compute the standarddeviation of the pixel values:

o- = 0.6052 x med i a n ( — Xi-2 + 2x i — x i+ 2 ) (5)

where i is the vector index of the pixel within eachrow of the image, and x i is the value of the ith

pixel. The median value is computed over all thepixels in each row of the image. In the limitingcase where the pixel values have a Gaussian dis-tribution, this formula converges to same value asthe Standard Deviation of the pixels. Note thatthis formula is not affected by linear intensity gra-dients across the image (which are canceled out bythe first and third terms), and the use of the me-dian makes the result insensitive to the presenceof outlying large pixel values. For example, if onerandomly sets 5% of the pixels in an image withGaussian distributed noise to very large values tosimulate the effects of cosmic rays, then the MADnoise estimate only increases by about 20%.

Once the noise level has been calculated, thefloating-point pixels are quantized into scaled in-tegers where the quantized levels are spaced atsome user-specified fraction, q, of the noise, sothat the spacing is given by Q /q. Normalizing thequantization spacing to a is a convenient way toproduce similar quality compressed images regard-less of the intrinsic noise level in the image. Thescaled integers are then compressed using the de-fault Rice algorithm, or one of the other optionalcompression algorithms. Finally, the compressedstream of bytes is stored in a FITS binary tablestructure as defined in the FITS tiled image com-pression convention (Pence et al. 2000; Seaman etal. 2007) .

Since the noise a in the array of quantized in-tegers is simply equal to q, the expected imagecompression ratio, from equation 3. is given by

R = BITPIX/(109 2 (q) -4- 1.792 ± K) (6)

For example, quantizing an image using q values of1 or 4 will produce compression ratios of about 10or 6.4, respectively, when using the Rice algorithmthat has K ^_ 1.2. In most cases, the compressionratio does not depend very much on the distri-bution of objects or structures within the image

itself. Note that this formula tends to break downfor q values much less than 1 because many com-pression algorithms become less effective on verylow entropy images and because the size of theFITS file header, which remains uncompressed,becomes relatively more significant.

One important caveat with the use of this quan-tizing method is that if the noise level in the imageis significantly overestimated for some reason (forour purposes it is generally sufficient if it is accu-rate to within about a factor of 2), then the im-age may be inadvertently quantized more coarselythan expected for a given q value, thus possiblycausing a loss of information in the compressedimage. Circumstances where the MAD algorithmcould overestimate the noise include the following:

If a large fraction of the pixels in an im-age tile are covered by bright objects andhave values (and noise) which is many timesgreater than in the fainter pixels, then theMAD noise estimate will be representativeof those bright pixels and not that of thefainter "background" pixels., This may causethe fainter pixels to be more coarsely quan-tized than would be desired.

If a significant amount of the pixel-to-pixelvariation in the image is real and not justdue to noise, (e.g., if there are large pixel-to-pixel variations in the detector sensitiv-ity, or if the observed object contains signifi-cant structure on a pixel size scale), then thenoise level may be overestimated.

If the noise in an image is anti-correlatedbetween adjacent pixels (e.g., after certaintypes of image convolutions) then the noisewill be overestimated. Note, however, thatequation 5 is a function only of the values inevery other pixel in each row of the image, sothe anti-correlation scale length must extendover at least 2 pixels to affect the MAD noiseestimate.

Our fpack compression program (see X2.4) offersseveral options for dealing with this issue. Onesimple method is to just compress the image us-ing a larger q value to compensate for the pos-sible -\-,IAD noise overestimate, although this cannegatively affect the overall compression ratio of

the image. Another option in cases where onlya small fraction of the rows in an image mightbe affected is to compress the entire image as asingle tile, rather than using the default row-by-row tiling pattern, so that the MAD noise estimatethen more accurately reflects the noise in the back-ground regions of the image as a whole. Finally,fpack users do not need to rely on the MAD noiseestimate at all, and instead can directly specifythe desired spacing between the quantization lev-els. This latter option is especially appropriatefor projects that generate large amounts of rela-tively homogeneous images because it ensures thatall the images will be compressed using the samequantization factor. This method has the addedadvantage that it will improve the compressionspeed because it is not necessary to compute theMAD noise value in this case.

2.2. Benefits of Dithering

The effects of linear quantization are naturallygreater in the fainter areas of an image than for thebrighter pixels where the Poissonian uncertaintyof the photon counts can be much larger than thequantized spacing. Measurement of the local `sky'background around the image of a star or galaxycan be especially vulnerable to the effects of quan-tization, in part because it is usually necessary toidentify and exclude pixels that are affected byother objects or by defects in the detector. If animage is coarsely quantized then most of the back-ground pixels will have a value equal to one of the 2quantization levels that bracket the true sky level.Any algorithm for rejecting outlying contaminatedpixels will tend to reject more of the pixels thathave the value that is further from the true skylevel, and thus the sky level that is derived fromthe remaining pixels will tend to be biased towardsthe value of the quantized level that lies closest tothe true sky level.

The issue of detecting small amplitude signalsin a. quantized system is a well-studied problemin engineering and communications fields wherethe phenomenon is known as "stochastic reso-nance". The somewhat counter-intuitive solutionfor improving the signal-to-noise of measurementsof quantized data is to add a moderate amountof noise into the system. When applied to im-ages, this technique is commonly called "dither-ing". Widrow & Kollar (2008) devote 2 chapters of

their book to the theory and practice of ditheringand recommend using a clever "subtractive dither-ing" technique, first proposed by Roberts (1962).This technique overcomes the drawback of havingto add noise to the image: a dither is added to thequantizer input, and the same dither is subtractedagain from the quantizer output. The dither thusbehaves as a catalyst which makes the processwork better but does not appear in the output im-age. We have adopted this technique when quan-tizing floating-point images by adding a randomdither, Ri, with a value uniformly distributed be-tween 0 and 1 during the scaling process,

h = round (Fi /ScaleFactor + Ri — 0.5) (7)

where Fi is the original floating-point value and 1iis the the quantized integer value. The interest-ing trick that distinguishes subtractive ditheringfrom ordinary dithering methods is that exactlythe same random dither value is subtracted whenconverting back to the quantized float value:

Fi = (Ii + 0.5 — Ri) x ScaleFactor (8)

The net effect of this subtractive dithering oper-ation is to shift the entire grid of linearly spacedintensity levels up or down by a random amounton a pixel by pixel basis. It should be noted thatthe dithered values are uniformly distributed be-tween the quantized levels in this implementation.One possible future enhancement may be to use atriangular or Gaussian dither (Widrow & Kollar2008) which may more closely replicate the actualdistribution of pixel values in the image.

In order to use this subtractive ditheringmethod it is necessary to define a specific pseudorandom number generator (PRNG) algorithm foruse by both the compressor and the uncompres-sor so that the same predictable sequence of ran-dom numbers is used in both cases. We adoptedthe PRNG algorithm described by Park & Miller(1988) which has been shown to produce statis-tically independent random numbers uniformlydistributed between 0 and 1. However, for prag-matic reasons we do not compute a unique randomnumber for every single image pixel because (a)it would add significant computational overheadrelative to the very efficient Rice compression al-gorithm, and (b) true randomness is not requiredfor this purpose since even crude dithering pat-terns would still help to mitigate the measurement

biases in quantized images. Our compromise so-lution is to calculate a look-up-table of 10000 ran-dom numbers using the above mentioned PRNG,and then repeatedly recycle through this LUTwhen calculating the amount of dither for eachpixel in the image. As a precaution against in-troducing regular cyclical noise patterns into theimage, a pseudo random starting offset is com-puted each time when recycling through the LUT.Finally, 1 out of a possible 10000 initial seed val-ues for the entire dithering process is computedbased on the system clock time (or optionally thechecksum of the first row of pixel values in theimage) to ensure that the same dithering patternis not used in every image. This initial seed valueis stored in the header of the compressed imagefor reuse when uncompressing the image.

2.3. Quantization Noise

While quantization reduces the entropy in therepresentation of the pixel values (thus improv-ing the image compression ratio), any quantizationoperation, even with subtractive dithering, modi-fies the pixel values and thus inherently increasesthe RMS noise in the pixel-to-pixel variations inthe image by an amount given by:

aq = as + 0 2 /12 (9)

where the total variance aq in the quantized im-age is the quadrature sum of the variance cointhe original image plus the quantization noise vari-ance, and where A is the intensity spacing betweenthe quantized levels. The factor 1/12 comes fromthe variance of a uniform random distribution withunit width (Janesick 2001, also see the appendixto Paper 1). Substituting A = a0 /q, the fractionalincrease in the noise caused by quantizing the im-age can be expressed as

(7q/CO = 1 + 1/(12g2) ( 10)

Figure 1 shows how both the fractional noise ra-tio (from equation 10) and the compression ratio(from equation 6) depend on q. To foreshadow theresults of the experiments that will be describedin §3, this figure shows that q values in the rangeof 4 to I provide a good combination of high com-pression ratio with relatively little added noise.

It should be cautioned that if a floating-point image is repeatedly compressed and uncom-pressed, then the cumulative quantization noise

0.5

.75

q =

1

1.52

86 4 3

10 15

Compression Ratio (R)

Fig. l.— Relationship between the compressionratio and the fractional increase in the backgroundnoise, for a range of q quantization parameter val-ues.

will be given by

°'g lao = V'I + N/(12g2 ) ( 11)

where N is the number of compression and uncom-pression cycles. (This assumes that the ditheringis re-randomized during each cycle, which is al-ways the case in our implementation of the sub-tractive dithering algorithm). The amplitude ofthis effect depends strongly on the q value. Toput this in practical terms, an image can be com-pressed and uncompressed at least 16 times usingq > 4 before the noise would increase to the samelevel equivalent to compressing the image oncewith q = 1. But if the image is compressed multi-ple times using q = 1, the noise would increase toscientifically unacceptable levels after just a fewcycles. Ideally, an image should only be com-pressed once, and then all the subsequent dataanalysis should be performed directly on the tile-compressed FITS file. If the software cannot readthe compressed format directly, then it should op-erate on an uncompressed version of the originalcompressed file, which then should not be recom-pressed.

15

10b

V

v'0 5Ze

2.4. (pack and funpack Compression Util-ity Programs

3.1. Data Model and Measurement Meth-ods

In order to make the image quantization andcompression techniques that are described in theprevious sections more widely available to theastronomical community, we have developed apair of general purpose utility programs, calledfpack and funpack (Seaman et al. 2007), whichcan be used to compress and uncompress anyFITS image, in integer or floating-point format.These utilities rely on the underlying CFITSIOlibrary (Pence 1999) to perform the quantiza-tion and compression operations. The fpackand funpack utility programs were used in theexperiments that are described in the follow-ing sections to quantize and compress the im-ages. Further information about fpack and fun-pack is available from the HEASARC web site athttp://heasarc.gsfc.nasa.gov/fitsio/fpack/.

3. Experiments on Quantized Images

In this section we present the results of experi-ments designed to show how the measurements ofobjects in an image are affected as the image isquantized by varying degrees. In particular, wewill verify that the noise in the image increases asa function of q by the amount predicted by equa-tion 10, and more significantly, that the statisti-cal errors on the magnitude and position measure-ments of faint objects in an image, which are lim-ited mainly by the background noise, also increaseby a similar factor. These results will provide gen-eral guidelines for achieving the greatest amountof image compression while still preserving the re-quired level of scientific precision in the image.

Section 3.1 describes the method of construct-ing the simulated CCD images that are used inthe first 2 experiments. The first experiment, in§3.2, examines how quantization affects the uncer-tainties of measurements of single star images, andthe second experiment, in §3.3, examines the casewhere many quantized images are added togetherto detect sources far below the detection thresholdof a single image. Finally, the third experiment,in §3.4, is performed on a set of actual astronomi-cal images to verify the results obtained from thesynthetic images.

In order to determine how quantization affectsthe precision of measurements of objects in an im-age, we generated a large sample of realistic CCDstar images with known input positions and mag-nitudes. This allows us to precisely calculate theerrors on the measured positions and magnitudesin the quantized images. All the stars have circu-lar Gaussian profiles with a = 1.0 and FWHM =2.35 pixels. This is typical of the spatial resolutioncommonly found in astronomical CCD images andis adequate to avoid the difficulties when analyz-ing spatially undersampled images. The centrallocation of the star images, relative to the pixelgrid, was varied so as to average out any subtlebiases in the subsequent star detection and mea-surement steps that might depend on the exactposition. The total integrated flux in the stars cov-ered a range of 10 magnitudes (a factor of 10000in intensity) in 0.5 magnitude increments. Finally,the sky background was simulated by adding 1000counts to each pixel.

Poissonian-distributed shot noise and Gaussiandistributed "read-out" noise was then added toeach of these star images to simulate real CCDimages. The shot noise in each pixel was ran-domly calculated using a a equal to the squareroot of that pixel value (which implicitly assumesthat the "gain" of the simulated CCD has beenset to 1 electron per analog-to-digital readoutcount), and the readout noise was calculated us-ing a = 10. The read-out noise in these imagesis relatively small compared to the shot noise inthe sky background, which is usually the casefor real astronomical CCD images that have amoderately bright background level. The totalnoise in the background areas of these images has

= 1000 + 10 2 = 33.2. Different starting ran-dom seed values were used so that the actual noisedistribution varies in every image.

The widely used SExtractor source extractionprogram (Bertin & Arnouts 1996) was employedto objectively detect and measure the position andmagnitude of the stars in these simulated, noisyCCD images. SExtractor calculated the positions(given by the XIMAGE and YIMAGE outputparameters) from the flux-weighted centroid of allthe pixels in each star image above an empirically

7

determined flux detection threshold. The totalmagnitude of each star (given by the MAG-APERparameter) was measured within a 7-pixel diame-ter aperture (i.e., out to 3.5 a of the Gaussian pro-file) around the centroid position. We calculatedthe actual errors on these measurements from thedifferences between the known input position andflux of each star in the synthetic image. For ref-erence, the faintest stars that SExtractor couldreliably detect in these images contained about1000 net counts, which is coincidentally equal tothe mean sky counts in each pixel. We arbitrarilyset the zero point of the instrumental magnitudescale,

m = —2.5log(flux) + ZeroPoint (12)

so that these faintest detected stars have a mag-nitude of 20.0.

We repeated the SExtractor measurements ofthe simulated CCD images after quantizing andcompressing each image with fpack using q val-ues ranging from 8 to 0.25, both with and with-out the subtractive dithering option. Since theSExtractor program cannot directly read imagesin the compressed FITS format, it was necessaryto convert them back into the standard FITS im-age format using funpack (but the pixel valuesremain quantized of course). In the case wheredithering was not applied, we also selected theoption to compress the entire image as a singlelarge tile so that all the pixels are quantized intothe same fixed grid of intensity values. If we hadused the default row-by-row tiling option instead,it effectively would have introduced some dither-ing of the quantized levels between adjacent rows,and the results would be intermediate between thedithered and non-dithered cases presented below.

Figure 2 shows the dramatic effect that subtrac-tive dithering has on the histogram of the pixelvalues in a q = I quantized image; the ditheredimage histogram is nearly identical to the that ofthe original image, whereas the non-dithered his-togram clearly shows the coarse intensity binning.Without dithering, the image breaks up into dis-crete "bands" of constant intensity which makes itmore difficult to detect faint features in the image.This effect is shown in Figure 3, (using an evenmore coarsely quantized q = 0.5 image to enhancethe effect), where the majority of the backgroundpixels all have the same (medium gray) intensity

900 950 1000 1050 1100

Pixel Value

Fig. 2.— Histograms of the pixel intensity dis-tribution in one of our test images which demon-strates the beneficial effects of subtractive dither-ing when quantizing the image with q = 1. Whendithering is applied (the finely stepped histogram),the pixel distribution closely matches that in theoriginal image (the continuous curve). Whendithering is not applied (the coarsely stepped his-togram) then subtle gradations in image intensityare lost.

Fig. 3.— Images of a pair of faint (m = 20) ar-tificial stars (left panel) and the same image afterquantizing the image with q = 0.5 with dither-ing (middle panel) and without dithering (rightpanel). Without dithering, the background breaksup into broad patches of constant intensity level,and the star images become harder to detect.

104

N

5000

value.

3.2. Experiment 1: Single Images

In this first experiment, we examine how theuncertainties of the photometric and astrometricmeasurements of individual stars depend on theq quantization factor. To measure this effect, wecomputed the Standard Deviation of the magni-tude and position errors (i.e., the value computedby SExtractor minus the known input magnitudeor position value) for 6250 simulated star imagesat each 0.5 magnitude increment. Figure 4 showshow the magnitude uncertainties decrease as thebrightness of the star increases (towards the right).The lower, thicker line in the figure was derivedfrom the original, unquantized images and showsthat the statistical uncertainty on the magnitudesdecreases from a = 0.23 for the faintest detectablestars (with m = 20), to a = 0.018 for stars thatare 3 magnitudes brighter. The line has a slopeclose to 1.0 (in log - log coordinates) as expectedin the limiting case where the noise is dominatedby the sky, and hence the signal-to-noise ratio ofthe measurement increases in direct proportion tothe signal.

The other 3 lines in Figure 4 were derived fromthe same images after they were first quantizedand compressed with successively coarser q valuesof 1.0, 0.5, and 0.25, when applying the subtrac-tive dithering option. The vertical displacementof these lines shows that the measurement errorsare systematically larger in the quantized imagesas compared to the measurements in the originalimage. This increase is relatively small for q >1, but increases rapidly for coarser quantizationvalues.

Figure 5 shows a similar plot of the StandardDeviation of the stellar centroid measurements, inunits of pixels, as a function of the magnitude ofthe star and the q quantization factor. In the origi-nal unquantized images, the positional uncertaintydecreases from a = 0.24 pixels for the m = 20stars, to a = 0.035 pixels for the stars that are 3magnitudes brighter. The positional measurementuncertainties are larger in the quantized image byabout the same factor as the increase in the mag-nitude uncertainties shown in Figure 4.

In order to better quantify the effects shownin these 2 figures, Table 1 summarizes how the

0.2

Withc

0.1q

7 0.25"10.059

0.02

0.01 2019 18 17

Magnitude

Fig. 4.— The effect of q on the measured magni-tude uncertainties when dithering is applied.

x 0.2 With dithering

q0.1

0.25

712 s0G0.05

20 19 18 17

Magnitude

Fig. 5.— The effect of q on the measured posi-tional uncertainties when dithering is applied.

noise and the measurement uncertainties increaseas the images are quantized more coarsely. Thepercentage increase in the measured MAD noise

level in the quantized images is given in column 2and agrees exactly with the predicted value fromequation 10. More strikingly, the magnitude andposition uncertainties for the faintest stars also in-crease by about the same factor as the noise, as

shown in columns 3 and 4. This means that equa-tion 10 also provides a good estimate of how muchthe measurement uncertainties of the faintest ob-jects in an image, which are limited by the back-ground noise, will increase when the image isquantized.

Finally, columns 5 and 6 in Table 1 give thecorresponding increase in measurement uncertain-ties for stars that are 5 magnitudes, or a factor

of 100, brighter than the image detection thresh-old. As expected, these brighter objects are rel-atively less affected by quantization because theinherent Poissonian noise in the brighter pixels islarger than the spacing between the quantized lev-els. Also, the small formal statistical errors onthe magnitudes and positions of the brighter stars(which are less than 0.001 of a magnitude or pixel,respectively, in this case) are often insignificantcompared to the systematic errors in the absolutecalibration of the measurements.

For comparison, we also measured the posi-tion and magnitude uncertainties in images thatwere quantized but without subtractive dithering.These results are shown in Figures 6 and 7. It isapparent from comparison with Figures 4 and 5that the uncertainties in the undithered quantizedimages are much larger than in the dithered case.In order to achieve the same level of photometricprecision in this test, the undithered images mustbe quantized by about a factor of 10 finer spacingthan in the dithered case, which greatly reducesthe resulting compression ratio of the image. Aswas discussed previously in §2, one reason for thisdifference is that dithering the pixel values has alarge effect on the measured background sky in-

tensity level. This is illustrated in Figure 8, whichshows the average difference between the magni-tudes measured in the original image and in a q= 0.5 quantized version of the same image. In thenon-dithered case, even stars that are 3 – 4 mag-

nitudes brighter than the detection threshold are

still significantly biased. Note that the amount of

0.2

No dithering

q0.1

a^

,b 0.05

X0.02

0.01 2019 18 17

Magnitude

Fig. 6.— Same as Figure 4 but without dithering.The magnitude uncertainties are much larger thanin the dithered case.

No dithering

ZZ

q1

19 18 17

Magnitude

Fig. 7.— Same as Figure 5 but without dithering.

0.2

a

c

. 0.1

b

.0.05u

20

10

TABLE 1

INCREASED NOISE AND MEASUREMENT UNCERTAINTIES AS FUNCTION OF Q

q aQf6o Ama92 Apos2 Amag5 Apos5

4 0.26% 0.31% 0.18% 0.10% 0.20%2 1.04% 1.1% 0.93% 0.25% 0.83%1 4.08% 5.6% 4.1% 1.7% 3.4%

0.5 15.5% 19% 15% 5.8% 12%

bias shown here depends on how closely the near-est quantized level happens to lie to the true skylevel in the image.

The benefits of dithering are less pronouncedon the astrometric measurements, as can be seenby comparing Figures 5 and 7. The non-ditheredimage only needs to be quantized with about afactor of 2 finer spacing to give the same positionalprecision as in the dithered image.

3.3. Experiment 2: Co-added Images

The second experiment investigates how wellfaint star images that are below the detectionthreshold of a single image can be measured af-ter co-adding many images together. To first or-der, one would expect that the noise in the co-added image will decrease by a factor equal tothe square root of the number of co-added im-ages. For this experiment we co-added 250 star im-ages, similar to those generated in the first exper-iment, which should allow detection of stars thatare about 2.5 log 250 = 3.0 magnitudes fainterthan in the single images. We repeated this co-adding experiment multiple times, using differentstarting seed values for the random backgroundnoise distribution, to generate a. total of 250 simu-lated co-added star images in each of the 0.5 mag-nitude bins. The magnitude and position of eachof these co-added stars were then measured withthe SExtractor program, just as in the first exper-iment.

Figure 9 shows the results of the photometricmeasurements in the co-added images. Other thanthe fact that the lines in the plots exhibit morestatistical scatter, since they were derived from asample of only 250 star measurements in each 0.5magnitude bin instead of 6250, these results arevery similar to the results from the single images

0.6

0q = 0.5

W 0.4 Not ditheredb

0.2

0Dithered

20 19 18 17 16

Magnitude

Fig. 8.— Mean errors on the measured magni-tudes of the fainter stars when coarsely quantizingan image with q = 0.5, with and without dither-ing. The large errors in the non-dithered case arecaused by systematic biases in the intensity mea-surements of the sky background. The amplitudeof the errors shown here are only representativeof one specific case and depend on how closelythe nearest quantized background level (withoutdithering) happens to match the true sky inten-sity level in the image.

11

With dithering

q0.25

0.5

22 21 20

Magnitude

0.2bY,

G 0.1

Vr.x0.05b8b4X0.02

0.01 L23

Fig. 9.— The effect of q on the measured magni-tude uncertainties in the sum of 250 images, withdithering.

0.2nTa 0.1

V

X0.05b

abn:'10.02

0.01 L23 22 21 20

Magnitude

Fig. 10.— Same as Figure 9, but without dither-ing. Dithering the 250 individual images has littleeffect on the precision of the magnitude measure-ments in the summed image.

shown in Figure 4, after allowing for the expectedshift of the horizontal axis by 3 magnitudes. Thissimilarity confirms that the photometric measure-ments in the co-added images have the same de-pendence on q as in the individual images. In par-ticular, it confirms that information about faintstars that are well below the detection thresholdin a single image is still preserved in the quantizedimages and can be detected after co-adding manyquantized images together.

We also confirmed that the astrometric preci-sion in the co-added images shows the same de-pendence on q as the measurements in the singleimages. A plot of the positional uncertainty as afunction of magnitude (not shown here) is almostidentical to Figure 5, except for the 3 magnitudeshift of the horizontal axis.

Finally, Figure 10 shows the magnitude uncer-tainties when the quantized images are not sub-tractively dithered before co-adding them. Unlikethe case for single images (as compared in Figures4 and 6), dithering has little effect on the pre-cision of the photometry in the co-added image.This is because the co-adding process effectivelyintroduces its own form of pixel dithering if thequantized levels in the different images are ran-domly offset with respect to each other. Thus,there is little added benefit by also dithering thepixel values within each image. Price-Whelan &Hogg (2010) demonstrate an even more extremecase where accurate photometry and astrometrycould be performed on a co-added sum of 1024images, each quantized with q = 1/16 = 0.0625without any dithering.

No dithering

q0.25

12

Fig. 11.— Sample region in one of the test im-ages taken with the Isaac Newton Telescope at LaPalma. This shows about 10% of the total imagearea.

3.4. Experiment 3: Real AstronomicalImages

In the third and final experiment we performedtests on a set of real astronomical CCD imagesto confirm the previous results on synthetic im-ages. We used a sequence of 20 similar CCDimages that show a random distribution of faintstars and galaxies taken with the 2.5-m IsaacNewton Telescope at La Palma (Figure 11). Allthe exposures were for 600 seconds through a V-band filter with the same pointing on the sky(12h51-, 26°24'). These images are publicly avail-able through the virtual observatory portal athttp://portal-nvo.lioao.edu .

We performed 2 tests to measure the effects ofquantization on these images. In the first test wequantized one of the images using q = 1 (with sub-tractive dithering), by compressing it with fpackand then uncompressing it again with (unpack.As predicted by equation 6, the compression ratioachieved by fpack on these images depends almostentirely on the q value that is used. We then com-pared the SExtractor magnitude measurements inthe original image to those derived from the quan-tized image. Since we do not know the true mag-nitudes of the stars in this image (unlike in theexperiments on the synthetic stars), we can onlycompare the measurements of the stars in the 2

images, both of which have measurement uncer-tainties. The background noise increased by 4.1 %,from a = 22.79 in the original image to a = 23.73in the quantized image, as predicted by equation10. Also as expected, the statistical errors (as cal-culated by SExtractor) on the magnitude measure-ments in the quantized image increased by a sim-ilar amount, 3.8% on average, over the same mea-surements in the original image. The top panel ofFigure 12 shows the difference between the 2 mag-nitude measurements for each star as a function ofthe magnitude of the star. The lower panel showsthat the corresponding relative errors (the mag-nitude difference divided by the statistical erroron that magnitude as calculated by SExtractor).The RIMS value of all the relative errors is 0.37v;The fact that this is much less than the ^ la thatone would expect when comparing the magnitudesderived from 2 identical CCD images of the samestars, confirms that quantizing the image with q= I (which results in a compression ratio of 10)has not introduced statistically significant photo-metric errors.

In the second test we co-added the 20 CCDimages, before and after quantizing them with q= 1, and then calculated the differences in themeasured magnitudes in the 2 co-added images.The results, shown in Figure 13, are similar tothose in Figure 12, derived from a single image,except that the faintest detected stars are now

2.51og 20 = 1.6 mag fainter, as expected fromco-adding 20 images. The RMS value of the rel-ative errors shown in the lower panel is 0.34a inthis case, which again demonstrates that quantiz-ing the images with q = 1 has not produced anysignificant photometric errors, even in objects thatare fainter than the detection threshold in a singleimage.

4. Summary and Discussion

The main purpose of this work has been to finda more effective compression method for floating-point astronomical images than is provided by thelossless methods that are commonly used (suchas GZIP). These floating-point images typicallydo not compress well with lossless algorithms be-cause a large fraction of the bits in each pixelvalue representation contain no significant infor-mation and are effectively filled with uncompress-

13

Mean of 20 images, with q I

-4V__

%

0.05

0

—0.05

0

ible noise. We have adopted the method of elim-

inating some of this noise by quantizing the pixel

values into a set of discrete, linearly spaced inten-

sity levels, which are represented by scaled integer

values. The scaled integers can then be efficiently

compressed using the very fast Rice algorithm. In

order to make these compression techniques more

widely available, we have produced a pair of util-

ity programs called fpack and funpack that can be

used to compress any FITS format image.

For convenience, we define a quantization pa-

rameter, q, which is equal to the measured RMS

noise in background regions of the image divided

by the spacing between the quantized intensity

levels. Given a particular q value when quantizing

and compressing an image, one can calculate from

fundamental principles the expected image com-

pression ratio, as given by equation 6. Coarser

quantization (i.e., smaller q values) gives greater

image compression, but at the same time it in-

creases the RXIS pixel-to-pixel noise by an amount

given by equation 10 which also tends to degrade

the precision of the magnitude and position mea-

surements of the objects in the image.

Our series of experiments on simulated and on

real astronomical CCD images demonstrate that

the noise equation 10 also gives a good estimate

of the increase in measurement uncertainties for

objects near the detection threshold in a quan-

tized image (which are noise limited). For many

practical applications, q values between 1 and 4

provide a good combination of high compression

and low increased noise: q = 1 gives a compres-

sion ratio of about 10 while increasing the noise

and the measurement uncertainties by about 4%,

and q = 4 gives a compression ratio of 6 with a

negligible 0.26% increase in

the noise and mea-

surement uncertainties. In 'quick-look' types of

applications, where high scientific accuracy is not

of primary importance, even greater compression

can be obtained by using q values less than 1.

One necessary requirement for achievin g these

results, however, is that the pixel values must be

dithered during the quantization process to im-

prove the signal-to-noise in measurements of faint

signals in the image. Without dithering, the faint

areas of coarsely quantized images tend break up

into broad plateaus of constant intensity, and it

becomes more difficult to detect the faintest ob-

jects in the image. In astronomical images this ef-

0.05

Single image, with q I

0

—0.05

^ 0

22 21 20 19 18 17

Magnitude

Fig. 12.— Differences in star magnitudes mea-

sured in a CCD image and a q = I quantized ver-

sion of the same image. The upper panel shows

the absolute magnitude differences, and the lower

panel shows the relative differences after dividing

by the statistical error.

23 22 21 20 19

Magnitude

Fig. 13.— Same as Figure 12, except the magni-

tudes are measured in

the sum of 20 CCD images

and in the sum of the q = I quantized version of

each of the images.

14

feet can seriously impact intensity measurementsof the local sky background which can lead tosystematic biases in photometric measurements inthe quantized image. In our experiments usingSExtractor, a dithered image can be quantized byabout a factor of 10 coarser than a non-ditheredimage (thus resulting in a much higher compres-sion ratio) while still preserving the same amountof photometric precision.

Several other recent studies have reached sim-ilar conclusions about the benefits quantizing as-tronomical images to reduce the amount of noiseand improve the compression ratio. Price-Whelan& Hogg (2010) performed photometric and astro-metric measurements on stars in simulated CCDimages (using a slightly different methodologythan used here) and concluded that no significanterrors were introduced if the image is quantized(without dithering) with q > 2. Bernstein et al.(2010) found that using q = 1, along with square-root scaling does not induce any significant biasin weak-lensing shape measurements of galaxies(i.e., the second statistical moment of the image)in simulated images from future space-based imag-ing missions. Caldwell et al. (2010) describe howthe Kepler mission is performing on-board quan-tization of the images with q values as low as 1.15to obtain the necessary amount of compression ofthe downlinked telemetry while still preserving therequired high precision in the differential photom-etry measurements of stars. While we recognizethat applying any sort of lossy compression schemeto scientific data tends to go against the inclina-tion of scientists and data archive professionals,hopefully the results of the experiments performedhere and in the other cited references will help toalleviate these concerns.

While we believe our test results should be ap-plicable to a wide range of floatin g-point astro-nomical images, we strongly encourage users toperform their own tests to verify that this com-pression technique is appropriate for their owndata. Users can easier create a quantized anddithered version of any floating-point FITS imageby compressing it with (pack and then uncompress-ing it with funpack. This image can then then beprocessed just like the original image to see if thereare any significant differences in the results.

REFERENCES

Bernstein, G. M., Bebek, C., Rhodes, J,Stoughton, C., Vanderveld, A., & Yeh, P. 2010,PASP, 122, 336 arXiv:0910.4571vl

Bertin, E & Arnouts, S. 1996, A&AS, 117, 393,http: //www.astromatic.net/software/sextractor

Caldwell, D. A., et al. 2010, ApJLetters, 713, L92

Gailly, J.L. & Adler, M, 1992, in RFC 1952, avail-able from http://www.ietf.org/rfc/rfcl952.txt& http://www.gzip.org .

Goodrich, J. et al. 2004, Proc. SPIE, 5493, 538

Gowen, R. A., & Smith, A. 2003, Rev. ScientificInstr., 74, 3853

Hanisch, R. et al. 2001, A&A, 376, 359

Harvey, J. W. et al. 1996, Science, 272, 1284

Janesick, J. R. 2001, Scientific Charge-CoupledDevices (Bellingham, WA: SPIE Press)

Lindstrom, P. & Isenburg, M. 2006, IEEE Trans.on Visualization and Computer Graphics, 12,No. 5, 1245

Nicula, B., Berghmans, D. & Hochedez, J. 2005,Solar Phys., 228, 253

Nieto-Santisteban, M. A., Fixsen, D. J., Offen-berg, J. D., Hanisch, R. J. & Stockman, H. S.1999, in ASP Conf. Ser., Vol. 172, Astronomi-cal Data Analysis Software and Systems VIII,eds. D. M. Mehringer, R. L. Plante, & D. A.Roberts (San Francisco: ASP), 487

Park, S. K. & Miller, K. W. 1988, Comm. ACM,31,1192

Pence, W. D. 1999, in ASP Conf. Ser., Vol. 172,Astronomical Data, Analysis Software and Sys-tems VIII, eds. D. M. Mehringer, R. L. Plante,& D. A. Roberts (San Francisco: ASP), 487

Pence, W. D., Seaman, R., & White, R. L.2009,PASP, 121, 414

Pence, W. D., White, R. L., Greenfield, P., &Tody, D. 2000, in ASP Conf. Ser., Vol. 216,Astronomical Data Analysis Software and Sys-tems IX, eds. N. Manset, C. Veillet, & D. Crab-tree (San Francisco: ASP), 551

15

Price-Whelan, A. M., & Hogg, D. W. 2010, PASP,122, 207 arXiv:0910.2375v2

Rice, R. F., Yeh, P.-S., & Miller, W. H. 1993, inProc. of the 9th AIAA Computing in AerospaceConf., (AIAA-93-4541-CP), American Instituteof Aeronautics and Astronautics

Roberts, G. L. 1962, "Picture coding usingpseudo-random noise", IRE Transactions on In-formation Theory IT-8(2), 145

Seaman, R., Pence, W. D., White, R., Dickin-son, M., Valdes, F., & Zarate, N. 2007, in ASPConf. Ser., Vol. 376, Astronomical Data Analy-sis Software and Systems XVI, eds. R. A. Shaw,R. Hill, & D. J. Bell (San Francisco: ASP), 483

Seaman, R., White, R., & Pence, W. D. 2009, inASP Conf. Ser., Vol. 411, Astronomical DataAnalysis Software and Systems XVIII, eds. D.Bohlender, D. Durand, & P. Dowler (San Fran-cisco: ASP), 101

Shannon, C.E. 1948, A Mathematical Theory ofCommunication, Bell System Technical Jour-nal, 27, 379 & 623

Stoehr, F. et al. 2007, ST-ECF Newsletter. 42, 4

Valdes, F. G. & Swaters, R. A. 2007, in ASPConf. Ser., Vol. 376, Astronomical Data Analy-sis Software and Systems XVI, eds. R. A. Shaw,F. Hill, & D. J. Bell (San Francisco: ASP), 273

Watson, A. M. 2002, Rev. Mex. AA, 38, 233

White, R. L. & Becker, I. 1998, in SPIE Proc., Vol.3356, Space Telescopes and Instruments V, eds.P. Y. Bely & J. B. Breckinridge (Bellingham:SPIE), 823

White, R. & Greenfield, P. 1999, in ASP Conf.Ser., Vol. 172, ADASS VIII, eds. D. M.Mehringer, R. L. Plante, & D. A. Roberts (SanFrancisco: ASP). 125

Widrow, B. & Kolldr, L 2008, "QuantizationNoise: Roundoff Error in Digital Computa-tion. Signal Processing, Control, and Commu-nications", Cambridge University Press, Cam-bridge, UK

This 2-column preprint was prepared with the AAS LkT)EXmacros v5.2.

16