Halftoning-Inspired Methods for Halftoning-Inspired Methods for Foveation in Variable Acuity Foveation in Variable Acuity Superpixel Imager CamerasSuperpixel Imager Cameras
Thayne R. CoffmanThayne R. Coffman1,21,2
Prof. Brian L. EvansProf. Brian L. Evans11 (presenting)(presenting)Prof. Alan C. BovikProf. Alan C. Bovik11
1 1 Center for Perceptual SystemsCenter for Perceptual Systems Department of Electrical and Computer Department of Electrical and Computer EngineeringEngineering The University of Texas at AustinThe University of Texas at Austin http://www.cps.utexas.eduhttp://www.cps.utexas.edu
222121stst Century Century Technologies, Inc.Technologies, Inc. Austin, TexasAustin, Texas
November 2, 2005, IEEE Asilomar Conference on Signals, Systems, and Computers
Motivation: Foveated ImageryMotivation: Foveated Imagery Foveated imagery has Foveated imagery has
variable spatial variable spatial resolutionresolution Human visual systemHuman visual system
Provides simultaneousProvides simultaneous Wide field of viewWide field of view High resolution on High resolution on
regions of interestregions of interest Low bandwidthLow bandwidth
19% bandwidth means 19% bandwidth means 19% of “superpixels”19% of “superpixels”
No compression in No compression in talktalk Full resolutionFull resolution
(100% bandwidth)(100% bandwidth)Variable resolutionVariable resolution(19% bandwidth)(19% bandwidth)
Motivation: VASI™ CamerasMotivation: VASI™ Cameras Variable Acuity Superpixel Imager (VASI) camerasVariable Acuity Superpixel Imager (VASI) cameras
Generate foveated images by sharing charges on focal plane arrayGenerate foveated images by sharing charges on focal plane array Achieve 1000-4000 frames/sec (e.g. to measure engine RPMs)Achieve 1000-4000 frames/sec (e.g. to measure engine RPMs) Pixel sharing reconfigured to achieve a particular frame ratePixel sharing reconfigured to achieve a particular frame rate
Use of 1x1, 2x2, and 4x4 pixel sharing Use of 1x1, 2x2, and 4x4 pixel sharing [McCarley [McCarley et alet al., 2002]., 2002]
VASI is a trademark of Nova Sensors, Inc.VASI is a trademark of Nova Sensors, Inc. Images from [McCarley Images from [McCarley et alet al., 2002]., 2002]
The CatchThe Catch Desired spatial acuity (resolution) is usually Desired spatial acuity (resolution) is usually
specified as specified as a continuous amplitude function on the range (0,1]a continuous amplitude function on the range (0,1]
Translate desired resolution function to VASI™ Translate desired resolution function to VASI™ binary share/no-share control signal at very high binary share/no-share control signal at very high frame ratesframe rates
Foveation like the human eye (left pixelation)Foveation like the human eye (left pixelation)Two fovea (right pixelation)Two fovea (right pixelation)
Halftoning for VASI Control Halftoning for VASI Control SignalsSignals
Select a small number of test imagesSelect a small number of test images Manually specify desired resolution (using Gaussians)Manually specify desired resolution (using Gaussians) Evaluate halftoning methods to control signal Evaluate halftoning methods to control signal
translationtranslation Figures of merit to predict object recognition Figures of merit to predict object recognition
performanceperformance Peak SNR (PSNR)Peak SNR (PSNR) Weighted SNR (WSNR)Weighted SNR (WSNR) Universal Quality Index (UQI)Universal Quality Index (UQI) Percentage of Bandwidth (PBW)Percentage of Bandwidth (PBW) Control
signal for
charge sharing
at a pixel X
X
Shared up Shared left
Halftoning Methods ExploredHalftoning Methods Explored Classical screeningClassical screening
9-level clustered dot9-level clustered dot 9-level dispersed dot9-level dispersed dot
Block error Block error diffusiondiffusion
Floyd-Steinberg Floyd-Steinberg error diffusionerror diffusion
Blue noise ditheringBlue noise dithering White noiseWhite noise Specialized (non-Specialized (non-
general) methodsgeneral) methods vasiHalftonevasiHalftone vasiHalftone2vasiHalftone2
Dispersed dot screeningDispersed dot screeningF-S error diffusionF-S error diffusion
White noiseWhite noise vasiHalftonevasiHalftone
Specialized MethodsSpecialized Methods Generate semi-regularly spaced squaresGenerate semi-regularly spaced squares Square size varies with inverse of desired bandwidthSquare size varies with inverse of desired bandwidth Side is 2Side is 2KK in vasiHalftone & unconstrained in in vasiHalftone & unconstrained in
vasiHalftone2vasiHalftone2Full-resolution Full-resolution imageimage
Binarized Binarized control control (sharing) (sharing) signalsignal
Foveated imageFoveated imageContinuous Continuous desired desired resolution resolution signalsignal
Nontrivial Translation of Nontrivial Translation of Control SignalControl Signal
Halftoning algorithms aim to achieve a specific ratio of white Halftoning algorithms aim to achieve a specific ratio of white or black pixels, e.g.or black pixels, e.g. For constant I(r,c)=0.1, 10% of pixels will be white (“don’t share”)For constant I(r,c)=0.1, 10% of pixels will be white (“don’t share”) For constant I(r,c)=0.8, 80% of pixels will be white (“don’t share”)For constant I(r,c)=0.8, 80% of pixels will be white (“don’t share”)
But bandwidth and resolution are functions of geometry alsoBut bandwidth and resolution are functions of geometry also
Example 1Example 1 Example 2Example 2
50% of pixels don’t share charge: 1% bandwidth50% of pixels don’t share charge: 1% bandwidth46% of pixels don’t share charge: 15% bandwidth46% of pixels don’t share charge: 15% bandwidth
Control signalControl signal Resulting imageResulting image Control signalControl signal Resulting imageResulting image
Nontrivial Translation of Control Nontrivial Translation of Control SignalSignal
Relationship between percent of “don’t Relationship between percent of “don’t share” pixels and bandwidth is different for share” pixels and bandwidth is different for every halftoning methodevery halftoning method Eliminate nonlinearity by Eliminate nonlinearity by
applying an inverse functionapplying an inverse function Implemented with lookupImplemented with lookup
tables storing x = ftables storing x = f-1-1(y)(y) Given target bandwidth andGiven target bandwidth and
halftoning method, findhalftoning method, findaverage value (average value (xx-axis) to use-axis) to usein continuous control signalin continuous control signal
Stairstep patterns inStairstep patterns inrelationship limitrelationship limitcontrol over bandwidthcontrol over bandwidth
Floyd-Steinberg gives piecewise linear map Floyd-Steinberg gives piecewise linear map and best bandwidth controland best bandwidth control
Nontrivial Translation of Nontrivial Translation of Control SignalControl Signal
Results are greatly improvedResults are greatly improved Better bandwidth controlBetter bandwidth control Better foveation resultsBetter foveation results
Floyd-Steinberg (F-S) results belowFloyd-Steinberg (F-S) results belowDesired bandwidth Desired bandwidth =11.9% from ideal =11.9% from ideal control signalcontrol signal
UncompensatUncompensated control ed control signalsignal
Achieved Achieved bandwidth = bandwidth = 2.6%2.6%
Compensated Compensated control signalcontrol signal
Achieved Achieved bandwidth = bandwidth = 12.5%12.5%
Results: F-S Error DiffusionResults: F-S Error Diffusion Good performance and good bandwidth controlGood performance and good bandwidth control
Good SNR in foveae means accurate object recognitionGood SNR in foveae means accurate object recognition Good SNR in periphery means good object detectionGood SNR in periphery means good object detection Good bandwidth control means precise VASI frame Good bandwidth control means precise VASI frame
rate controlrate controlOriginalOriginal Sharing SignalSharing Signal Resulting ImageResulting Image
PSNR = 17.5 dB (33.3 dB in ROI)PSNR = 17.5 dB (33.3 dB in ROI)WSNR = 16.4 dB (33.8 dB in ROI)WSNR = 16.4 dB (33.8 dB in ROI)
Desired BW = 11.6%Desired BW = 11.6%Actual BW = 12.1%Actual BW = 12.1%Inflation = 4%Inflation = 4%
Results: vasiHalftone and Results: vasiHalftone and vasiHalftone2vasiHalftone2
For a given desired resolution signal, methods For a given desired resolution signal, methods consistentlyconsistently Had better PSNR & WSNR than other methodsHad better PSNR & WSNR than other methods Overshot desired bandwidth by ~30-100%Overshot desired bandwidth by ~30-100%
Essentially “cheating” by using extra bandwidthEssentially “cheating” by using extra bandwidthOriginalOriginal Sharing SignalSharing Signal Resulting ImageResulting Image
PSNR = 13.3 PSNR = 13.3 dBdB
WSNR = 16.9 WSNR = 16.9 dBdB
Desired BW = Desired BW = 9.6%9.6%
Actual BW = Actual BW = 18.8%18.8%
Inflation = 97%Inflation = 97%
Results: Other Halftoning Results: Other Halftoning MethodsMethods
MethodMethod Performance Performance (SNR)(SNR)
Bandwidth Bandwidth controlcontrol
Block error Block error diffusiondiffusion
PoorPoor GoodGood
Classical Classical screeningscreening
DecentDecent PoorPoor
Stochastic Stochastic methodsmethods
PoorPoor ““Catastrophic Catastrophic gray-out”gray-out”
OriginalOriginal Blue noiseBlue noiseBlock error diffusionBlock error diffusion
OriginalOriginal Clustered dotClustered dot Dispersed dotDispersed dot White noiseWhite noise
ConclusionsConclusions Floyd & Steinberg error diffusion gives the best Floyd & Steinberg error diffusion gives the best
results while still being able to control bandwidth results while still being able to control bandwidth preciselyprecisely
vasiHalftone and vasiHalftone2 vasiHalftone and vasiHalftone2 Consistently the best PSNR, WSNRConsistently the best PSNR, WSNR Poor bandwidth control – overshot specifications by 30-Poor bandwidth control – overshot specifications by 30-
100%100% Bandwidth inflation means it’s not a fair comparison Bandwidth inflation means it’s not a fair comparison
(they’re cheating)(they’re cheating) Stochastic methods (white & blue noise) perform Stochastic methods (white & blue noise) perform
poorlypoorly Outperformed by deterministic approachesOutperformed by deterministic approaches Susceptible to “catastrophic gray-out”Susceptible to “catastrophic gray-out”
Classical screening performs marginally Classical screening performs marginally andand has has poor bandwidth controlpoor bandwidth control
Recent WorkRecent Work vasiHalftone3 and vasiHalftone4vasiHalftone3 and vasiHalftone4
Extensions to eliminate simplifying assumption that Extensions to eliminate simplifying assumption that VASI™ shareUp and shareLeft signals are equalVASI™ shareUp and shareLeft signals are equal
This eliminates single-pixel artifacts in non-foveal This eliminates single-pixel artifacts in non-foveal regionsregions
Eliminated lookup table (LUT) in F-S approach Eliminated lookup table (LUT) in F-S approach by determining closed-form inverse by determining closed-form inverse relationshiprelationship Significant speedupSignificant speedup
Greatly shrank LUT in vasiHalftone & Greatly shrank LUT in vasiHalftone & vasiHalftone2 approachesvasiHalftone2 approaches Leveraged “stairstep” form of inverse relationshipLeveraged “stairstep” form of inverse relationship 10x speedup in vasiHalftone, 4x speedup in 10x speedup in vasiHalftone, 4x speedup in
vasiHalftone2vasiHalftone2 2121stst Century Technologies and Nova Sensors Century Technologies and Nova Sensors
are actively collaborating on further workare actively collaborating on further work Sponsored by U.S. Air Force Research LaboratorySponsored by U.S. Air Force Research Laboratory
Background ReferencesBackground References B.E. Bayer, “An optimum method for two level rendition B.E. Bayer, “An optimum method for two level rendition
of continuous-tone pictures,” of continuous-tone pictures,” Proc. IEEE Int. Conf. on Proc. IEEE Int. Conf. on Communications, Conf. Rec.Communications, Conf. Rec., pp. (26-11)-(26-15), 1973., pp. (26-11)-(26-15), 1973.
R. Floyd and L. Steinberg, “An adaptive algorithm for R. Floyd and L. Steinberg, “An adaptive algorithm for spatial grayscale,” spatial grayscale,” Proc. SID’76Proc. SID’76, pp. 75-77, 1976. , pp. 75-77, 1976.
P. McCarley, M. Massie, and J.P. Curzan, “Large format P. McCarley, M. Massie, and J.P. Curzan, “Large format variable spatial acuity superpixel imaging: visible and variable spatial acuity superpixel imaging: visible and infrared systems applications,” infrared systems applications,” Proc. SPIE, Infrared Proc. SPIE, Infrared Technology and Applications XXXTechnology and Applications XXX, vol. 5406, pp. 361-, vol. 5406, pp. 361-369, Aug 2002. 369, Aug 2002.
V. Monga, N. Damera-Venkata, and B.L. Evans, V. Monga, N. Damera-Venkata, and B.L. Evans, Halftoning Toolbox for MatlabHalftoning Toolbox for Matlab.. Version 1.1 released Version 1.1 released November 7, 2002. Available online at November 7, 2002. Available online at http://http://www.ece.utexas.edu/~bevans/projects/halftoningwww.ece.utexas.edu/~bevans/projects/halftoning//. .
R.A. Ulichney, “Dithering with blue noise,” R.A. Ulichney, “Dithering with blue noise,” Proc. IEEEProc. IEEE, , vol. 76, pp. 56-79, Jan 1988. vol. 76, pp. 56-79, Jan 1988.
Z. Wang, A.C. Bovik, and L. Lu, “Wavelet-based Z. Wang, A.C. Bovik, and L. Lu, “Wavelet-based foveated image quality measurement for region of foveated image quality measurement for region of interest image coding,” interest image coding,” Proc. IEEE Int. Conf. Image Proc. IEEE Int. Conf. Image Proc.Proc., vol. 2, pp. 89-92, Oct 2001. , vol. 2, pp. 89-92, Oct 2001.
Top Related