Dual-sensor foveated imaging system

Hong Hua* and Sheng LiuCollege of Optical Sciences, University of Arizona, 1630 E. University Boulevard, Tucson, Arizona 85721, USA

*Corresponding author: hhua@optics.arizona.edu

Received 25 September 2007; revised 16 November 2007; accepted 17 November 2007;posted 28 November 2007 (Doc. ID 87895); published 14 January 2008

Conventional imaging techniques adopt a rectilinear sampling approach, where a finite number of pixelsare spread evenly across an entire field of view (FOV). Consequently, their imaging capabilities arelimited by an inherent trade-off between the FOV and the resolving power. In contrast, a foveationtechnique allocates the limited resources (e.g., a finite number of pixels or transmission bandwidth) as afunction of foveal eccentricities, which can significantly simplify the optical and electronic designs andreduce the data throughput, while the observer’s ability to see fine details is maintained over the wholeFOV. We explore an approach to a foveated imaging system design. Our approach approximates thespatially variant properties (i.e., resolution, contrast, and color sensitivities) of the human visual systemwith multiple low-cost off-the-shelf imaging sensors and maximizes the information throughput andbandwidth savings of the foveated system. We further validate our approach with the design of a compactdual-sensor foveated imaging system. A proof-of-concept bench prototype and experimental results aredemonstrated. © 2008 Optical Society of America

OCIS codes: 110.0110, 220.4830, 330.7338, 330.1800, 230.4685.

1. Introduction

Real-time acquisition of high-resolution, wide field ofview (FOV), and high dynamic range (HDR) images isessential for many vision-based applications, rangingfrom the macroscale imaging in surveillance [1], tele-conferencing [2], and robot navigation [3], to micro-meter or submicrometer-scale imaging in biomedicalimaging [4] and vision-guided microfabrication [5].Although detector technologies have improved dra-matically over the past decades, many imaging sys-tems are limited by their performances on FOV,resolution, speed, and dynamic range. Most conven-tional imaging techniques spread a finite number ofpixels evenly across the entire FOV, and thus theirimaging capabilities are constrained by a well-knowninherent trade-off between the FOV and the resolvingpower; the higher the resolving power, the smallerthe FOV. Consider a surveillance camera with a typ-ical National Television System Committee (NTSC)CCD detector of 640 � 480 pixels, as an example. Adiagonally 60° FOV only offers an angular resolutionof 6.4 arc min per pixel at the maximum. If mounted

onto a 20 ft ceiling, the camera can only deliver ahuman face image of about 20 � 16 pixels, which areneither sufficient for object tracking nor face identi-fication.

This resolution-FOV trade-off problem may be re-lieved with the advent of higher-resolution sensors andthe development of sensor tiling or image mosaicingschemes [4,5]. For instance, in an adaptive scanningoptical microscope system [5], a 40 mm observing fieldis formed by mosaicing 25 � 25 images, each havinga resolution of 512 � 512 pixels. Thus the full-fieldimage will be 12800 � 12800 pixels. However, pro-cessing and analysis of such high-resolution images iscomputationally demanding. Furthermore, capturing25 � 25 images to reconstruct a full-field image at atolerable frame rate (e.g., 10 fps) not only requires animage sensor with a very high frame rate, but alsodemands a fast scanning mechanism. Moreover, withthe increasing interests in teleoperation and tele-medicine, real-time transmission of these imagesover a network can be very challenging.

Rather than demanding more expensive (botheconomically and computationally), higher resolu-tion, and faster image sensors as well as associatedprocessing technologies. In this paper, we presentan approach to a foveated imaging system design,

0003-6935/08/030317-11$15.00/0© 2008 Optical Society of America

317 APPLIED OPTICS � Vol. 47, No. 3 � 20 January 2008

aiming to improve the information throughput oflow-cost off-the-shelf detectors. In Section 3, we willpresent a comprehensive analysis and explorationon how the spatially variant properties of thehuman visual system (HVS), including the well-explored domain of spatially variant resolution[6–8] and the less-explored spatial variances of con-trast and color sensitivities, may be utilized to max-imize the information throughput and reduce databandwidth requirements of an imaging system. InSection 4, we will describe a schematic design of acompact, actively foveated dual-sensor imagingsystem, which mimics several aspects of the HVS.Finally, a bench prototype and experiments are pre-sented in Section 5, which demonstrates that 96.4%bandwidth savings can be achieved at the stage ofimage capturing in a dual-sensor foveated imagingsystem.

2. Related Work

Foveation techniques, inspired by the foveation prop-erties of the human eye, can be generally character-ized as a method that dynamically tracks a user’sattended area of interest (A-AOI) by means of a gazetracker, or other mechanism, and treats the A-AOIdifferently from the peripheral area. A great amountof effort has been made to explore foveation tech-niques in imaging and display applications, whichare sometimes referred to as gaze-contingent or eye-tracked multiresolution techniques. Examples in-clude spatially variant compression ratio in imageand video processing [6–8], variable levels of detailin three-dimensional (3D) rendering [9–11], vari-able pixel resolution in imaging or display systems[12,13], and adaptive correction of optical aberrationsin imaging or display systems [14,15]. More applica-tion examples and reviews can be found in [9,16].

The prior work on foveation techniques falls intoone of three categories, as follows. The first categoryof work relates to experimental research on percep-tion and cognition to understand visual processingand perceptual artifacts produced by simulated mul-tiresolution images or display systems [17–22]. Forinstance, researchers investigated perceptual arti-facts in foveated multiresolution displays, such asperceptible image blur and image motion, which havethe potential to distract users [17,20–22]. Ideally, onewould like to maximize the bandwidth savings of fo-veation techniques, while minimizing perception ar-tifacts and performance costs.

The second category is the algorithmic approachin which foveation techniques are applied primarilyto image processing [2,7], video encoding [6,8], andgraphics rendering [9–11] to achieve real-timevideo communication through low-bandwidth net-works and to save data transmission bandwidthand processing resources. For instance, Geisler andPerry demonstrated a three times greater compres-sion ratio in multiresolution images�videos [8];Ienaga et al. described a stereoscopic video trans-mission system with embedded high-resolution fo-vea images to accelerate the image processing and

to reduce the bandwidth requirement [2]; severalresearchers applied perceptually driven foveationtreatments to 3D graphics rendering such a level-of-detail management and polygon reduction in 3Dmodels based on the view eccentricity [9–11]. Mur-phy and Duchowski demonstrated that the render-ing speed can be significantly accelerated by onlyrendering high levels of detail of a 3D scene arounda user’s gaze direction in virtual environments [10];Luebeke et al. demonstrated that 2–6 times fewerpolygons can produce the similar rendering effect inperceived model resolution [11].

The third category of work takes a hardware ap-proach, in which various imaging sensors or displayswith spatially varying resolution are developed toreduce requirements for high-resolution detectorsand displays or high-quality and complex optical sys-tems [3,12–15,23–25]. For instance, Sandini et al.described the implementation of a retina-like compli-mentary metal-oxide semiconductor (CMOS) sensorcharacterized by spatially variant resolution similarto that of the human retina [12]. They demonstratedthat 35 times fewer pixels were needed in the space-variant resolution sensor as compared with a con-stant high-resolution image of 1100 � 1100 pixels.Martinez and Wick presented the designs of foveatedimaging systems in which a liquid crystal (LC) spa-tial light modulator (SLM) was used to dynamicallycorrect the aberrations of a simple-structure wideFOV optics at the foveated region of interest (FRoI)[14,15]. Godin described a multiprojector system toachieve a dual-resolution 3D display [24]. Rollandpresented the design of a head-mounted display withhigh-resolution insets [13]. Several groups of re-searchers demonstrated the use of a camera pair withdifferent angular resolutions in robot navigation sys-tems to achieve foveated images [3,25].

Although both the algorithmic and hardware ap-proaches can achieve foveated images, the hardwareapproach is advantageous in the sense that itachieves foveated imaging at a more fundamentallevel. It captures foveated images with spatiallyvariant resolution by designing novel imaging sen-sors and systems, rather than postprocessing full-resolution images by applying foveated filtering andencoding techniques. It potentially requires less com-plex optical and electronic designs and reduces therequirements on data bandwidth and computationalresources, while the postprocessing approach ismostly limited to the benefits of information storageand transmission.

3. Foveated Imaging with Spatially Varying Resolutionand Chromaticity

In the HVS, only a narrow region around the foveaoffers exceptional resolving power, contrast, and colorsensitivities, while these properties fall off rapidlywith an increasing retinal eccentricity. The essentialgoal of designing a foveated imaging system is toimprove the sampling efficiency and to reduce therequirements for data bandwidth as well as compu-

20 January 2008 � Vol. 47, No. 3 � APPLIED OPTICS 318

tational resources by mimicking such spatially vary-ing properties of the HVS.

Rather than pursuing specially designed sensorswith retina-like properties [12,23], we use multiplelow-cost sensors to approximate the spatially vari-ant properties of the HVS, including the well-explored domain of spatially variant resolution[6–8] and the less-explored spatial variances of con-trast and color sensitivities. Our approach signifi-cantly improves the information throughput oflow-cost sensors and reduces the total number ofrequired pixels. The reduction in total pixel countsdirectly translates to less storage space and com-munication bandwidth, which offers great opportu-nities to break the technical barrier in designingwide FOV systems with much simpler optical de-signs, to achieve system miniaturization, and todevelop distributed sensing networks with limitedbandwidths.

This section details the analytical methods forachieving spatially varying resolution and spatial-chromatic foveation. The analysis predicts that over80% of bandwidth saving can be easily achieved witha dual-resolution dual bit depth (DRDB) foveated im-aging system, compared to a traditional rectilinearsampling approach.

A. Spatially Variant Sampling and Multiresolution ImagingSystems

We characterize the sampling scheme of an imagingsystem with its relative spatial resolution distribu-tion as a function of field angles. The distribution,denoted as F�ex, ey�, is the normalized reciprocal of theangular resolution at a given visual field �ex, ey�,where ex and ey are the horizontal and vertical eccen-tricities measured in degrees, respectively. The an-gular resolution here is measured by arcminutes perpixel. The amount of raw data acquired by the systemcan be calculated as the integration of its relativeresolution response across the visual field, given as

B ����X�2

�X�2 ���Y�2

�Y�2

F2�ex, ey�dexdey, (1)

where �X and �Y are the full FOVs covered by thesystem along the horizontal and vertical directions,respectively. The amount of raw data characterizesthe bandwidth requirement.

The sampling efficiency of a given system may beevaluated against a reference system whose spatialresolution distribution represents the just-adequateresolvability required for a particular task. Systemswith the same or better resolvability than that oftheir reference system are considered to be perceptu-ally equivalent. Without loss of generality, we use thevisual acuity response of the human eye, FHVS�ex, ey�,as the reference system throughout the paper. Com-pared against the reference system providing thesame FOV and the same peak resolution, the sam-pling efficiency, E, of a given system is defined as

E �

���X�2

�X�2 ��Y�2

�Y�2

FHVS2�ex, ey�dexdey

���X�2

�X�2 ���Y�2

�Y�2

F2�ex, ey�dexdey

. (2)

E also characterizes the relative information through-put of the system. For a single-resolution imaging(SRI) system with uniform sampling characterizedby FSRI�ex, ey� � 1, Eq. (2) is simplified as ESRI �

1�X�Y

���X�2�X�2 �

�Y�2�Y�2 FHVS

2�ex, ey�dexdey.

In Fig. 1 we plot the sampling efficiency of a SRIsystem as a function of the overall FOV. The relativevisual acuity response of the eye was modeled asFHVS�ex, ey� � 2.3��2.3 � �ex

2�ey2�, adopted from [8,19].

A 4:3 aspect ratio was assumed for the overall field.As shown in Fig. 1, the efficiency monotonically de-creases with an increasing FOV. For instance, theefficiency is as low as 3% for a 60° system and is lessthan 0.5% for a 200° system. The amount of redun-dant raw data produced by a 60° SRI system is almost35 times more than that of the perceptually equiva-lent reference system, and over 200 times more for a200° SRI system.

Aiming to improve the sampling efficiency of a SRIsystem depicted in Fig. 1, we explored an approach,where multiple low-cost sensors are utilized to ap-proximate the spatial-variant properties of the ret-ina. Let us consider approximating the eccentricityresponse of the visual acuity with two levels of spatialresolution as shown in Fig. 2(a). One sensor, referredto as the foveated imager, covers the central region ofthe overall visual field, while the other sensor coversthe peripheral region. The foveated imager uniformlymaintains a maximum resolution within the range of��XF and ��YF degrees in the horizontal and verticaldirections, respectively. In order to avoid discernibleartifacts, the relative resolvability of the peripheralimager is set to be FDRI��XF, �YF�, where FDRI��XF, �YF�� max�FHVS��XF, 0�, FHVS�0, �YF�) for rectangular vi-sual fields. Compared with a SRI system, the band-width saving of a dual-resolution imaging (DRI)

Fig. 1. (Color online) Sampling efficiency of a SRI system as afunction of the overall FOV.

319 APPLIED OPTICS � Vol. 47, No. 3 � 20 January 2008

system is given as

SDRI �BSRI � BDRI

BSRI

��X�Y � ��X�Y � �XF�YF�FDRI

2��XF, �YF� � �XF�YF

�X�Y.

(3)

In Fig. 2(b), we plot the bandwidth saving ratio ofa DRI system as a function of the foveated FOV andthe overall visual field. A 4:3 aspect ratio was as-sumed for both the overall and foveated FOVs. For agiven overall FOV, the bandwidth saving ratio can bemaximized by varying the foveated FOV. Figure 2(c)plots the maximum bandwidth saving ratio as a func-tion of the overall FOV. In the same figure we furtherplot the optimal ratio of the foveated FOV to theoverall FOV.

Based on Fig. 2(c), by correctly choosing the fove-ated FOV relative to the overall FOV, we can max-imize the bandwidth reduction and informationthroughput of a dual-sensor foveated imaging systemwhile maintaining indiscernible image quality degra-dation. Consider a system requiring a total of 120°diagonal FOV and 1 arc min of peak angular resolu-tion. A SRI system with uniformly 1 arc min resolu-tion requires a single sensor with 5760 � 4320 pixels.In contrast, in a dual-sensor foveated system, therelationship in Fig. 2(c) suggests that a 25° foveatedFOV maximizes the spatial sampling efficiency of a120° system. The combination of the 25° foveatedFOV in 1 arc min resolution with a 95° peripheralFOV in 4.25 arc min of resolution yields a bandwidthreduction ratio of about 90.4%, which directly trans-lates to almost ten times fewer pixels than those fora SRI design.

The above sampling approach can be extended tomore than two resolution levels using a similarscheme. Maximizing the overall sampling efficiencyof multisensors will involve optimizing the FOV cov-erage of these sensors and correctly selecting theirresolution levels so that the resulting system pre-sents a multiresolution image with indiscerniblequality degradation while minimizing the amount ofredundant information.

B. Contrast�Color Sensitivity and Image Bit Depth

Similar to the property of spatially variant resolu-tion, the contrast and color sensitivities of an im-aging system can be encoded with the greatestprecision at the center of the area of interest andwith decreasing precision as the distance from thecenter (the eccentricity) increases. Several re-searchers have suggested that exploring peripheralchromatic degradation can potentially lead to sig-nificant bandwidth saving [9,16]. In this subsection,we explore the spatial variance of the contrast sen-sitivity function of the HVS to examine the minimalbit depth and color channels, which yield imageswith indiscernible peripheral chromatic degrada-tion.

The contrast sensitivity function (CSF) of the HVSat different retinal eccentricities can be adequatelydescribed by the formula [8,19]

CSF�e, �� �1

CT�e, ��� CSF�0, ��exp����

e2 � ee2

�,(4)

Fig. 2. (Color online) Optimization of a dual-resolution foveatedimaging system: (a) eccentricity of visual acuity and dual-resolution sampling scheme; (b) bandwidth saving ratio as a func-tion of peripheral and foveated FOVs; (c) maximum bandwidthsaving in percentile (blue curve with square marker) and optimalratio of foveated FOV to overall FOV (green curve with � marker).

20 January 2008 � Vol. 47, No. 3 � APPLIED OPTICS 320

where e � �ex2�ey

2 is the retinal radial eccentricityin degrees, � is the spatial frequency in cycles perdegree (cpd), � is the spatial frequency decay con-stant (� 0.045), e2 is the retinal eccentricity,where spatial resolution falls to half of what it is inthe center of the fovea �e2 2.3°�, and CSF(0, �)models the contrast sensitivity of the fovea as afunction of spatial frequency (please see [26] fordetailed explanations).

The decreasing contrast sensitivity with retinal ec-centricity suggests that the bit depth of an image maybe reduced monotonically as the eccentricity in-creases without creating perceptible banding effects.Let us consider the DRI system discussed in Subsec-tion 3.A. Here we assume the foveated imager, with abit depth of N, maintains a uniform spatial resolutionof f arc min within the foveated FOV ���XF, ��YF�.The peripheral imager maintains a uniform spatialresolution of p arc min with a lower bit depth, M. Toavoid perceivable resolution degradation, p is deter-mined by p f�FDRI��XF, �YF� as discussed in Sub-section 3.A. Furthermore, to minimize artifactscaused by the reduction of bit depth, we set up aconstraint for the reduced bit depth, M, of the periph-eral imager, written as

�Ci,j�N, M��max CT��F, � 60

f � p�

�i � 0, 1, . . . 2N � 1, j � 0, 1, . . . 2M � 1�, (5)

where �Ci,j�N, M�|max is the maximum contrast be-tween the original and remapped pixel values whena pixel value at the ith gray level in a N-bit imageis remapped to the jth gray level in a M-bit sensor.This constraint ensures that a user is unable todiscern the bit reduction at the boundaries of thetwo imagers when fixating on the center of the fo-veal image.

Figure 3(a) plots the HVS contrast sensitivity as afunction of spatial frequency and eccentricity. Underthe conditions of f � 1 arc min and N � 8 bits percolor channel, in Fig. 3(b) we plot �Ci,j�N, M�|max as afunction of the bit depth, M, of the peripheral imager.On the same graph, we also show the contrast mod-ulation thresholds for �10°, �15°, and �20° of fove-ated FOVs for 6 cpd spatial frequency. For instance,the contrast modulation threshold at �10° of eccen-tricity is 0.0385, and the bit depth for the peripheralimager may be reduced down to 4 without noticeableboundary artifacts.

Compared to a SRI system with a uniform resolu-tion and single bit depth, the percentage of band-width saving of a DRDB design is given as

For a given overall FOV, the foveated FOV can beselected to maximize the sampling efficiency based onthe relationship in Fig. 2. As a result, the magnitudeof bandwidth saving essentially depends on the over-all FOV and the bit depth of the foveal and peripheralimagers. Figure 4 plots the bandwidth saving ratio ofthe DRDB system as a function of the overall FOVand reduced peripheral bit depth M, where we as-sumed a 4:3 aspect ratio for the visual fields andeight bits per color channel for the foveated imager�N � 8�. As an example, to design a system of 120°diagonal FOV, the bandwidth saving ratio is in-creased to 95.65% with a 25° foveated FOV in eightbits per color channel, and the remaining 95° periph-eral FOV in five bits per color channel. This resultdirectly translates to 23 timesless storage space andcommunication bandwidth. The plots in Fig. 4 fur-ther suggest that the benefit of bit depth reductiondiminishes dramatically for M less than 5.

SDRDB �BSRI � BDRDB

BSRI�

2N · �X�Y � 2M · ��X�Y � �XF�YF�FDRI2��XF, �YF� � 2N · �XF�YF

2N · �X�Y. (6)

Fig. 3. (Color online) Contrast sensitivity and image bit depth: (a)the HVS contrast sensitivity as a function of spatial frequency andeccentricity; (b) maximum contrast modulation change as a func-tion of reduced bit depth.

321 APPLIED OPTICS � Vol. 47, No. 3 � 20 January 2008

Besides reducing the image bit depth, the band-width savings can be further improved by exploringthe color sensitivity of the HVS and reducing thenumber of color channels for the peripheral imager,for instance, from a full-color system to a gray scale asthe eccentricity increases [9]. For example, a 120°dual-sensor system described above can be reducedfrom a 15 bit color imager for the peripheral to a fivebit gray scale imager. The resultant bandwidth sav-ing ratio is 95.66% compared with a single-sensordesign with perceptually equivalent quality. How-ever, further human factor research is needed to con-struct a chromaticity eccentricity function that can beutilized to create a spatially varying color system thatis just imperceptibly different from a full-color image.

From the point of view of bandwidth reduction,reducing color channels on top of spatially varyingresolution and bit depth appears to only provide mar-ginal benefits, while the combination of applying spa-tially varying resolution and reducing bit depth percolor channel for the peripheral imager yields muchmore effective reduction. However, from the imple-mentation point of view, reducing color channels re-quires much less hardware modification and achievessimilar efficiency to bit depth reduction. In applica-tions where peripheral color sensation is not criticalfor the aimed tasks, spatially varying resolutionmethods can be sensibly combined with spatiallyvarying color schemes.

C. Simulation Results

Based on the eccentricity properties discussed in Sub-sections 3.A and 3.B, a multisensor system can bedesigned with maximum sampling efficiency andbandwidth savings. Figures 5(a)–5(c) present a set ofsimulation results to demonstrate the visual effects ofresolution, bit depth, and color channel reduction.Figure 5(a) is the original 24 bit full color image with2400 � 1800 pixels. This image provides approxi-mately 50° diagonal FOV with an angular resolutionof about 1 arc min when viewed on a 30 in. monitor ata distance of 32 in. It is worth noting that, in general,a uniform angular resolution in the object space doesnot map into a uniform pixel density on the image

plane. Although sensors with spatially varying pixeldensities may be custom-made, in implementationsthroughout the paper, we choose to use regular sen-sors with a uniform pixel density in which the worstangular resolution of the pixels satisfies the angularresolution requirements.

Figure 5(b) simulates a DRI with a 24 bit centerregion of 691 � 518 color pixels and a peripheral areaof 766 � 574 pixels with 5 bits per color channel. Thecenter region covers about 14.4° of the overall FOV in1 arc min resolution when viewed on the same mon-itor. Each enlarged image pixel of the peripheral areais equivalent to 3 � 3 pixels on the monitor, whichcorresponds to an angular resolution of 3.3 arc min.Figure 5(c) simulates a similar image with a 24 bitcenter region of 691 � 518 pixels but an eight bitblack�white (B�W) peripheral area of 766 � 574 en-larged pixels. Due to the reduced size of the imagesshown in Fig. 5, it is hard to tell their resolutiondifferences. Figures 5(d)–5(f) show enlarged views ofa narrow region of the images corresponding to Figs.5(a)–5(c), respectively.

Compared with the original image in Fig. 5(a), thetotal number of pixels for both Figs. 5(b) and 5(c) wasreduced from 4.32 � 106 to 0.798 � 106. The overallbandwidth saving ratios of Figs. 5(b) and 5(c) are91.71% and 91.73%, respectively. From the point ofview bandwidth reduction, they are almost equiva-lently efficient. If these simulated images are viewedunder the same condition mentioned above, and theuser fixates at the center of the images, Figs. 5(b) and5(c) should provide the about same spatial resolvabil-ity perceptually equivalent to Fig. 5(a), while thecolor shift in Fig. 5(b) would not be as noticeable asFig. 5(c).

4. Design of a Dual-Sensor Fovated Imaging System

When the foveation approach discussed in Section 3is combined with a dynamic tracking and scanning

Fig. 4. (Color online) Bandwidth saving ratio of a DRDB systemas a function of overall FOV and reduced bit depth.

Fig. 5. (Color online) Simulated foveated images: (a) original SRI(2400 � 1800 24 bit pixels); (b) DRDB image (691 � 518 24 bitpixels in the center and 766 � 574 15 bit pixels in the peripheral);(c) DRDB image (691 � 518 24 bit pixels in the center and 766� 574 8 bit gray-scale pixels in the peripheral); (d)–(f) enlargedviews of a narrow region of images (a)–(c).

20 January 2008 � Vol. 47, No. 3 � APPLIED OPTICS 322

mechanism, we can make the highest resolution, bestcontrast, and best color images available, where theuser is mostly interested in, while using considerablyless hardware bandwidth and computational re-sources. In this section, we present a design of afoveated imaging system based on the dual-sensorapproach. The system operates by capturing two im-ages of the scene simultaneously with two relativelylow-resolution detectors: one covers the entire sceneand the other covers a narrow area of interest. Thetwo imagers, sharing the same entrance pupil,are optically coupled together through a two-axismicroelectro-mechanical systems (MEMS) scanner,which is driven by fovea tracking algorithms. De-pending on the light levels of the entire scene, andlocal features of interest, the imagers may be adap-tively modulated to acquire HDR images by imple-menting the pixel-by-pixel modulation approachdescribed in [27]. The ability to dynamically steer thefovea region toward a detected stimulus via instan-taneous eye movements is referred to as adaptivefoveation; the ability to adapt the pupil opening inresponse to a large magnitude of illumination levelvariations is referred to as adaptive illuminationsensing.

A. Conceptual Design

Conceptually, our targeted system mainly consists offive components: a foveated imager, a peripheral im-ager, a fast 2D scanner, adaptive aperture modula-tors, and fovea tracking and high-dynamic rangecontrol algorithms. The foveated imager mimics thefovea pit of the eye and provides the fine high-contrast details and color sensation of a narrow fove-ated region for target recognition. The peripheralimager mimics the peripheral vision of the eye andcaptures an extended field with relatively low reso-lution. It provides the peripheral context for stimulusdetection. The scanning system mimics the eye move-ments and provides the dynamic capability for targettracking. The adaptive aperture modulator consistsof two elements. The first modulator can mimic theeffect of pupil size adaptation to the scene luminancevariation. It dynamically adjusts the active apertureof the imaging system based on the overall light lev-els detected from the peripheral and foveated imag-ers. The second modulator is to enhance the dynamicrange of the imaging system by using a method byGao et al. [27], in which the image exposure of a sceneis controlled on a pixel-by-pixel basis to deal withscenes with a wide dynamic range locally [27]. Foveatracking algorithms can be implemented to dynami-cally detect the salient region of interest and stimu-lus events from the peripheral image.

The dual-sensor design differs from the existingmethods of foveated imaging in several aspects[12,14,15,23]. First of all, the dual-sensor approachoffers real-time peripheral awareness and potentiallythe capability of event detection and tracking. Sec-ond, compared to the single-sensor foveated imagingapproach by Wick et al. [14,15], or the retina-likesensor approach [12,23], the dual-sensor method

yields high information throughputs for low-cost de-tectors without the need for custom-made specialsensors. It is worth noting that the single-sensor fo-veated systems by Wick et al. capture both the fove-ated and peripheral areas on the same detector. Theangular coverage per pixel remains approximatelyconstant across the entire field if a regular sensorwith a constant pixel density is utilized. The low-resolution appearance of their systems is due to op-tical aberrations [14,15]. It has the advantage ofusing simple optics for imaging but compromises thedetector efficiency. Furthermore, a retina-like sensorwith space-variant pixel density imposes a highercost and lacks the ability to achieve dynamic fove-ation unless the spatial-variant pixel arrangementscan be dynamically controlled. Finally, the proposeddesign can potentially incorporate adaptive aperturemodulators for advanced HDR imaging [27].

B. Schematic Design

The schematic design of an actively foveated imagingsystem is shown in Fig. 6. To achieve compactness,the foveated and peripheral imagers share the sameobjective lens, which forms the first intermediate im-age plane. The objective lens itself contains a physicalstop near its front focal plane for the entire imagingsystem. A beam splitter placed after the first inter-mediate image plane separates the paths for the twoimagers. A scanner lens collects the light transmittedthrough the beam splitter. It forms an intermediatepupil plane at which a two-axis analog MEMS mirroris placed to achieve a compact design. Other types ofscanning mirror with lower cost may be used if com-pactness is less critical. By tilting the mirror instan-taneously toward the direction of interest, rays fromthe interested FOV are redirected toward the opticalaxis. To achieve compactness, the same scanner lenscollects a narrow field of the light reflected by themirror and forms a high-resolution foveated imagefor the region of interest. In the meantime, a camerawith a relay lens is placed on the reflected path of thebeam splitter to form a low-resolution peripheral im-age of the entire visual field. An SLM may be placedat the first intermediate image plane to facilitate the

Fig. 6. (Color online) Schematic design of a dual-sensor foveatedimaging system.

323 APPLIED OPTICS � Vol. 47, No. 3 � 20 January 2008

capture of HDR images by applying the mask ap-proach [27]. A second SLM may replace the stop foractive aperture control over the entire FOV.

The first-order parameters of the foveated imagerpath are annotated in Fig. 6. The effective focallengths of the objective and scanner lenses are de-noted as f1 and f2, respectively. The entrance pupil isoffset by a distance of �t from the front focal plane ofthe objective lens. The front focal point of the scannerlens is offset by a distance of �t� from the back focalpoint of the objective lens. All distances and anglesutilize the well-accepted sign conventions [28].

When the mirror is scanned by an angle of �m fromits reference orientation, the central angle of the vi-sual fields instantaneously imaged by the foveatedimager is given by

�c � arctan�f1f2 tan�2�m�f1

2 � �t�t� �, (7)

where �c corresponds to the scene point that is imagedonto the center of the foveated detector. The half FOVof the foveated imager, �r, is given by

�r � arctanDF�f12 � �t�t��2f1

2f2�, (8)

where DF is the diagonal dimension of the foveateddetector. Further, mp � �f1f2��f1

2 � �t�t�� is the mag-nification between the diameters of the mirror andthe entrance pupil.

Providing that the MEMS mirror is capable of me-chanically scanning within a range of ��max degrees,the maximum visual field covered by scanning theMEMS is given by

�F�max � 2 arctanf1f2 tan�2�max � �r�f1

2 � �t�t� �. (9)

At a given visual field angle in the object space, thecorresponding image height on the peripheral imageris given by

yF �f1

2f2 tan�2�m � �2�f1

2 � �t�t�, (10)

where a 1:1 relay from the intermediate imageplane is assumed, and �2 is the chief ray angle ex-iting the scanner lens and is given as �2 � arctan��mp tan ��.

The mirror is conjugate to the entrance pupil, andits location is characterized by the distance LXP fromthe back focal point of the scanner lens,

LXP �f2

2�t

f12 � �t�t�

. (11)

As the projection area of the mirror surface on theintermediate pupil plane varies with its scanning an-gle, to ensure that the effective F-number of the fo-veated imager is independent of the mirror scanningangles, the entrance pupil diameter, DEP, should sat-isfy the condition of

DEP DMEMS

�mp�, (12)

where DMEMS is the effective diameter of the mirrordevice.

There are several advantages associated with theabove schematic design. First of all, the design en-sures that the foveated and peripheral imagers sharethe same entrance pupil. Satisfying the condition ofsingle-viewpoint imaging allows an easy mosaic ofthe two images independent of the scene depth. Sec-ond, as shown in Eq. (9), with an appropriate choiceof focal lengths for the objective and scanning lenses,the design can effectively relax the requirements forthe mirror scanning angle. Fabricating a large-size,fast steering mirror with large tip-tilt capability isquite challenging with current technologies. Third, itoffers flexible pupil controls and can be enhancedwith adaptive HDR image capturing methods [27].Finally, the design is compact and low-cost. The fold-ing design of the scanner lens helps reduce the overalllength of the foveated beam. The system is exemptfrom using high-resolution detectors and complex op-tics while achieving high resolving power across arelatively wide visual field.

5. Experimental Prototype and Results

A. Prototype Design

Based on the schematic layout in Fig. 6, we builta bench prototype, as shown in Fig. 7(a), using off-the-shelf optical components to demonstrate thepotential system capability. In the prototype, weutilized two low-cost CCD sensors, one 1�4 in. full-color CCD (640 � 480 pixels) and one 1�4 in. B�WCCD (640 � 480 pixels), for foveated and peripheralimaging, respectively. Considering the factors of com-pactness, scanning range, and cost, we implementedthe scanning element with a two-axis analog MEMSchip by Texas Instruments as shown in Fig. 7(b). Thechip has an elliptical active area measured between3.2 mm and 3.6 mm along its minor and major axes,respectively. Its mechanical scanning range is �5°around two orthogonal axes, and it has a mechanicalresonant frequency of 130 Hz. The limited scanningrange of the mirror requires |mp| � 1 between theobjective and scanner lenses to dynamically scan thefoveated image across a wide peripheral visual field.The mirror dimensions along with the magnificationrequirements set up an upper limit to the entrancepupil diameter of the system [Eq. (12)].

An off-the-shelf 28 mm Erfle eyepiece assemblywas utilized as the objective lens. It produces a lowaberration intermediate image plane, where we can

20 January 2008 � Vol. 47, No. 3 � APPLIED OPTICS 324

potentially insert a SLM for pixel-level HDR imagecapturing. Another 28 mm lens was used togetherwith the objective lens for the peripheral imagingpath. As a result, the overall peripheral FOV is 44.6°diagonally with an angular resolution of 3.3 arc minper pixel at the best. The actual resolution of theperipheral imager may be further reduced by usinglower-cost simpler optics.

Based on the plot in Fig. 2(c), a foveated imagerwith a total of 12° FOV will maximize the bandwidthsaving ratio while maintaining imperceptible imagedegradation in the peripheral. A 60 mm doublet wasused as the scanner and imaging lens. Its combina-tion with objective lens yields a magnification powerof about 2.1. The instantaneous FOV of the foveatedimager is 8.5° diagonally with an angular resolutionof 0.48 arc min per pixel at the best.

Combining with the optics above, the MEMS mir-ror is capable of sweeping the 8.5° foveated FOVacross the full peripheral field by scanning theMEMS mirror within �4°. Compared with a single-sensor design that maintains uniformly high resolu-tion across the entire FOV, the bandwidth saving

ratio of the above dual-sensor design is 96.4%, whichis higher than the optimal value in Fig. 2. However,due to the use of off-the-shelf components, the com-promise is the perceptible image degradation in theperipheral when gazing at the center of the foveatedimage.

B. Experimental Results

Image registration is a crucial postprocessing step tocreate a seamless mosaic of the foveated and periph-eral images as the MEMS scans the foveated FOVacross the overall visual field. It requires deliberatecalibration of the MEMS device and the distortioncoefficients of the imaging systems. We calibrated theMEMS tilt angle responses versus the voltages ap-plied to the x- and y-axes of the chip. In both direc-tions, the tilt angles demonstrated excellent linearityfor voltages within �4.5 V. The distortion coefficientof the peripheral imager, independent of MEMSscanning angles, was calibrated by applying a well-established camera calibration technique [29]. Mean-while, the distortion of the foveated imager dependson MEMS scanning and cannot be calibrated by areadily available method. We have developed a cali-bration technique that accurately models and cali-brates the distortion of the foveated images as afunction of MEMS scanning angles. By unwarpingthe distorted images captured at a given scanningangle, we are able to create a seamless foveated im-age [Fig. 8(a)] with a 44.6° diagonal FOV. The high-resolution foveated region with 8.5° FOV is shown inFig. 8(b). In this example, a printed map of TucsonCity was used as the scene object placed at the nearrange of the depth of field (�300 mm away from theentrance pupil).

The MEMS mirror was driven by applying voltagesfrom �3.5 V to �3.5 V with 0.5 V increments in bothorthogonal directions and scans a range from �4° to�4° with approximately 0.6° increments, which al-lowed us to steer the 8.5° foveated FOV across theentire 44.6° peripheral FOV for dynamic foveation. Asequence of high-resolution image samples was cap-tured at different MEMS tilting angles. The mosaic ofthese foveated images with the peripheral back-ground is shown in Fig. 9(a). Two enlarged views fora region near the center and the top-left corner areshown in Figs. 9(b) and 9(c), respectively. They dem-

Fig. 8. (Color online) Registration and mosaic of (a) a peripheralimage with (b) a foveated image.

Fig. 7. (Color online) (a) Bench prototype of a dual-sensor fove-ated imaging system and the pupil conjugation on the MEMS chip(inset); (b) the MEMS chip.

325 APPLIED OPTICS � Vol. 47, No. 3 � 20 January 2008

onstrated that the foveated foreground images werewell aligned with the corresponding regions on theperipheral background with simple postprocessingsteps.

C. Image Quality Characterization

We used the slanted edge target on an ISO12233standard resolution chart to characterize the modu-lation transfer function (MTF) of the prototype. TheImatest software (http://www.imatest.com) was uti-lized for MTF analysis. At each of the MEMS scan-ning angles, the slanted edge on the resolution chartwas positioned to the center of the foveated FOV, andthe corresponding image was analyzed. The mea-sured MTF performance of the bench prototype isshown in Fig. 10 as a function of the field angle �c ofthe foveated image. Here MTF50 measures the spa-tial frequency, where MTF drops to 0.5. In bothtangential and sagittal directions, the MTF50 per-formances remain constant around 55 cycles�mmwithin �15° of the visual field. The degradation be-yond that range is mainly due to the use of off-the-shelf lenses.

6. Conclusion and Future Work

Acquiring high-resolution wide FOV and HDR im-ages in real time is essential for many vision-basedapplications. The efficiency of the conventional recti-linear sampling schemes is as low as 3% for a 60°FOV system. In this paper, we presented an approachfor the design of a foveated imaging system, aiming tomaximize the sampling efficiency and informationthroughput of low-cost off-the-shelf image sensors, byexploring the spatially variant properties of the HVS.We demonstrated that a dual-sensor system with a120° total FOV can achieve a 90.4% bandwidth re-duction ratio, requiring ten times less pixels than auniformly sampled single resolution system. Thebandwidth reduction ratio can be further increased to95.7% by applying spatial variances of contrast andcolor sensitivities, which directly translates to 23times less communication bandwidth. We presentedan analytical method to maximize the bandwidthsaving of a multisensor foveated system. We furtherdescribed the schematic design of a compact dual-sensor system and demonstrated the implementationof a bench prototype and experimental results. Theprototype captures foveated images with 96.4% ofbandwidth reduction compared to a single resolutionsystem. In the future, we plan to custom-design anoptimized system with an overall FOV of over 120°and optimum information throughput ratio based onthe analysis in Section 3. We will further developfovea tracking algorithms that dynamically detectthe salient region of interest from the peripheral im-age and adaptively steer the fovea region toward adetected stimulus via the scanner unit. Such a capa-bility can be particularly beneficial to applicationssuch as surveillance and robot navigation.

This work is partially funded by the NationalScience Foundation grant award 04-11578 and 05-34777.

References1. G. Scotti, L. Marcenaro, C. Coelho, F. Selvaggi, and C. S.

Regazzoni, “Dual camera intelligent sensor for high definition360 degrees surveillance,” IEE Proc. Vision Image Signal Pro-cess. 152, 250–257 (2005).

2. T. Ienaga, K. Matsunaga, K. Shidoji, K. Goshi, Y. Matsuki, andH. Nagata, “Stereoscopic video system with embedded highspatial resolution images using two channels for transmis-sion,” in Proceedings of ACM Symposium on Virtual RealitySoftware and Technology (2001), pp. 111–118.

3. A. Ude, C. Gaskett, and G. Cheng, “Foveated vision systemswith two cameras per eye,” in Proceedings of IEEE Interna-tional Conference on Robotics and Automation (IEEE, 2006),pp. 3457–3462.

4. C. Wang, P. Shumyatsky, F. Zeng, M. Zevallos, and R. R.Alfano, “Computer-controlled optical scanning tile micro-scope,” Appl. Opt. 45, 1148–1152 (2006).

5. B. Potsaid, Y. Bellouard, and J. T. Wen, “Adaptive scanningoptical microscope (ASOM): a multidisciplinary optical micro-scope design for large field of view and high resolution imag-ing,” Opt. Express 13, 6504–6518 (2005).

6. M. Böhme, M. Dorr, T. Martinetz, and E. Barth, “Gaze-contingent temporal filtering of video,” in Proceedings of ACM

Fig. 9. (Color online) Mosaic of a sequence of foveated and pe-ripheral images.

Fig. 10. (Color online) Image quality characterization: MTF50versus the field angle of foveated image center.

20 January 2008 � Vol. 47, No. 3 � APPLIED OPTICS 326

Symposium on Eye Tracking Research & Applications (2006),pp. 109–116.

7. W. Zhou and A. C. Bovik, “Embedded foveation image coding,”IEEE Trans. Image Process. 10, 1397–1410 (2001).

8. W. S. Geisler and J. S. Perry, “Real-time foveated multireso-lution system for low-bandwidth video communication,” Proc.SPIE 3299, 294–305 (1998).

9. A. T. Duchowski and A. Çöltekin, “Foveated gaze-contingentdisplays for peripheral LOD management, 3D visualization,and stereo imaging,” ACM Trans. Multimedia Comput. Com-mun. Appl. 3, 1–21 (2007).

10. H. Murphy and A. T. Duchowski, “Hybrid image-�model-basedgaze-contingent rendering,” in Proceedings of ACM Sympo-sium on Applied Perception in Graphics and Visualization(2007), pp. 1–8.

11. D. Luebke and B. Hallen, “Perceptually driven simplificationfor interactive rendering,” in Proceedings of the 2001 Euro-graphics Workshop on Rendering (2001), pp. 223–234.

12. G. Sandini, P. Questa, D. Scheffer, and A. Mannucci, “A retina-like CMOS sensor and its applications,” in Proceedings ofIEEE Workshop on Sensor Array and Multichannel SignalProcessing (IEEE, 2000), pp. 514–519.

13. J. P. Rolland, A. Yoshida, L. D. Davis, and J. H. Reif, “High-resolution inset head-mounted display,” Appl. Opt. 37, 4183–4193 (1998).

14. D. V. Wick, T. Martinez, S. R. Restaino, and B. R. Stone,“Foveated imaging demonstration,” Opt. Express 10, 60–65(2002).

15. T. Martinez, D. V. Wick, and S. R. Restaino, “Foveated, widefield-of-view imaging system using a liquid crystal spatial lightmodulator,” Opt. Express 8, 555–560 (2001).

16. E. M. Reingold, L. C. Loschky, G. W. McConkie, and D. M.Stampe, “Gaze-contingent multiresolutional displays: an inte-grative review,” Hum. Factors 45, 307–328 (2003).

17. L. C. Loschky and G. S. Wolverton, “How late can you updategaze-contingent multiresolutional displays without detec-tion?,” ACM Trans. Multimedia Comput. Commun. Appl. 3(2007).

18. W. S. Geisler, J. S. Perry, and J. Najemnik, “Visual search: therole of peripheral information measured using gaze-contingentdisplays,” J. Vision 6, 858–873 (2006).

19. L. C. Loschky, G. W. McConkie, H. Yang, and M. E. Miller,“The limits of visual resolution in natural scene viewing,” Vi-sual Cognition 12, 1057–1092 (2005).

20. M. Dorr, M. Böhme, T. Martinetz, and E. Barth, “Visibility oftemporal blur on a gaze-contingent display,” in Proceedings ofthe ACM Symposium on Applied Perception, Graphics & Vi-sualization (2005), pp. 33–36.

21. D. J. Parkhurst and E. Niebur, “Variable-resolution displays:a theoretical, practical, and behavioural evaluation,” Hum.Factors 44, 611–629 (2002).

22. L. C. Loschky and G. W. McConkie, “Investigating spatialvision and dynamic attentional selection using a gaze-contingent multiresolutional display,” Q. J. Exp. Psychol. A 8,99–117 (2002).

23. R. Etienne-Cummings, J. Van der Spiegel, P. Mueller, andM. Z. Zhang, “A foveated silicon retina for two-dimensionaltracking,” IEEE Trans. Circuits Syst. II 47, 504–517 (2000).

24. G. Godin, P. Massicotte, and L. Borgeat, “High-resolution in-sets in projector-based stereoscopic displays: principles andtechniques,” Proc. SPIE 6055, 60550F (2006).

25. S. J. D. Prince, J. H. Elder, Y. Hou, and M. Sizinstev, “Pre-attentive face detection for foveated wide-field surveillance,” inIEEE Workshops on Application of Computer Vision (IEEE,2005), pp. 439–446.

26. Peter G. J. Barten, Contrast Sensitivity of the Human Eye andIts Effects on Image Quality (SPIE Optical Engineering Press,1999).

27. C. Gao, N. Ahuja, and H. Hua, “Active aperture control andsensor modulation for flexible imaging,” in Proceedings of In-ternational Conference on Computer Vision and Pattern Rec-ognition (2007).

28. J. E. Greivenkamp, Field Guide to Geometrical Optics (SPIEPress, 2004).

29. Camera calibration toolbox, http://www.vision.caltech.edu/bouguetj/calib_doc/index.html.

327 APPLIED OPTICS � Vol. 47, No. 3 � 20 January 2008