Imager Design using Object-Space Prior Knowledge · PDF fileImager Design using Object-Space...
Transcript of Imager Design using Object-Space Prior Knowledge · PDF fileImager Design using Object-Space...
Imager Design using Object-Space Prior KnowledgeM. A. Neifeld
University of Arizona
OUTLINE
1. The Last Slot
2. Introduction
3. PSF Engineering
4. Feature-Specific Imaging
Neifeld IMA 2005
Introduction: objects are not iid pixels.- Conventional cameras are designed to image iid pixels
à impulse-like point-spread-functions (identity transformation)
à generic metrics such as resolution, field of view, SNR, etc.
- Real objects are not iid pixels so don’t estimate pixels
- This keeps the compression guys employed!
- (106 pixels)(3 colors/pixel)(8 bits/color) = 2.4x107 bits
- (1011 people)(4x109 years)(109 images/year) = 4x1029 images à <100 bits
- The set of “interesting” objects is small
- Many ways to characterize “interesting” objects: power spectra, principal components, Markov fields, wavelet projections, templates, task-specific models, finite alphabets, etc.
Information depends upon task:q Option 1 - this is a random image è I = 107 bits
q Option 2 – this is a “battlefield” image è I = ? bits
… how to quantify PDF!
q Option 3 – this image either contains a tank or not è I = 1bit
… task-specific source model
Neifeld IMA 2005
Introduction: post-processing exploits priors.- Linear Restoration: de-noising and de-blurring exploit noise statistics, object power
spectra, principal components, wavelets, …
- Nonlinear Restoration: super-resolution uses finite support, positivity, finite alphabet, power spectra, wavelets, principal components, isolated points, …
- Recognition: features, templates, image libraries, syntax, invariance, …
- Finite Alphabet Post-Processing Examples
LADAR Multi-Frame Super-Resolution
Object
10
15
20
25
30
35
40
45
50
55
largest returnrm
se= 7.3m
m
Wiener
rmse = 5.8m
m
10
15
20
25
30
35
40
45
50
55
Viterbirm
se = 0.6mm
Object Measurement IBP – 28%
IBPP – 24% 2D4 - 2%
Optical blur = 1.5 and pixel-blur = 2. Reconstruction from 2 images, s = 1%
Axial extent of target = Temporal pulse width = 30mm. Target feature size = Scan step size = 4.6mm
Neifeld IMA 2005
Imager Goals:Imager Goals:uu Estimate point source position(s): Estimate point source position(s): { }uu Conventional image may be formed as a postConventional image may be formed as a post--processing stepprocessing step
Source Source volumevolume
ImagerImager
r1
rM
r2
z
yx
Strong Object Model:Strong Object Model:
rMr1 r2 …
M : Number of point sources
Conventional image
®® Fluorescent markersFluorescent markers®® Distant “bright” objects: aircraft, missile, starsDistant “bright” objects: aircraft, missile, stars
uu EqualEqual--intensity monochromatic point sourcesintensity monochromatic point sourcesuu Scene is completely specified by sources positions:Scene is completely specified by sources positions:
rMr1 r2 …
rMr1 r2 …
Introduction: plausibility of a single pixel imager.
uu Measure only what you want to knowMeasure only what you want to know
Neifeld IMA 2005
1
2
3
Source Volume
12
3
d1 ,h1LensDetector
phasemask
q Optimize imager based on information metric.qMaximize measurement entropy.q Select detector sizes and positions based on measurement pdf.
Measurement log-pdf
random phase
Measurement log-pdf
cubic phase
Measurement log-pdf
Introduction: information-based design.
source power = 0.5mWNEP=2nW
40cm
1cm3
Neifeld IMA 2005
89%74%36%Two detectors in two apertures
74%54%30%Two detectors in one aperture
65%39%21%One detector in one aperture
RPMCPMConventionalê Detector(s) : Imager Type è
Multiple Sources in VolumeMultiple Sources in VolumeSingle Source in VolumeSingle Source in Volume
Introduction: single pixel imager results.
q Object-space prior knowledge should inform the optical designq Let’s utilize this viewpoint in a more useful problem domain
Neifeld IMA 2005
PSF ENGINEERING
Neifeld IMA 2005
w Imagers for which pixel size > optical spot size. .
w Large pixels result in under-sampling/aliasing.
w Sub-pixel shifted measurements to resolve ambiguity. spatial ambiguity
Frame 1
shift camera
Frame 2
…..
Frame K
PSF Engineering: Under-Sampled Imagers
w Optical degrees of freedom not exploited.
wWe consider engineering optical point spread function.
Neifeld IMA 2005
Sensor details:w Pixel = 7.5 µmw Under-sampling = 15xw Full well capacity = 49ke-
w Spectral bandwidth = 10nmw Center wavelength = 550nm
Optics details:w Resolution = 0.2mrad/1µmw Field of view = 0.1 radw Thickness = 5mmw Aperture = 2.75mmw F/# = 1/1.8
M = 34x34
Object: f Imaging operator: H Measurements: g
N = 512x512 Sub-pixel shifts
…..
…..
w Single frame signal to noise ratio: SNR = 10log[sqrt(Ne)] = 23.3dBw SNR can be improved via multi-frame averaging ~ sqrt(K)w Total photon-count is kept constant over multiple-frames.
Phase-mask
Imaging ModelNeifeld IMA 2005
u Linear imaging model: g = Hf + n (note: n is AWGN)
u Block-wise shift-invariant imaging operator H is M x N
u Problem: M << N (e.g., M=N/15)
u Linear minimum mean square error (LMMSE) reconstruction: f = Wg
u LMMSE operator: W = RfHt(HRfHt+Rn)-1
u No Priors = flat PSD
u Priors = power law PSD or triangle PSD
^
Exa
mpl
e tr
aini
ng o
bjec
ts
PSD model
Power Lawà PSD(f) = 1/fη
Linear ReconstructionNeifeld IMA 2005
u Root Mean Squared Error:( )
[%]255
ˆ100RMSE
2
ff −×=
u Angular resolution:
Point Objectf = δ(r)
CompositeChannel
Hc
Reconstruction to Diffraction-limited
sinc2+
gn
f̂
RMSE=8.6%
ObjectCompositeChannel
Hc
LMMSEReconstruction+
gn
2
2 ˆsincminarg
−
∆
→∆∆
fx
θ
θθ
∆θ=0.4mrad
Performance MeasuresNeifeld IMA 2005
Resolution for TOMBO
sub-pixel shift Sub-pixel shiftedmeasurements
TOMBO ImagerConventional Imager
Shift-sensor
Conventional/TOMBO Imager Results
RMSE for TOMBO
Neifeld IMA 2005
u Consider use of extended point spread function(PSF)
extended PSF
impulse-like PSF
u Design issue #1: retain full optical bandwidthu Design issue #2: tradeoff SNR for condition number
u Pseudo-Random Phase masks for extended PSF
Modulation Transfer Function
( )ρ
γσασ
φ
∆⋅=
−⋅∆⋅= ,exp
4 2
2xR x ∆ - mask roughness ρ - mask correlation length
Realization of a spatial Gaussian random process.
Pseudo-Random Phase mask Enhanced Lens (PRPEL)
Example PSF(∆=0.5λ ,ρ=10 λ)
Alternate PSFNeifeld IMA 2005
uAll designs use optimal roughness.
u Note more rapid convergence of PRPEL compared to TOMBO.
u Higher resolution achieved by PRPEL at reduced number of frames.
u PRPEL achieves 0.3mradresolution at K=5 compared to K=12 for TOMBO.
Resolution Results
Resolution for PRPEL and TOMBO
Neifeld IMA 2005
RMSE for PRPEL and TOMBO
u PRPEL makes effective use of prior knowledge at K=1
u Note more rapid convergence of PRPEL.
u PRPEL consistently out-performs TOMBO.
K=1
PRPEL
K=2
K=3
TOMBOK=1
K=2
K=3
RMSE ResultsNeifeld IMA 2005
45K
PRPELTOMBOImager Type→Number of Frames↓
4% RMSE requirement
3.9%4.2%RMSE
PRPELTOMBOImager Type →(K=4)
RMSE achieved at M=N/4
0.35mrad0.60mradResolution
PRPELTOMBOImager Type →(K=4)
Resolution achieved at M=N/4
u PRPEL imager achieves 60% improvement in resolution.
u PRPEL imager obtains 22% improvement in RMSE.
K
Imager Type→Number of Frames↓
512
PRPELTOMBO
0.3mrad Resolution requirement
PRPEL SummaryNeifeld IMA 2005
u Sine-Phase mask Enhanced Lens(SPEL) :
( )θωαφ ii
N
ii xx += ∑
=sin)(
1
Amplitude Spatial-frequency Phase offset
u Pick N=3: yields 12 free parameters for optimization.
u Optimization criteria: ( )
[%]255
ˆ100RMSE
2
ff −×=
u RMSE computed over object class using LMMSE operator.
u PSF is optimized for each value of K.
Phase-maskωα
PSF Engineering via SPELNeifeld IMA 2005
K=1
K=2
u Note smaller support of SPEL PSF compared to PRPEL PSF.
u SPEL PSF also contains sub-pixel structure.
u SPEL PSF has more efficient photon-distribution.
u PSF support reduces with increasing K.
u SPEL PSF is array of delta pulses.
Observations
Observations
Optimized PSFNeifeld IMA 2005
K=16
u SPEL PSF converges to delta pulses as K increases.
u In limit Kà16 we observe that SPEL PSF to converge to TOMBO-like PSF.
Observations
Optimized PSF: System ImplicationsNeifeld IMA 2005
RMSE for SPEL, PRPEL, and TOMBO
RMSE : Power law PSD
uSPEL provides best use of prior knowledge for K=1uSPEL outperforms TOMBO by 47% in terms of RMSE(K=8).uSPEL improves RMSE by 35% compared to PRPEL (K=8).
K=2
PRPELK=1
K=3
K=2
SPELK=1
K=3
ResultsNeifeld IMA 2005
Angular resolution Resolution for SPEL,PRPEL and TOMBO
u Note PSF optimization was performed over RMSE.
u SPEL out-performs TOMBO.
u SPEL performance compared to PRPEL improves with increasing K.
Results
q PSF engineering can exploit weak object prior knowledge to improve performanceq Stronger object prior knowledge can enable non-traditional image measurement
Neifeld IMA 2005
FEATURE-SPECIFIC IMAGING
Neifeld IMA 2005
noisy image
Feature extraction Features Task
Feature-specific optics Features Task
Conventional imaging system PCA, ICA, Fisher, Wavelet, etc.
noise
noiseFeature-specific imaging
system (FSI)
Restoration, recognition,compression, etc.
u Feature-Specific Imaging (FSI) is a way of directly measuring linear features (linear combinations of object pixels).
u Attractive solution for tasks that require linear projections of object spaceu Let’s consider a case for which task = pretty picture
Passive Feature-Specific Imaging: MotivationNeifeld IMA 2005
2
11
ˆ
ˆmin {|| || }
|| || max{ | |} 1m
iji j
E
subject to f
ε
=
= =
= −
= =∑
y Fx x My
x x
F
Noise-free reconstruction:
= pca pca pcaTF M F
PCA solution :
1( )general−= T
x xM R F FR F
is any invertable matrixpca=F AF
A
General solution :
Result using PCA features:
FSI for Reconstruction
u PCA features provide optimal measurements in the absence of noise
photon count constraint
Neifeld IMA 2005
2 -1( I) Wiener - operatoropt σ= +T Tx xM R F FR F
2 2 -1{ ( I) } { }Tr Trε σ= − + +T Tx x xFR F FR F R
MyxnFxy =+= ˆ
)}||{||
log(10 2
2
σxE
SNR=1
|| ||
pca
pca
=F
FF
• Object block size = 4x4• Noise = AWGN• We use stochastic tunneling to optimize/search
optF
RMSE = 11.8
RMSE = 124
RMSE = 12
RMSE = 12.9
Optimal Features in Noise
u PCA features are not optimal in presence of noise
Noise-free problem statement:
Note: PCA error is no longer monotonic in the number of features à trade-off between truncation error and photon count constraint
11
|| || max{ | |} 1m
iji j
subject to f=
= =∑F
Neifeld IMA 2005
u Error increases as number of feature increases for PCA solution
u Reconstructed is improved significantly by using optimal solution
u Optical implementation requires non-negative projections
Optimal Features in NoiseNeifeld IMA 2005
u Optimal FSI is always superior to conventional imagingu Non-negative solution is a good experimental system candidate
Passive FSI Result SummaryNeifeld IMA 2005
Passive FSI for Face Recognition
• Face recognition from grayscale image feature measurements
• Class of 10 faces, 600 images per face
• Training = 3000 faces and testing = 3000 faces
• Features: wavelet, PCA, Fisher, …
• Recognition algorithms:
- k – nearest neighbor based on Euclidean distance metric
- 2-layer neural networks batch trained using back-propagation with momentum
Sample images from face database [Each image is 128x96]
First Wavelet feature of the above images [Each feature is 8x6]
Comparison of PCA recognition with AWGN
0
10
20
30
40
50
60
70
80
90
100
0 250 500 750 1000 1250 1500 1750 2000AWGN standard deviation
Rec
ogni
tion
perf
orm
ance
[%]
0 mux0 conv0_1 mux0_1 conv
Conventional
FSI
Neifeld IMA 2005
Passive FSI Optical ImplementationsNeifeld IMA 2005
• What is active illumination ?
• Project known structure onto scene
• Additional degrees of freedomimprove imager performance
• Past work on active illumination focused on:• Obtain depth-information for 3D objects
• Enhanced resolution for 2D objects
• Our goals:• Improve object- and/or task-specific performance
• Simplify light collection hardware
Projector
Object
Illumination pattern
Conventional cameras
Active Feature-Specific Imaging: MotivationNeifeld IMA 2005
ece
• Illumination patterns are eigenvectors (refer as PCA - FSAI)
• Advantages
• Small number of detectors
• High measurement SNR
• Task is to produce object estimate using these values
Object GLight Collection
Photodetector noise (AWGN)
Sequence of illumination patterns
GP )]([ idiag
iidiag α̂~)](][[∑ GPH
iii ndiagr += ∑ GPH )](][[
H (optics operator)
=
Mr
rr
.
.2
1
R
Vector of Measurements
(Estimate of feature weight)
16 × 16 replication of eigenvector P1
P2PM
64 × 64
16 × 16detector
FSAI System Flow DiagramNeifeld IMA 2005
ece
• Post-processing operator W is obtained by minimizing J
[ ]
matrix. covariance noise matrix,n correlatioobject
)](][[~]~~
,]~~
[~ˆ
2
2
1,
1
==
==
+=
∑=
×
−
nG
N
njniji,NMji,
nT
GT
G
and
diagandwhere
RR
PHHH[H
RHRHHRW
=
Mr
r
r
.
.2
1
R
)(]})ˆ)(ˆ[({ errorsquaremeanRWGRWGtraceEJ T−−=
Measurementvector
Linear post-processing
W
G = W R ?
• The MMSE operator is given by:
∑ ii Pα̂
(suboptimal in noise)
• Metric to evaluate reconstructions :
∑ ∑= =
−=objectsofnumber
k
N
iikikNobjectsofnumber
RMSE1 1
22
2
)ˆ(11 GG
N 2 = number of pixels, M = number of patterns
FSAI Post-ProcessingNeifeld IMA 2005
ece
• PCA vectors are not optimal in presence of noise
i
K
1ii
2PCA PaG with|GG|iswhichJ ∑
=
=−≠ ˆˆ)(]})ˆ)(ˆ[({ noisecontainsRRWGRWGtraceEJ T−−=
• Minimize the residual MMSE (JMMSE) with respect to both Pi’s and Ti
’s
( ) ( ) ( )1
2
2
22
2
21
2
111
11
,....,)...()...()...(
),.....,,.....(ˆ
~~~−
+=
M
T
MGM
T
MG
MM
TTTdiagPPRPPPPR
TTPPWwhereσσσHHH
}~ˆ{),....,,...( 11 GGMMMMSE RHWRTraceTTPPJ −=
SNR = 26 dB
M = 4 M = 8
optimaloptimal
PCAPCA
• Optimal features depend on M, SNR
Illumination Using Optimal PatternsNeifeld IMA 2005
ece
SNR = 26 dB (LOW NOISE)
M = 4
M = 8
Original object
PCA-FSAI(uniform T)
PCA-FSAI (optimal T) Optimal FSAI
• Minimum from PCA-FSAI
RMSE = 0.0633
• Minimum from optimal FSAI
RMSE = 0.0465
0 2 2 6 8 10 12 14 160.04
0.1
0.2
N u m b e r o f f e a t u r e s
Ave
rag
e R
MS
E (
LO
G S
CA
LE
)
U n i f o r m i l l u m i n a t i o n
P C A - F S A I(u n i f o r m T)
P C A - F S A I (n o n-u n i f o r m T)O p t i m a l
f e a t u r e s
FSAI ResultsNeifeld IMA 2005
ece
54 %31 %
Improvement of optimal FSAI compared to uniform illumination
0.07 (M = 16)0.0465 (M = 16)Optimal features
0.0768 (M > 2 )0.063 (M > 4)PCA – FSAI (nonuniform T)
0.0768 (M = 2)0.063 (M = 4)PCA – FSAI (uniform T)
0.151 (M =1)0.067 (M = 1)Uniform illumination
SNR = 16 dBSNR = 26 dBAlgorithm
FSAI Results SummaryNeifeld IMA 2005
Conclusions
q Objects are not iid pixels
èPixel-fidelity should not be the goal of an imager
èNeed new non-traditional design metrics
q Design should reflect prior knowledge of objects
èObject-specific imagers (e.g., SPEL)
èJoint design of optics and post-processing
q Design should reflect prior knowledge of application
èTask-specific imagers (e.g., FSI)
Neifeld IMA 2005