ADAPTIVE, OPTIMAL-RECOVERY IMAGE INTERPOLATION

D. Darian Muresan and Thomas W. Parks

Schoolof ElectricalandComputerEngineeringCornellUniversity

darian,parks@ee.cornell.edu

ABSTRACT

We considerthe problemof imageinterpolationusingadaptiveoptimalrecovery. We adaptively estimatethelocalquadraticsignalclassof our imagepixels. We thenuseop-timal recovery to estimatethemissinglocal samplesbasedon this quadraticsignalclass.Thisapproachtendspreserveedges,interpolatingalongedgesandnot acrossthem.

1. INTRODUCTION

Imageinterpolationis becomingan increasinglyimportanttopic in digital imageprocessing,especiallyas consumerdigital photographyis becomingever morepopular. Fromenlarging consumerimagesto creatinglargeartisticprints,interpolationis at theheartof it all. It hasbeenknown forsometimethatclassicalinterpolationtechniquessuchaslin-earandbi-cubicinterpolationarenotgoodperformerssincethesemethodstendto blur andsmoothedges.

Waveletshave beensuccessfullyusedin interpolation[1, 4, 6]. Thesemethodsassumetheimagehasbeenpassedthrougha low passfilter beforedecimationandthentry toestimatethemissingdetails,or waveletcoefficientsfrom thelow resolutionscalingcoefficients. Onedrawbackto theseapproachesis that they assumethe knowledgeof the lowpassfilter.

Directional interpolationalgorithmstry to first detectedgesandtheninterpolatealongedges,avoiding interpola-tion acrossedges[5]. In this class,therearealgorithmsthatdo not requirethe explicit detectionof edges.Rather, theedgeinformationis built into the algorithmitself. For ex-ample,[3] usesdirectionalderivativesto generateweightsusedin estimatingthemissingpixels from theneighboringpixels. In [2], the local covariancematrix is usedfor esti-matingthemissingpixels.This interpolationtendsto adjustto anarbitrarily orientededge.

In this paperwe presenta new directionalinterpolationtechniquebasedon optimalrecovery. Theresultsof our in-terpolationapproachcanbe thoughtof asan extensionto[2]. In regions of high frequency our approachprovides

Thiswork wassupportedby NSFMIP9705349,TI andKodak

PSfragreplacements�

Fig. 1. GeometricDiagramof Ellipsoid Class

slightly better resultsthan [2] and in somecasesoutper-forms[9].

2. OPTIMAL-RECOVERY

In this sectionwe briefly review the theoryof optimal re-coveryasappliedto theinterpolationproblem[8]. We thenapplythis theoryto developanew adaptiveapproachto im-ageinterpolation.Theinterpolationproblemmaybeviewedas a problemof estimatingmissingsamplesof an image.This latterproblemcanbeexaminedusingthetheoryof op-timal recovery. The theoryof optimal recovery providesabroadersetting,which illuminatestheprocessof interpola-tion, by providing errorboundsandallowing calculationofworst-caseimageswhich achievethesebounds.

Locally, at location � (Fig. 2), we modelthe imageasbelongingto acertainellipsoidalsignalclass����������� ��������������� (1)

where � is derivedfrom thelocal imagepixelsasshown insection3. Vector � is any subsetof theimagecontainingthemissingpixel � . Vector � is chosensuchthatany � linearfunctionals( ��� �"!#�%$&��'('�')� � ) of � areassumedknown. Ifwe notethe actualvaluesof the functionalsby *+� we have� � , ��-.�/* � . In this paperwe assumethat the functionalsare basedon derivativesand/oractualpixel valuesof thedecimatedimage. The known functionals � � , in the localimage,determineahyper-plane0 (Fig. 1).

The intersectionof the hyper-planeand ellipsoid is ahyper-circlein 0 . Theintersectiondependsupontheknown

functionalsof thelocal imageandwe call it 132 . Formally,1425�6���70%�&� � , ��-8�9* � �;:<��:)=��9$&�>� (2)

For a linear mapping ? , the imageof 1 2 under ? is therangeof valuesthat ?�* can take. The optimal recoveryproblemis to selectthevaluein 0 which is a bestapproxi-mationoverall ?�* in ?@1 2 . We wantto minimizeA �CB�D+E2GFH=I2 JLK�5MN� JTheChebyshev centerachievesthisminimization.TheCheby-shev centerhasbeenshown to betheminimum O -normsig-nal on the hyper-planedeterminedby the known samples.Thesolutionto thisproblemis well-known: see[8, 7].

If the collection of known functionalsis * � , the mini-mum norm signalis PQ . Signal PQ is the uniquesignalin 0with theproperty, :RPQ : = � SUTRVW&XZY 2�[]\I^ X :)��: = (3)

Our estimatessignal PQ mustsatisfy � � , PQ -_�`* � andwe areestimating� , PQ -a�b* . As shown in [8] thereexist vectorsc � ced ��'('�'(� c�f suchthat � , KQ -8� , c �&PQ - = and��� , KQ -g� , c � ��PQ - = (4)

wherethe parenthesesdenotea Q dot product. Vectorsc �

areknownastherepresentors.From[8] thesolutionisgivenby PQ � fh � \ dji � c � (5)

wherethe constantsi � aredeterminedfrom the constraintof equation(4).

An advantageof this approachis not only that we canminimize the distance

A �bB�D�E 2;Fk=I2 JlK�mMn� J, but we also

obtainboundson themaximumerrorA

andwe canfind theimagewhichachievesthismaximumerror.

We now dealwith theproblemof determiningO adap-tively from the imagedata. To make this explanationassimpleandasstraightforwardaspossible,we demonstrateourmethodwith asimpletoy example.

3. ADAPTIVE, OPTIMAL-RECOVERYINTERPOLATION

Ouradaptively determinedquadraticsignalclass,or O , willbeameasureof how well thelocaldatamatchesthealreadyknown functionals � � . We want to find an adaptive signalclass� of theform:���o�G�� ��� � �p�q�n��� (6)

To bestunderstandthis process,let’s look at Fig. (2). In

a r1 a r

2 s b 1

a r4 a r

3 t b 4

d u

1 d u

2 c v 1

d u

4 w d

u3 t c v 4

w

y x 1 y x 2 s y x 3

ty x 4 y x y x 6

yy x 7 z y x 8

{ y x 9 |

b 2 s

b 3 t

c v 2

c v 3 t

Fig. 2. Interpolatepixel � . The only known pixelsarethegraypixels.

this small example,the problemis thatof estimatingpixel� . Our first stepis to choosea signal � that containsthemissingpixel � . For reasonsthatwill beclearin a moment,let �@�~} � d �&�4�&�3�k�3���4���3���>�4���4� d �&�4�&�4�&�<� �Next, we assumethat thereexists weight � d �('('�')�"��� suchthat locally, eachpixel can be estimatedby the weightedsumof thefour closestdiagonalpixels.With: K� : � ��BpS�T� : � : �ourmeasureof how well thedatamatchesthenearbypointsis

K� �������������

� d� �� �����<�����...

�(�����������

M������������

� d �k���k���������� d �����k�� d �<� �<� ���� d � d � d � d� � � � � � � �� � � � � � � d� � � � � d � �...

......

...

�(�����������

��� � d�������3�

�(�� (7)

or equivalently K� ���@M �#¡Thenormsquaredof

K� is givenbyK� � K� �¢� �n£¥¤ M �9¦Z� � ��§+¨ d � �j© �Thus, �C� £¥¤ M �9¦Z� � ��§ ¨ d � � © (8)

The problemof finding estimateK� which minimizes

A �B�D+E 2;Fk=I2 JLK�ªMa� Jis equivalentto finding � whichminimizes� � �p� giventheknown functionals� � .

Assuming� is full rank,matrix Q hasfour zeroeigen-valueswith therestof thembeingall one.Thenull spaceofQ is spannedby thecolumnvectorsof � . At first glance,it

seemsthatthesolutionto this problemmight beany vectorin thenull space,sincethatwill give zeroerror. Thathow-ever, is nottruesincethesolutionmustalsosatisfythegivenfunctionals.Unlessthereareonly four known functionals,oursolutionwill not bein thenull spaceof Q .

Known functionalscanbepixel valuesof thedecimatedimage,derivative assumptionsor any otherlinear function-als of the high resolutionimage. In our toy example,thelinear functionalsarethegivenpixels �k�&�«���>� � d � ��� andas-sumptionsaboutthederivatives.In particular, welook atthederivativesin thedirections�>�¬M ��� and �k�¬M­� d . We chosethe directionwith the smallestchangeandassumethat thederivativesof theunknown pixelsin thatdirectionareequalwith thederivativesof theknown pixels in thesamedirec-tion. For example,if ����M���� hasthesmallestdifference,thederivativebasedfunctionalswouldbe�&��MN�p�9�<��M ��� and �&�¬MN�p���>�_M­�>�Whenthe known functionalsareonly the decimatedpixelvalues,this methodsimplifies to the methodpresentedin[2].

Theformulationof ourproblemandtheadaptive Q ma-trix is alsoquiteusefulwhenweassumethattheimagewentthroughalow passfilter, beforedecimation.In thiscase,ourassumptionis that thepixel valuesof Fig. 2 aresamplesofthefilteredimage.If we let ® bea filtering matrix andweassumethattheimagebeforefiltering is ¯����®7¯ (9)

then Q of (8) becomes�°�±® � Oa® . Thenew Q will stillhave four zeroeigenvalues,but the othereigenvalueswillno longerbeone.

Theapproximationof thesignalclass,andthereforetheinterpolationresults,canbefurtherimprovedby aniterativeprocessasfollows:

1. Interpolatethe missing pixels with the methodde-scribedabove.

2. Using the interpolatedpixels, return to equation(2)andaddthecalculatedpixelsat thehigherresolutionasextra functionals.

We haven’t provedconvergencehere,but from our experi-mentalresults,repeatingthis processthreetimesseemstobeenough.

4. RESULTS

In obtainingour resultswe first startedwith a high resolu-tion image.We thenfilteredthehigherresolutionimagebya low passfilter (Daubechies1), to simulatecameralenses,and decimatedby two. We then reconstructedthe imageusingdifferentinterpolationapproaches.

For our resultssectionwe have comparedtheadaptive,optimal recovery imageinterpolationalgorithmagainstthealgorithm presentedin [2], againstbi-cubic interpolationandagainsta commerciallyavailablealgorithm[9]. Whencomparedagainst[2] the algorithmoutperformedslightly,especiallyaroundsharpand/orthinedges.Thealgorithmal-waysoutperformedbi-cubicinterpolation.Whencomparedagainst[9] therewereplaceswherethe adaptive, optimal-recoveryinterpolationoutperformed[9], but therewerealsoplaceswhereit under-performed.Somesampleimagesareincludedat theendof this section,but thereaderis encour-agedto view TIF imagesatww.ee.cornell.edu/² splab.

Finally, we would like to thankXin Li for providing uswith his interpolationalgorithm[2].

5. REFERENCES

[1] D. DarianMuresanandThomasW. Parks,“Predic-tion of ImageDetail,” IEEE ICIP, Vancouver, Sept.2000.

[2] Xin Li and Michael T. Orchard, “New Edge Di-rectedInterpolation,” IEEE International Conferenceon Image Processing, Vancouver, Sept.2000.

[3] RonKimmel, “Demosaicing:ImageReconstructionfrom Color CCD Samples,” IEEE Transactions onImage Processing, Vol. 8, No. 9, (September1999),pp.1221-1228.

[4] W. Knox Carey, Daniel B. Chuangand Sheila S.Hemami, “Regularity-PreservingImage Interpola-tion,” IEEE Transactions on Image Processing, Vol.8, No. 9 (Sept.1999),1293-1297.

[5] J. Allebach and P. W. Wong “Edge-directedInter-polation,” IEEE Proceedings of ICIP, 1996,pp.707-710.

[6] S. GraceChang,ZoranCvetkovic, andMartin Vet-terli, “Resolution Enhancementof Images UsingWavelet Transform Extrema Interpolation,” IEEEICASSP, (May 8-12,1995),pp.2379-2382.

[7] C.A. Michelli andT. J.Rivlin, “ A Survey of OptimalRecovery,” in Optimal Estimation in ApproximationTheory, C. A. Michelli and T. J. Rivlin, Eds.NewYork: Plenum1976,pp.1-54.

[8] M. GolombandH. F. Weinberger, “Optimal approx-imationandError Bounds,” On Numerical Approxi-mation, R. E. Langered.,TheUniversityof Wiscon-sinPress,Maddison,pp.117-190,1959.

[9] Commercial,FractalBasedInterpolationAlgorithm(www.genuinefractals.com)

Fig. 3. Altamira (top), Cubic (center),Optimal-Recovery(bottom).

Fig. 4. Altamira (top), Cubic (center),Optimal-Recovery(bottom).