A Model for Image Splicing

Tian-Tsong Ng, Shih-Fu ChangDepartment of Electrical EngineeringColumbia University, New York, USA

OutlineReview

Problem and MotivationOur ApproachDefinition: BicoherenceWhy Bicoherence good for splicing detection? Previous Hypothesis

Bicoherence FeaturesMagnitude featurePhase feature

Proposed Image Splicing ModelBipolar Perturbation HypothesisBicoherence of bipolar signalBipolar perturbation effect on magnitude featureBipolar perturbation effect on phase feature

Problem & Motivation: How much can we trust digital images?

General problem: Image Forgery DetectionImage Forgery: Images with manipulated or fake content(In)Famous examples:

March 2003: A Iraq war news photograph on LA Times front page was found to be a photomontageFeb 2004: A photomontage showing John Kerry and Jane Fonda together was circulated on the Internet

Adobe Photoshop: 5 million registered users

Definitions: Photomontage and Spliced Image

Specific problem: Image Splicing Detection

Photomontage: A paste-up produced by sticking together photographic images, possibly followed by post-processing (e.g. edge softening and adding noise).Spliced Image (see figures): Splicing of image fragments without post-processing. A simplest form of photomontage.

Why interested in detecting image splicing?Image splicing is a basic and essential operation in the creation of photomontage

Therefore, a comprehensive solution for photomontage detection includes detection of post-processing operations and intelligent techniques for detecting internal scene inconsistencies

spliced

spliced

Image Forgery Detection ApproachesPassive and blind approach:

Without any prior information (e.g. digital watermark or authentication signature), verifying whether an image is authentic or fake.Advantages: No need for watermark embedding or signature generation at the source side

Active approach: Fragile/Semi Fragile Digital Watermarking: Inserting digital watermark at the source side and verifying the mark integrity at the detection side.Authentication Signature:Extracting image features for generating authentication signature at the source side and verifying the image integrity by signature comparison at the receiver side.Effective when there is

A secure trustworthy cameraA secure digital watermarking algorithmA widely accepted watermarking standard

What are the qualities of authentic images?

Image AuthenticityNatural-imaging Quality

Entailed by natural imaging process with real imaging devices, e.g. camera and scannerEffects from optical low-pass, sensor noise, lens distortion, demosicking, nonlinear transformation.

Natural-scene QualityEntailed by physical light transport in 3D real-world scene with real-world objectsResults are real-looking texture, right shadow, right perspective and shading, etc.

Examples:Computer graphics and photomontages lack in both qualities.

Computer Graphics

photomontage

Definition: BicoherenceBicoherence = normalized bispectrum (3rd order moment spectra)Definition (Bicoherence) The bicoherence of a signal x(t)with its Fourier transform being X(ω) is given by:

1 2

*( ( , )1 2 1 2

1 2 1 22 21 2 1 2

[ ( ) ( ) ( )]( , ) ( , )

[ ( ) ( ) ] [ ( ) ]j bX

X X

E X X Xb b e

E X X E Xω ωω ω ω ω

ω ω ω ωω ω ω ω

Φ+= =

+

MagnitudePhase

Numerator =Bispectrum

Normalized by the Cauchy-Schwartz Inequality upper bound

2

2 2

Cauchy-Schwartz Inequality

Hilbert space, : is a random variable satisfying [ ]

[ ] [ ] [ ] ( , )

x x E x

E xy E x E y x y∗

Κ= <∞

≤ ∈Κ

Notations:

( )magnitude

phase⋅ =

Φ ⋅ =

Why BIC is Good for Splicing Detection? Hypothesis I [Farid99]

Quadratic Phase Coupling (QPC)A phenomena where quadratic related frequencies

has the same quadratic relationship1 2 1 2, and ω ω ω ω+1 2 1 2, and φ φ φ φ+ Phases are quadratic coupled

(not independent)!

*1 2 1 2

1 2 2 21 2 1 2

[ ( ) ( ) ( )]( , )[ ( ) ( ) ] [ ( ) ]

X

X X

E X X XbE X X E X

ω ω ω ωω ωω ω ω ω

+=+

If (ω1 , ω2 , ω1+ω2) are quadratic phase coupled,

1 2*

1 2 1 2

1 2 1 2

1 2 1 2

1 2

1) [ ( , )] would be 0

[ ( ) ( ) ( )][ ( )] [ ( )] [ ( )]

( ) 02) ( , ) would be close to unity (imagine X now becomes positive RV)

b

X X XX X X

b

ω ω

ω ω ω ωω ω ω ω

φ φ φ φω ω

Φ

Φ +=Φ +Φ −Φ += + − + =

If (ω1 , ω2 ,ω1+ω2) have statistically independent phase,

1 2

1 2

1) ( , ) would be 0 due to statistical averaging2) [ ( , )] would be random

b

b

ω ω

ω ωΦ

Hypothesis I (cont.)

1 1 2 2( ) cos( ) cos( )x t t tω φ ω φ= + + +1 1

1 1 2 22 2

1 2 1 2

1 2 1 2

1 1 2 2

( ) cos(2 2 ) cos(2 2 )cos(( ) ( ))cos(( ) ( ))

cos( ) cos( ) 1

y t t ttt

t t

ω φ ω φω ω φ φω ω φ φ

ω φ ω φ

= + + +

+ + + ++ − + − +

+ + + +

Argument [Farid99]: Quadratic-linear operation gives rise to QPC and a nonlinear function, in Taylor expansion, contains quadratic-linear term. As splicing is a nonlinear operation, hence bicoherence is good at detecting splicing.

Problems:1. No detailed analysis was given. 2. The quadratic-linear operation here is a point-wise operation, it is not

clear how splicing can be related to a point-wise operation?

2

Quadratic linear Operation( ) ( ) ( )y t x t x t= +

OutlineReview

Problem and MotivationOur ApproachDefinition: BicoherenceWhy Bicoherence good for splicing? Quadratic Phase Coupling Hypothesis

Bicoherence FeaturesMagnitude featurePhase feature

Proposed Image Splicing ModelBipolar Perturbation HypothesisBicoherence of bipolar signalBipolar perturbation effect on magnitude featureBipolar perturbation effect on phase feature

Columbia Image Splicing Detection Evaluation Dataset

933 authentic and 912 spliced image blocks (128x128 pixels)Extracted from

Berkeley’s CalPhotos images (contributed by photographers) which we assume to be authentic

Splicing is done by cut-and-paste of arbitrary-shaped objects and also vertical/horizontal strip.

Authentic

Samples

Spliced

TexturedSmooth

TexturedTextured

SmoothSmoothTextured Smooth

Download URL: http://www.ee.columbia.edu/dvmm/newDownloads.htm

Definition: Phase HistogramPhase histogram (normalized)

1 22

1 21 2 1 2 1 2

1( ) 1 [ ( , )] , ,...,

where 1 1 otherwise 0

2 2 , | , ; , 0,..., 1

(2 1) (2 1) | (2 1) (2 1)

i i

i

p b i N NM

truem m m m M

M Mi iN N

ω ω

π πω ω ω ω

π πφ φ

ΩΨ = Φ ∈Ψ =−

=

Ω= = = = −

− +Ψ = ≤ ≤+ +

∑

Strong phase concentration at ±90°

Symmetry Property: For real-valued signal, bicoherence phase histogram is symmetrical, i.e.,

( ) ( )i ip p −Ψ = Ψ

Bicoherence FeaturesDefinition: Phase feature

Definition: Magnitude feature

( ) log ( ) where ( ) is phase histogramP i i iif p p p= Ψ Ψ Ψ∑

1 21 22 ( , )

1 ( , )Mf bM ω ω

ω ω∈Ω

= ∑

Additional Results on BicoherenceFeatures [ISCAS’04]

0.50.55

0.60.65

0.70.75

0.80.85

0.9

Accuracy Mean Average Percision Average Recall

Plain BICPrediction ResiduePlain BIC + Prediction ResiduePlain BIC + EdgePrediction Residue + EdgePlain BIC + Prediction Residue + Edge

72%62%

OutlineReview

Problem and MotivationOur ApproachDefinition: BicoherenceWhy Bicoherence good for splicing? Quadratic Phase Coupling Hypothesis

Bicoherence FeaturesMagnitude featurePhase feature

Proposed Image Splicing ModelBipolar Perturbation HypothesisBicoherence of bipolar signalBipolar perturbation effect on phase featureBipolar perturbation effect on magnitude feature

Hypothesis II: Bipolar perturbation model

Original image signal is relatively smooth due to the low-pass anti-aliasing operation in camera or scanner.Spliced image signal can have arbitrary discontinuity

difference

Definition (Bipolar signal):

1 2 1 2

1 2

( ) ( ) ( ) ( ) exp( ) exp( ( ) )

where ( ) is a delta function, 0, 0

o o o oFouriertransform

d x k x x k x x D k jx k j x

k k

δ δ ω ω ω

δ

= − + − −∆ ⇔ = − + − +∆

⋅ < ∆>

( )d x

( )s x

( )

( ) ( )ps x

s x d x= +

( ) ( )ps x s x−k1

k2∆

xo

Bicoherence of Bipolar Signal

Results: Bicoherence phase of bipolar signal is concentrated at ±90°:

When there is phase coherency, bicoherence magnitude is close to unity

[ ]31 2 1 2 1 1 1 2( ) ( ) ( ) 2 sin( ) sin( ) sin( ( ))D D D k jω ω ω ω ω ω ω ω∗ + = ∆ + ∆ − ∆ +

Resulting in ±90° phase bias

Bipolar Perturbation Effect on Phase Feature

Bipolar perturbation( ) ( ) ( ) ( ) ( ) ( )p pFourier

Transform

s x s x d x S S Dω ω ω= + ⇔ = +

[ ]1 2 1 2 1 2 1 2

31 2 1 1 1 2

( ) ( ) ( ) ( ) ( ) ( )

( , , , ) 2 sin( ) sin( ) sin( ( ))p p PS S S S S S

C k k j

ω ω ω ω ω ω ω ω

ω ω ω ω ω ω

∗ ∗+ = + +

∆ + ∆ + ∆ − ∆ +

Numerator of the perturbed signal bicoherence:

Cross term involves both S(ω) and D(ω), hence we

assume that it has no consistent phase across all (ω1, ω2) frequency pair

Consistently contributing to the ±90° phase, for every (ω1, ω2) frequency pair.

The contribution depends on k, the magnitude of the bipolar

Empirical Support for Bipolar Perturbation Model

Spliced averaged phase histogram - Authentic averaged phase histogram

More Spliced image blocks have large phase feature value

Spliced average phase histogram hasSignificantly greater 90 deg phase bias

Effect of Bipolar Perturbation on Magnitude Feature

])()(

[])]()(

[)]()(

[[

)]]()(

[)]()(

[)]()(

[[

),(2

21*212

2

22

114

21*21

*

22

113

21

ωωωω

ωω

ωω

ωωωω

ωω

ωω

ωω

+++

+⋅+

+++

⋅+⋅+

=

Gk

SkEG

kS

Gk

SkE

Gk

SG

kS

Gk

SkE

b

( )( ) ( ) ( ) ( ) ( ) ( ) ( ( ))p pFourierTransform

Ss x s x d x S S D k Gkωω ω ω ω= + ⇔ = + = +

Markov Inequality:[ ( ) ]( )p( )

E SSk k

ωω εε

≥ ≤ For energy signal (finite power)

( )lim ( ) 0k

Spkω ε

→∞≥ =

2( )S ω <∞∑

Effect of Bipolar Perturbation on Magnitude Feature (cont.)

0]|)([|]|)]()([|

|)]()()([|),(lim

221

*221

21*

2121 =

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛≥

+

+−

∞→ε

ωωωω

ωωωωωω

DEDDE

DDDEbP

k

( )lim ( ) 0k

Spkω ε

→∞≥ =

Close to 1, due to phase coherency of bipolar signal

More Spliced image blocks have large magnitude feature value

ConclusionsWe propose a bipolar perturbation model for explaining the effectiveness of bicoherence in detecting image splicingThe prediction of the model matches empirical observations (90 deg phase bias)Columbia Dataset for Image Splicing Detectionhttp://www.ee.columbia.edu/dvmm/newDownloads.htmRecent related work in using image phase information for estimating perceptual image blur:

Local phase coherence and the perception of blurZ Wang and E P Simoncelli.Neural Information Processing Systems, December 2003 (NIPS 2003).

Thank You

Proof of Markov Inequality

[ ]1( ) ( ) ( ) ( )x x

x E xp x p x dx p x dx x p x dx

ε ε

εε ε ε

+∞

≥ ≥ −∞

≥ = ≤ ≤ =∫ ∫ ∫

Proof of Cauchy-Schwartz Inequality

2 2 22

2 2 2

2 , 0, where is a scalarNote, the above expression is a quadratic polynomial of Then:

4 , 4 0

, , ,

tf g t f t f g g tt

f g f g

f g f g f f g g

+ = + + ≥

− ≤

≤ =

Effect of Bipolar Perturbation on Magnitude Feature

Recall the correlation of bipolar signal:

[ ]31 2 1 2 1 1 1 2( ) ( ) ( ) 2 sin( ) sin( ) sin( ( ))D D D k jω ω ω ω ω ω ω ω∗ + = ∆ + ∆ − ∆ +

( ) exp( ) exp( ( ) ) exp( )(1 exp( ))o o oD k jx k j x k jx jω ω ω ω= − + − +∆ = − − − ∆

In ideal case, if bipolar signal at every segment in the averaging term is identical (having same k, xo and ∆) …..

Then, for every (ω1, ω2) frequency pair:2 2*

1 2 1 2 1 2 1 2

1 2 1 2

1 2

[ ( ) ( ) ( )] [ ( ) ( ) ] [ ( ) ]

( ) ( ) ( ) where c=constant( , ) 1

E D D D E D D E D

D D cDb

ω ω ω ω ω ω ω ω

ω ω ω ωω ω

+ = +

⇔ = +⇒ = Reach Cauchy-Schwartz

Inequality Upper BoundBicoherence

magnitude is 1!

Goal: Image Splicing Detection using Natural-imaging Quality (NIQ)

NIQ: Authentic images comes directly from camera and have low-pass property due to camera optical anti-aliasing low-passDeviations from NIQ: Image splicing introduces arbitrarily rough edges/discontinuities in image signalWe characterize such NIQ using bicoherence

Extraction of BIC Features from image

128128-points DFT(with zero padding and Hanning windowing)

⎟⎠⎞

⎜⎝⎛ +⎟⎠⎞

⎜⎝⎛

+=

∑∑

∑

k kk kk

k kkk

Xk

XXk

XXXkb

221

221

21*

21

21

)(1)()(1

)()()(1

),(ˆ

ωωωω

ωωωωωω

2 21 1( ) ( )h v

Horizontal VerticalMi MiN Ni i

M f f= +∑ ∑

2 21 1( ) ( )h v

Horizontal VerticalPi PiN Ni i

P f f= +∑ ∑

3

4

55

6

6

64Overlapping segments

21

Negative Phase Entropy (P)

( ) log ( )P n nnf p p= Ψ Ψ∑

21 2

11 2( , )

Magnitude mean, ( , )Mf bω ω

ω ω∈ΩΩ

= ∑

*

* To reduce noise effect, phase histogram is obtained from the BIC components with magnitude exceeding a threshold

Bipolar Perturbation Effect on Phase Feature (cont.)

Estimation:

The strength of the final ±90° degree phase bias also depends on

% segments in the averaging term having bipolar

⎟⎠

⎞⎜⎝

⎛ +⎟⎠

⎞⎜⎝

⎛

+=

∑∑∑

k kk kk

k kkk

Xk

XXk

XXXkb

221

221

21*

2121

)(1)()(1

)()()(1

),(ˆ

ωωωω

ωωωωωω