Post on 31-May-2020
Attributing and Authenticating Evidence
Attribution Forensic source identification
Link multimedia content to the acquisition device
2
Authentication
Computer generated images? http://www.businessinsider.com/photorealistic
-3d-images-2013-2 The scientists found that 97% of test
subjects were fooled into believing that the digital renderings were real photographs and that real photos were CGI.
3
Attribution
4
Authentication Fake photo?
Tampering detection
5 6
Multimedia Forensics
Application of scientific methods to the investigation and prosecution of a crime Outcomes of a forensic analysis may
serve as probative facts in court Detect: source of multimedia data Detect forgeries
Copy-move forgery Hide undesired objects/replicate
similar objects Copying another region of the same image
Cell Phone Camera
Standalone Camera
Scanner
Computer Generated
7
Examples
A tampered image appeared in press in July 2008
4 Iranian missiles: 3 are real Red/purple: copy-move forgeries
2007: Fars News Agency, Tehran copy-move forgeries
Recapturing problem
10
Applications
https://www.youtube.com/watch?v=3bZvtWA7qGQ
11
Hash function An algorithm
Input: files (word document, pdf, image, …) Output: a pre-fixed length string
Purpose: ensure data integrity Property
Hashed result unique One way function
Good for authenticating word/pdfdocuments 12
Hash function Example:
http://www.fileformat.info/tool/hash.htm
13
d41d8cd98f00b204e9800998ecf8427e
7dbc9f235835a899880f3e9a7ae1f393
Hashing function To see if images are modified
Compare hash values Too strict for multimedia data
Images: transmitted through sharing platform Compression content / meaning: doesn’t change
Video: wireless loss
14
Example Images taken by smart phone send
through whatsapp compression
15 16
Feature-based Image Hashing
Feature: invariant under perceptually insignificant distortion corners? (Harris
corner detection) For each corner:
find average brightness feature 17
bitmap JPEG
rotate Local scale change
Illustration
Resizing Color Featureextraction
Hash GenerationImage hash
original whatsapp
Resize -> 8x8
Grayscale
originalwhatsapp
Color feature extraction
Average color (mean)93.42 92.39
If (intensity)> average 1Otherwise 0
Feature-based Image Hashing
Features: color features For RGB and YCbCr color spaces: 6 color
components For each color component, calculate the
statistical information Mean, variance, Moment values http://www.naturalspublishing.com/files/published/
54515x71g3omq1.pdf Concatenate moment values of the six color
components to form a feature vector
21
Feature-based Image Hashing
Form signature based on the extracted features concatenate all features together Different methods to form the hash
Represent them by using certain number of bits Take Fourier Transform, consider the magnitude and the
phase as features and represented using certain number of bits
http://www.brainflux.org/java/classes/FFT2DApplet.html
22
Feature-based Image Hashing
23
Feature: invariant under perceptually insignificant distortion
Evaluation
Hash length Robustness towards different
changes Brightness adjustment Contrast adjustment JPEG compression Addition of noise Lowpass filtering …
24
Evaluation
Large degree of compression? Share through social media
Miss detection?25 26
Active approach for data authentication
27
Digital watermark
10011010 …
© Copyright …
Active approach for data authentication
Active approach Digital watermark
Epson PhotoPC 3000Z, 700/750Z, 800/800Z (discontinued) Watermark is invisible Requires optional software to embed and
view watermark Kodak DC-200, 260, 290
(discontinued) Watermark is invisible Watermark capabilities built into
camera
Active approach for data authentication
Active approach Kodak DC-200, 260, 290
(discontinued) Watermark is invisible Watermark capabilities built into
camera
Active approach for data authentication
Active vs Passive Active approach
Addition of extra data More powerful, end-to-end protocol Not popular
Passive approach: Detect intrinsic image regularity or tampering
artifacts Wider application, less powerful
31
Forgery detection techniques http://www.izitru.com/ Three levels of assumption
Rules and models of the physics of the scene Inconsistency a basis for forgery detection Size inconsistency, lighting directions, shadow
inconsistencies, reflection inconsistencies Inherent characteristics of the acquisition system
(camera components, imaging pipeline) Statistics of natural images
32
Demonstration Web platform
https://29a.ch/photo-forensics/ Python:
http://www.sourcecodeonline.com/details/copy-move_forgery_detection_in_images.html
Purchase: http://belkasoft.com/forgery-detection
33 34
Example: JPEG Forensics Quantization tables:
Transform to frequency domain (Discrete cosine transform), divide each F[u,v] by a constant q[u,v]
Eye: more sensitive to low frequencies Most software use standard quantization Some software (Photoshop)
Have their own quantization table Camera manufacturers have their own table Clue for manipulation 35
Example: JPEG Forensics Quantization tables:
www.dfrws.org/sites/default/files/session-files/paper-using_jpeg_quantization_tables_to_identify_imagery_processed_by_software.pdf
36
Examples : Quantization
37
Photo1_SamsungA7
Examples: Quantization
38
Photo1_SamsungA7 Standard JPEG table, quality=96
Examples: Quantization
39
Photo1_SamsungA7_modified (software) Standard JPEG table, quality=90
Examples: Quantization
40
Photo1_SamsungNote Non Standard JPEG table, quality=97
Examples: Quantization
41
Photo1_Nikkon Standard JPEG table, quality=80
Example: Clone detection Samsung A7 photos: combine
42
Example: Clone detection Clone: Copied regions in an image Similarity: the similarity between the
copied regions and the original Minimal detail:
Blocks with less detail are not considered in searching for copied regions
Cluster size: how many copied regions need to be found in order for them to show up as results
43
Example: Clone detection Increase “Minimum similarity”
44
http://www.imageforensic.org/
45
http://www.imageforensic.org/
46
Forgery detection techniques General two classes of techniques
Non-source identification related Lighting direction, shape of the light source specific tampering anomalies
Source identification related Features:
sensor noise pattern, dust patterns, demosaicingregularity, statistical regularities, chromatic aberration
47
Non-source identification methods
Tampering characteristics Different tampering methods different
characteristics Copy-move forgery
Highly correlated regions
Splicing Sharp discontinuity boundary
Double JPEG compression Periodicity in DCT coefficient histogram Uneven JPEG blocking artifacts 48
Copy-move ForgeryCopying regions of the original image and pasting into other areas.The yellow area has been copied and moved to conceal the truck.2 types of techniques
Block-basedKeypoint-based 49 50
Detection of Copy-move ForgeryBlock-based
Feature extraction
Find Similar blocks
N
B
B
N
B
B
Generate
(N-B+1)(N-B+1) Blocks
52
155 155 155 158 158 156 158 159
155 155 155 158 158 156 158 159
155 155 155 158 158 156 158 159
155 155 155 158 158 156 158 159
155 155 155 158 158 156 158 159
151 151 151 154 157 156 156 156
155 155 155 156 157 158 156 153
149 149 149 153 155 154 153 154
Original image
155 155 155 158
155 155 155 158
155 155 155 158
155 155 155 158
155 155 158 158
155 155 158 158
155 155 158 158
155 155 158 158
…158 156 158 159
157 156 156 156
157 158 156 153
155 154 153 154
Block size : 4 × 4
Detection of Copy-move Forgery: Features: DCT
Detection of Copy-move Forgery: DCT
Discrete cosine transform From spatial domain to frequency domain
53
155 155 155 158
155 155 155 158
155 155 155 158
155 155 155 158
DCT Transform
Original blockDCT coefficient block
420.8 37.7 -3.3 4.2
-3.0 0.9 2.2 -0.3
-0.3 -5.4 0.8 -0.7
2.6 0.7 -0.6 0.6
Features: coefficients or histogram
Detection of Copy-move Forgery
Block-based
Feature extraction
Find Similar blocks
[05, 0.6, …]
[08, 0.7, …]Similar condition : 4
2
1_ ( , ) ( )k k
i i j i i j similark
m match A A v v D
2 2( , )i i j i i j i i j dd V V x x y y N
Results
55 56
Detection of Copy-move Forgery: Block-based
High computational complexity Lots of blocks compute features, find
matching blocks Geometric manipulation
Scaling, rotation
Detection of Copy-move Forgery: keypoints
Keypoint-based
Descriptors for each keypoint
Associate similar keypoints
[05, 0.6, …]
[08, 0.7, …]
Review of SIFT-based approach Steps:
Scale-space extreme detection Search over multiple scales DoG: difference of Gaussian
Gaussian filtering
Downsampling & Gaussian filtering
Downsampling & Gaussian filtering
Difference
Difference
Review of SIFT-based approach
Steps: Scale-space extreme detection Keypoint localization
Local extrema in the DoG pyramid Cleaning: remove low contrast points
Orientation assignment Compute best orientation for each
keypoint Achieve rotation invariance
Review of SIFT-based approach
Steps: Scale-space extreme detection Keypoint localization Orientation assignment
Find orientation of intensity gradients
36 bins (10 degrees) histogram Keypoint orientation = histogram peak
, , , , ,L x y G x y I x y
1 , 1 ( , 1), tan
( 1, ) ( 1, )L x y L x y
x yL x y L x y
Review of SIFT-based approach
Steps: keypoint descriptors
16x16 image patch descriptors Center: keypoint, origin axis: orientation
Form 4x4 sub-patches Sub-patch: histogram (8 bin) of
gradient orientation Local image gradients: 4x4x8 = 128
values
Copy-move Forgery Detection Keypoint matching
1 2
,1 ,2 ,
, ,,
, ,
j j
j
j m
j j j n
x x x xx
x x
d d d
F F F FD X
F F
Small distance: similarity of keypoints
similar objects matching
Forgery detection techniques General two classes of techniques
Non-source identification related Lighting direction, specific tampering anomalies
Source identification related Legal system:
Accepts the forensic analysis of digital image evidence if the attribution techniques are unbiased, reliable, non-destructive and widely accepted by experts in the field
Features: Hardware defects (lens distortion) Sensor defects (sensor noise pattern, dust patterns) Processing regularities (CFA, JPEG) 65
Forensic work flow
66
Image authentication Two-step process
Exam for the reliability of the evidence (image tampering and forgery detection)
Analysis to determine its probative value regarding to source camera and image metadata
67
Example Prosecuting attorneys claim:
Series of images discovered on a suspect’s computer are potentially an evidence of a crime
Possible that a third party has access to the suspect’s computer, but no evidence of such access
Desirable if forensic evidence examiner provides info about: The consistency of these images with a specific
digital camera discovered in the suspect’s house68
Digital Image Generation
69
Example: Image Acquisition
70
Example: Image Acquisition Lens: focus the light of scene on sensor Filters: filter out invisible part of light (infra-red,
ultraviolet) CFA: color filter array (on top of the sensor)
Common: only one sensor for detecting all three colors (red, green blue)
71
CFA / Demosaicing
72
Example: Image Acquisition Sensor: CCD/CMOS
Photosensitive pixels capture photons and convert them into charge
CFA interpolation To generate image with full resolution for all
colors At each sensor pixel, only one color is
measured The other two colors have to be estimated from
neighboring pixels 73
Example: Image Acquisition Post processing:
Apply enhancement technique to eliminate unwanted artifacts, degradations or noise
Color-artifact removal (introduced during CFA interpolation), edge enhancement
Storing EXIF JPEG format
74
Source-based forgery detection or model identification Discover traces left by hardware
component or software process during image generation process Image artifacts: 2 types Hardware-related
Caused by lens, sensor imperfections (noise) software-related
Introduced through camera processing
75
Image artifacts
76
hardware
software
Opticalaberrations
Sensor
Processing statistics
Lens radial distortionChromatic aberration
Processing regularities
Sensor noiseSensor dust pattern
Model statisticsHigh order statistics
CFA arrayJPEG compression
Hardware: Optical defects Optical aberrations
Radial lens distortion Straight lines appear curved in an image
Serious in low-cost wide-angle lenses The degree of distortion changes with focal length
77
Hardware: Optical defects Order-2 model
(xD,yD): distorted image coordinate (x,y): undistorted coordinate (a,b): optical centre r = sqrt((x-a)^2 + (y-b)^2)
Find distorted lines to estimate k1 and k278
2 41 2
2 41 2
( )(1 )
( )(1 )
x xD a kr k r ay yD b kr k r b
Hardware: Optical defects More likely to be used for forgery
detection Less likely to be used for source camera
attribution Built with the same/similar lenses similar
characteristics Scene content dependency: difficult to
estimate distortion in images with flat scene content
79
Hardware: Optical defects Less likely to be used for source camera
attribution Camera setting dependency
Change with focal length, focal distance, aperture size, illumination, etc
Images captured with one device but different zooming different distortions
80
Hardware: sensor defects Sensor imperfections:
Sensor defects, sensor pattern noise, sensor dust
Sensor defects / pixel defects Dead pixels:
not responding to light, appear as a black spot Rarely exist in new manufactured camera or be
removed during post-processing
81
Hardware: sensor defects Sensor pattern noise
Most sever type of sensor artifacts Photo-response non-uniformity: generated
based upon the sensitivity of pixels Sensitivity: measured by determining the
light intensity Effect of inhomogeneity of silicon wafer and the
imperfection of the sensor manufacturing process
82
Hardware: sensor defects
83
Output image Original
image
PRNU
= + +
Noisy Output Noise free input PRNU Noise Other Noise
84
Photo response non-uniformity noise (PRNU)
This pattern noise will survive for every image that taken by the same camera.
Unique for each individual
deviceDevice 1 Device 2
Hardware: sensor defects PRNU:
Can be used to identify individual device used for taking the image
Is able to distinguish cameras from same model and brand
Has been used to solve court cases when the query image was tested to verify the claimed camera device Device linking
85
Hardware: sensor defects Dust pattern on lens
Cameras with interchangeable lens Dust particles remain in front of the imaging
sensor Produce a constant pattern in all captured
images Results:
High classification accuracy Problem: user cleaned the lens?
Positive result is conclusive, but negative result is inconclusive 86
Software: processing statistics
Identify statistical artifacts left by different cameras https://www.dpreview.com/reviews/studioco
mpare.asp Color characteristics
Color reproduction of the camera with respect to each color band
image quality Measure quality of the scene reproduction by the
optical system 87 88
Software: processing statistics
Example statistical features Average pixel value per RGB and RGB pairs correlation Pixel difference Use filters to decompose RGB band to three
sub-bands determine mean, variance etc Discrete cosine transform, wavelet transform,
ridgelet, contourlet, …
89
Software: processing statistics
Challenges Difficult to achieve large inter-model similarity
for devices of the same brand sharing similar hardware and processing components
Camera setting dependency: focal length, indoor/outdoor illumination/flash
Scene content dependency: Images captured by 2 cameras in different environments
90
Software: processing regularities
Examine processing artifacts CFA configuration: specific arrangement of
color filters across the sensor plane
91 92
93
Software: processing regularities
Examine processing artifacts CFA interpolation algorithms
Used to estimate missing color from surrounding samples of the raw pixel
Use different size for interpolation (number of surrounding samples)
Adopt different methods to estimate the missing color Simple averaging, weighted averaging, image content
dependent averaging
94
Software: processing regularities
Bilinear interpolation
95
GA
GB
GL GR
)(41
ABRLI GGGGG
GI
Software: processing regularities
Bilinear interpolation
96
97original interpolated
98
artifacts Appear at edges / regions with high freq
99
Features: Study the relationship among neighboring pixels
100
Image artifacts + machine learning
101
hardware
software
Opticalaberrations
Sensor
Processing statistics
Lens radial distortionChromatic aberration
Processing regularities
Sensor noiseSensor dust pattern
Model statisticsHigh order statistics
CFA arrayJPEG compression
Machine learning approach Used to analyze large amounts of data Black Box Approach:
Collect all features from a large number of multimedia data
Use the machine learning approach for grouping / classifying these features
102
103
Machine learning approach 2 types
Supervised Make predictions based on a given
set of features Unsupervised
Learn the data and organize the data by the algorithm
104
Machine learning approach Examples:
Support vector machine clustering algorithm artificial neural networks nearest neighbors Deep learning algorithm
105
Example: Tampering detection using demosaicing regularity
106
Tools for source camera attribution
Amped software: authenticate https://ampedsoftware.com/authenticate Qualified government/law enforcement
agencies Software package for forensic image
authentication and tamper detection on digital photos
107
Error level analysis Multiple JPEG compression
PRNU identification Create PRNU
PRNU tampering Find inconsistencies in PRNU noise
Clone Blocks 108
109Multiple compression 110
Inconsistence of sensor noise
111Multiple compression
Applications Insurance companies
Use forensics to cut fraud and abuse (save time)
Car crash: minor dents and scratches Upload a picture/video to the insurance company
to save time Findings: use photo editing software to create
fake photo evidence
112
Applications: insurance
113
Forensic Image analyser http://www.forensic-
pathways.com/forensic-image-analyser/
Identifies if the image was taken by a suspected device
identifies which images in a set were taken by the same device and which were taken by other devices 114
Read about the real court case in the web site
Other tools: Fourandsix Technologies http://www.fourandsix.com/ 115
Other approaches Photos: mostly come with EXIF header
Consistency between information (ISO Speed rating, exposure time, focal length) with the image content? Estimate camera setting from the image content
and compare with the data found in the EXIF header
116
Consistency checking
117