Dealing with the Complexities of Camera ISP Tuning...Dealing with the Complexities of Camera ISP...
Transcript of Dealing with the Complexities of Camera ISP Tuning...Dealing with the Complexities of Camera ISP...
Dealing with the Complexities of Camera ISP Tuning
Clément Viard, Sr Director, R&DFrédéric Guichard, CTO, [email protected]
1© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Dealing with the Complexities of Camera ISP Tuning
2
> Basic camera image processing
> First revolution: optical and sensor defects corrections
> Second revolution: miniaturization
> ISP, A key differentiator in image and video processing
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
3
Introduction
« Basic » digital image processing in a camera
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
RAW data from Sensor
4© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After « demosaicing »
5© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After exposure adaptation
6© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After « white ballance »
7© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After color rendering
8© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After sharpening
9© DxO Labs 2016 | PREPARED FOR: AutoSens conference
10
A digital revolution in cameras
Optical and sensor defect corrections
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Geometric distortion
11© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Geometric distortion
12© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Volume anamorphos – perspective error with wide angle lenses
13© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Volume anamorphos – perspective error with wide angle lenses
14© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Optical vignetting
15© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Chromatic aberation
16© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Chromatic aberation
17© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Tone mapping
18© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Tone mapping
19© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Tone mapping (with HDR sensor)
20© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Tone mapping (with HDR sensor)
21© DxO Labs 2016 | PREPARED FOR: RENAULT
Clipping – Recovering saturated area
22© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Clipping – Recovering saturated area
23© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Roll
Yaw
Pitch
X
Y
Z
Stabilisation and Electronic Rolling Shutter (ERS) effect
Translation Z
Yaw (Rotation)
Translation X
Pitch (Rotation)
Translation Y
ERS horizontal
ERS vertical
24© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Range of Digital Corrections with Advanced ISPs
25
> Optical aberrations> Distortion> Blur (spherical aberration, field
curvature, coma, astigmatism, motion)> Chromatic aberrations> Vignetting> Flare, veiling glare
> Light & Sensor> Noise> Dynamic range> Contrast> Atmospheric haze
> Sensor limitations> Field non uniformity (color, black
level,…)> Defective pixels, Dust> Clipping> Metamerism
> Others> Video stabilization> Electronic rolling shutter
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
26
Miniaturization – the second camera revolution
Will they eventually reach same performance?
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Sensor Miniaturization Challenge
27
22 times less light 4.5 stops
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Performance improvement over 10 years
28
> CMOS Sensor improvement : +1.5 stops
> Digital processing gain: +3 to +4 stops
DxOMark sensor score, APS-C cameras
+1.5 stops
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Nikon D70s, ISO 3200 – jpg
29© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v3
30© DxO Labs 2016 | PREPARED FOR: AutoSens conference
31
Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v7
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
32
Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v9
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
33
Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v11
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
34
Nikon D70s, ISO 3200 – jpg
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Sensor color response changes with CRA
Photosite
Color response as a function of the angles350 450 550 650 750
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
36
ISP
A key differentiator in image and video processing
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
« ISP » as a key differentiator
37
> Enables camera hardware and scene artefact corrections
> Only remaining differentiator since access to best sensor and lens is now ubiquitous
> Paradigm shift, cameras are designed considering possible digital corrections
> Computing power requirement consistently increasesDelivering 2,000 Ops/pixel with 240 Mpix/s (24 frames at 10Mpix per second) requires 500 GOps/s
> Very significant investment within the Mobile industry(e.g. iPhone camera engineering team ~ 800 people)
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Same HW, same ISP, same BOM same image quality?
DxO Mark Mobile score (Photos and Videos)38© DxO Labs 2016 | PREPARED FOR: AutoSens conference
« Camera Tuning »
39
For correcting each image, ISP requires 10,000+ register settings to be adapted to the situation.
> Per serie Calibration
> Per unit Calibration
> On the fly parameter estimations
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Image Quality Evaluation Challenge
> What is “Image Quality”?> Perception of how a picture “looks good”> In essence a subjective matter…that can be modeled with engineering tools
> What influences “Image Quality”> Shooting conditions: illumination, illuminant, dynamic> Content has a strong influence on Image Quality perception and image processing> Image Quality perception is different between imaging expert, professional
photographers and consumer
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Image Quality Evaluation Challenge - Subjective vs. objective
> Objective evaluation> Objective means a measurement that is neutral, operator independent: “2
cm, 15 meters, 150 kg, 2 g, -5°C, 50°C, …”> A device must provide figures (metrics) that are related to image quality> Normalizations may be necessary to have comparable metrics (cameras
with different resolutions)> Only addresses a set of metrics (some artifacts may be ignored)
> Subjective evaluation> “Subjective” means real people give an opinion like: “big, small, heavy, light,
cold, hot, …”> People can judge the quality of photographs> Methodology is key to get non-biased results
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Camera IQ assessments
42
Temporal behaviorDynamic behavior Photogrammetry and 3D
Thousands of videos/photos are required to characterize IQ along differentdimensions
Specifying AND verifying image quality targets are tough challenges
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Just Noticeable Difference (JND), a Matter of Statistics
> The smallest statistically measurable difference of perception, e.g., smallest perceivable distance between 2 parallel lines
> Typically, defined when half of the people perceive a difference and the other half are guessing (50% JND)
50% perceive
a change 50% guessing
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Anchored Scaling Concept
C D E F
Test image
??
Anchors
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Objective/subjective correlation example: Sharpness JND mapping
After extensive visual experiments, one could show that the Acutance objective metric linearly correlates with perception of sharpness (expressed with JND unit)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
5
10
15
20
25
30
35
Acutance
JN
D
0
20
40
60
80
100
120
0.0 0.1 0.2 0.3 0.4 0.5
MTF
(%
)
Frequency (cycle/pixel)
CSF
MTF
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Back to the real world!
> Unfortunately the assumptions to build a complete multi-variate analysis based scoring system are not met > Only few metrics are linearly correlated to perception> They are not strictly orthogonal to each others> New issues comes with every technological improvement
> Since we can’t build a unified scoring system based on a set of perceptual metrics, a Multi-Modal approach is required
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Recommendation for building a relevant Image or video score
47
> Multi-modal testing approach> Objective measurements in lab environment> Perceptually-correlated metrics when available> Perceptual analysis from natural scenes
> Multi-level testing results> Overall, Photo, and Video scores> Top-level scores (open scale) non technical> Detailed reports for engineering
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Example of natural scene set used for mobile camera application
48© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Conclusion
49
> Eventually, all industries would benefit from Smartphone technology breakthroughs
> ISP can fix many optical and sensor defects
> ISP technology improvements are a key enabler for miniaturization
> Mastering tuning complexity and IQ evaluation is a key differentiator
> DxO has developed robust methods to deal with this problem
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
1 / DxO USA 1407 Minnesota Street San Francisco, CA [email protected]: +1 415 805 8567
2 / DxO France3, rue Nationale 92100 Boulogne-Billancourt France [email protected] Tel: +33 1 55 20 55 99
3 / DxO Korea16th Floor, Gangnam Building 1321-1 Seocho-dong, Seocho gu Seoul [email protected]: +82 2 2190 3825
4 / DxO TaiwanTai Centre 2/F-A170 Dunhua North Road Taipei 10548Taiwan Tel: +886 2 5574 0429
5 / DxO Japan3F Metlife Kabutocho Bldg 5-1 Kabutocho Chuo-ku Nihonbashi, [email protected]: +81 (0)3 6861 7565
Thank you!
50
© DxO Labs 2016 | PREPARED FOR: AutoSens conference