2D to 3D conversion of formula 1 footage
-
Upload
serge13 -
Category
Technology
-
view
390 -
download
0
Transcript of 2D to 3D conversion of formula 1 footage
![Page 1: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/1.jpg)
2D to 3D conversion of Formula 1 footage
Serge Hendrickx
1
![Page 2: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/2.jpg)
Presentation Overview
• Thesis objective
• Conversion process
• Conclusion and result 2
• 3D rotation
• Car detection and tracking
• Inpainting
• Overlay detection and conversion of background
![Page 3: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/3.jpg)
Objective: 2D to 3D conversion of Formula 1 footage
2D recorded and broadcasted images 3D experience
3
![Page 4: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/4.jpg)
Objective: 2D to 3D conversion of Formula 1 footage
4
What is special about Formula 1?
Undeformable objects with near perfect 3D models -> Investigate how these could be used in this conversion process -> Not just generating best-looking 3D
![Page 5: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/5.jpg)
Introduction to Formula1 footage Example footage
5
![Page 6: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/6.jpg)
Presentation Overview
• Thesis objective
• Conversion process
• Conclusion and result 6
• 3D rotation
• Car detection and tracking
• Inpainting
• Overlay detection and conversion of background
![Page 7: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/7.jpg)
3D effect
• Translation + rotation
7
Retinal Disparity
![Page 8: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/8.jpg)
3D effect
• Translation + rotation
8
Retinal Disparity
1) Nearest-neighbor
2) 3D Projection
![Page 9: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/9.jpg)
3D effect
• Translation + rotation
9
Retinal Disparity
1) Nearest-neighbor
2) 3D Projection
![Page 10: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/10.jpg)
Nearest neighbor rotation
10
For each pixel: 1) Find corresponding RGB 2) Detect nearest match in
rotated view 3) Copy pixelvalue
XYZ represented by RGB during rendering
![Page 11: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/11.jpg)
11
Nearest neighbor rotation
![Page 12: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/12.jpg)
3D effect
• Translation + rotation
12
Retinal Disparity
1) Nearest-neighbor
2) 3D Projection
![Page 13: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/13.jpg)
3D projection
13
Traverse rendering pipeline
(X,Y) –coordinate in new view
(X,Y,Z) coordinate
(X,Y) –coordinate in known view
Using ModelView and Projection matrices of new view
Using ModelView and Projection matrices of known view
![Page 14: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/14.jpg)
3D projection
14
Traverse rendering pipeline
Source:
Generated:
![Page 15: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/15.jpg)
3D rotation
• 3D projection : + Fast + Produces clear image - Holes
15
• Neighborhood: - Very slow - Artifacts + Fills in holes
![Page 16: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/16.jpg)
3D rotation
• 3D projection : + Fast + Produces clear image - Holes
16
• Neighborhood: - Very slow - Artifacts + Fills in holes
-> Combine both methods
![Page 17: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/17.jpg)
17
Source:
3D rotation: final result
Rotated 10 degrees:
![Page 18: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/18.jpg)
18
• Translation + rotation
Retinal Disparity
3D effect
![Page 19: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/19.jpg)
19
• Translation + rotation
Retinal Disparity
3D effect
-> Exact position and pose of car needed
![Page 20: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/20.jpg)
Presentation Overview
• Thesis objective
• Conversion process
• Conclusion and result 20
• 3D rotation
• Car detection and tracking
• Inpainting
• Overlay detection and conversion of background
![Page 21: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/21.jpg)
Car detection and tracking
• Car detection
• Tracking throughout fragment
21
![Page 22: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/22.jpg)
Car detection and tracking
• Car detection
• Tracking throughout fragment
22
![Page 23: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/23.jpg)
Car detection
23
Sliding window -> binary classifier: contains car yes or no?
![Page 24: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/24.jpg)
Car detection
24
Sliding window -> binary classifier: contains car yes or no?
![Page 25: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/25.jpg)
Car detection
25
Sliding window -> binary classifier: contains car yes or no?
![Page 26: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/26.jpg)
Car detection
26
Sliding window -> binary classifier: contains car yes or no?
![Page 27: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/27.jpg)
Render car from all different angles
27
Car detection
![Page 28: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/28.jpg)
Car detection
28
Sliding window -> binary classifier: contains car yes or no?
-> Does it match one of the renders?
![Page 29: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/29.jpg)
Car detection
29
How to compare? Pixel per pixel -> Not resistant to illumination changes and other differences Solution: Histogram of Oriented Gradients (HoG)
![Page 30: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/30.jpg)
HoG based object detection
30
Histogram of Oriented Gradients (HoG)
Gradient computation
Gradient binning
![Page 31: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/31.jpg)
HoG based object detection
31
Parts-based detection
Star-based representation:
Real example:
![Page 32: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/32.jpg)
HoG based object detection
32
![Page 33: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/33.jpg)
Analysis of detection method on F1 footage
33
Analysis of distibution of certainty-scores of 1 particular render on 1 frame
Example: best certainty-score is 10 -> only 4% of matches score > 9 -> ony 14% of matches score > 5
![Page 34: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/34.jpg)
Analysis of detection method on F1 footage
34
Score deduction in neighboring renders
![Page 35: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/35.jpg)
Car detection and tracking
• Car detection
• Tracking throughout fragment
35
![Page 36: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/36.jpg)
Car tracking
36
36,000 renders (360*100 angles) * 500 frames = 18,000,000 detections -> coarse selection of renders and frames for first pass.
Tier 1
![Page 37: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/37.jpg)
Car tracking
37
36,000 renders (360*100 angles) * 500 frames = 18,000,000 detections -> coarse selection of renders and frames for first pass.
Tier 1
![Page 38: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/38.jpg)
Car tracking
38
Tier 2 Starting from best detection: 1) Rerun detection on best frame 2) Detect surrounding frames
Surrounding frames: - Car is at nearly same position - Car viewed under nearly same angle
![Page 39: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/39.jpg)
Car tracking
39
Tier 3: smoothing Smooth angles Smooth boudingbox location and size
Horizontal angle
Vertical angle
![Page 40: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/40.jpg)
Car tracking
40
Tier 3: smoothing Smoothed angles can still be used Exact size and position needed
Horizontal angle
Vertical angle
![Page 41: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/41.jpg)
Car tracking
41
Tier 3: smoothing Keypoint tracking
Track points using feature-tracking algorithm Calculate new positions using rendering pipeline
![Page 42: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/42.jpg)
Car tracking
42
Tier 3: smoothing Keypoint tracking
![Page 43: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/43.jpg)
Car tracking
43
Tier 3: smoothing Car rotation and template matching
1) Best match
2) Rotate to desired angle 3) Find rotated car in cropped image
![Page 44: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/44.jpg)
Car tracking
44
Tier 3: smoothing Car rotation and template matching
![Page 45: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/45.jpg)
Car tracking
45
Tier 3: smoothing Car rotation and template matching
1) Detect car 2) Rotate car to desired angle 3) Average multiple rotated cars
![Page 46: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/46.jpg)
Car tracking
46
Tier 3: smoothing
![Page 47: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/47.jpg)
Presentation Overview
• Thesis objective
• Conversion process
• Conclusion and result 47
• 3D rotation
• Car detection and tracking
• Inpainting
• Overlay detection and conversion of background
![Page 48: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/48.jpg)
48
• Translation + rotation
Retinal Disparity
3D effect
![Page 49: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/49.jpg)
Inpainting
49
Single frame methods:
Geometrical inpainting: Patch-based inpainting:
![Page 50: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/50.jpg)
Inpainting
50
Single frame methods:
Geometrical inpainting: Patch-based inpainting:
![Page 51: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/51.jpg)
Inpainting
51
Single frame methods:
Objective comparison Only outermost mixels important
![Page 52: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/52.jpg)
Inpainting
52
Video inpainting
![Page 53: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/53.jpg)
Inpainting
53
Video inpainting
![Page 54: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/54.jpg)
Inpainting
54
Video inpainting
![Page 55: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/55.jpg)
Inpainting
55
Video inpainting
![Page 56: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/56.jpg)
Inpainting
56
Video inpainting
![Page 57: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/57.jpg)
Presentation Overview
• Thesis objective
• Conversion process
• Conclusion and result 57
• 3D rotation
• Car detection and tracking
• Inpainting
• Non-object based 3D conversion
![Page 58: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/58.jpg)
Non-object based 3D conversion
• Overlay detection
• Conversion of background to 3D
58
![Page 59: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/59.jpg)
Non-object based 3D conversion
• Overlay detection
• Conversion of background to 3D
59
![Page 60: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/60.jpg)
Overlay detection
60
Graphical overlay: - Not always present - If present, always at fixed location
Template: averaged over 100 frames
![Page 61: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/61.jpg)
Non-object based 3D conversion
• Overlay detection
• Conversion of background to 3D
61
![Page 62: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/62.jpg)
Conversion of background to 3D
62
Make3D algorithm - Texture gradients - Texture variations - Reduced contrast
![Page 63: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/63.jpg)
Presentation Overview
• Thesis objective
• Conversion process
• Conclusion and result 63
• 3D rotation
• Car detection and tracking
• Inpainting
• Non-object based 3D conversion
![Page 64: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/64.jpg)
Overview of 3D conversion process
64
1) Inpaint car and overlay 2) Generate depthmap 3) Shift background 4) Inpaint background 5) Add overlay 6) Add car 7) Merge to 3D
![Page 65: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/65.jpg)
Presentation Overview
• Thesis objective
• Conversion process
• Conclusion and result 65
• 3D rotation
• Car detection and tracking
• Inpainting
• Non-object based 3D conversion
![Page 66: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/66.jpg)
Result and conclusion
66
Is it a feasible method of 2D-3D conversion?
Yes, but 1) very time-consuming -> also the reason why most of this thesis featured 1 single fragment 2) differences between chosen render-viewpoint and car can become noticeable -> can be solved/prevented by using realistically rendered car instead of the actual car from the frame Car angle, size and location known at each frame -> could be used for other purposes, for example advanced graphical overlays
Result: video on 3D tv
![Page 67: 2D to 3D conversion of formula 1 footage](https://reader034.fdocuments.in/reader034/viewer/2022042908/58ec5c151a28ab52278b45b9/html5/thumbnails/67.jpg)
Questions?
67