Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and...
-
Upload
gary-underwood -
Category
Documents
-
view
215 -
download
2
Transcript of Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and...
![Page 1: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/1.jpg)
Consistent Visual Information Processing
Axel Pinz
EMT – Institute of Electrical Measurement and Measurement Signal Processing
TU Graz – Graz University of Technology
[email protected]://www.emt.tu-graz.ac.at/~pinz
![Page 2: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/2.jpg)
“Consistency”• Active vision systems / 4D data streams
• Imprecision
• Ambiguity
• Contradiction
• Multiple visual information
![Page 3: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/3.jpg)
This Talk: Consistency in
• Active vision systems:– Active fusion– Active object recognition
• Immersive 3D HCI:– Augmented reality– Tracking in VR/AR
![Page 4: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/4.jpg)
AR as Testbed
Consistent perceptionin 4D:
• Space– Registration– Tracking
• Time– Lag-free– Prediction
![Page 5: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/5.jpg)
Agenda
• Active fusion
• Consistency
• Applications– Active object recognition– Tracking in VR/AR
• Conclusions
![Page 6: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/6.jpg)
Active Fusion
fusion, contro lin teraction
w orld
w orlddescription
sceneselection
scene
exposure
im age
im age proc.,segm entation
im age
description
grouping,3D m odeling
scenedescription
integration
Simple top level decision-action-fusion loop:
![Page 7: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/7.jpg)
Active Fusion (2)
• Fusion schemes– Probabilistic– Possibilistic (fuzzy)– Evidence theoretic (Dempster & Shafer)
![Page 8: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/8.jpg)
Probabilistic Active Fusion
N measurements, sensor inputs: mi
M hypotheses: oj , O = {o1, …, oM }
Bayes formula:
),...,(
)()|,...,(),...,|(
1
11
N
jjNNj
mmP
oPommPmmoP
Use entropy H(O) to measure the quality of P(O)
)(log)()(1
j
M
j
j oPoPOH
![Page 9: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/9.jpg)
Probabilistic Active Fusion (2)Flat distribution: P(oj )=const. Hmax
• Measurements can be:• difficult,• expensive,
• N can be prohibitively large, …
Find iterative strategy to minimize H(O)
Pronounced distribution:P(oc ) = 1; P(oj ) = 0, j c H = 0
)(log)()(1
j
M
j
j oPoPOH
![Page 10: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/10.jpg)
Probabilistic Active Fusion (3)
Start with A 1 measurements: P(oj|m1, … ,mA), HA
Iteratively take more measurements: mA+1, … ,mB
Until: P(oj|m1, … ,mB), HB Threshold
![Page 11: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/11.jpg)
Summary: Active Fusion
• Multiple (visual) information, many sensors, measurements,…
• Selection of information sources
• Maximize information content / quality
• Optimize effort (number / cost of measurements, …)
Information gain by entropy reduction
![Page 12: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/12.jpg)
Summary: Active Fusion (2)
• Active systems (robots, mobile cameras)– Sensor planning– Control– Interaction with the scene
• “Passive” systems (video, wearable,…)– Filtering– Selection of sensors / measurements
![Page 13: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/13.jpg)
Consistency
• Consistency vs. Ambiguity– Unimodal subsets Ok
• Representations– Distance measures
![Page 14: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/14.jpg)
Consistent Subsets
Hypotheses O = {o1 ,…, oM }
Ambiguity: P(O) is multimodal
Consistent unimodal subsets Ok O
• Application domains
• Support of hypotheses
• Outlier rejection
Benefits:
![Page 15: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/15.jpg)
Distance Measures
Depend on representations, e.g.:
• Pixel-level SSD, correlation, rank• Eigenspace Euclidean• 3D models Euclidean• Feature-based Mahalanobis, …• Symbolic Mutual information• Graphs Subgraph isomorphism
![Page 16: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/16.jpg)
Mutual Information
Shannon´s measure of mutual information:
O = {o1 ,…, oM }A O, B O
I(A,B) = H(A) + H(B) – H(A,B)
![Page 17: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/17.jpg)
Applications
• Active object recognition– Videos– Details
• Tracking in VR / AR– Landmark definition / acquisition– Real-time tracking
![Page 18: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/18.jpg)
Active vision laboratory
![Page 19: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/19.jpg)
Active Object Recognition
![Page 20: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/20.jpg)
Active Object Recognitionin Parametric Eigenspace
• Classifier for a single view
• Pose estimation per view
• Fusion formalism
• View planning formalism
• Estimation of object appearance at unexplored viewing positions
![Page 21: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/21.jpg)
![Page 22: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/22.jpg)
![Page 23: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/23.jpg)
![Page 24: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/24.jpg)
![Page 25: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/25.jpg)
![Page 26: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/26.jpg)
![Page 27: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/27.jpg)
![Page 28: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/28.jpg)
Applications
Active object recognition– Videos– Details
Control of active vision systems
• Tracking in VR / AR– Landmark definition / acquisition– Real-time tracking
Selection, combination, evaluation Constraining of huge spaces
![Page 29: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/29.jpg)
Landmark Definition / Acquisition
corners blobs natural landmarks
What is a “landmark” ?
![Page 30: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/30.jpg)
Automatic Landmark Acquisition
• Capture a dataset of the scene:– calibrated stereo rig
– trajectory (by magnetic tracking)– n stereo pairs
• Process this dataset– visually salient landmarks for tracking
![Page 31: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/31.jpg)
Automatic Landmark Acquisitionvisually salient landmarks for tracking
• salient points in 2D image• 3D reconstruction• clusters in 3D:
– compact, many points– consistent feature descriptions
• cluster centers landmarks
![Page 32: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/32.jpg)
Processing Scheme
![Page 33: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/33.jpg)
Office Scene
![Page 34: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/34.jpg)
Office Scene - Reconstruction
![Page 35: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/35.jpg)
Office Scene - Reconstruction
![Page 36: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/36.jpg)
Unknown Scene
Real-Time Tracking
LandmarkAcquisition
![Page 37: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/37.jpg)
Real-Time Tracking
• Measure position and orientation of object(s) • Obtain trajectories of object(s)
• Stationary observer – “outside-in” – Vision-based
• Moving observer, egomotion – “inside-out”– Hybrid
• Degrees of Freedom – DoF– 3 DoF (mobile robot)– 6 DoF (head and device tracking in AR)
![Page 38: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/38.jpg)
Outside-in Tracking (1)
stereo-rigIR-illumination
• wireless
• 1 marker/device:3 DoF
• 2 markers: 5 DoF• 3 markers: 6 DoF
devices
![Page 39: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/39.jpg)
2D B
lob
Tra
ckin
gE
pipo
lar
Geo
met
ry3D
Cor
resp
onde
nce
3D O bjects and Pose
2D Backpro jection
Epipolar G eom etry
C onstra in ts
3D C orrespondence
3D Prediction
B lob D etection
Tile Q uantisation
Prediction
B lob D etection
Tile Q uantisation
Prediction
W orkspace
O bject M odels
LEFT IM AG E R IG H T IM AG E
Outside-inTracking (2)
![Page 40: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/40.jpg)
Consistent Tracking (1)
• Complexity– Many targets– Exhaustive search vs. Real-time
• Occlusion– Redundancy (targets | cameras)
• Ambiguity in 3D– Constraints
![Page 41: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/41.jpg)
Consistent Tracking (2)
• Dynamic interpretation tree– Geometric / spatial consistency
• Local constraints– Multiple interpretations can happen– Global consistency is impossible
• Temporal consistency– Filtering, prediction
![Page 42: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/42.jpg)
Consistent Tracking (3)
![Page 43: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/43.jpg)
![Page 44: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/44.jpg)
![Page 45: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/45.jpg)
Hybrid Inside-Out Tracking (1)
• 3 accelerometers• 3 gyroscopes• signal processing• interface
Inertial Tracker
![Page 46: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/46.jpg)
Hybrid Inside-Out Tracking (2)
• complementary sensors
• fusion
![Page 47: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/47.jpg)
Summary: Consistency in
• Active vision systems:– Active fusion– Active object recognition
• Immersive 3D HCI:– Augmented reality– Tracking in VR/AR
![Page 48: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/48.jpg)
Conclusion
Consistent processing of visual informationcan significantly improve
the performance ofactive and real-time vision systems
![Page 49: Consistent Visual Information Processing Axel Pinz EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6b5503460f94b692df/html5/thumbnails/49.jpg)
Acknowledgement
Thomas Auer, Hermann Borotschnig, Markus Brandner, Harald Ganster, Peter Lang, Lucas Paletta, Manfred Prantl, Miguel Ribo, David Sinclair
Christian Doppler Gesellschaft, FFF, FWF, Kplus VRVis, EU TMR Virgo