Introduction to Security Ms. Tommie Abrams Security Branch PERS-534C Wood Hall, Bldg 769, Room 184.
3D Light-field Technology in Combination with Deep-Learning for...
Transcript of 3D Light-field Technology in Combination with Deep-Learning for...
![Page 1: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/1.jpg)
3D Light-field Technology in Combination with Deep-Learning for Industry
Philipp Schmid – Head Robotics & Automation
![Page 2: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/2.jpg)
Copyright 2019 CSEM | Page 1
Our mission
Development and transfer of microtechnologies to the industrial sector – in Switzerland, as a priority – in order to
reinforce its competitive advantage.
Cooperation agreements with established companies
Encouraging the creation of start-ups
![Page 3: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/3.jpg)
Copyright 2018 CSEM | Big Data – Smart Data – Fachgruppe Smart Systems; 12. Oktober 2018 | Alexander Steinecker| Page 2
CSEM – A TECHNOLOGY ORGANIZATION IN SWITZERLAND
The CSEM is a Swiss RTO with over 450 emps. Thecompany develops advanced microtechnologiesfor the industry. Thus it creates innovations.Focus areas are Indusry 4.0, e-health, advancedmanufacturing, clean energy.CSEM’s head quarters are located in Neuchâtel,Further branches are in Muttenz, Landquart,Alpnach and Zürich. CSEM is connected toInternational RTO’s such as FhG, VTT, CEA.In the past 10 years CSEM created over 40 Startup companies active in numerous markets.
![Page 4: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/4.jpg)
Copyright 2018 CSEM | Big Data – Smart Data – Fachgruppe Smart Systems; 12. Oktober 2018 | Alexander Steinecker| Page 3
CSEM at a glance
437Persons
43New ventures
175Industrialclients
203Patent families
64Europeanprojects
43Nationalities
83.0Turnover(mio CHF)
![Page 5: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/5.jpg)
Copyright 2019 CSEM | Page 4
CSEM – Swiss Research for Industry
N N A L M Z
N A L
MZ
N
CSEM Zürich
CSEM Muttenz
CSEM Neuchâtel
CSEM Alpnach
CSEM Landquart
![Page 6: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/6.jpg)
Copyright 2019 CSEM | Page 5
Technology platforms to foster innovation
Advanced micro-manufacturing
Printableelectronics
Bio-surface engineering
MEMS
Nano-surface engineering
Scientific instrumentation
Medical technologies
Automation
Energy systems
Emerging & thin-film PV
PV cells& Modules
Vision systems
System-on-chip
Wireless
Industry 4.0
IoT
e-health
e-energy
Advanced manufacturing
Functionalpackaging
![Page 7: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/7.jpg)
Copyright 2019 CSEM | Page 6
Robotics & Automation
Robotics
Vision
Control
Deep Learning
![Page 8: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/8.jpg)
![Page 9: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/9.jpg)
Copyright 2019 CSEM | Page 8
History
1908Gabriel Lippmann
![Page 10: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/10.jpg)
Copyright 2019 CSEM | Page 9
100 years later…
2008Gabriel Lippmann
Market entrance 2010
Light field cameras for industry and research
Patented MLA design for optimal depth-of-field
![Page 11: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/11.jpg)
Copyright 2019 CSEM | Page 10
Principle
Standard Camera Lightfield Camera
![Page 12: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/12.jpg)
Copyright 2019 CSEM | Page 11
Lightfield (Plenoptic) Camera – 3D Vision
GPU Power
![Page 13: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/13.jpg)
Copyright 2019 CSEM | Page 12
3D Lightfield camera
![Page 14: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/14.jpg)
Copyright 2019 CSEM | Page 13
3D Lightfield camera
![Page 15: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/15.jpg)
Copyright 2019 CSEM | Page 14
Extended depth-of-field
Lens Type 1 Lens Type 2 Lens Type 3
![Page 16: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/16.jpg)
Copyright 2019 CSEM | Page 15
Multi Focus
Image plane Microlens plane
Lens Type 1
Lens Type 2
Lens Type 3
![Page 17: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/17.jpg)
Copyright 2019 CSEM | Page 16
Extended Depth-of-field
Raytrix Lightfield Camera R5μStandard 4MP Camera
![Page 18: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/18.jpg)
Copyright 2019 CSEM | Page 17
Occlusion
Stereo Camera Lightfield Camera
![Page 19: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/19.jpg)
Copyright 2019 CSEM | Page 18
3D in production
Total Focus Image
3D imageRoundness
Statistics
![Page 20: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/20.jpg)
Copyright 2019 CSEM | Page 19
Difficult Parts
![Page 21: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/21.jpg)
Copyright 2019 CSEM | Page 20
Screws
![Page 22: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/22.jpg)
Copyright 2019 CSEM | Page 21
BinPicking for small parts (under development)
• 3D Lightfield Camera
• CSEM Libraries
• Fast and Robust
![Page 23: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/23.jpg)
Copyright 2019 CSEM | Page 22
Dental
![Page 24: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/24.jpg)
Copyright 2019 CSEM | Page 23
3D Microscopy
FoV ~ 1x1mm Total Focus 3D
![Page 25: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/25.jpg)
Copyright 2019 CSEM | Page 24
Solder Bumps
![Page 26: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/26.jpg)
Copyright 2019 CSEM | Page 25
Product range
![Page 27: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/27.jpg)
Copyright 2019 CSEM | Page 26
Quality Control Today
![Page 28: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/28.jpg)
Copyright 2019 CSEM | Page 27
Traditional Image Processing…
Model
World
≠
Algorithm
![Page 29: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/29.jpg)
Copyright 2019 CSEM | Page 28
![Page 30: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/30.jpg)
Copyright 2019 CSEM | Page 29
How does our Brain work?
![Page 31: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/31.jpg)
DEEP LEARNINGAT CSEM
10+ years experience
Support humans
Industry proven
![Page 32: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/32.jpg)
Copyright 2019 CSEM | Page 31
Deep Learning since 2008 => Industrially validated solutions
CSEM
technology
transfer
![Page 33: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/33.jpg)
Copyright 2019 CSEM | Page 32
![Page 34: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/34.jpg)
Copyright 2019 CSEM | Page 33
Missing Link
![Page 35: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/35.jpg)
Copyright 2019 CSEM | Page 34
Integrated Robotic Software Framework «VISARD»
PC-based, one stop solution
Flexible, extendable,
reusable and user-friendly
Database logging and
automatic management
report generation
Hardware independent
Online process
development with scripting
tool (similar IEC 61131)
Benefits for the users
![Page 36: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/36.jpg)
Copyright 2019 CSEM | Page 35
Resolution
Parameters Overview system High-resolution system
Working distance 48 mm 20 mm
Field of view 7.8 mm x 7.8 mm 0.65 mm x 0.44 mm
Lateral resolution 3 m 0.7 m
Depth of field 1.1 mm 0.06 mm
Depth resolution 2.5 m 0.4 m
Framerate (max) 81 fps 30 fps
www.citcom.eu
Funded by the European Union’s HORIZON 2020 programme, Grant Agreement no. 768883
![Page 37: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/37.jpg)
Copyright 2019 CSEM | Page 36
3D Vision Demo
![Page 38: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/38.jpg)
Copyright 2019 CSEM | Page 37
CSEM Vision Benefits
• CSEM source code – no third party libraries (Cognex, Halcon, Matrox)
• No third party license fees
• No black box, not depended on external support
• CSEM has control over every bit and byte -> perfect adjusted solution
• Use of latest machine learning technics
• Modular software in C# .NET (supporting many camera brands)
• Easy to customize
• Full data logging (folder structured or database)
• Connection to PLC (ADS, OPC, …), TCP/IP, RS232, …
![Page 39: 3D Light-field Technology in Combination with Deep-Learning for …b6b0987f-a944-4582-8749-534c... · 2021. 1. 12. · Contact: Philipp Schmid Head Robotics & Automation psd@csem.ch](https://reader031.fdocuments.in/reader031/viewer/2022012008/611e22dd37c0da409e190b5b/html5/thumbnails/39.jpg)
Contact:
Philipp Schmid
Head Robotics & Automation
T +41 41 672 75 62
M +41 78 769 86 78
CSEM SA
Untere Gründlistr. 1 | 6055 Alpnach Dorf
www.csem.ch