Leveraging AI for Self-Driving Cars at GM -...
Transcript of Leveraging AI for Self-Driving Cars at GM -...
![Page 1: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/1.jpg)
Leveraging AI for Self-Driving Cars at GM
Efrat Rosenman, Ph.D.
Head of Cognitive Driving Group
General Motors – Advanced Technical Center, Israel
![Page 2: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/2.jpg)
Agenda
• The vision
• From ADAS (Advance Driving Assistance Systems) to AV (Autonomous Vehicles)
• AI for Self-Driving cars
• ADAS, AV and in-between
• Summary
10/19/2017 General Motors 2
![Page 3: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/3.jpg)
The Vision
3
?
• Mobility – one of the most significant revolutions of modern times
• Self-driving cars will take mobility to a completely new phase…
”Zero Crashes, Zero Emissions, Zero Congestion” (Mary Barra, GM CEO)
10/19/2017 General Motors
![Page 4: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/4.jpg)
The Vision
4
Increase Mobility: anywhere, anytime Increase Car Sharing & Reduce Road Capacity and Parking needs
10/19/2017 General Motors
Increase Safety Increase Productivity
![Page 5: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/5.jpg)
5
From ADAS to AV
10/19/2017 General Motors
L5:Full
automation
Level 4: High automation
Level 3: Conditional automation
Level 2: Partial automation
Level 1: Driver assistance
Level 0: Driver in full control Info, warnings
Cruise control, lane position
Traffic jam assist
Anywhere, anytime
Fully autonomous specific scenarios
Highway driving (driver takes control with notice)
![Page 6: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/6.jpg)
6
From ADAS to AV
• Will incremental steps get us to the top of this pyramid?
10/19/2017 General Motors
![Page 7: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/7.jpg)
10/19/2017 General Motors 7
Sensing Mapping PerceptionDecision Making
Control
Components of self driving cars
![Page 8: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/8.jpg)
Components of self driving cars
AI AGENT serves as the “brain” of the car
10/19/2017 General Motors 8
PerceptionDecision Making
Control
![Page 9: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/9.jpg)
AI for Self-Driving Cars
9
![Page 10: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/10.jpg)
AI in Perception
• Unsupervised learning
• Finding structure in point clouds
• Feature learning
• Supervised learning
• Object detection
• 2D object recognition (Classification)
• 3D scene understanding and modeling (3D objects pose)
• Semantic segmentation (boundaries of objects, free space)
10/19/2017 General Motors 11
![Page 11: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/11.jpg)
AI in Perception - E2E trend
• Classification:
• Scene understanding:
• Perception:
10/19/2017 General Motors 12
Pixels Key Points ModelSIFT features Labels
Sensors 2D object detection
Pose estimationDepth estimation
3D World state
Pixels Segmentation Contextual relations
Object detection
Scene description
![Page 12: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/12.jpg)
AI in Perception - E2E trend
• Classification:
• Scene understanding:
• Perception:
10/19/2017 General Motors 13
Pixels Key Points ModelSIFT features Labels
Sensors 2D object detection
Pose estimationDepth estimation
3D World state
Pixels Segmentation Contextual relations
Object detection
Scene description
DNN
DNN
DNN
![Page 13: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/13.jpg)
Towards E2E: Sensors Fusion
10/19/2017 General Motors 14
• All sensors contribute
• Enables learning of complex dependencies “optimally”
• Sparse Vs. dense sensors
• Larger models, harder to learn
• Utilizes domain knowledge
• Model is explainable
• Based on tailored rules
• Suboptimal performance
Low Level: raw data combined in input stage
High Level: tailored hierarchy between sensors
![Page 14: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/14.jpg)
Towards E2E: Multi-Task Learning
• Most our outputs are inter related• Objects, free space, lanes, etc.
• Cross regularization allows reaching a better local minima
• TPT• Major parts of the Deep Net are used for multiple tasks
• Data Efficiency
10/19/2017 General Motors 15
Mask R-CNN Facebook AI Research (FAIR); Apr 2017
![Page 15: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/15.jpg)
What about data?
16
![Page 16: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/16.jpg)
Automatic Data Annotation
• Data is the key contributor to perception accuracy – With no visible saturation
• How can we create annotated data• Manual annotation – Expensive and inaccurate
• Automatically
10/19/2017 General Motors 17
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era, Google 2017
![Page 17: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/17.jpg)
Automatic Data Annotation
• Technology• High end sensors (Lidar, IMU, etc.)
• High accuracy detectors (on behalf of computation time)
10/19/2017 General Motors 18
![Page 18: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/18.jpg)
Example – AGT for StixelNet
• StixelNet - Monocular obstacle detection• Based on stixel representation
• Identify road free space
• Ground truthing is based on Lidar
10/19/2017 General Motors 19
Dan Levi, Noa Garnett, Ethan Fetaya. StixelNet : A Deep Convolutional Network
for Obstacle Detection and Road Segmentation. In BMVC 2015.Lidar (Velodyne HDL32) is used to identify obstacle on each stixel in the image
[Badino, Franke, Pfeiffer 2009]
Compact, local representation
![Page 19: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/19.jpg)
Is Perception “solved”?
• Challenge of Cost• Sensors
• Mapping
• Computation
• Challenge of false positive & false negative• Data uncertainty (noise)
• Model uncertainty (confidence)
10/19/2017 General Motors 20
Label: Cyclist RGB: Pedestrian (0.56)
![Page 20: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/20.jpg)
Decision Making
10/19/2017 General Motors 21
PerceptionDecision Making
Control
![Page 21: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/21.jpg)
Learning Decision Making
Decision Making cannot learn from static examples
Need interactive domain
- > Reinforcement Learning (RL)
RL has seen some major successes in the recent years:
10/19/2017 General Motors 22
Go[Google deepmind] source: uk business insider
Poker [Bowling et al] source: wikipedia
Autonomous Helicopter Flight [Ng et al] source: ai.stanford.edu
Atari[Google Deepmind] source: nbcnews
![Page 22: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/22.jpg)
RL challenges in Self-Driving agents
• Learn to act in a very high dimensional space
• Plan sequences of driving actions
• Predict long term behaviors of other road users
• Few sec
• Complicated situations
• Negotiate with other road user
• Guarantee safety
10/19/2017 General Motors 23
![Page 23: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/23.jpg)
Simulation
• Advanced simulations are required• Multi-agent
• Various conditions
• Focus on “interesting miles”
• Drive billions of “virtual miles” (fuzzing)
“Any system that works for self driving cars will be a combination of more than 99 percent simulation.. plus some on-road testing.” [Huei Peng director of Mcity, the University of Michigan’s autonomous- and connected- vehicle lab]
10/19/2017 General Motors 24
Waymo simulation:https://www.engadget.com/2017/09/11/waymo-self-driving-car-simulator-intersection/
![Page 24: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/24.jpg)
Safety Guarantees - From ADAS to AV
Will incremental steps get us to the top of this pyramid?
The technological heart is different in kind
10/19/2017
![Page 25: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/25.jpg)
What’s the difference?
• For ADAS – Safety guarantee is based on the driver
• For autonomous – Safety guarantee should come from the system itself
10/19/2017 General Motors 26
![Page 26: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/26.jpg)
Example: Highway Driving in Super Cruise™
10/19/2017 General Motors 27
The 2018 Cadillac CT6 will feature Super Cruise™ - a hands-free driving technology for the highway
It includes an Exclusive driver attention system to support safe operation
![Page 27: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/27.jpg)
Safe Driving for level 4/5
• System should handle 100% of the cases
• Redundancy requires at all levels• Sensing• Algorithm• Computing• Control• Fallback strategies
• Guarantee of Safety is a must to the acceptance of AV• Statistical data-driven approach [miles-per-interrupts] requires driving billions of
miles to validate an agent• Should be repeated with every SW version
• Need safety constrains (rule-based/model-based)
10/19/2017 General Motors 28
![Page 28: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/28.jpg)
Summary
• Advances in AI are key to success of self-driving cars
• AI-based features can bring ADAS to a new level in terms of accidence avoidance, productivity gain and saving in human lives
• Level 4/5 AV should be a parallel effort focus on redundancy and safety constrains
10/19/2017 General Motors 29
![Page 29: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/29.jpg)
GM Advanced Technical Center in Israel (ATCI)
![Page 30: Leveraging AI for Self-Driving Cars at GM - NVIDIAon-demand.gputechconf.com/gtc-il/2017/presentation/... · The Vision 3? •Mobility –one of the most significant revolutions of](https://reader034.fdocuments.in/reader034/viewer/2022042316/5f0450c57e708231d40d5f09/html5/thumbnails/30.jpg)
Thank you
10/19/2017 General Motors 31