Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 ·...
Transcript of Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 ·...
![Page 1: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/1.jpg)
Donkey CarMake an RC car drive itself with an
end-to-end neural network.
Open Source (MIT)Python
Raspberry PiKeras / Tensorflow
OpenCV
![Page 2: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/2.jpg)
One year and one day of Donkey Cars.
How the hardware and software works?
Tips to work with neural network autopilots.
What's next for DIY self driving?
Agenda
![Page 3: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/3.jpg)
Will RoscoeSoftware
Adam ConwayHardware
![Page 4: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/4.jpg)
Roscoe makes pitch for competition to demonstrate AutoBART.
Donkey software refactored to be modular.
October, 2016
![Page 5: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/5.jpg)
Chris Anderson hosts first DIYRobocar meetup in Carl Bass’ workshop.
November, 2016
![Page 6: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/6.jpg)
First time DIYRobocars raced in Oakland Pipe Factory.
January, 2017
![Page 7: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/7.jpg)
Improved hardware with Donkey2
March, 2017
![Page 8: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/8.jpg)
Many more people build donkey.
July - August, 2017
![Page 10: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/10.jpg)
We are approaching human level racing.
![Page 11: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/11.jpg)
How does it work?
![Page 12: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/12.jpg)
Donkey Car - Instructions
donkeycar.com
Hardware instructions.
Software Docs
Help Brawl
![Page 13: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/13.jpg)
Donkey Car - Hardware Steps
1. Get Parts ($200) 2. Assemble Parts (1 hour)
![Page 14: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/14.jpg)
Donkey Car - Hardware Architecture
PCA9685
5V, 0
0s m
A
i2c
Steering Servo
ESC5v 3A
● Simplest possible hardware that we could build for a camera driven car.
● Enables Side-quests○ Lidar○ Odometry○ Joystick controller○ IMU
![Page 15: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/15.jpg)
Donkey Car - Software Goal
Image (120 x 160)
Steering (-1 to 1)Throttle (-1 to 1)Drive Mode (manual / auto)
Drive Perfectly.
![Page 16: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/16.jpg)
Get user input.
Take picture.
Update servo.
Update motor.
Get autopilot
inputSave data.
Donkey Car - Software Architecture
Run the “vehicle loop” 30 times per second.
![Page 17: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/17.jpg)
Donkey Car - Other Software Architectures
ControllerCamera Servo MotorAutopilot Datastore
Message Queue
Publisher / Subscriber
CAN Bus
Cars
![Page 18: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/18.jpg)
Get user input.
Take picture.
Update servo.
Update motor.
Get autopilot
inputSave data.
Donkey Car - Software Architecture
But.... KISS
![Page 19: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/19.jpg)
Get user input.
Take picture.
Update servo.
Update motor.
Get autopilot
inputSave data.
Donkey Car - Software Architecture
Camera
vehicle state
![Page 20: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/20.jpg)
Get user input.
Update servo.
Update motor.
Get autopilot
inputSave data.
Donkey Car - Software Architecture
Take picture.
Webserver
Browser
Http POST
vehicle state
![Page 21: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/21.jpg)
Get user input.
Take picture.
Update servo.
Update motor.
Get autopilot
inputSave data.
Donkey Car - Software Architecture
Neural net
Steering & Throttle
vehicle state
![Page 22: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/22.jpg)
Get user input.
Take picture.
Update servo.
Update motor.
Get autopilot
inputSave data.
Donkey Car - Software Architecture
Servo
PWMSignal
PWMSignal
ESC
Motor
Motor Controller
PWM Value
PWM Value
Example PWM Signals
vehicle state
![Page 23: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/23.jpg)
Get user input.
Take picture.
Update servo.
Update motor.
Get autopilot
inputSave data.
Donkey Car - Software Architecture
vehicle state
.jpg .jsonuser/steering: .3user/throttle: .4
![Page 24: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/24.jpg)
Donkey Car - Software Installation
Get your donkey driving.
1. Clone donkey disk image.2. donkey createcar --path ~/d23. python d2/manage.py drive4. Go to d2.localhost:8887 in your
browser.
![Page 25: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/25.jpg)
Get training data.
Drive around the track ~20 times to collect 10-50k records.
![Page 26: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/26.jpg)
Prepare training data.
1. Transfer records to computer.2. Remove bad data.
![Page 27: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/27.jpg)
Train an autopilot using Keras / Tensorflow.
![Page 28: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/28.jpg)
Test drive… repeat.
![Page 29: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/29.jpg)
Collect Data Train Pilot
Test Pilot
![Page 30: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/30.jpg)
Keras / Tensorflow Autopilots
![Page 31: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/31.jpg)
Neural network.Image Array
Steering + Throttle
120 pixels high160 pixels wide 3 RGB channels
Convolution layers
Fully connected layers
![Page 32: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/32.jpg)
Can we test an autopilot without driving?
![Page 33: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/33.jpg)
What’s next for Donkey.
![Page 34: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/34.jpg)
More competitions.
diyrobocars.com
Next race is this weekend in Oakland 10am - 2pm.
2 day Hackathon in Vegas Nov 26-27
![Page 35: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/35.jpg)
Simulators
![Page 36: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/36.jpg)
Additional data.
More sensors.Pooling data from
multiple cars..
![Page 37: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/37.jpg)
New Environments
![Page 38: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/38.jpg)
Better Autopilots
Modular Nets Maps Stateful
![Page 39: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/39.jpg)
Join us.
![Page 40: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/40.jpg)
1 - Build your own car or try to drive the simulator.
Web Computer Vision Deep Learning Hardware
2 - Hack on any part that interests you.
![Page 41: Make an RC car drive itself with an Open Source (MIT) Python Raspberry Pi … · 2018-01-11 · Donkey Car Make an RC car drive itself with an end-to-end neural network. Open Source](https://reader033.fdocuments.in/reader033/viewer/2022041611/5e378646f0c9eb1ef65b6868/html5/thumbnails/41.jpg)
Instructions to build your own car.donkeycar.com
Race events / meetups.diyrobocars.com
Stay in touch:
Will Roscoe: @dataduce Adam Conway @acb0t (with a zero)