Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision...
Transcript of Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision...
![Page 1: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/1.jpg)
Julian Kent - 21/06/2019
Frontier of Computer Vision
![Page 2: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/2.jpg)
DISCUSSION
Who has done CV development?
(even toy projects)
![Page 3: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/3.jpg)
The Frontier of Computer Vision3
Computer Vision - Motivation
● Enables autonomy
● Rich, high bandwidth sensor data
● Easy for humans to understand the raw data
![Page 4: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/4.jpg)
The Frontier of Computer Vision4
Computer Vision - Current Industry Status
On drones, CV functionality is limited to roughly:
● Collision Prevention
● VIO
● Follow-me tracking or other use-case specific applications
![Page 5: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/5.jpg)
The Frontier of Computer Vision5
Computer Vision - Current Industry Status
On drones, CV functionality is limited to roughly:
● Collision Prevention SONAR / laser distance sensors
● VIO On-chip Optical Flow (think optical mouse)
● Follow-me tracking Target GPS tracking
The cheap, lightweight (sometimes more reliable) solution
![Page 6: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/6.jpg)
The Frontier of Computer Vision6
Computer Vision - Current PX4 Status
On drones, CV functionality is limited to roughly:
● Collision Prevention SONAR / laser distance sensors
● VIO On-chip Optical Flow (think optical mouse)
● Follow-me tracking Target GPS tracking
The cheap, lightweight (sometimes more reliable) solution
![Page 7: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/7.jpg)
The Frontier of Computer Vision7
Computer Vision - Current PX4 Status
Safe landing planner
![Page 8: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/8.jpg)
The Frontier of Computer Vision8
Computer Vision - Current PX4 Status
Obstacle avoidance in mission
![Page 9: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/9.jpg)
The Frontier of Computer Vision9
Computer Vision - What can it do
● Human use of vision○ Localization
○ Recognition and object tracking
○ Environment awareness & map building
○ Minimum risk path planning
○ Safe stopping locations
○ Facial expressions, body language
![Page 10: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/10.jpg)
The Frontier of Computer Vision10
Computer Vision - What can it do
● Academic projects, but not industry:○ SLAM - build maps and localize the robot dynamically via the robot’s movement
○ Safety analysis beyond simple landing systems
○ Object classification
○ Playback of exact flight paths based on visual markers
○ Learned safe flight patterns from dashcam videos
○ Accurate landings on dynamic platform
○ Many many more
![Page 11: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/11.jpg)
The Frontier of Computer Vision11
Computer Vision - What can it do
● Academic projects, but not industry:○ SLAM - build maps and localize the robot dynamically via the robot’s movement
○ Safety analysis beyond simple landing systems
○ Object classification
○ Playback of exact flight paths based on visual markers
○ Learned safe flight patterns from dashcam videos
○ Accurate landings on dynamic platform
○ Many many more
![Page 12: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/12.jpg)
The Frontier of Computer Vision12
Computer Vision - What can it do
● Academic projects, but not industry:○ SLAM - build maps and localize the robot dynamically via the robot’s movement
○ Safety analysis beyond simple landing systems
○ Object classification
○ Playback of exact flight paths based on visual markers
○ Learn safe flight patterns from dashcam videos
○ Accurate landings on dynamic platform
○ Many many more
![Page 13: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/13.jpg)
The Frontier of Computer Vision13
Computer Vision - Why isn’t it everywhere
Expensive
● Weight
● Power
● Hardware prices
These are being solved in newer hardware generations
![Page 14: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/14.jpg)
The Frontier of Computer Vision14
Computer Vision - Why isn’t it everywhere
Reliability
● Depends on lighting conditions
Historical deployments of CV/ML systems are in controlled environment
● Eg. factory assembly lines for component alignment and QA
Partially solved with better cameras - industrial still catching up with consumer
● Active sensors - IR pattern projectors
![Page 15: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/15.jpg)
The Frontier of Computer Vision15
Computer Vision - Why isn’t it everywhere
Other problems:
● Concentrated R&D Costs
● Systems are complex and require large upfront investment, with risk of failure
● Customers and regulators don’t trust black boxes
Open Source solves these problems
![Page 16: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/16.jpg)
The Frontier of Computer Vision16
Computer Vision - So what are we waiting for
● Purpose designed hardware○ Higher bandwidth between FCU and companion computer WIP
○ Cheap enough to deploy in scale Depending on platform
○ Widely available Depending on platform
● The frameworks to effectively re-use libraries○ Reliable PX4 ← → ROS[2] communication ROS1 works, ROS2 WIP
○ Integration of vision interfaces into PX4 architecture Works, but extensions missing
○ Making vision components with better integration WIP
![Page 17: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/17.jpg)
DISCUSSION
Who has ideas for CV projects they’d like to see integrated into or created for the
PX4 ecosystem?
![Page 18: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/18.jpg)
The Frontier of Computer Vision18
Computer Vision - Features I’d like to see
● Minimum risk path planning
● Full SLAM with collision prevention in position mode○ Give pilot additional environmental awareness based on generated map
● RTL following visual landmarks if GPS is lost, including precision landing
● Structure following for inspection
● Full integration of compatible ROS[2] packages with PX4
![Page 19: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/19.jpg)
The Frontier of Computer Vision19
Computer Vision - Integrating with the FCU
● Offboard mode - great for research, poor for end users
● As of 1.9 we can inject trajectories while in auto modes
● Integrating more fully requires PX4 flight mode awareness
○ Make a standard ROS[2] node to do this
![Page 20: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/20.jpg)
The Frontier of Computer Vision20
Conclusions
● Computer vision is hard, but the community can help
● Hardware is finally good enough for (some) drones
● We still need to customize things, the software isn’t just ‘plug and play’
![Page 21: Julian Kent - 21/06/2019 - PX4 autopilot€¦ · The Frontier of Computer Vision 18 Computer Vision - Features I’d like to see Minimum risk path planning Full SLAM with collision](https://reader035.fdocuments.in/reader035/viewer/2022071217/604b471fd69efc3b6176ed8f/html5/thumbnails/21.jpg)
Thank you!