CV inRobotics

1/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Computational Vision in Mobile Robotics

Dr. Julian Ryde

SUNY at Buffalo

January 14, 2013

CV inRobotics

2/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Abstract/Lecture Overview

A more realistic set of motivating application examplesbased in Mobile Robotics

Motivations for mobile roboticsChallenges of mobile roboticsExamples involving pattern recognitionNot going to talk about Industrial/static robotics

Even thought there are many applications of patternrecognition in this area of robotics

CV inRobotics

3/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

What is Robotics?

DefinitionsIntelligent connection of perception to action

Perception - Acquisition and interpretation of sensordataIntelligence?

Decision makingLearning

Actuation - Effecting changes in the physical worldLocomotionManipulation

Roles of pattern recognition in roboticsPattern recognition also has a role to play in intelligencePattern recognition forms part of the interpretation

CV inRobotics

4/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Robotics Areas

StaticIndustrial roboticsControlled environmentsMore commercially successful

MobileIndoor Robotics

Personal roboticsOffice robots

Remote telepresenceDomestic robots

Roomba - most successful consumer mobile robot?Field Robotics

MiningMilitary

Health careHome care for the elderly (Especially Japan)

Wherever it is Dirty, Dull or Dangerous

CV inRobotics

5/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Willow Garage

Long term funded company that does not need to makea profitAdheres to the open source development modelRobot Operating System (ROS)OpenCV

Computer VisionMachine Learning algorithms

PR2 RobotPR2 Beta Overview Video

CV inRobotics

6/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Sensors

Internal sensorsGlobal position system - GPSInertial Measurement Unit - IMU

Vision basedStandard CamerasOmnidirectional

Range sensorsTime of flightTriangulation based

Structured lightStereo vision

UltrasonicKinect

Structured light 3D depth sensorReturns RGBD - Red, Green, Blue and DepthInexpensive at 150$

CV inRobotics

7/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Pattern Recognition in 3D Data versus Pictures

More difficult for humansMore suitable for computers?

3D data is more independent of observer positionMany objects are rigid or articulated so 3D shape isimportant and invariantFor features invariant is good

implies reliable/repeatable extraction

Camera images susceptible toillumination changesloss of information in conversion from 3D to 2D

CV inRobotics

8/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Considerations specific to mobile robotics

AutonomyThere can be no guarantee of continuous connection.Therefore a certain level of autonomy is required forreliable continuous operation.

Power consumptionRestricted processing power

Especially for indoor and smaller robots that run onbatteries

Real-time constraintsPedestrian recognition has to be fast and reliable forautomotive applicationsBatch algorithms not really suitable

For long term operation robot ultimately has to processdata as fast as it receives it.

CV inRobotics

9/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Localisation and Mapping

LocalisationFor most applications need to have an awareness oflocationAugment the robots workspace with features

Possible but expensive for factories and other controlledenvironments

MappingLearning a model of the operatingworkspace/environmentAids localisationAids path planning

Simultaneous Localisation and Mapping (SLAM)The process of both inferring location from a map whilstsimultaneously updating the map with new observedinformation.

CV inRobotics

10/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Simultaneous Localisation and Mapping(SLAM)

SLAM is vital for indoor robotsInterestingly not used for the iRobot roomba

Outside not as necessary where GPS coverage isavailableVarious flavours of SLAM

MetricTopological

Matching2D/3D scan matchingImage

CV inRobotics

11/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Loop Closing

Closing of an open loop in a large (80m by 25m) cyclicenvironment from Konolige and Gutmann (1999)Improving map accuracy through loop closing

Have I been here before?Recognising previously visited locations

CV inRobotics

12/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Scan matching

Two scans taken from different posesFind features present in each scanDetermine the correspondence between featuresUse this information to align the scansThis gives the pose change between scansMuch research done with 2D range scansExample with 3D scans or an indoor environment

Scan matching Illustration

CV inRobotics

13/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

3D mapping indoors

3D map fly around video

CV inRobotics

14/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Probabilistic robotics

Return not only an answer but some idea of theprobability distribution

For example the pose distribution can be multimodalMany ways of representing these probablity densityfunctionsThus you have some idea of how likely the result is tobe correct

Algorithms in robotics need to be robustRobust statistics e.g. median

contrast with sufficient statisticsInsensitive to outliers

Sufficient statistics e.g. meanMathematically analyticComputational convenientSensitive to outliers

CV inRobotics

15/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

DARPA Grand and Urban Challenges

Grand Challenge$2 million Prize awarded to Stanford Racing Team132 mile desert course in just under 7 hours

Urban challengeFollow on from the easier grand challengeAutonomous vehicle drive through an urbanenvironment

Drive in trafficManeuvers merging, passing, parking, negotiatingintersections

Featured the first autonomous car crashReliable perception and interpretation vitalFinal highlights

DARPA Urban challenge highlights video

Key EnablersVelodyne 3D laser scannerNear to far learning

CV inRobotics

16/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Learning for mobile autonomous systems

Pattern recognition enables systems to learn andgeneralise from new data.Interaction with humans provides plenty of opportunityfor supervised learning.Amazon’s mechanical turk

Humans working for the robots!DARPA urban challenge videos

near to far learning in images e.g. using laser data andcorresponding vision results to train and so classify farimage pixels.

Self supervised learningTraining data might arrives at a later time for somecasesTraining data provided by others sensors

e.g. Near to far learning

CV inRobotics

17/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Near to Far Learning

Image and research from Raia HadsellOnline machine learning

Either laser or stereo provides reliable near range 5mclassification (driveable or not based on smoothness)Then used as training data for appearance basedclassifier for pixels beyond 5m

CV inRobotics

18/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Solutions in Perception Challenge

GoalRigid objects recognized and their 6DOF posedetermined with the Kinect sensor

Kinect sensorBest solutions likely to combine conventional imageinformation with depth informationIdeal opportunity to put into practice what you learn inthis course

CV inRobotics

19/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Field robotics

Work undertaken at the Australian CommonwealthScientific and Industrial Research Organisation(CSIRO)Autonomous earth moving

CV inRobotics

20/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Autonomous Skid Steer loader (Bobcat)

Earth moving taskMove soil material from one location to another

Ultimate goal is the design the terrain you want in aCAD package and have it replicated in the physicalworld by autonomous excavation and earth movingmachines

CV inRobotics

21/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Reliable Human/Novelty Detection

Learn the operational environmentHighlight those areas that have changedEssential for safety reasons

If unknown objects detected or the environment hasbeen tampered with need to potentially stop operation

Handle noise, how much variation constitutes change?

CV inRobotics

22/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Some active research questions related toPattern Recognition

SLAMMonocular camera based visual SLAMSLAM with RGBD sensors

Correspondence problemFinding corresponding points in multiple imagesFor stereo vision

Object recognition in 3D data3D shape based retrieve and lookupView and occlusion invariant hashingLocality sensitive hashing

Object recognition from a moving camera sensorDifferent to object recognition in random imagesBackground subtraction for moving cameras

CV inRobotics

23/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Recap

What is (mobile) robotics?Example applications involving pattern recognition

LocalisationMapping

Scan matchingLoop closing

Terrain classification for path planningNear to far learningSelf supervised machine learning

Field roboticsAutonomous Earth moving

Current work in the research communitySolutions in Perception Challenge

CV inRobotics

24/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Questions for consideration?

What is holding back mobile robotics?Was the lack of low cost depth sensors, but now withthe Kinect?Low cost manipulator platforms? The PR2 is $400,000

Motivated by the recent debut of the Kinect how canexisting computer vision pattern recognition algorithmsbe applied to RGBD data?