Seashell Effect Pretouch Sensing for Robotic Grasping

Liang-Ting Jiang and Joshua R. Smith

Abstract— This paper introduces seashell effect pretouch sens-ing, and demonstrates application of this new sensing modalityto robot grasp control, and also to robot grasp planning.

“Pretouch” refers to sensing modalities that are intermediatein range between tactile sensing and vision. The novel pretouchtechnique presented in this paper is effective on materials thatprior pretouch techniques fail on. Seashell effect pretouch isinspired by the phenomenon of “hearing the sea” when aseashell is held to the ear, and in particular relies on theobservation that the “sound of the sea” changes as the distancefrom the seashell to the head varies. To turn the familiar seashelleffect into a sensor for robotic manipulation, a cavity andmicrophone was built into a robot finger. The sensor detectschanges in the spectrum of ambient noise that occur when thefinger approaches an object. Environmental noise is amplifiedmost (attenuated least) at the cavity’s resonant frequency, whichchanges as the cavity approaches an object.

After introducing the sensing modality and characterizing itsperformance, the paper describes experiments performed withprototype sensors integrated into a Willow Garage PR2 robot’sgripper. We explore two primary applications: (1) reactive graspcontrol and (2) pretouch-assisted grasp planning. For reactivegrasp control, we demonstrate the advantage of pretouch overpressure sensing for sensing objects that are too compliant forthe PR2’s tactile sensors to detect during the grasp execution.We demonstrate the benefit of seashell effect pretouch overprior pretouch methods by demonstrating that the new methodsucceeds with materials on which prior methods fail. Forpretouch-assisted grasp planning, the pointcloud from the PR2’stexture-projection / narrow stereo camera is augmented withadditional points collected by recording the position of therobot’s end effector when the pretouch sensor detects theobject. This method can compensate for points that are missingfrom the pointcloud either because of depth camera failure orocclusion. For this application, pretouch is advantageous overcontact sensing because contact sensing tends to unintentionallydisplace the object.

I. INTRODUCTION AND RELATED WORK

Long range non-contact sensors such as RGB cameras,depth cameras, and laser scanners are widely used in roboticsfor object recognition and pose estimation. These long rangesensors typically provide the data used for grasp planningprior to manipulation. Contact sensors such as tactile pres-sure sensors are commonly used during the process ofmanipulation to provide force feedback information to agrasp controller. “Pretouch” sensors are non-contact sensorswith properties intermediate between these two: the rangeof pretouch sensors is shorter than vision but longer thancontact-based tactile sensing. This paper introduces “seashell

Liang-Ting Jiang is with the Department of Mechanical Engineering, Uni-versity of Washington, Seattle, WA. jianglt@u.washington.edu

Joshua R. Smith is with the Departments of Computer Scienceand of Electrical Engineering, University of Washington, Seattle, WA.jrs@cs.washington.edu

Fig. 1. The seashell effect pretouch sensing fingertip on the PR2 robotgripper. Two different designs are presented for different applications: (a)the sensor on the finger surface for sensing extremely compliant objects,and (b) the sensor on the fingertip for adding pretouch pointcloud at theunknown portions of the object before grasp planning.

effect pretouch,” a novel form of pretouch that works ona wide variety of materials; the paper demonstrates theapplication of seashell effect pretouch to both grasp controland grasp planning.

A. Robotic Pretouch Sensing

The hypothesis that pretouch sensing can be useful formanipulation is being explored actively [1][2][3][4]. It isviewed as potentially beneficial for manipulation because itprovides reliable geometric information in the last centimeterbefore contact. The disadvantage of using tactile sensing forcollecting local geometric information (as in [5], [6]) is thatcontacting the object may displace it—an outcome that isparticularly likely when the object’s geometry is initiallyuncertain. An advantage of pretouch over vision is thatpretouch is not subject to the problem of the hand occludingthe camera, because the sensor is integrated into the hand;another is that there are no camera-to-hand registrationerrors since the sensor is in the coordinate frame of thehand. Because of these challenges for vision sensing, graspsplanned over point cloud data are typically executed openloop: new point cloud data is usually not collected duringexecution of a grasp. By contrast, like tactile sensor data,pretouch sensor values can easily be collected during graspexecution and used by a grasp controller. Finally, depthsensors typically provide incomplete point clouds, droppingmany points because of failure modes such as: challeng-ing optical properties (transparency, specularity); algorithmicfailure to find stereo correspondences; and failure to findprojected structure (visible or IR texture or grids).

A further distinction between pretouch and depth sensingtechnologies is that the latter typically fail below someminimum distance. For example, here is a list of depthsensing technologies and their minimum specified range:

Microsoft Kinect, 120 cm [7]; PR2 textured narrow stereocamera, 60 cm [8]; Hokuyo UTM-30LX laser scanner, 10cm [9]; Parallax PING ultrasound sensor, 2 cm [10].

Compared to tactile sensing, one can think of pretouchas a sensor that detects surface proximity, but has no lowerlimit on detectable force, and thus is able to sense arbitrarilycompliant objects.

Electric field pretouch has many desirable properties, butonly works with materials that are conductive or have highdielectric constant. Optical pretouch depends on surfacealbedo, and thus fails in some of the same cases as thelong range vision sensors: transparent or specular objects.Thus optical pretouch may fail to complement the 3D visualsensors used to plan grasps: since they rely on similarphysical phenomena, they are likely to fail in a correlatedfashion. Seashell effect pretouch has the desirable propertiesof other pretouch mechanisms, but does not depend onelectrical or optical material properties. Thus it works onmaterials that are difficult for electric-field pretouch, opticalpretouch, and conventional vision / depth sensing.

B. Seashell Effect

There is a well-known folk myth that if one holds aseashell to the ear, one can hear the sound of the ocean.The rushing sound that one hears is in fact the noise ofthe surrounding environment, resonating within the cavityof the shell. The same effect can be produced with anyresonant cavity, such as an empty cup or even by simplycupping one’s hand over one’s ear. The resonator is simplyamplifying (attenuating) the ambient environmental noise,in a frequency dependent fashion that is determined by theresonant modes of the specific cavity [11]. It is easily verifiedthat the perceived sound depends on the position of theseashell with respect to the head. Inspired by this seashelleffect, we propose to measure the distance to nearby objectsby detecting changes in the ambient noise spectrum insidean acoustic cavity. A cavity with a microphone is integratedinto a robot finger; as the finger approaches an object, thespectrum of the noise collected by the microphone changes.

C. Robot Grasping

Robotic grasping in unstructured human environmentsis a highly challenging problem for robotics. Besides theinaccuracy of the end-effector motion control, a key difficultyis the lack of reliable perception, caused by sensor calibrationerrors, or simply the limitation of sensors. One approachto accommodate it is to improve the grasp selection andexecution algorithm for unknown objects which is able todeal with uncertainty [12][13][14]. Another approach is toimprove the quality of the depth sensor. Depth sensing cam-era systems, including textured stereo cameras and structuredinfrared depth cameras, have dramatically improved recently.However, these sensors still frequently fail to provide points,either because of geometric difficulties (such as occlusion),or material difficulties (transparency or specularity). In thispaper, we demonstrate the use of pretouch to augment thepoint clouds provided by depth cameras, with the aim of

improving the grasps that can be planned for un-modeledobjects.

Petrovskaya [5][6] demonstrated the use of tactile sensorsfor global object localization, and proposed an efficientMonte Carlo algorithm for realtime applications. Since Petro-vskaya relies on tactile sensing, the approach is subject todifficulties such as unintentional displacement of the objectby the robot manipulator. Another feature distinguishing ourapproach is our integration of pretouch sensing with opticaldepth sensing. This paper demonstrates the use of seashelleffect pretouch for both grasp control and grasp planning.

II. SEASHELL EFFECT PRETOUCH SENSOR

The seashell effect pretouch sensor is essentially an open-ended pipe attached on a microphone. The detailed sensordesign and characterization will be described in this section.

A. Acoustic Theory

The radiation impedance of an open-ended pipe has asmall but finite value. Its imaginary part acts as an endcorrection to the geometrical pipe length from which theresonance frequency of the pipe is calculated. Dalmont [16]found an empircal formula for the end correction in this kindof pipe-surface configuration by fitting a function to valuesproduced by a finite element model.

The end correction δobj caused by the approaching objectis given by:

δobj =a

3.5(h/a)0.8(h/a+ 3w/a)−0.4 + 30(h/d)2.6

where a, w are the inner radius and wall thickness of thepipe; h is the distance between the object and pipe opening,and d is the radius of the object. The end correction δpipecaused by the opening geometry of the pipe is given by:

δpipe = 0.6133a1 + 0.044(ka)2

1 + 0.19(ka)2

for ka < 3.5, where k is wavenumber, which is determinedby the frequency of acoustic wave. The fundamental fre-quency of a one-sides open tube, is approximately given by:

f0 =1

4

c

(L+ δobj + δpipe)

where L is the original length of the pipe, and c is the soundof the speed. The formulas above show that the approachingobject will increase the end correction, and thus increase theeffective acoustical length of the pipe, which results in adecrease in the pipe’s resonance frequency. In order to knowthe accuracy of the empirical formulas and understand theeffect of pipe geometry, the computed resonance frequenciesof different geometry are compared with our experimentaldata and shown in Figure 2 (L is the length of the pipe).

Fig. 2. Comparison of the theoretical and experimental fundamental moderesonance frequencies. The figure also shows the effect of cavity geometryon resonance frequencies. Note that the sensors in this figure have not beenoptimized for performance: they do not use the reference microphone of ourfinal system. This figure is meant to illustrate the effect of cavity geometry.

1 2 3 4 5 6 7 8 9 10Distance (mm)

7500

8000

8500

9000

9500

10000

10500

11000

Res

onan

ceFr

eque

ncy

(Hz)

Contrast to Noise Ratio (CNR) = 22.04

Pretouch Thresold

Fig. 3. The box and whisker plot of 1000 estimated resonance frequenciesat each distance. It represents the sensor characteristics of the sensor withlength L=5 mm and radius a=2.5 mm integrated on the robot fingertip.The blue lines at the bottom and top of each box are the 25th and 75th

percentile (the lower and upper quartiles, respectively), and the red bandnear the middle of the box is the 50th percentile (the median). The lowerand upper black horizonal lines are the lowest and highest samples stillwithin 1.5× (upperquartile− lowerquartile) of the lower quartile andthe upper quartile, respectively. The green crosses are outlier samples.

B. Sensor System Design

To make usable sensors based on the principle describedabove, our sensor system is comprised of both hardware andsoftware components. Figure 4 shows the schematic of thesystem. The cavity amplifies the ambient noise in a certainfrequency response. The sound in the cavity is collected byan omnnidirectional electret condenser microphone cartridge(WM-61A, Panasonic) with sampling rate (Fs) of 44,100Hz. The signal is amplified by 50dB and converted to digitalthrough an audio interface. An additional microphone is usedto collect the sound outside the cavity as the reference signalin order to capture only the frequency response of cavity.

The power spectrum of the signal is estimated every 0.05second (N = 2205) using Welch spectrum estimation (Ns

Microphone I(Signal Channel)

A/D Converter

Microphone II(Reference Channel)

Ambient Sound Signal

Sound Signal in the cavity

-+Mic

Preamp

WelchSpectrum

Estimation

SubtractedSpectrum

Spectral PeakEstimation [16]

EstimatedResonanceFrequency

KalmanFilter [17]

FilteredResonanceFrequency

Fig. 4. The system architecture of the seashell effect pretouch sensor.

= 1024; overlap ratio = 70%; Hanning data taper). Thespectrum of the reference channel is subracted from thespectrum of the sensor signal before finding the peak power,thus the effect of the noise in the environment is rejected. Thepeak-finding algorithm is finding the frequency having max-imum power. Due to the sampled nature of spectra obtainedusing the Fast Fourier Transform, the peak (location andheight) found by finding the maximum-magnitude frequencybin is only accurate to within half a bin. A bin representsa frequency interval of Fs/Ns Hz (in this case is 43Hz).Therefore, an accurate and computationally simple approx-imation method [18] is used to estimate the actual spectralpeak frequency. The frequency spectral peak is then filteredby a Kalman filter [19] with process variance 10−5 andmeasurement variance 10−4. The paramters are experimentedand determined by the balance of fast response time andthe measurement stability. The distance measurement can becarried out by mapping the filtered frequency to the pre-calibrated frequency-distance function.

C. Integration with the PR2 Robot

We use Willow Garage PR2 robot as our experimentplatform. The current implementation is detached fromthe exsisting PR2 finger electrical interface. Two differentconfigurations of the seashell effect pretouch sensors areintegrated on the PR2 gripper shown in Figure 1. The designin Figure 1(b) is for the pretouch-assisted grasp planningapplication. The sensor is located on the fingertip for addingpointcloud at the location of fingertip. The design in Figure1(a) is for the application of grasping. The sensor is locatedon the surface of the finger, enabling sensing extremelycompliant objects during the grasp execution.

The cavity used in our system is selected to a 2.5 mmradius / 5 mm length cylindrical pipe attached with the 3mmthick microphone, which has compact size to be embeddedon the PR2 gripper fingertip.

D. Sensor System characterization

The performance of the sensor on the fingertip is evaluatedby collecting 1000 sensor readings (filtered spectral peakfrequency) at various distance from 1mm to 10mm. A box-and-whisker plot is plotted to present the performance ofthe sensor (Figure 3). A contrast to noise ratio (CNR) forevaluating the sensor performance is defined as

CNR =mean(f10)−mean(f1)

mean(std(fi)), i = 1..10

where i is the distance in millimeter. The box plot showsthe filtered resonance frequency starts to decrease from 6mm. Based on that, we select the threshold to 9500Hz (thelower quartile at 3 mm) as the first distance being able to bemeasured with confidence. In this way, the upper quartileat 3 mm is smaller than the lower quartile at 6 mm, sothe sensor is not getting confused. All the software in thiswork is implmented in ROS (Robot Operation System). Thesignal processing and frequency estimation is implementedin Python as a ROS node, which continuously publishesthe estimated resonance frequency, total signal energy, anddistance in a ROS message at the rate of 20 Hz.

E. Material sensitivity

Our sensor does not depend on optical or electrical mate-rial properties. Instead, it depends on mechanical or acousticproperties. Thus it is likely to complement long range opticaldepth sensors. For example, it can sense transparent andextremly light-reflective materials which are difficult foroptical sensors.

The only materials we have found so far on which thesensor fails are open foams (such as very thin bubble wrap),certain rough fabrics, and fur.

III. APPLICATION I: REACTIVE GRASPING OFCOMPLIANT OBJECTS

Tactile sensors have been widely researched for lo-cal perception of objects by monitoring the contact force[12][15][5]. One of the cases in which this approach mightfail is when the object is too compliant, so the pressuresensor cannot detect it. The first application of our seashellpretouch sensor is exploring the feasibility of sensing theseextremely compliant objects during grasp execution. A pre-grasp execution experiment was performed, in which thepressure sensor built in to the PR2’s gripper was compared toour seashell effect pretouch sensor for the ability to detect theobjects. The two sensors were applied in a pre-grasp task tofour different objects with different compliance. The objectsused were a cookie box, a disposable cup, a folded paperbox, and a folded aluminum foil box.

In the force sensing trial, the PR2’s pressure sensorfingertips are installed on both right and left fingers onthe PR2 right gripper. The standard PR2 gripper sensorcontroller is used to detect contact with objects. In order toincrease the sensitivity of the pressure sensors, we loweredthe contact detection threshold from the default values (0.1N for the high-pass filtered pressure readings, and 1.2 N forthe unfiltered pressure) to the lowest values (0.05 N for thehigh-passed pressure readings, and 0.4 N for the unfilteredpressure) such that false positive detection is avoided.

In the pretouch trial, the seashell effect pretouch sensoris installed on one of the fingers on the PR2 right gripper.The gripper is commanded to close until the fingertip issufficiently close to the object according to the seashellsensor.

A trial is defined as successful if the gripper stops beforesqeezing so much that it breaks the object. The pressure

Fig. 5. (a) The pressure sensor doesn’t sense the aluminum foil box andthus squeezes the object. (b) The pretouch sensor on the right finger sensesthe aluminum foil box and stops before touching the object. (c) The pressuresensor doesn’t sense the paper box and thus squeezes the object. (d) Thepretouch sensor on the right finger senses the paper box and stops beforetouching the object.

TABLE IRESULTS OF THE SENSING FOR COMPLIANT OBJECTS EXPERIMENTS.

(∨: success; ×: failed)Objects Pressure Seashell

pretouchcookie box ∨ ∨

disposable cup ∨ ∨folded paper box × ∨

folded aluminum foil box × ∨

sensor is able to detect the contact for cookie box anddisposable cup, but it is not able to sense the contact withthe extremely compliant folded paper and aluminum foilboxes, thus breaks them. On the other hand, the seashelleffect pretouch sensor is able to sense all four objects andstop the gripper at an approppriate distance from the object,which shapes the pre-grasp pose (Figure 5 and Table I). Aninteractive mode in which the gripper dynamically adjustsits opening based on the seashell effect pretouch sensor hasalso been implemented. This experiment shows the pretouchsensor can be a good complement to tactile sensing whengrasping compliant objects.

IV. APPLICATION II: PRETOUCH-ASSISTEDGRASP PLANNING

3D perception is becoming more and more important inrobotics, as well as in other fields. Pointcloud data structuresare widely used to represent the sensed spatial and colorinformation. A large community is developing a dedicatedopen source library, PCL (Point Cloud Library) [17], provid-ing algorithms to process pointclouds efficiently. In roboticgrasping, an incomplete pointcloud can result in poor grasp

Fig. 6. Pretouch pointcloud grasping workflow. Two new components,pretouch motion planner and pointcloud server, are added to the PR2grasping framework. The robot can choose to use the pretouch sensor ornot to use it. The concatenated pointcloud will be used for grasp planning,followed by the collision map processing and grasp execution.

plans, simply because the planner is operating with incorrectinformation about the object geometry. Although depth sens-ing hardware has dramatically improved recently, it is stillprone to dropping a many points, and there are still casesin which key object shape information cannot be collectedby the depth camera because of geometric constraints; forexample, in highly constrained environments, the robot maynot be able to move its head enough to collect additionalviews. We propose to use pretouch sensing in conjunctionwith the PR2s existing depth sensors, to provide data onportions of the object where information is missing, eitherbecause of occlusion or depth sensor failure. Given a partialpointcloud collected by a depth sensor, our pretouch sensor(combined with robot kinematic data) will add additionalpoints to the pointcloud, prior to grasp planning. In thissection, we will describe the methdology and how additionalpointcloud could be beneficial to robotic grasping. Somepractical examples will also be discussed.

Currently, our grasping framework is built on top ofthe PR2 object manipulation framework proposed in [12].We add two novel components to the framework: (1) apointcloud server to store both the pointcloud generatedfrom the depth sensor (stereo cameras in this case) andthe pretouch pointcloud we are generating later; and (2) apretouch motion planner to command the robot end-effector

to probe at the location of interest. Both components areimplemented as ROS nodes.

The initial pointcloud is generated by the depth dataobtained from a single frame of the narrow-field-of-viewstereo camera on the PR2. This pointcloud is used to segmentthe objects from the table, and the pointcloud cluster of thesegmented object is then saved on the pointcloud server. Ifpretouch pointcloud is desired, the pretouch motion plannerwill use the pointcloud to decide the initial probe locationand then probe around the area.

The completed workflow of the object grasping withpretouch pointcloud is shown in Figure 6, and each key stepwill be described in detail below.

A. Pretouch Motion Planning

The pretouch motion planner is a component that decideswhere the robot should move its end-effector to and lookfor the object surface. The current pretouch motion planningalgorithm assumes the object is convex and fits an ellipiccylinder from the segmented object pointcloud, and then theinitial probe pose is determined by the farthest point the end-effector can reach in the unknown area of the object allowedby the PR2 arm’s workspace. The algorithm works for mostof objects with round and rectangular shapes. The fingertipalways approaches in the direction normal to the periphery ofthe fitted ellipse with a constant velocity. When the pretouchsensor senses the distance closer than the threshold (whenthe estimated resonance frequency lower than the threshold9500 Hz), W points are added on the pointcloud server at thelocation translated by 3 mm in the X-axis from the fingertipframe. The offset 3mm is corresponding to the threshold9500 Hz determined by the sensor characterization in Figure3. W is the weight of the point generated by the pretouchsensor. It determines how much the grasp planner will takethese new points into account when doing grasp planning.Adding a single point at each location will not alter the graspplannng results much because the pretouch pointcloud issparse. The end-effector will continue lowering and probingwith a fixed interval in the Z (vertical) axis before the gripperhitting the table. Once the height is too low, the grippermoves back to the initial height and also moves toward tothe robot body. The same process continues until the newlydetected points are close enough to the existing pointcloudgenerated from the camera.

In our particular implementation, PCL [17] is used for theprocessing of pointclouds, and a Jacobian inverse controlleris used as a low level controller to control the end-effectorpose and velocity in the cartesian space. Figure 7 shows anexample of the PR2 probing a Coke bottle. The incompletepointcloud due to the transparency and reflection is filled bythe pretouch sensor.

B. Grasp Planning and Execution

The grasp planner used in this paper relies solely onthe observed pointcloud of an object versus fitting objectmodels. The observed pointcloud can be either with thepretouch points added or not. The algorithms presented in

Fig. 7. (a) PR2 robot probing the unseen area of the bottle and addingpretouch pointcloud before grasp planning. (b) The 3-D visualization inrviz. The red points are generated from the narrow stereo camera , andthe yellow points are generated by the pretouch sensor during the pretouchmotion. (c) The view from the left-narrow stereo camera overlayed with thevisualization markers.

detail in [12]. This planner uses a set of heuristics, such ashand alignment with the cluster bounding box, size of thebounding box compared to gripper aperture, and number ofobserved points inside the gripper, to assess the quality ofgrasp, and also to populate a list of grasp candidates.

When the pointcloud generated by the pretouch sensor isprovided, it is concatenated with the pointcloud from thecameras before being fed to the grasp planner. If pretouchpointcloud is not provided, only the pre-stored pointcloudfrom camera will be used to do the planning. The graspcandidate with the highest quality is used to execute theobject grasping. After collision map processing, the robotwill try to grasp the object in the planned pose.

C. Experiment and Results

In order to understand how the additional pointcloudadded by the pretouch sensor can affect the planned grasps,object grasping experiments are performed on 4 objects bothwith and without using the pretouch-assisted grasp planningapproach. Figure 8 shows the 3D visualization in rviz, theimage seen from the narrow stereo camera on the PR2sensor head overlayed with the visualization markers, andthe planned grasp pose in the real world.

(a) Coke bottle: Without pretouch sensor, the narrow-viewstereo camera only captures the pointcloud of the wrapperdue to the transparency and the reflection of the light. Alsothe right and back sides of the red wrapper is missing becauseof self-occlusion. Since the grasp is planned only using thepointcloud from the vision (red), the robot do not knowthe complete shape of the bottle, thus the planned grasp isoff-centered. This particular grasp failed becuase the robottries to grasp the bottle from the inclined portion of thebottle. After adding pretouch pointcloud (yellow points), thefitted bounding box now matches the actual shape of thebottle better. The final grasp planned using the concatenatedpointcloud is centered and thus successful. This exampleshows the pretouch pointcloud is useful for grasp planningwhen the object is self-occluded due to the view of camera.

(b) Lego blocks: The robot intends to grasp the yellowblock on the table. In this configuration, ideally the a front-sided grasp should be more safe and reliable. However, thereis another block in front of the target, which prevents anyfront grasp. When the robot is forced to do a side or overheadgrasp, the lack of information on the backside of the objectmay result in the wrong decision. such as in this example, therobot decides to grasp from the top and it locates the gripperwithout knowing the bumps in the back, thus hits the bumpduring the grasp execution. With the help of the pretouchpointcloud, the robot fingers are located on the side wallsin the overhead grasp, which avoids dealing with the bumpybackside. In this example, the depth sensor works properly,but due to the collision constraints, the additional informationprovided by the pretouch sensor plays an important roll ingrasp planning.

(c) milk box: Due to the particular inclined shape andthe view of the camera, the entire back side of the milkbox is occluded. Similar to the case of lego block, there arechances that the planned grasp will collide with the backsideof the object without knowing there are things on the back.The added pretouch pointcloud again helps plan a centeredgrasping pose in this case.

(d) Snapple bottle: The transparency and reflection of theplastic material cause the missing information on most of theportions except the wrapper on the bottle. Also the openingof the bottle is outside of the camera’s view, so the initiallyplanned overhead grasp is off-centered. The pretouch sensoradds additional useful information and shift the graspingpose to the center, which makes the grasp sucessful. Thisexample demonstrates the pretouch pointcloud is useful whenthe object is self-occluded and partially out of the view.

We demonstrate general applicability of the approach onsome critical cases that the pretouch pointcloud becomes use-ful to correct the grasp planning by reducing the uncertaintyin object’s geometry in the cases of occlusion, depth sensorfailure, or when in constrained environment.

V. CONCLUSIONS AND FUTURE WORK

A. Conclusions

This paper demonstrates a novel acoustic pretouch modal-ity for robotic grasping inspired by the well-known seashell

(b)(a)

(c) (d)

Pointcloud from camera Pointcloud from pretouch sensor

Vision Only Vision + Pretouch Vision Only Vision + Pretouch

3-DVisualization

CameraView

Real WorldGrasp

3-DVisualization

CameraView

Real WorldGrasp

Fig. 8. Examples of pretouch-assisted grasping. Compare the planned grasp with and without the additional pointcloud generated using the pretouchsensor on 4 objects: (a) Coke bottle, (b) Lego blocks, (c) milk box, (d) Snapple bottle. For each object, the first column shows the results with visiononly, and the second column shows results with the additional pointcloud from pretouch sensor. The red pointcloud are from stereo camera, the yellowpointcloud are from pretouch sensor, the green block is the fitted bounding box given the pointcloud, and the gripper model shows the final planned graspto be excuted. In (a) and (d), part of the incomplete pointcloud in the transparent portions is filled by the pretouch sensor. In (b) and (d), part of theincomplete pointcloud in the backside is filled by pretouch sensor.

effect. As far as we know, this effect has not previouslybeen used to build proximity sensors. Prototypes basedon this priciple were built and installed on PR2 gripper.We demonstrate that this technique is useful for sensingextremely compliant objects, which can not be detectedby the standard pressure sensors on the PR2 fingertip. Wealso demonstrate that using pretouch to add points to apointcloud collected by a depth camera can enable improvedgrasp plans. Seashell effect pretouch can be used to collectspatial information in regions where optical depth sensinghas failed. Unlike existing pretouch modalities, seashelleffect pretouch complements optical depth sensing becausethey rely on fundamentally different physical mechanisms:mechanical/acoustic in the case of seashell effect pretouch,and electromagnetic/optical in the case of depth cameras,electric field pretouch, and optical pretouch.

Videos of both pretouch-assisted grasp planning and reac-tive grasping applications can be seen at:

http://sensor.cs.washington.edu/pretouch/icra12/

B. Future Work

It is natural to wonder whether seashell effect pretouchcould be improved by actively generating sound. It is likelythat the answer is yes, but the passive scheme reportedhere works well enough for the applications we describe,and avoiding active audio generation allows for apparatusthat is simpler, smaller, and therefore more easily integratedinto robot fingers. Also, our current implementation operateswithin the band of human hearing, so an active scheme mightbe annoying to humans.

The integration of the seashell effect pretouch fingertip tothe existing SPI interface on the PR2 gripper is in progress.Our next generation system will have seashell effect sensorson both fingers without external wires, and have embeddedsignal processing capability. (In our initial implementation,an external cable to the sensor is used, and only one fingerhas the pretouch sensor.) Once the pretouch sensors are well-integrated into the robot, we plan to make them available toother PR2 users.

With the sensors in both fingers, we will be able toapply seashell effect pretouch to actually grasp extremelycompliant objects (taking the next step beyond the pre-shaping demonstrated here). This will require moving theentire arm to align the two fingers with the object.

We also plan to experiment with multiple sensors onthe same finger, to acquire surface orientation informationin addition to the distance information we currently have.This will facilitate a new mode of pretouch pointcloudcollection, in which the hand moves continuously, servoingto the pretouch data. This is expected to be more efficientthan the current discrete pointcloud sampling scheme. Also,a continuously executed pretouch servoing scheme can bereadily integrated with grasp control / execution: the graspcontroller would try to keep the fingers near the object dur-ing grasp execution, making pretouch measurements duringgrasp execution. This could allow both collection of new

pointcloud information during tentative grasping motions,and could also be used to falsify or invalidate the currentpointcloud during the normal reach motions in the case ofobject motion or depth camera error. For example, if thepretouch sensor detects unexpected obstacles during graspexecution, the grasp could either be aborted or modified. Inthe first case, the existing pointcloud would be discarded, anew pointcloud would be captured with the depth camera andpretouch, and a new grasp would be planned. In the secondcase, the points newly detected by pretouch would be addedto the existing pointcloud, and realtime grasp re-planningwould occur.

A further extension of this work would be to also make useof the 2-D image from the PR2’s stereo camera and forearmcamera. Even in regions where depth sensing has failed, the2-D images might provide valuable suggestions for usefultrajectories for pretouch exploration.

Finally, pretouch-assisted depth perception could also beused to obtain an object’s complete 3-D model, or used forobject recognition before grasping.

REFERENCES

[1] B. Mayton, L. LeGrand, J.R. Smith, An Electric field pretouch systemfor grasping and co-manipulation, in ICRA, 2010.

[2] R. Wistort, J.R. Smith, Electric field servoing for robotic manipulation,in IROS, 2008.

[3] K. Hsiao, P. Nangeroni, M. Huber, A. Saxena, A. Ng, ReactiveGrasping Using Optical Proximity Sensors, in ICRA, 2009.

[4] R. Platt, A.H. Fagg, R. Grupen, Null space grasp control: Theory andexperiments, in IEEE Transactions on Robotics, vol. 26(2), pp. 282-295, 2010.

[5] A. Petrovskaya, O. Khatib, S. Thrun, A.Y. Ng, Bayesian estimationfor autonomous object manipulation based on tactile sensors, in ICRA,2006.

[6] A. Petrovskaya, O. Khatib, Global Localization of Objects via Touch,in IEEE Transactions on Robotics, vol. 27(3), pp. 569-585, 2011.

[7] http://www.msxbox-world.com/xbox360/kinect/faqs/305/kinect-technical-specifications.html

[8] This figure is from our own experimental evaluation of the PR2’snarrow stereo camera.

[9] http://www.hokuyo-aut.jp/02sensor/07scanner/utm_30lx.html

[10] http://www.parallax.com/tabid/768/ProductID/92/Default.aspx

[11] D.H. Ashmead, R.S. Wall, Auditory perception of walls via spectralvariations in the ambient sound field, in J. Vis. Impair. Blindn., vol.9, pp. 615, 1998.

[12] K. Hsiao, S. Chitta, M. Ciocarlie, E.G. Jones, Contact-ReactiveGrasping of Objects with Partial Shape Information, in ICRA, 2010.

[13] K. Hsiao , M. Ciocarlie, P. Brook, Bayesian Grasp Planning, in ICRA,2011.

[14] P. Brook, M Ciocarlie, K. Hsiao, Collaborative Grasp Planning withMultiple Object Representations, in ICRA, 2011.

[15] J.M. Romano, K. Hsiao, G. Niemeyer, S. Chitta, K.J. Kuchenbecker,Human-Inspired Robotic Grasp Control with Tactile Sensing, in IEEETransaction on Robotics, 2011.

[16] J.P. Dalmont, C.J. Nederveen, N. Joly, Radiation impedance of tubeswith different flanges: numerical and experimental investigations, inJ. Sound and Vibration, vol. 244(3), pp. 505-534, 2001.

[17] R.B. Rusu, S. Cousins, 3D is here: Point Cloud Library (PCL), inICRA, 2011.

[18] E. Jacobsen, P. Kootsookos, Fast, Accurate Frequency Estimators, inIEEE Signal Processing Magazine, vol. 24(3), pp. 123-125, 2007.

[19] R.E. Kalman, A New Approach to Linear Filtering and PredictionProblems, in Journal of Basic Engineering, vol. 82(D), pp. 35-45,1960.