Self-Repair and Self-Extension by Tightening Screws based onPrecise Calculation of Screw Pose of Self-Body with CAD Data

and Graph Search with Regrasping a Driver

Takayuki Murooka Kei Okada and Masayuki Inaba

Abstractmdash In this paper we propose methods for tighteningscrews of self-body using a driver which enable self-repair andself-extension There are two difficulties for tightening screwsof self-body First the precise calculation of the screw pose isneeded When calculation with visual images using a camerathe observation error is so high The merit of the robot isthat the robot has CAD data of self-body There we calculatethe precise screw pose with self CAD data Second becauseof the small closed links when tightening screws of self-bodythat the robot cannot move the driver for rotating around thescrew sometimes happens because inverse kinematics cannot besolved To solve this problem we propose a method of tighteningmotion generation with regrasping a driver if inverse kinematicscannot be solved With these methods humanoid robots PR2and HIRO realized self-repair and self-extension by tighteningscrews of self-body

I INTRODUCTION

Humanoid robots are expected to work continuously in thehome environment or disaster sites However the hardware ofthe robot sometimes breaks and needs to be repaired So it isessential for humanoid robots to obtain self-repair methodsWe human can become aware of our own abnormalitieswith visual information plan repairing and execute thatplan using appropriate tools if needed Like this we wanthumanoid robots to repair self-body by oneself We comeup with loose screws of the robot which occurs sometimesand needs to be repaired In this study we focus on the robottightening loose screws by oneself as self-repair In additionthe robot can realize self-extension by tightening screws sowe also focus on self-extension

One of the advantages of robots including humanoidrobots is that they have their own softwarehardware designinformation This time by using its own CAD data whichdescribe hardware information the robot realizes calculatingthe screw pose more precisely than vision-based calculationAs a format of CAD data STEP file is widely used as anintermediate file so we use this format We extract each partfrom STEP file using FreeCAD [1]

In the case of tightening the screw of an external objectrather than the robot itself the robot can move relatively tothe object whereas in the case of tightening the screw of self-body the solvability of inverse kinematics is considerablylimited because of the small closed links when tighteningthe screw of self-body So in this research when inverse

T Murooka K Okada and M Inaba are with Department of Mechano-Infomatics The University of Tokyo 7-3-1 Hongo Bunkyo-ku Tokyo 113-8656 Japan t-murooka at jskimiiu-tokyoacjp

Fig 1 The robot tightens the screw of self-body

kinematics can not be solved it is made to be able to solveinverse kinematics by changing the grasping pose withoutchanging the pose of the tool This we call regraspingCombining these methods we aim that robots tighteningscrews of self-body without any help

II RELATED WORKS AND PROPOSED SYSTEM

The topics about self-repair self-heal or self-fix have beenwidely researched There are modular robots which canreconfigure by oneself [2] [3] [4] However there are limitsof strength precision and mechanical and electrical stabilitywith modular robots Other solutions of the self-repair systemlike mechanism improvement to the screw itself [5] [6] orSH (self-healing) soft materials [7] [8] were developed

However it is hard to incorporate these mechanisms ormaterials into the existing robots because it needs much costand efforts to remodel So it is better for robots to repairself-body using hands like human especially if the robot isa humanoid robot About the research of tightening screwswith hands the system which enables tightening screw witha robot arm [9] or with human-robot interaction [10] weredeveloped but in this case a driver is fixed with the robot

The problem of the task of tightening screws is thatthe precise information about the screw pose is neededIn the task which needs precise information like tighteningscrews or peg-in-hole the method using a force-torque sensorwas proposed [11] [12] or vision-based method was alsoproposed [13] [14] However calculating the screw posedirectly from a force-torque sensor is of course impossibleso the planner to insert the screw or the peg is needed andthe vision-based system depends on the performance of thecamera of the robot and that camera is not always preciseenough to calculate the precise screw pose

Real Robot

RGBD Camera amp Microphone

(1) Tighten Screw for Self-Repair (2) Tighten Screw for Self-Extension

Attach a hook

(A) Examples of Perception of Loose Screw

Self Point Cloud Extractor

People Pose Estimator

Abnormal Points Detector

Voice Recognizer

Pointed Position Calculator

(B) Screw Pose Calculator (C) Motion Generator with RegraspRobot CAD

Voice

TextldquoThis screw is looserdquo

Robot CAD

119953 119904119888119903119890119908119887119886119904119890

119953 119904119888119903119890119908119903119900119906119892ℎ

Robot CAD

Mask Image

Robot Point Cloud

120637

Fig 2 Overview of tightening screws system

A Overview of Tightening Screws System

We show the overview of the tightening screw system inFig 2 The robot thinks about taking up tightening screws ofself-body with some triggers For example the robot checkswhether there is any abnormality of self-body as a dailycheck or perceives the loose screws by the pose of linksof the robot which is out of ideal pose or by other peoplersquosindication from the abnormality Then from the informationabout that trigger the robot calculates the position of thescrew to tighten roughly (A) With self CAD data therobot calculates the precise screw pose to tighten (B) Aftercalculating the precise screw pose the robot generates themotion of tightening with regrasping a driver if it is neededto tighten (C)

III PERCEPTION OF LOOSE SCREW

In this section we will show two examples of perceivingthe loose screws One is that the robot perceives the loosescrew by oneself and another is that the robot perceivesthe loose screw by other peoplersquos indication with interactivecommunication

A Perception by Oneself

The robot can calculate the pose of each link of the robotby calculating joint angles with joint encoders One exampleof the bad result caused by loose screws which is muchvisible is that the pose of a link is out of ideal pose likeFig 3 In detail applying a mask image of self-body whoseblack region means the robot and whose white region meansothers to a RGBD image observed with the RGBD camerathe RGBD image which describes only the robot can becalculated Then the robot reconstructs point cloud from thatRGBD image then the robot can get the point cloud of self-body If there are some points out of ideal the pose of alink is decided as out of ideal That means there may be theloose screw so this information is sent to the next step oflsquoScrew Pose Calculatorrsquo in Fig 2

B Perception by Other Peoplersquos Indication

Of course the robot cannot perceive all loose screwsby oneself so it is important to be taught that there is aloose screw by other peoplersquos indication with interactivecommunication like Fig 4 In this study we can teach theloose screw position to the robot by pointing with a finger

If the pose of a link is

out of ideal hellip

Fig 3 The robot perceives the loose screw of self-body byoneself Yellow points are points observed with the RGBDcamera and not far from the geometric model of the robotwhich means not abnormal Red points are points observedwith the RGBD camera and far from the geometric modelof the robot which means abnormal If there is a link out ofideal the red points appears like the right figure

and the voice like lsquoThere is a loose screwrsquo We implementedcalculating people pose with OpenPose [15] Then the robotperceives that there is a loose screw and calculates theposition of the loose screw roughly with the tip of the fingerposition of pointing The robot sends this information to thenext step of lsquoScrew Pose Calculatorrsquo in Fig 2

pointing and voice

Fig 4 The robot perceives the loose screw by pointed outwith a voice by people

IV CALCULATION OF SCREW POSE

A Screw Judgement

We aim to detect the screw which is closest to thedesignated position which is calculated with Sec III At firstwe describe an algorithm to judge whether the part whichconstructs the robot is a screw or not in Alg 1 Each part inthe STEP file is expressed in units of shell and each shellhas information of edges and vertices that constitute the part

We describe functions in Alg 1 in the following lmax lminare shown in Fig 6

Algorithm 1 Judging Whether Shell is Screw or not1 function JUDGEWHETHERSCREW(shell)2 Clarr []3 for edge in shellEdges do4 if IsCircle(edge) then5 radiuslarr calcRadiusO fCircle(edge)6 dlarr calcDirectionO fCircle(edge)7 pcenter larr calcCenterO fCircle(edge)8 circle in f olarr (radiusdpcenter)9 push(circle in f oC)

10 end if11 end for12 if Clength lt threshold1 then13 return False14 end if15 if calcConcentricMaxRatio(C)lt threshold2 then16 return False17 end if18 Cconcentriclarr calcConcentricCircles(C)19 lmaxlarr calcMaxLength(Cconcentric)20 lminlarr calcMinLength(Cconcentric)21 if lmaxlmin lt threshold3 then22 return True23 else24 return False25 end if26 end function

bull isCircle(edge) This function judges whether the edge is circle or notEach edge has that information

bull calcRadiusO fCircle(edge) This function calculates the radius of thecircular edge

bull calcDirectionO fCircle(edge) This function calculates the verticaldirection of the circular edge

bull calcCenterO fCircle(edge) This function calculates the center posi-tion of the circular edge

bull calcConcentricMaxRatio(list) This function calculates the max ratioof the circles which are concentric each other in all the circles in list

bull calcConcentricCircles(list) This function calculates the circles whichare concentric each other and the ratio of the circles in all the circlesis maximum in list

bull calcMaxLength(list) This function calculates the most far distanceof the centers of the circular edges whose radius is maximum in allthe circles in list

bull calcMinLength(list) This function calculates the most far distance ofthe centers of the circular edges whose radius is minimum in all thecircles in list

Using this screw judgement algorithm we extractedscrews from three parts UR10 (Universal Robot 10)1 theleft shoulder of PR2 and the base parts we designed Theresults are shown in Fig 5 All results show that the screw iscorrectly extracted Some thresholds in Alg 1 must be tunedaccording to the characteristic of screws of the parts but wecan extract screws of Fig 5 with the same thresholds Atlast the closest screw among screws extracted with Alg 1to the designated points of Sec III we determine the targetscrew to tighten

B Calculation of Screw Pose

The flow of calculation of the screw pose is shown inFig 7 Now we aim to find the screw pose with the leftshoulder of the robot pb

a describes the pose of a in thecoordinate system of b and bTa describes the translation

1httpswwwuniversal-robotscomproductsur10-robot

Fig 5 Extracting the screws of UR10 the left shoulder ofPR2 and the base parts we designed from left to right withAlg 1 Red parts are screws and blue parts are others

119949119950119938119961 119949119950119946119951

119955119950119946119951119955119950119938119961

Fig 6 Information about screw

matrix from the coordinate system of b to the coordinatesystem of a The robot calculates baseTlink with each jointangle and the geometric model of the robot and alsocalculates linkTscrew with d and pcenter of the target screwcalculated by Alg 1 We can get pbase

screw by calculation ofbaseTscrew =base Tlink

linkTscrew

119953 119897119894119899119896

119953 119897119894119899119896

119953 119887119886119904119890

Screw119953 119904119888119903119890119908

Fig 7 Calculating the target screw pose from the coordinatesystem of the base link

V MOTION GENERATION OF TIGHTENING SCREWS WITHREGRASPING

A Inverse Kinematics Considering a Driver and a Screw

After grasping the driver inverse kinematics is calculatedso that the pose of the tip of the driver and the pose of thescrew head will match Assuming that the pose of the tipof the driver is rdriver and the Jacobian of the robot linkrelated to the movement of the pose of the tip of the driver(rdriver) is Jdriver Similarly assuming that the pose of thescrew head is rscrew and the Jacobian of the robot link relatedto the movement of the pose of the screw head (rscrew) isJscrew We can solve whole-body inverse kinematics whichcombines the robot links of not only a sole arm but also botharms with a driver and a screw head by iterative calculationby using ∆θ as Eq (1) The result of inverse kinematics isdescribed in Fig 8

∆rdriver = Jdriver∆θdriver∆rscrew = Jscrew∆θscrew

(1)

Suppose ∆rdriver∆rscrew as follows∆rdriver = rscrewminusrdriver∆rscrew = rdriverminusrscrew

(2)

IK

119953 119904119888119903119890119908119953 119889119903119894119907119890119903

120637 119941119955119946119959119942119955

119921 119941119955119946119959119942119955

120637 119956119940119955119942119960

119921 119956119940119955119942119960

Fig 8 Whole-body inverse kinematics considering thedriver and the screw

119891ℎ119886119899119889

119898 ℎ119886119899119889

119899 119904119888119903119890119908 119899 119904119888119903119890119908

RotationalRregrasp

feasible

119891ℎ119886119899119889

119898 ℎ119886119899119889

119899 119904119888119903119890119908 119899 119904119888119903119890119908

TranslationalRegrasp

infeasible

driverdriver

screw screw

handhand

Driver detachesfrom screw

x

y

Fig 9 Example of regrasping a driver which is feasibleand infeasible The left figure is rotational regrasping andfeasible The right figure is translational regrasping andinfeasible which causes detaching from the screw

B Regrasping of Driver

In this subsection we aim to judge the regrasping isfeasible or not like Fig 9 We describe changing graspingpose while the robot grasps the driver with the constraintthat the driver does not move on a two-dimensional planeof Fig 9 as regrasping a driver This regrasping is used bymotion generation of tightening a screw by graph search inthe next subsection We assume regrasping motion is limitedto within two-dimensional plane like Fig 9 The force andmoment applied to the driver from the robotrsquos hand are fhandand mhand The force and moment applied to the driver fromthe screw are fscrew and mscrew The position vector fromthe screw to the grasping points is rhand nscrew describesthe unit vector of the direction of the screw Then we willjudge regrasping is feasible or infeasible We assume theconstraint that the driver does not move like Eq (3) withapproximated physical model When the driver contacts thescrew the situation that the direction of fhand is opposite tonscrew is hard to think unless the static friction of the driverand the screw is satisfied mmax is a limit of moment appliedto the screw head of static friction microscrew is the translationalcoefficient between the screw and the driver The coordinatesystem is like Fig 9

fscrew middotnscrew gt 0 or microscrew|fscrewx|gt |fscrewy||mscrew|lt mmax

(3)

Then we will judge the regrasping is feasible or not whenregrasping is translational or rotational We donrsquot considerthe regrasping which combines translational regrasping androtational regrasping The balance of force and moment ofthe driver is below

fhand +fscrew =Omscrew +mhand +(rhandxfhandyminusrhandyfhandx) = 0 (4)

When using the robot which has a force-torque sensor onthe robotrsquos limbs the robot can judge whether regrasping isfeasible or not by Eq (3) Eq (4) while executing regraspingbecause the robot knows fhand and mhand However the robotwhich doesnrsquot have a force-torque sensor cannot judge feasi-bility only with Eq (3) Eq (4) So we consider constraintsfor the robot which doesnrsquot have a force-torque sensor withsome approximation in the following

1) When Translational Regrasping To convert the con-straint of force and moment to the constraint of positionand orientation we formulate below equations microt is thetranslational coefficient of static friction between the robotrsquoshand and the driver elowast describes the unit vector of lowastvector ∆rpos is a two-dimensional unit vector describing thedirection of translational regrasping movement N is force ofgrasping a driver

|fhand |= microt Nefhand = e∆rpos

(5)

Then we apply strong approximation obtained empiricallyWhen regrasping motion is only translational we assumethat mhand equals zero So we can formulate fscrewmscrewwith the pose of the robotrsquos hand information by combiningEq (3) Eq (4) Eq (5) N is assumed to be known

2) When Rotational Regrasping To convert the constraintof force and moment to the constraint of position andorientation we formulate below equation micror is the rotationalcoefficient of static friction ∆rori is a value whose absolutevalue is 1 describing the direction of rotational regraspingmovement sgn(val) returns sign of val

|mhand |= microrNsgn(mhand) = sgn(∆rori)

(6)

Then we apply strong approximation obtained empiricallyWhen regrasping motion is only rotational we assume thatmscrew equals zero So we can formulate fscrewmscrew withthe pose of the robotrsquos hand information by combiningEq (3) Eq (4) Eq (6) N is assumed to be known

To verify the approximate constraint described upper weconducted experiments of regrasping with PR2 like Fig 10The robot which doesnrsquot have a force-torque sensor didtranslational regrasping and rotational regrasping which isconsidered to be feasible according to the approximateconstraint 20 times respectively The direction of regraspingis random The movement of translational regrasping is 1cmand the rotation angle of rotational regrasping is 15 degreeThe result of the experiment is shown in TabI Successmeans that regrasping is realized without the driver movingIt is considered that the approximation that mscrew equalszero is strict

This regrasping is not needed if the axis of rotation of therobotrsquos grasping hand and the axis of rotation of the driveris on the same line If that matching is difficult because ofthe less solvability of inverse kinematics of the small closedlinks this regrasping is effective

TABLE I Result of experiment of regrasping

translational regrasp rotational regraspsuccess 18 15fail 2 5total 20 20success rate 09 075

Feasible Regrasping Infeasible Regrasping

Driver detached from the screw

Driver moved(rotated)

Fig 10 The two figures of the left side are feasible regrasp-ing which the driver doesnrsquot move while regrasping The twofigures of the right side are infeasible regrasping which thedriver rotated or the driver detached from the screw

C Motion Generation of Tightening Screw by Graph Search

We aim to generate motion of tightening a screw Becauseof the small closed link when tightening screws of self-bodythat the robot cannot move the driver to rotate around thescrew sometimes happens That means inverse kinematicsof Sec V-A cannot be solved To solve this problem weconsider inverse kinematics with the regrasping with someconstraints described in Sec V-B

First the rotation of the driver is advanced from the initialrotation angle of the driver (ϕstart ) If inverse kinematicscannot be solved the robot searches the next grasping posewith the constraints of Sec V-B If inverse kinematics canbe solved with the next grasping pose the rotation of thedriver is advanced and if not the robot searches the nextgrasping pose This search is executed as a depth-first searchconsidering two kinds of infeasibility inverse kinematics andregrasping We show this search in Fig 11 and the algorithm

119944 0

φ 119904119905119886119903119905

119944 1

φ 119892119900119886119897

119944

φIK infeasible

Regraspinfeasible

119944 119899

Fig 11 Graph search of tightening motion A vertical axisdescribes the grasping pose of the driver and a horizontalaxis describes the rotation angle of the driver of tighteningmotion Blue nodes are feasible to transit and red nodes areinfeasible to transit With regrasping the rotation angle isrealized to advance from ϕstart to ϕgoal

We show the examples of motion of regrasping a driverin Fig 12The robot succeeded rotating a driver one roundwith regrasping twice

regrasped

① ② ③ ④ ⑤

Fig 12 Tightening a screw with regrasping a driver Therobot does regrasping from 3⃝ to 4⃝

VI EXPERIMENTS OF SELF-REPAIR ANDSELF-EXTENSION

With the proposed system shown in Fig 2 we conductedexperiments of self-repair and self-extension with a real life-size humanoid robot using a driver designed for human

A Self-repair as Daily Check by HIRO

① ② ③ ④

Grasping a driver

Fig 13 HIRO tightened a screw of self-body as a dailycheck

The humanoid robot HIRO executes self-repair as a dailycheck HIRO calculated a screw pose with self CAD data andmoved a proper driver to the screw then tightened Shown inFig 13 HIRO succeeded in tightening a screw of self-body

B Self-repair by Other Peoplersquos Indication by PR2

① ② ③

④ ⑤ ⑥

Grasping a driver

Regrasped

Fig 14 PR2 perceived a loose screw by other peoplersquosindication and executed tightening the screw with a properdriver PR2 executed regrasping from 3⃝ to 4⃝

The humanoid robot PR2 executes self-repair by otherrsquosindication A people noticed the loose screw of PR2 and saidlsquoThis screw is loosersquo Then PR2 perceived the loose screwcalculated the rough position of the screw and also calculatedprecise the screw pose with self CAD data After that PR2started tightening the screw with a proper driver We showthe snapshots of the two examples when PR2 tightened ascrew of self-body with regrasping in Fig 14

① ② ③ ④ ⑤

Grasping a driver

Grasping a hook

⑥Attach

the hook

Fig 15 PR2 wants something which can contain objects then attaches the hook to his body and hang a bag PR2 startedtightening the screw ( 2⃝) then the people put a lot of cans in the bag ( 4⃝) and put the bag on PR2rsquos shoulder ( 5⃝) NowPR2 can have a lot of cans with the bag on the shoulder without PR2rsquos hand that means PR2 can manipulate various tasks

C Self-extension by Attaching the Hook by PR2

When PR2 wants to have a lot of things the only twohands are not enough to realize that So we let PR2 to usea bag the same as we put it on our shoulder PR2 startedattaching the hook whose pose is calculated with self CADdata with a driver on his shoulder in order to put a bag on hisshoulder PR2 finished attaching the hook and the people puta lot of cans in a tote bag and put it on PR2rsquos shoulder Asshown in Fig 15 PR2 realized using a bag like us human

VII CONCLUSION

This paper dealt with self-repair and self-extension bytightening screws of self-body In conclusion we proposebelow ideas and methodsbull We proposed an idea of self-repair and self-extension

system by tightening self-screws for humanoid robotsbull We proposed a method of calculating the precise screw

pose with self CAD databull We proposed a method of judging feasible regrasping

of a driver in order to solve inverse kinematics of thesmall closed links when tightening a screw

bull We proposed a method of generating tightening motionwith graph search considering regrasping a driver

In order to verify the methods we did some experiments ofself-repair and self-extension with a real humanoid robot Asa future work to make this system more general we want toestablish the unified method of perceiving loose screws Alsomanaging the tendency of loose screws is important Whenthe robot tightens screws of self-body the robot memorizesit By keeping storing the information the robot knows whichscrews tend to loosen and which not In addition the error ofgrasping pose when grasping the driver is so high because theactual grasping pose and the reference grasping pose is oftendifferent which we ignored in this paper So the methodto recalculate the grasping pose by observing the pose ofthe driver to the robotrsquos hand after grasping the driver isneeded The tightening system including those methods willhelp humanoid robots

REFERENCES

[1] R Juergen M Werner and H Yorik van ldquoFreecad (version0166712)rdquo [Online] Available [Software]Availablefromhttpwwwfreecadweborg

[2] M Yim W-M Shen B Salemi D Rus M Moll H LipsonE Klavins and G S Chirikjian ldquoModular self-reconfigurable robotsystems [grand challenges of robotics]rdquo IEEE Robotics amp AutomationMagazine vol 14 no 1 pp 43ndash52 2007

[3] M Yim D G Duff and K D Roufas ldquoPolybot a modular recon-figurable robotrdquo in ICRA 2000 pp 514ndash520

[4] B Salemi M Moll and W-M Shen ldquoSuperbot A deployable multi-functional and modular self-reconfigurable robotic systemrdquo in 2006IEEERSJ International Conference on Intelligent Robots and Systems2006 pp 3636ndash3641

[5] L A Mateos and M Vincze ldquoLammos-latching mechanism based onmotorized-screw for reconfigurable robotsrdquo in 2013 16th InternationalConference on Advanced Robotics (ICAR) 2013 pp 1ndash8

[6] G Park and D Inman ldquoSmart bolts an example of self-healingstructuresrdquo Smart Materials Bulletin vol 2001 no 7 pp 5ndash8 2001

[7] S Terryn J Brancart D Lefeber G Van Assche and B Vander-borght ldquoSelf-healing soft pneumatic robotsrdquo Science Robotics vol 2no 9 p eaan4268 2017

[8] S Nakashima T Shirai Y Asano Y Kakiuchi K Okada and M In-aba ldquoResistance-based self-sensing system of active self-melting bolttowards autonomous healing structurerdquo in 2018 IEEE InternationalConference on Soft Robotics (RoboSoft) 2018 pp 88ndash93

[9] A Ono and S Fukumoto ldquoTightening systemrdquo Mar 28 2017 uSPatent 9604329

[10] A Cherubini R Passama P Fraisse and A Crosnier ldquoA unifiedmultimodal control framework for humanndashrobot interactionrdquo Roboticsand Autonomous Systems vol 70 pp 106ndash115 2015

[11] C H Kim and J Seo ldquoShallow-depth insertion Peg in shallow holethrough robotic in-hand manipulationrdquo IEEE Robotics and AutomationLetters vol 4 no 2 pp 383ndash390 2019

[12] B Lara K Althoefer and L D Seneviratne ldquoAutomated robot-based screw insertion systemrdquo in IECONrsquo98 Proceedings of the 24thAnnual Conference of the IEEE Industrial Electronics Society (CatNo 98CH36200) vol 4 1998 pp 2440ndash2445

[13] H H Chen ldquoA screw motion approach to uniqueness analysis ofhead-eye geometryrdquo in Proceedings 1991 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition 1991 pp145ndash151

[14] M Yokomae Y Itsuzaki K Horikami K Okumura et al ldquoMethodof recognizing a screw hole and screwing method based on therecognitionrdquo Sept 19 2000 uS Patent 6122398

[15] Z Cao T Simon S-E Wei and Y Sheikh ldquoRealtime multi-person2d pose estimation using part affinity fieldsrdquo in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition 2017pp 7291ndash7299

Real Robot

RGBD Camera amp Microphone

(1) Tighten Screw for Self-Repair (2) Tighten Screw for Self-Extension

Attach a hook

(A) Examples of Perception of Loose Screw

Self Point Cloud Extractor

People Pose Estimator

Abnormal Points Detector

Voice Recognizer

Pointed Position Calculator

(B) Screw Pose Calculator (C) Motion Generator with RegraspRobot CAD

Voice

TextldquoThis screw is looserdquo

Robot CAD

119953 119904119888119903119890119908119887119886119904119890

119953 119904119888119903119890119908119903119900119906119892ℎ

Robot CAD

Mask Image

Robot Point Cloud

120637

Fig 2 Overview of tightening screws system

A Overview of Tightening Screws System

We show the overview of the tightening screw system inFig 2 The robot thinks about taking up tightening screws ofself-body with some triggers For example the robot checkswhether there is any abnormality of self-body as a dailycheck or perceives the loose screws by the pose of linksof the robot which is out of ideal pose or by other peoplersquosindication from the abnormality Then from the informationabout that trigger the robot calculates the position of thescrew to tighten roughly (A) With self CAD data therobot calculates the precise screw pose to tighten (B) Aftercalculating the precise screw pose the robot generates themotion of tightening with regrasping a driver if it is neededto tighten (C)

III PERCEPTION OF LOOSE SCREW

In this section we will show two examples of perceivingthe loose screws One is that the robot perceives the loosescrew by oneself and another is that the robot perceivesthe loose screw by other peoplersquos indication with interactivecommunication

A Perception by Oneself

The robot can calculate the pose of each link of the robotby calculating joint angles with joint encoders One exampleof the bad result caused by loose screws which is muchvisible is that the pose of a link is out of ideal pose likeFig 3 In detail applying a mask image of self-body whoseblack region means the robot and whose white region meansothers to a RGBD image observed with the RGBD camerathe RGBD image which describes only the robot can becalculated Then the robot reconstructs point cloud from thatRGBD image then the robot can get the point cloud of self-body If there are some points out of ideal the pose of alink is decided as out of ideal That means there may be theloose screw so this information is sent to the next step oflsquoScrew Pose Calculatorrsquo in Fig 2

B Perception by Other Peoplersquos Indication

Of course the robot cannot perceive all loose screwsby oneself so it is important to be taught that there is aloose screw by other peoplersquos indication with interactivecommunication like Fig 4 In this study we can teach theloose screw position to the robot by pointing with a finger

If the pose of a link is

out of ideal hellip

Fig 3 The robot perceives the loose screw of self-body byoneself Yellow points are points observed with the RGBDcamera and not far from the geometric model of the robotwhich means not abnormal Red points are points observedwith the RGBD camera and far from the geometric modelof the robot which means abnormal If there is a link out ofideal the red points appears like the right figure

and the voice like lsquoThere is a loose screwrsquo We implementedcalculating people pose with OpenPose [15] Then the robotperceives that there is a loose screw and calculates theposition of the loose screw roughly with the tip of the fingerposition of pointing The robot sends this information to thenext step of lsquoScrew Pose Calculatorrsquo in Fig 2

pointing and voice

Fig 4 The robot perceives the loose screw by pointed outwith a voice by people

IV CALCULATION OF SCREW POSE

A Screw Judgement

We aim to detect the screw which is closest to thedesignated position which is calculated with Sec III At firstwe describe an algorithm to judge whether the part whichconstructs the robot is a screw or not in Alg 1 Each part inthe STEP file is expressed in units of shell and each shellhas information of edges and vertices that constitute the part

We describe functions in Alg 1 in the following lmax lminare shown in Fig 6

Algorithm 1 Judging Whether Shell is Screw or not1 function JUDGEWHETHERSCREW(shell)2 Clarr []3 for edge in shellEdges do4 if IsCircle(edge) then5 radiuslarr calcRadiusO fCircle(edge)6 dlarr calcDirectionO fCircle(edge)7 pcenter larr calcCenterO fCircle(edge)8 circle in f olarr (radiusdpcenter)9 push(circle in f oC)

10 end if11 end for12 if Clength lt threshold1 then13 return False14 end if15 if calcConcentricMaxRatio(C)lt threshold2 then16 return False17 end if18 Cconcentriclarr calcConcentricCircles(C)19 lmaxlarr calcMaxLength(Cconcentric)20 lminlarr calcMinLength(Cconcentric)21 if lmaxlmin lt threshold3 then22 return True23 else24 return False25 end if26 end function

bull isCircle(edge) This function judges whether the edge is circle or notEach edge has that information

bull calcRadiusO fCircle(edge) This function calculates the radius of thecircular edge

bull calcDirectionO fCircle(edge) This function calculates the verticaldirection of the circular edge

bull calcCenterO fCircle(edge) This function calculates the center posi-tion of the circular edge

bull calcConcentricMaxRatio(list) This function calculates the max ratioof the circles which are concentric each other in all the circles in list

bull calcConcentricCircles(list) This function calculates the circles whichare concentric each other and the ratio of the circles in all the circlesis maximum in list

bull calcMaxLength(list) This function calculates the most far distanceof the centers of the circular edges whose radius is maximum in allthe circles in list

bull calcMinLength(list) This function calculates the most far distance ofthe centers of the circular edges whose radius is minimum in all thecircles in list

Using this screw judgement algorithm we extractedscrews from three parts UR10 (Universal Robot 10)1 theleft shoulder of PR2 and the base parts we designed Theresults are shown in Fig 5 All results show that the screw iscorrectly extracted Some thresholds in Alg 1 must be tunedaccording to the characteristic of screws of the parts but wecan extract screws of Fig 5 with the same thresholds Atlast the closest screw among screws extracted with Alg 1to the designated points of Sec III we determine the targetscrew to tighten

B Calculation of Screw Pose

The flow of calculation of the screw pose is shown inFig 7 Now we aim to find the screw pose with the leftshoulder of the robot pb

a describes the pose of a in thecoordinate system of b and bTa describes the translation

1httpswwwuniversal-robotscomproductsur10-robot

Fig 5 Extracting the screws of UR10 the left shoulder ofPR2 and the base parts we designed from left to right withAlg 1 Red parts are screws and blue parts are others

119949119950119938119961 119949119950119946119951

119955119950119946119951119955119950119938119961

Fig 6 Information about screw

matrix from the coordinate system of b to the coordinatesystem of a The robot calculates baseTlink with each jointangle and the geometric model of the robot and alsocalculates linkTscrew with d and pcenter of the target screwcalculated by Alg 1 We can get pbase

screw by calculation ofbaseTscrew =base Tlink

linkTscrew

119953 119897119894119899119896

119953 119897119894119899119896

119953 119887119886119904119890

Screw119953 119904119888119903119890119908

Fig 7 Calculating the target screw pose from the coordinatesystem of the base link

V MOTION GENERATION OF TIGHTENING SCREWS WITHREGRASPING

A Inverse Kinematics Considering a Driver and a Screw

After grasping the driver inverse kinematics is calculatedso that the pose of the tip of the driver and the pose of thescrew head will match Assuming that the pose of the tipof the driver is rdriver and the Jacobian of the robot linkrelated to the movement of the pose of the tip of the driver(rdriver) is Jdriver Similarly assuming that the pose of thescrew head is rscrew and the Jacobian of the robot link relatedto the movement of the pose of the screw head (rscrew) isJscrew We can solve whole-body inverse kinematics whichcombines the robot links of not only a sole arm but also botharms with a driver and a screw head by iterative calculationby using ∆θ as Eq (1) The result of inverse kinematics isdescribed in Fig 8

∆rdriver = Jdriver∆θdriver∆rscrew = Jscrew∆θscrew

(1)

Suppose ∆rdriver∆rscrew as follows∆rdriver = rscrewminusrdriver∆rscrew = rdriverminusrscrew

(2)

IK

119953 119904119888119903119890119908119953 119889119903119894119907119890119903

120637 119941119955119946119959119942119955

119921 119941119955119946119959119942119955

120637 119956119940119955119942119960

119921 119956119940119955119942119960

Fig 8 Whole-body inverse kinematics considering thedriver and the screw

119891ℎ119886119899119889

119898 ℎ119886119899119889

119899 119904119888119903119890119908 119899 119904119888119903119890119908

RotationalRregrasp

feasible

119891ℎ119886119899119889

119898 ℎ119886119899119889

119899 119904119888119903119890119908 119899 119904119888119903119890119908

TranslationalRegrasp

infeasible

driverdriver

screw screw

handhand

Driver detachesfrom screw

x

y

Fig 9 Example of regrasping a driver which is feasibleand infeasible The left figure is rotational regrasping andfeasible The right figure is translational regrasping andinfeasible which causes detaching from the screw

B Regrasping of Driver

In this subsection we aim to judge the regrasping isfeasible or not like Fig 9 We describe changing graspingpose while the robot grasps the driver with the constraintthat the driver does not move on a two-dimensional planeof Fig 9 as regrasping a driver This regrasping is used bymotion generation of tightening a screw by graph search inthe next subsection We assume regrasping motion is limitedto within two-dimensional plane like Fig 9 The force andmoment applied to the driver from the robotrsquos hand are fhandand mhand The force and moment applied to the driver fromthe screw are fscrew and mscrew The position vector fromthe screw to the grasping points is rhand nscrew describesthe unit vector of the direction of the screw Then we willjudge regrasping is feasible or infeasible We assume theconstraint that the driver does not move like Eq (3) withapproximated physical model When the driver contacts thescrew the situation that the direction of fhand is opposite tonscrew is hard to think unless the static friction of the driverand the screw is satisfied mmax is a limit of moment appliedto the screw head of static friction microscrew is the translationalcoefficient between the screw and the driver The coordinatesystem is like Fig 9

fscrew middotnscrew gt 0 or microscrew|fscrewx|gt |fscrewy||mscrew|lt mmax

(3)

Then we will judge the regrasping is feasible or not whenregrasping is translational or rotational We donrsquot considerthe regrasping which combines translational regrasping androtational regrasping The balance of force and moment ofthe driver is below

fhand +fscrew =Omscrew +mhand +(rhandxfhandyminusrhandyfhandx) = 0 (4)

When using the robot which has a force-torque sensor onthe robotrsquos limbs the robot can judge whether regrasping isfeasible or not by Eq (3) Eq (4) while executing regraspingbecause the robot knows fhand and mhand However the robotwhich doesnrsquot have a force-torque sensor cannot judge feasi-bility only with Eq (3) Eq (4) So we consider constraintsfor the robot which doesnrsquot have a force-torque sensor withsome approximation in the following

1) When Translational Regrasping To convert the con-straint of force and moment to the constraint of positionand orientation we formulate below equations microt is thetranslational coefficient of static friction between the robotrsquoshand and the driver elowast describes the unit vector of lowastvector ∆rpos is a two-dimensional unit vector describing thedirection of translational regrasping movement N is force ofgrasping a driver

|fhand |= microt Nefhand = e∆rpos

(5)

Then we apply strong approximation obtained empiricallyWhen regrasping motion is only translational we assumethat mhand equals zero So we can formulate fscrewmscrewwith the pose of the robotrsquos hand information by combiningEq (3) Eq (4) Eq (5) N is assumed to be known

2) When Rotational Regrasping To convert the constraintof force and moment to the constraint of position andorientation we formulate below equation micror is the rotationalcoefficient of static friction ∆rori is a value whose absolutevalue is 1 describing the direction of rotational regraspingmovement sgn(val) returns sign of val

|mhand |= microrNsgn(mhand) = sgn(∆rori)

(6)

Then we apply strong approximation obtained empiricallyWhen regrasping motion is only rotational we assume thatmscrew equals zero So we can formulate fscrewmscrew withthe pose of the robotrsquos hand information by combiningEq (3) Eq (4) Eq (6) N is assumed to be known

To verify the approximate constraint described upper weconducted experiments of regrasping with PR2 like Fig 10The robot which doesnrsquot have a force-torque sensor didtranslational regrasping and rotational regrasping which isconsidered to be feasible according to the approximateconstraint 20 times respectively The direction of regraspingis random The movement of translational regrasping is 1cmand the rotation angle of rotational regrasping is 15 degreeThe result of the experiment is shown in TabI Successmeans that regrasping is realized without the driver movingIt is considered that the approximation that mscrew equalszero is strict

This regrasping is not needed if the axis of rotation of therobotrsquos grasping hand and the axis of rotation of the driveris on the same line If that matching is difficult because ofthe less solvability of inverse kinematics of the small closedlinks this regrasping is effective

TABLE I Result of experiment of regrasping

translational regrasp rotational regraspsuccess 18 15fail 2 5total 20 20success rate 09 075

Feasible Regrasping Infeasible Regrasping

Driver detached from the screw

Driver moved(rotated)

Fig 10 The two figures of the left side are feasible regrasp-ing which the driver doesnrsquot move while regrasping The twofigures of the right side are infeasible regrasping which thedriver rotated or the driver detached from the screw

C Motion Generation of Tightening Screw by Graph Search

We aim to generate motion of tightening a screw Becauseof the small closed link when tightening screws of self-bodythat the robot cannot move the driver to rotate around thescrew sometimes happens That means inverse kinematicsof Sec V-A cannot be solved To solve this problem weconsider inverse kinematics with the regrasping with someconstraints described in Sec V-B

First the rotation of the driver is advanced from the initialrotation angle of the driver (ϕstart ) If inverse kinematicscannot be solved the robot searches the next grasping posewith the constraints of Sec V-B If inverse kinematics canbe solved with the next grasping pose the rotation of thedriver is advanced and if not the robot searches the nextgrasping pose This search is executed as a depth-first searchconsidering two kinds of infeasibility inverse kinematics andregrasping We show this search in Fig 11 and the algorithm

119944 0

φ 119904119905119886119903119905

119944 1

φ 119892119900119886119897

119944

φIK infeasible

Regraspinfeasible

119944 119899

Fig 11 Graph search of tightening motion A vertical axisdescribes the grasping pose of the driver and a horizontalaxis describes the rotation angle of the driver of tighteningmotion Blue nodes are feasible to transit and red nodes areinfeasible to transit With regrasping the rotation angle isrealized to advance from ϕstart to ϕgoal

We show the examples of motion of regrasping a driverin Fig 12The robot succeeded rotating a driver one roundwith regrasping twice

regrasped

① ② ③ ④ ⑤

Fig 12 Tightening a screw with regrasping a driver Therobot does regrasping from 3⃝ to 4⃝

VI EXPERIMENTS OF SELF-REPAIR ANDSELF-EXTENSION

With the proposed system shown in Fig 2 we conductedexperiments of self-repair and self-extension with a real life-size humanoid robot using a driver designed for human

A Self-repair as Daily Check by HIRO

① ② ③ ④

Grasping a driver

Fig 13 HIRO tightened a screw of self-body as a dailycheck

The humanoid robot HIRO executes self-repair as a dailycheck HIRO calculated a screw pose with self CAD data andmoved a proper driver to the screw then tightened Shown inFig 13 HIRO succeeded in tightening a screw of self-body

B Self-repair by Other Peoplersquos Indication by PR2

① ② ③

④ ⑤ ⑥

Grasping a driver

Regrasped

Fig 14 PR2 perceived a loose screw by other peoplersquosindication and executed tightening the screw with a properdriver PR2 executed regrasping from 3⃝ to 4⃝

The humanoid robot PR2 executes self-repair by otherrsquosindication A people noticed the loose screw of PR2 and saidlsquoThis screw is loosersquo Then PR2 perceived the loose screwcalculated the rough position of the screw and also calculatedprecise the screw pose with self CAD data After that PR2started tightening the screw with a proper driver We showthe snapshots of the two examples when PR2 tightened ascrew of self-body with regrasping in Fig 14

① ② ③ ④ ⑤

Grasping a driver

Grasping a hook

⑥Attach

the hook

Fig 15 PR2 wants something which can contain objects then attaches the hook to his body and hang a bag PR2 startedtightening the screw ( 2⃝) then the people put a lot of cans in the bag ( 4⃝) and put the bag on PR2rsquos shoulder ( 5⃝) NowPR2 can have a lot of cans with the bag on the shoulder without PR2rsquos hand that means PR2 can manipulate various tasks

C Self-extension by Attaching the Hook by PR2

When PR2 wants to have a lot of things the only twohands are not enough to realize that So we let PR2 to usea bag the same as we put it on our shoulder PR2 startedattaching the hook whose pose is calculated with self CADdata with a driver on his shoulder in order to put a bag on hisshoulder PR2 finished attaching the hook and the people puta lot of cans in a tote bag and put it on PR2rsquos shoulder Asshown in Fig 15 PR2 realized using a bag like us human

VII CONCLUSION

This paper dealt with self-repair and self-extension bytightening screws of self-body In conclusion we proposebelow ideas and methodsbull We proposed an idea of self-repair and self-extension

system by tightening self-screws for humanoid robotsbull We proposed a method of calculating the precise screw

pose with self CAD databull We proposed a method of judging feasible regrasping

of a driver in order to solve inverse kinematics of thesmall closed links when tightening a screw

bull We proposed a method of generating tightening motionwith graph search considering regrasping a driver

In order to verify the methods we did some experiments ofself-repair and self-extension with a real humanoid robot Asa future work to make this system more general we want toestablish the unified method of perceiving loose screws Alsomanaging the tendency of loose screws is important Whenthe robot tightens screws of self-body the robot memorizesit By keeping storing the information the robot knows whichscrews tend to loosen and which not In addition the error ofgrasping pose when grasping the driver is so high because theactual grasping pose and the reference grasping pose is oftendifferent which we ignored in this paper So the methodto recalculate the grasping pose by observing the pose ofthe driver to the robotrsquos hand after grasping the driver isneeded The tightening system including those methods willhelp humanoid robots

REFERENCES

[1] R Juergen M Werner and H Yorik van ldquoFreecad (version0166712)rdquo [Online] Available [Software]Availablefromhttpwwwfreecadweborg

[2] M Yim W-M Shen B Salemi D Rus M Moll H LipsonE Klavins and G S Chirikjian ldquoModular self-reconfigurable robotsystems [grand challenges of robotics]rdquo IEEE Robotics amp AutomationMagazine vol 14 no 1 pp 43ndash52 2007

[3] M Yim D G Duff and K D Roufas ldquoPolybot a modular recon-figurable robotrdquo in ICRA 2000 pp 514ndash520

[4] B Salemi M Moll and W-M Shen ldquoSuperbot A deployable multi-functional and modular self-reconfigurable robotic systemrdquo in 2006IEEERSJ International Conference on Intelligent Robots and Systems2006 pp 3636ndash3641

[5] L A Mateos and M Vincze ldquoLammos-latching mechanism based onmotorized-screw for reconfigurable robotsrdquo in 2013 16th InternationalConference on Advanced Robotics (ICAR) 2013 pp 1ndash8

[6] G Park and D Inman ldquoSmart bolts an example of self-healingstructuresrdquo Smart Materials Bulletin vol 2001 no 7 pp 5ndash8 2001

[7] S Terryn J Brancart D Lefeber G Van Assche and B Vander-borght ldquoSelf-healing soft pneumatic robotsrdquo Science Robotics vol 2no 9 p eaan4268 2017

[8] S Nakashima T Shirai Y Asano Y Kakiuchi K Okada and M In-aba ldquoResistance-based self-sensing system of active self-melting bolttowards autonomous healing structurerdquo in 2018 IEEE InternationalConference on Soft Robotics (RoboSoft) 2018 pp 88ndash93

[9] A Ono and S Fukumoto ldquoTightening systemrdquo Mar 28 2017 uSPatent 9604329

[10] A Cherubini R Passama P Fraisse and A Crosnier ldquoA unifiedmultimodal control framework for humanndashrobot interactionrdquo Roboticsand Autonomous Systems vol 70 pp 106ndash115 2015

[11] C H Kim and J Seo ldquoShallow-depth insertion Peg in shallow holethrough robotic in-hand manipulationrdquo IEEE Robotics and AutomationLetters vol 4 no 2 pp 383ndash390 2019

[12] B Lara K Althoefer and L D Seneviratne ldquoAutomated robot-based screw insertion systemrdquo in IECONrsquo98 Proceedings of the 24thAnnual Conference of the IEEE Industrial Electronics Society (CatNo 98CH36200) vol 4 1998 pp 2440ndash2445

[13] H H Chen ldquoA screw motion approach to uniqueness analysis ofhead-eye geometryrdquo in Proceedings 1991 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition 1991 pp145ndash151

[14] M Yokomae Y Itsuzaki K Horikami K Okumura et al ldquoMethodof recognizing a screw hole and screwing method based on therecognitionrdquo Sept 19 2000 uS Patent 6122398

[15] Z Cao T Simon S-E Wei and Y Sheikh ldquoRealtime multi-person2d pose estimation using part affinity fieldsrdquo in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition 2017pp 7291ndash7299

We describe functions in Alg 1 in the following lmax lminare shown in Fig 6

Algorithm 1 Judging Whether Shell is Screw or not1 function JUDGEWHETHERSCREW(shell)2 Clarr []3 for edge in shellEdges do4 if IsCircle(edge) then5 radiuslarr calcRadiusO fCircle(edge)6 dlarr calcDirectionO fCircle(edge)7 pcenter larr calcCenterO fCircle(edge)8 circle in f olarr (radiusdpcenter)9 push(circle in f oC)

10 end if11 end for12 if Clength lt threshold1 then13 return False14 end if15 if calcConcentricMaxRatio(C)lt threshold2 then16 return False17 end if18 Cconcentriclarr calcConcentricCircles(C)19 lmaxlarr calcMaxLength(Cconcentric)20 lminlarr calcMinLength(Cconcentric)21 if lmaxlmin lt threshold3 then22 return True23 else24 return False25 end if26 end function

bull isCircle(edge) This function judges whether the edge is circle or notEach edge has that information

bull calcRadiusO fCircle(edge) This function calculates the radius of thecircular edge

bull calcDirectionO fCircle(edge) This function calculates the verticaldirection of the circular edge

bull calcCenterO fCircle(edge) This function calculates the center posi-tion of the circular edge

bull calcConcentricMaxRatio(list) This function calculates the max ratioof the circles which are concentric each other in all the circles in list

bull calcConcentricCircles(list) This function calculates the circles whichare concentric each other and the ratio of the circles in all the circlesis maximum in list

bull calcMaxLength(list) This function calculates the most far distanceof the centers of the circular edges whose radius is maximum in allthe circles in list

bull calcMinLength(list) This function calculates the most far distance ofthe centers of the circular edges whose radius is minimum in all thecircles in list

Using this screw judgement algorithm we extractedscrews from three parts UR10 (Universal Robot 10)1 theleft shoulder of PR2 and the base parts we designed Theresults are shown in Fig 5 All results show that the screw iscorrectly extracted Some thresholds in Alg 1 must be tunedaccording to the characteristic of screws of the parts but wecan extract screws of Fig 5 with the same thresholds Atlast the closest screw among screws extracted with Alg 1to the designated points of Sec III we determine the targetscrew to tighten

B Calculation of Screw Pose

The flow of calculation of the screw pose is shown inFig 7 Now we aim to find the screw pose with the leftshoulder of the robot pb

a describes the pose of a in thecoordinate system of b and bTa describes the translation

1httpswwwuniversal-robotscomproductsur10-robot

Fig 5 Extracting the screws of UR10 the left shoulder ofPR2 and the base parts we designed from left to right withAlg 1 Red parts are screws and blue parts are others

119949119950119938119961 119949119950119946119951

119955119950119946119951119955119950119938119961

Fig 6 Information about screw

matrix from the coordinate system of b to the coordinatesystem of a The robot calculates baseTlink with each jointangle and the geometric model of the robot and alsocalculates linkTscrew with d and pcenter of the target screwcalculated by Alg 1 We can get pbase

screw by calculation ofbaseTscrew =base Tlink

linkTscrew

119953 119897119894119899119896

119953 119897119894119899119896

119953 119887119886119904119890

Screw119953 119904119888119903119890119908

Fig 7 Calculating the target screw pose from the coordinatesystem of the base link

V MOTION GENERATION OF TIGHTENING SCREWS WITHREGRASPING

A Inverse Kinematics Considering a Driver and a Screw

After grasping the driver inverse kinematics is calculatedso that the pose of the tip of the driver and the pose of thescrew head will match Assuming that the pose of the tipof the driver is rdriver and the Jacobian of the robot linkrelated to the movement of the pose of the tip of the driver(rdriver) is Jdriver Similarly assuming that the pose of thescrew head is rscrew and the Jacobian of the robot link relatedto the movement of the pose of the screw head (rscrew) isJscrew We can solve whole-body inverse kinematics whichcombines the robot links of not only a sole arm but also botharms with a driver and a screw head by iterative calculationby using ∆θ as Eq (1) The result of inverse kinematics isdescribed in Fig 8

∆rdriver = Jdriver∆θdriver∆rscrew = Jscrew∆θscrew

(1)

Suppose ∆rdriver∆rscrew as follows∆rdriver = rscrewminusrdriver∆rscrew = rdriverminusrscrew

(2)

IK

119953 119904119888119903119890119908119953 119889119903119894119907119890119903

120637 119941119955119946119959119942119955

119921 119941119955119946119959119942119955

120637 119956119940119955119942119960

119921 119956119940119955119942119960

Fig 8 Whole-body inverse kinematics considering thedriver and the screw

119891ℎ119886119899119889

119898 ℎ119886119899119889

119899 119904119888119903119890119908 119899 119904119888119903119890119908

RotationalRregrasp

feasible

119891ℎ119886119899119889

119898 ℎ119886119899119889

119899 119904119888119903119890119908 119899 119904119888119903119890119908

TranslationalRegrasp

infeasible

driverdriver

screw screw

handhand

Driver detachesfrom screw

x

y

Fig 9 Example of regrasping a driver which is feasibleand infeasible The left figure is rotational regrasping andfeasible The right figure is translational regrasping andinfeasible which causes detaching from the screw

B Regrasping of Driver

In this subsection we aim to judge the regrasping isfeasible or not like Fig 9 We describe changing graspingpose while the robot grasps the driver with the constraintthat the driver does not move on a two-dimensional planeof Fig 9 as regrasping a driver This regrasping is used bymotion generation of tightening a screw by graph search inthe next subsection We assume regrasping motion is limitedto within two-dimensional plane like Fig 9 The force andmoment applied to the driver from the robotrsquos hand are fhandand mhand The force and moment applied to the driver fromthe screw are fscrew and mscrew The position vector fromthe screw to the grasping points is rhand nscrew describesthe unit vector of the direction of the screw Then we willjudge regrasping is feasible or infeasible We assume theconstraint that the driver does not move like Eq (3) withapproximated physical model When the driver contacts thescrew the situation that the direction of fhand is opposite tonscrew is hard to think unless the static friction of the driverand the screw is satisfied mmax is a limit of moment appliedto the screw head of static friction microscrew is the translationalcoefficient between the screw and the driver The coordinatesystem is like Fig 9

fscrew middotnscrew gt 0 or microscrew|fscrewx|gt |fscrewy||mscrew|lt mmax

(3)

Then we will judge the regrasping is feasible or not whenregrasping is translational or rotational We donrsquot considerthe regrasping which combines translational regrasping androtational regrasping The balance of force and moment ofthe driver is below

fhand +fscrew =Omscrew +mhand +(rhandxfhandyminusrhandyfhandx) = 0 (4)

When using the robot which has a force-torque sensor onthe robotrsquos limbs the robot can judge whether regrasping isfeasible or not by Eq (3) Eq (4) while executing regraspingbecause the robot knows fhand and mhand However the robotwhich doesnrsquot have a force-torque sensor cannot judge feasi-bility only with Eq (3) Eq (4) So we consider constraintsfor the robot which doesnrsquot have a force-torque sensor withsome approximation in the following

1) When Translational Regrasping To convert the con-straint of force and moment to the constraint of positionand orientation we formulate below equations microt is thetranslational coefficient of static friction between the robotrsquoshand and the driver elowast describes the unit vector of lowastvector ∆rpos is a two-dimensional unit vector describing thedirection of translational regrasping movement N is force ofgrasping a driver

|fhand |= microt Nefhand = e∆rpos

(5)

Then we apply strong approximation obtained empiricallyWhen regrasping motion is only translational we assumethat mhand equals zero So we can formulate fscrewmscrewwith the pose of the robotrsquos hand information by combiningEq (3) Eq (4) Eq (5) N is assumed to be known

2) When Rotational Regrasping To convert the constraintof force and moment to the constraint of position andorientation we formulate below equation micror is the rotationalcoefficient of static friction ∆rori is a value whose absolutevalue is 1 describing the direction of rotational regraspingmovement sgn(val) returns sign of val

|mhand |= microrNsgn(mhand) = sgn(∆rori)

(6)

Then we apply strong approximation obtained empiricallyWhen regrasping motion is only rotational we assume thatmscrew equals zero So we can formulate fscrewmscrew withthe pose of the robotrsquos hand information by combiningEq (3) Eq (4) Eq (6) N is assumed to be known

To verify the approximate constraint described upper weconducted experiments of regrasping with PR2 like Fig 10The robot which doesnrsquot have a force-torque sensor didtranslational regrasping and rotational regrasping which isconsidered to be feasible according to the approximateconstraint 20 times respectively The direction of regraspingis random The movement of translational regrasping is 1cmand the rotation angle of rotational regrasping is 15 degreeThe result of the experiment is shown in TabI Successmeans that regrasping is realized without the driver movingIt is considered that the approximation that mscrew equalszero is strict

This regrasping is not needed if the axis of rotation of therobotrsquos grasping hand and the axis of rotation of the driveris on the same line If that matching is difficult because ofthe less solvability of inverse kinematics of the small closedlinks this regrasping is effective

TABLE I Result of experiment of regrasping

translational regrasp rotational regraspsuccess 18 15fail 2 5total 20 20success rate 09 075

Feasible Regrasping Infeasible Regrasping

Driver detached from the screw

Driver moved(rotated)

Fig 10 The two figures of the left side are feasible regrasp-ing which the driver doesnrsquot move while regrasping The twofigures of the right side are infeasible regrasping which thedriver rotated or the driver detached from the screw

C Motion Generation of Tightening Screw by Graph Search

We aim to generate motion of tightening a screw Becauseof the small closed link when tightening screws of self-bodythat the robot cannot move the driver to rotate around thescrew sometimes happens That means inverse kinematicsof Sec V-A cannot be solved To solve this problem weconsider inverse kinematics with the regrasping with someconstraints described in Sec V-B

First the rotation of the driver is advanced from the initialrotation angle of the driver (ϕstart ) If inverse kinematicscannot be solved the robot searches the next grasping posewith the constraints of Sec V-B If inverse kinematics canbe solved with the next grasping pose the rotation of thedriver is advanced and if not the robot searches the nextgrasping pose This search is executed as a depth-first searchconsidering two kinds of infeasibility inverse kinematics andregrasping We show this search in Fig 11 and the algorithm

119944 0

φ 119904119905119886119903119905

119944 1

φ 119892119900119886119897

119944

φIK infeasible

Regraspinfeasible

119944 119899

Fig 11 Graph search of tightening motion A vertical axisdescribes the grasping pose of the driver and a horizontalaxis describes the rotation angle of the driver of tighteningmotion Blue nodes are feasible to transit and red nodes areinfeasible to transit With regrasping the rotation angle isrealized to advance from ϕstart to ϕgoal

We show the examples of motion of regrasping a driverin Fig 12The robot succeeded rotating a driver one roundwith regrasping twice

regrasped

① ② ③ ④ ⑤

Fig 12 Tightening a screw with regrasping a driver Therobot does regrasping from 3⃝ to 4⃝

VI EXPERIMENTS OF SELF-REPAIR ANDSELF-EXTENSION

With the proposed system shown in Fig 2 we conductedexperiments of self-repair and self-extension with a real life-size humanoid robot using a driver designed for human

A Self-repair as Daily Check by HIRO

① ② ③ ④

Grasping a driver

Fig 13 HIRO tightened a screw of self-body as a dailycheck

The humanoid robot HIRO executes self-repair as a dailycheck HIRO calculated a screw pose with self CAD data andmoved a proper driver to the screw then tightened Shown inFig 13 HIRO succeeded in tightening a screw of self-body

B Self-repair by Other Peoplersquos Indication by PR2

① ② ③

④ ⑤ ⑥

Grasping a driver

Regrasped

Fig 14 PR2 perceived a loose screw by other peoplersquosindication and executed tightening the screw with a properdriver PR2 executed regrasping from 3⃝ to 4⃝

The humanoid robot PR2 executes self-repair by otherrsquosindication A people noticed the loose screw of PR2 and saidlsquoThis screw is loosersquo Then PR2 perceived the loose screwcalculated the rough position of the screw and also calculatedprecise the screw pose with self CAD data After that PR2started tightening the screw with a proper driver We showthe snapshots of the two examples when PR2 tightened ascrew of self-body with regrasping in Fig 14

① ② ③ ④ ⑤

Grasping a driver

Grasping a hook

⑥Attach

the hook

Fig 15 PR2 wants something which can contain objects then attaches the hook to his body and hang a bag PR2 startedtightening the screw ( 2⃝) then the people put a lot of cans in the bag ( 4⃝) and put the bag on PR2rsquos shoulder ( 5⃝) NowPR2 can have a lot of cans with the bag on the shoulder without PR2rsquos hand that means PR2 can manipulate various tasks

C Self-extension by Attaching the Hook by PR2

When PR2 wants to have a lot of things the only twohands are not enough to realize that So we let PR2 to usea bag the same as we put it on our shoulder PR2 startedattaching the hook whose pose is calculated with self CADdata with a driver on his shoulder in order to put a bag on hisshoulder PR2 finished attaching the hook and the people puta lot of cans in a tote bag and put it on PR2rsquos shoulder Asshown in Fig 15 PR2 realized using a bag like us human

VII CONCLUSION

This paper dealt with self-repair and self-extension bytightening screws of self-body In conclusion we proposebelow ideas and methodsbull We proposed an idea of self-repair and self-extension

system by tightening self-screws for humanoid robotsbull We proposed a method of calculating the precise screw

pose with self CAD databull We proposed a method of judging feasible regrasping

of a driver in order to solve inverse kinematics of thesmall closed links when tightening a screw

bull We proposed a method of generating tightening motionwith graph search considering regrasping a driver

In order to verify the methods we did some experiments ofself-repair and self-extension with a real humanoid robot Asa future work to make this system more general we want toestablish the unified method of perceiving loose screws Alsomanaging the tendency of loose screws is important Whenthe robot tightens screws of self-body the robot memorizesit By keeping storing the information the robot knows whichscrews tend to loosen and which not In addition the error ofgrasping pose when grasping the driver is so high because theactual grasping pose and the reference grasping pose is oftendifferent which we ignored in this paper So the methodto recalculate the grasping pose by observing the pose ofthe driver to the robotrsquos hand after grasping the driver isneeded The tightening system including those methods willhelp humanoid robots

REFERENCES

[1] R Juergen M Werner and H Yorik van ldquoFreecad (version0166712)rdquo [Online] Available [Software]Availablefromhttpwwwfreecadweborg

[2] M Yim W-M Shen B Salemi D Rus M Moll H LipsonE Klavins and G S Chirikjian ldquoModular self-reconfigurable robotsystems [grand challenges of robotics]rdquo IEEE Robotics amp AutomationMagazine vol 14 no 1 pp 43ndash52 2007

[3] M Yim D G Duff and K D Roufas ldquoPolybot a modular recon-figurable robotrdquo in ICRA 2000 pp 514ndash520

[4] B Salemi M Moll and W-M Shen ldquoSuperbot A deployable multi-functional and modular self-reconfigurable robotic systemrdquo in 2006IEEERSJ International Conference on Intelligent Robots and Systems2006 pp 3636ndash3641

[5] L A Mateos and M Vincze ldquoLammos-latching mechanism based onmotorized-screw for reconfigurable robotsrdquo in 2013 16th InternationalConference on Advanced Robotics (ICAR) 2013 pp 1ndash8

[6] G Park and D Inman ldquoSmart bolts an example of self-healingstructuresrdquo Smart Materials Bulletin vol 2001 no 7 pp 5ndash8 2001

[7] S Terryn J Brancart D Lefeber G Van Assche and B Vander-borght ldquoSelf-healing soft pneumatic robotsrdquo Science Robotics vol 2no 9 p eaan4268 2017

[8] S Nakashima T Shirai Y Asano Y Kakiuchi K Okada and M In-aba ldquoResistance-based self-sensing system of active self-melting bolttowards autonomous healing structurerdquo in 2018 IEEE InternationalConference on Soft Robotics (RoboSoft) 2018 pp 88ndash93

[9] A Ono and S Fukumoto ldquoTightening systemrdquo Mar 28 2017 uSPatent 9604329

[10] A Cherubini R Passama P Fraisse and A Crosnier ldquoA unifiedmultimodal control framework for humanndashrobot interactionrdquo Roboticsand Autonomous Systems vol 70 pp 106ndash115 2015

[11] C H Kim and J Seo ldquoShallow-depth insertion Peg in shallow holethrough robotic in-hand manipulationrdquo IEEE Robotics and AutomationLetters vol 4 no 2 pp 383ndash390 2019

[12] B Lara K Althoefer and L D Seneviratne ldquoAutomated robot-based screw insertion systemrdquo in IECONrsquo98 Proceedings of the 24thAnnual Conference of the IEEE Industrial Electronics Society (CatNo 98CH36200) vol 4 1998 pp 2440ndash2445

[13] H H Chen ldquoA screw motion approach to uniqueness analysis ofhead-eye geometryrdquo in Proceedings 1991 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition 1991 pp145ndash151

[14] M Yokomae Y Itsuzaki K Horikami K Okumura et al ldquoMethodof recognizing a screw hole and screwing method based on therecognitionrdquo Sept 19 2000 uS Patent 6122398

[15] Z Cao T Simon S-E Wei and Y Sheikh ldquoRealtime multi-person2d pose estimation using part affinity fieldsrdquo in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition 2017pp 7291ndash7299

IK

119953 119904119888119903119890119908119953 119889119903119894119907119890119903

120637 119941119955119946119959119942119955

119921 119941119955119946119959119942119955

120637 119956119940119955119942119960

119921 119956119940119955119942119960

Fig 8 Whole-body inverse kinematics considering thedriver and the screw

119891ℎ119886119899119889

119898 ℎ119886119899119889

119899 119904119888119903119890119908 119899 119904119888119903119890119908

RotationalRregrasp

feasible

119891ℎ119886119899119889

119898 ℎ119886119899119889

119899 119904119888119903119890119908 119899 119904119888119903119890119908

TranslationalRegrasp

infeasible

driverdriver

screw screw

handhand

Driver detachesfrom screw

x

y

Fig 9 Example of regrasping a driver which is feasibleand infeasible The left figure is rotational regrasping andfeasible The right figure is translational regrasping andinfeasible which causes detaching from the screw

B Regrasping of Driver

In this subsection we aim to judge the regrasping isfeasible or not like Fig 9 We describe changing graspingpose while the robot grasps the driver with the constraintthat the driver does not move on a two-dimensional planeof Fig 9 as regrasping a driver This regrasping is used bymotion generation of tightening a screw by graph search inthe next subsection We assume regrasping motion is limitedto within two-dimensional plane like Fig 9 The force andmoment applied to the driver from the robotrsquos hand are fhandand mhand The force and moment applied to the driver fromthe screw are fscrew and mscrew The position vector fromthe screw to the grasping points is rhand nscrew describesthe unit vector of the direction of the screw Then we willjudge regrasping is feasible or infeasible We assume theconstraint that the driver does not move like Eq (3) withapproximated physical model When the driver contacts thescrew the situation that the direction of fhand is opposite tonscrew is hard to think unless the static friction of the driverand the screw is satisfied mmax is a limit of moment appliedto the screw head of static friction microscrew is the translationalcoefficient between the screw and the driver The coordinatesystem is like Fig 9

fscrew middotnscrew gt 0 or microscrew|fscrewx|gt |fscrewy||mscrew|lt mmax

(3)

Then we will judge the regrasping is feasible or not whenregrasping is translational or rotational We donrsquot considerthe regrasping which combines translational regrasping androtational regrasping The balance of force and moment ofthe driver is below

fhand +fscrew =Omscrew +mhand +(rhandxfhandyminusrhandyfhandx) = 0 (4)

When using the robot which has a force-torque sensor onthe robotrsquos limbs the robot can judge whether regrasping isfeasible or not by Eq (3) Eq (4) while executing regraspingbecause the robot knows fhand and mhand However the robotwhich doesnrsquot have a force-torque sensor cannot judge feasi-bility only with Eq (3) Eq (4) So we consider constraintsfor the robot which doesnrsquot have a force-torque sensor withsome approximation in the following

1) When Translational Regrasping To convert the con-straint of force and moment to the constraint of positionand orientation we formulate below equations microt is thetranslational coefficient of static friction between the robotrsquoshand and the driver elowast describes the unit vector of lowastvector ∆rpos is a two-dimensional unit vector describing thedirection of translational regrasping movement N is force ofgrasping a driver

|fhand |= microt Nefhand = e∆rpos

(5)

Then we apply strong approximation obtained empiricallyWhen regrasping motion is only translational we assumethat mhand equals zero So we can formulate fscrewmscrewwith the pose of the robotrsquos hand information by combiningEq (3) Eq (4) Eq (5) N is assumed to be known

2) When Rotational Regrasping To convert the constraintof force and moment to the constraint of position andorientation we formulate below equation micror is the rotationalcoefficient of static friction ∆rori is a value whose absolutevalue is 1 describing the direction of rotational regraspingmovement sgn(val) returns sign of val

|mhand |= microrNsgn(mhand) = sgn(∆rori)

(6)

Then we apply strong approximation obtained empiricallyWhen regrasping motion is only rotational we assume thatmscrew equals zero So we can formulate fscrewmscrew withthe pose of the robotrsquos hand information by combiningEq (3) Eq (4) Eq (6) N is assumed to be known

To verify the approximate constraint described upper weconducted experiments of regrasping with PR2 like Fig 10The robot which doesnrsquot have a force-torque sensor didtranslational regrasping and rotational regrasping which isconsidered to be feasible according to the approximateconstraint 20 times respectively The direction of regraspingis random The movement of translational regrasping is 1cmand the rotation angle of rotational regrasping is 15 degreeThe result of the experiment is shown in TabI Successmeans that regrasping is realized without the driver movingIt is considered that the approximation that mscrew equalszero is strict

This regrasping is not needed if the axis of rotation of therobotrsquos grasping hand and the axis of rotation of the driveris on the same line If that matching is difficult because ofthe less solvability of inverse kinematics of the small closedlinks this regrasping is effective

TABLE I Result of experiment of regrasping

translational regrasp rotational regraspsuccess 18 15fail 2 5total 20 20success rate 09 075

Feasible Regrasping Infeasible Regrasping

Driver detached from the screw

Driver moved(rotated)

Fig 10 The two figures of the left side are feasible regrasp-ing which the driver doesnrsquot move while regrasping The twofigures of the right side are infeasible regrasping which thedriver rotated or the driver detached from the screw

C Motion Generation of Tightening Screw by Graph Search

We aim to generate motion of tightening a screw Becauseof the small closed link when tightening screws of self-bodythat the robot cannot move the driver to rotate around thescrew sometimes happens That means inverse kinematicsof Sec V-A cannot be solved To solve this problem weconsider inverse kinematics with the regrasping with someconstraints described in Sec V-B

First the rotation of the driver is advanced from the initialrotation angle of the driver (ϕstart ) If inverse kinematicscannot be solved the robot searches the next grasping posewith the constraints of Sec V-B If inverse kinematics canbe solved with the next grasping pose the rotation of thedriver is advanced and if not the robot searches the nextgrasping pose This search is executed as a depth-first searchconsidering two kinds of infeasibility inverse kinematics andregrasping We show this search in Fig 11 and the algorithm

119944 0

φ 119904119905119886119903119905

119944 1

φ 119892119900119886119897

119944

φIK infeasible

Regraspinfeasible

119944 119899

Fig 11 Graph search of tightening motion A vertical axisdescribes the grasping pose of the driver and a horizontalaxis describes the rotation angle of the driver of tighteningmotion Blue nodes are feasible to transit and red nodes areinfeasible to transit With regrasping the rotation angle isrealized to advance from ϕstart to ϕgoal

We show the examples of motion of regrasping a driverin Fig 12The robot succeeded rotating a driver one roundwith regrasping twice

regrasped

① ② ③ ④ ⑤

Fig 12 Tightening a screw with regrasping a driver Therobot does regrasping from 3⃝ to 4⃝

VI EXPERIMENTS OF SELF-REPAIR ANDSELF-EXTENSION

With the proposed system shown in Fig 2 we conductedexperiments of self-repair and self-extension with a real life-size humanoid robot using a driver designed for human

A Self-repair as Daily Check by HIRO

① ② ③ ④

Grasping a driver

Fig 13 HIRO tightened a screw of self-body as a dailycheck

The humanoid robot HIRO executes self-repair as a dailycheck HIRO calculated a screw pose with self CAD data andmoved a proper driver to the screw then tightened Shown inFig 13 HIRO succeeded in tightening a screw of self-body

B Self-repair by Other Peoplersquos Indication by PR2

① ② ③

④ ⑤ ⑥

Grasping a driver

Regrasped

Fig 14 PR2 perceived a loose screw by other peoplersquosindication and executed tightening the screw with a properdriver PR2 executed regrasping from 3⃝ to 4⃝

The humanoid robot PR2 executes self-repair by otherrsquosindication A people noticed the loose screw of PR2 and saidlsquoThis screw is loosersquo Then PR2 perceived the loose screwcalculated the rough position of the screw and also calculatedprecise the screw pose with self CAD data After that PR2started tightening the screw with a proper driver We showthe snapshots of the two examples when PR2 tightened ascrew of self-body with regrasping in Fig 14

① ② ③ ④ ⑤

Grasping a driver

Grasping a hook

⑥Attach

the hook

Fig 15 PR2 wants something which can contain objects then attaches the hook to his body and hang a bag PR2 startedtightening the screw ( 2⃝) then the people put a lot of cans in the bag ( 4⃝) and put the bag on PR2rsquos shoulder ( 5⃝) NowPR2 can have a lot of cans with the bag on the shoulder without PR2rsquos hand that means PR2 can manipulate various tasks

C Self-extension by Attaching the Hook by PR2

When PR2 wants to have a lot of things the only twohands are not enough to realize that So we let PR2 to usea bag the same as we put it on our shoulder PR2 startedattaching the hook whose pose is calculated with self CADdata with a driver on his shoulder in order to put a bag on hisshoulder PR2 finished attaching the hook and the people puta lot of cans in a tote bag and put it on PR2rsquos shoulder Asshown in Fig 15 PR2 realized using a bag like us human

VII CONCLUSION

This paper dealt with self-repair and self-extension bytightening screws of self-body In conclusion we proposebelow ideas and methodsbull We proposed an idea of self-repair and self-extension

system by tightening self-screws for humanoid robotsbull We proposed a method of calculating the precise screw

pose with self CAD databull We proposed a method of judging feasible regrasping

of a driver in order to solve inverse kinematics of thesmall closed links when tightening a screw

bull We proposed a method of generating tightening motionwith graph search considering regrasping a driver

In order to verify the methods we did some experiments ofself-repair and self-extension with a real humanoid robot Asa future work to make this system more general we want toestablish the unified method of perceiving loose screws Alsomanaging the tendency of loose screws is important Whenthe robot tightens screws of self-body the robot memorizesit By keeping storing the information the robot knows whichscrews tend to loosen and which not In addition the error ofgrasping pose when grasping the driver is so high because theactual grasping pose and the reference grasping pose is oftendifferent which we ignored in this paper So the methodto recalculate the grasping pose by observing the pose ofthe driver to the robotrsquos hand after grasping the driver isneeded The tightening system including those methods willhelp humanoid robots

REFERENCES

[1] R Juergen M Werner and H Yorik van ldquoFreecad (version0166712)rdquo [Online] Available [Software]Availablefromhttpwwwfreecadweborg

[2] M Yim W-M Shen B Salemi D Rus M Moll H LipsonE Klavins and G S Chirikjian ldquoModular self-reconfigurable robotsystems [grand challenges of robotics]rdquo IEEE Robotics amp AutomationMagazine vol 14 no 1 pp 43ndash52 2007

[3] M Yim D G Duff and K D Roufas ldquoPolybot a modular recon-figurable robotrdquo in ICRA 2000 pp 514ndash520

[4] B Salemi M Moll and W-M Shen ldquoSuperbot A deployable multi-functional and modular self-reconfigurable robotic systemrdquo in 2006IEEERSJ International Conference on Intelligent Robots and Systems2006 pp 3636ndash3641

[5] L A Mateos and M Vincze ldquoLammos-latching mechanism based onmotorized-screw for reconfigurable robotsrdquo in 2013 16th InternationalConference on Advanced Robotics (ICAR) 2013 pp 1ndash8

[6] G Park and D Inman ldquoSmart bolts an example of self-healingstructuresrdquo Smart Materials Bulletin vol 2001 no 7 pp 5ndash8 2001

[7] S Terryn J Brancart D Lefeber G Van Assche and B Vander-borght ldquoSelf-healing soft pneumatic robotsrdquo Science Robotics vol 2no 9 p eaan4268 2017

[8] S Nakashima T Shirai Y Asano Y Kakiuchi K Okada and M In-aba ldquoResistance-based self-sensing system of active self-melting bolttowards autonomous healing structurerdquo in 2018 IEEE InternationalConference on Soft Robotics (RoboSoft) 2018 pp 88ndash93

[9] A Ono and S Fukumoto ldquoTightening systemrdquo Mar 28 2017 uSPatent 9604329

[10] A Cherubini R Passama P Fraisse and A Crosnier ldquoA unifiedmultimodal control framework for humanndashrobot interactionrdquo Roboticsand Autonomous Systems vol 70 pp 106ndash115 2015

[11] C H Kim and J Seo ldquoShallow-depth insertion Peg in shallow holethrough robotic in-hand manipulationrdquo IEEE Robotics and AutomationLetters vol 4 no 2 pp 383ndash390 2019

[12] B Lara K Althoefer and L D Seneviratne ldquoAutomated robot-based screw insertion systemrdquo in IECONrsquo98 Proceedings of the 24thAnnual Conference of the IEEE Industrial Electronics Society (CatNo 98CH36200) vol 4 1998 pp 2440ndash2445

[13] H H Chen ldquoA screw motion approach to uniqueness analysis ofhead-eye geometryrdquo in Proceedings 1991 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition 1991 pp145ndash151

[14] M Yokomae Y Itsuzaki K Horikami K Okumura et al ldquoMethodof recognizing a screw hole and screwing method based on therecognitionrdquo Sept 19 2000 uS Patent 6122398

[15] Z Cao T Simon S-E Wei and Y Sheikh ldquoRealtime multi-person2d pose estimation using part affinity fieldsrdquo in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition 2017pp 7291ndash7299

TABLE I Result of experiment of regrasping

translational regrasp rotational regraspsuccess 18 15fail 2 5total 20 20success rate 09 075

Feasible Regrasping Infeasible Regrasping

Driver detached from the screw

Driver moved(rotated)

Fig 10 The two figures of the left side are feasible regrasp-ing which the driver doesnrsquot move while regrasping The twofigures of the right side are infeasible regrasping which thedriver rotated or the driver detached from the screw

C Motion Generation of Tightening Screw by Graph Search

We aim to generate motion of tightening a screw Becauseof the small closed link when tightening screws of self-bodythat the robot cannot move the driver to rotate around thescrew sometimes happens That means inverse kinematicsof Sec V-A cannot be solved To solve this problem weconsider inverse kinematics with the regrasping with someconstraints described in Sec V-B

First the rotation of the driver is advanced from the initialrotation angle of the driver (ϕstart ) If inverse kinematicscannot be solved the robot searches the next grasping posewith the constraints of Sec V-B If inverse kinematics canbe solved with the next grasping pose the rotation of thedriver is advanced and if not the robot searches the nextgrasping pose This search is executed as a depth-first searchconsidering two kinds of infeasibility inverse kinematics andregrasping We show this search in Fig 11 and the algorithm

119944 0

φ 119904119905119886119903119905

119944 1

φ 119892119900119886119897

119944

φIK infeasible

Regraspinfeasible

119944 119899

Fig 11 Graph search of tightening motion A vertical axisdescribes the grasping pose of the driver and a horizontalaxis describes the rotation angle of the driver of tighteningmotion Blue nodes are feasible to transit and red nodes areinfeasible to transit With regrasping the rotation angle isrealized to advance from ϕstart to ϕgoal

We show the examples of motion of regrasping a driverin Fig 12The robot succeeded rotating a driver one roundwith regrasping twice

regrasped

① ② ③ ④ ⑤

Fig 12 Tightening a screw with regrasping a driver Therobot does regrasping from 3⃝ to 4⃝

VI EXPERIMENTS OF SELF-REPAIR ANDSELF-EXTENSION

With the proposed system shown in Fig 2 we conductedexperiments of self-repair and self-extension with a real life-size humanoid robot using a driver designed for human

A Self-repair as Daily Check by HIRO

① ② ③ ④

Grasping a driver

Fig 13 HIRO tightened a screw of self-body as a dailycheck

The humanoid robot HIRO executes self-repair as a dailycheck HIRO calculated a screw pose with self CAD data andmoved a proper driver to the screw then tightened Shown inFig 13 HIRO succeeded in tightening a screw of self-body

B Self-repair by Other Peoplersquos Indication by PR2

① ② ③

④ ⑤ ⑥

Grasping a driver

Regrasped

Fig 14 PR2 perceived a loose screw by other peoplersquosindication and executed tightening the screw with a properdriver PR2 executed regrasping from 3⃝ to 4⃝

The humanoid robot PR2 executes self-repair by otherrsquosindication A people noticed the loose screw of PR2 and saidlsquoThis screw is loosersquo Then PR2 perceived the loose screwcalculated the rough position of the screw and also calculatedprecise the screw pose with self CAD data After that PR2started tightening the screw with a proper driver We showthe snapshots of the two examples when PR2 tightened ascrew of self-body with regrasping in Fig 14

① ② ③ ④ ⑤

Grasping a driver

Grasping a hook

⑥Attach

the hook

Fig 15 PR2 wants something which can contain objects then attaches the hook to his body and hang a bag PR2 startedtightening the screw ( 2⃝) then the people put a lot of cans in the bag ( 4⃝) and put the bag on PR2rsquos shoulder ( 5⃝) NowPR2 can have a lot of cans with the bag on the shoulder without PR2rsquos hand that means PR2 can manipulate various tasks

C Self-extension by Attaching the Hook by PR2

When PR2 wants to have a lot of things the only twohands are not enough to realize that So we let PR2 to usea bag the same as we put it on our shoulder PR2 startedattaching the hook whose pose is calculated with self CADdata with a driver on his shoulder in order to put a bag on hisshoulder PR2 finished attaching the hook and the people puta lot of cans in a tote bag and put it on PR2rsquos shoulder Asshown in Fig 15 PR2 realized using a bag like us human

VII CONCLUSION

This paper dealt with self-repair and self-extension bytightening screws of self-body In conclusion we proposebelow ideas and methodsbull We proposed an idea of self-repair and self-extension

system by tightening self-screws for humanoid robotsbull We proposed a method of calculating the precise screw

pose with self CAD databull We proposed a method of judging feasible regrasping

of a driver in order to solve inverse kinematics of thesmall closed links when tightening a screw

bull We proposed a method of generating tightening motionwith graph search considering regrasping a driver

In order to verify the methods we did some experiments ofself-repair and self-extension with a real humanoid robot Asa future work to make this system more general we want toestablish the unified method of perceiving loose screws Alsomanaging the tendency of loose screws is important Whenthe robot tightens screws of self-body the robot memorizesit By keeping storing the information the robot knows whichscrews tend to loosen and which not In addition the error ofgrasping pose when grasping the driver is so high because theactual grasping pose and the reference grasping pose is oftendifferent which we ignored in this paper So the methodto recalculate the grasping pose by observing the pose ofthe driver to the robotrsquos hand after grasping the driver isneeded The tightening system including those methods willhelp humanoid robots

REFERENCES

[1] R Juergen M Werner and H Yorik van ldquoFreecad (version0166712)rdquo [Online] Available [Software]Availablefromhttpwwwfreecadweborg

[2] M Yim W-M Shen B Salemi D Rus M Moll H LipsonE Klavins and G S Chirikjian ldquoModular self-reconfigurable robotsystems [grand challenges of robotics]rdquo IEEE Robotics amp AutomationMagazine vol 14 no 1 pp 43ndash52 2007

[3] M Yim D G Duff and K D Roufas ldquoPolybot a modular recon-figurable robotrdquo in ICRA 2000 pp 514ndash520

[4] B Salemi M Moll and W-M Shen ldquoSuperbot A deployable multi-functional and modular self-reconfigurable robotic systemrdquo in 2006IEEERSJ International Conference on Intelligent Robots and Systems2006 pp 3636ndash3641

[5] L A Mateos and M Vincze ldquoLammos-latching mechanism based onmotorized-screw for reconfigurable robotsrdquo in 2013 16th InternationalConference on Advanced Robotics (ICAR) 2013 pp 1ndash8

[6] G Park and D Inman ldquoSmart bolts an example of self-healingstructuresrdquo Smart Materials Bulletin vol 2001 no 7 pp 5ndash8 2001

[7] S Terryn J Brancart D Lefeber G Van Assche and B Vander-borght ldquoSelf-healing soft pneumatic robotsrdquo Science Robotics vol 2no 9 p eaan4268 2017

[8] S Nakashima T Shirai Y Asano Y Kakiuchi K Okada and M In-aba ldquoResistance-based self-sensing system of active self-melting bolttowards autonomous healing structurerdquo in 2018 IEEE InternationalConference on Soft Robotics (RoboSoft) 2018 pp 88ndash93

[9] A Ono and S Fukumoto ldquoTightening systemrdquo Mar 28 2017 uSPatent 9604329

[10] A Cherubini R Passama P Fraisse and A Crosnier ldquoA unifiedmultimodal control framework for humanndashrobot interactionrdquo Roboticsand Autonomous Systems vol 70 pp 106ndash115 2015

[11] C H Kim and J Seo ldquoShallow-depth insertion Peg in shallow holethrough robotic in-hand manipulationrdquo IEEE Robotics and AutomationLetters vol 4 no 2 pp 383ndash390 2019

[12] B Lara K Althoefer and L D Seneviratne ldquoAutomated robot-based screw insertion systemrdquo in IECONrsquo98 Proceedings of the 24thAnnual Conference of the IEEE Industrial Electronics Society (CatNo 98CH36200) vol 4 1998 pp 2440ndash2445

[13] H H Chen ldquoA screw motion approach to uniqueness analysis ofhead-eye geometryrdquo in Proceedings 1991 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition 1991 pp145ndash151

[14] M Yokomae Y Itsuzaki K Horikami K Okumura et al ldquoMethodof recognizing a screw hole and screwing method based on therecognitionrdquo Sept 19 2000 uS Patent 6122398

[15] Z Cao T Simon S-E Wei and Y Sheikh ldquoRealtime multi-person2d pose estimation using part affinity fieldsrdquo in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition 2017pp 7291ndash7299

① ② ③ ④ ⑤

Grasping a driver

Grasping a hook

⑥Attach

the hook

Fig 15 PR2 wants something which can contain objects then attaches the hook to his body and hang a bag PR2 startedtightening the screw ( 2⃝) then the people put a lot of cans in the bag ( 4⃝) and put the bag on PR2rsquos shoulder ( 5⃝) NowPR2 can have a lot of cans with the bag on the shoulder without PR2rsquos hand that means PR2 can manipulate various tasks

C Self-extension by Attaching the Hook by PR2

When PR2 wants to have a lot of things the only twohands are not enough to realize that So we let PR2 to usea bag the same as we put it on our shoulder PR2 startedattaching the hook whose pose is calculated with self CADdata with a driver on his shoulder in order to put a bag on hisshoulder PR2 finished attaching the hook and the people puta lot of cans in a tote bag and put it on PR2rsquos shoulder Asshown in Fig 15 PR2 realized using a bag like us human

VII CONCLUSION

This paper dealt with self-repair and self-extension bytightening screws of self-body In conclusion we proposebelow ideas and methodsbull We proposed an idea of self-repair and self-extension

system by tightening self-screws for humanoid robotsbull We proposed a method of calculating the precise screw

pose with self CAD databull We proposed a method of judging feasible regrasping

of a driver in order to solve inverse kinematics of thesmall closed links when tightening a screw

bull We proposed a method of generating tightening motionwith graph search considering regrasping a driver

In order to verify the methods we did some experiments ofself-repair and self-extension with a real humanoid robot Asa future work to make this system more general we want toestablish the unified method of perceiving loose screws Alsomanaging the tendency of loose screws is important Whenthe robot tightens screws of self-body the robot memorizesit By keeping storing the information the robot knows whichscrews tend to loosen and which not In addition the error ofgrasping pose when grasping the driver is so high because theactual grasping pose and the reference grasping pose is oftendifferent which we ignored in this paper So the methodto recalculate the grasping pose by observing the pose ofthe driver to the robotrsquos hand after grasping the driver isneeded The tightening system including those methods willhelp humanoid robots

REFERENCES

[1] R Juergen M Werner and H Yorik van ldquoFreecad (version0166712)rdquo [Online] Available [Software]Availablefromhttpwwwfreecadweborg

[2] M Yim W-M Shen B Salemi D Rus M Moll H LipsonE Klavins and G S Chirikjian ldquoModular self-reconfigurable robotsystems [grand challenges of robotics]rdquo IEEE Robotics amp AutomationMagazine vol 14 no 1 pp 43ndash52 2007

[3] M Yim D G Duff and K D Roufas ldquoPolybot a modular recon-figurable robotrdquo in ICRA 2000 pp 514ndash520

[4] B Salemi M Moll and W-M Shen ldquoSuperbot A deployable multi-functional and modular self-reconfigurable robotic systemrdquo in 2006IEEERSJ International Conference on Intelligent Robots and Systems2006 pp 3636ndash3641

[5] L A Mateos and M Vincze ldquoLammos-latching mechanism based onmotorized-screw for reconfigurable robotsrdquo in 2013 16th InternationalConference on Advanced Robotics (ICAR) 2013 pp 1ndash8

[6] G Park and D Inman ldquoSmart bolts an example of self-healingstructuresrdquo Smart Materials Bulletin vol 2001 no 7 pp 5ndash8 2001

[7] S Terryn J Brancart D Lefeber G Van Assche and B Vander-borght ldquoSelf-healing soft pneumatic robotsrdquo Science Robotics vol 2no 9 p eaan4268 2017

[8] S Nakashima T Shirai Y Asano Y Kakiuchi K Okada and M In-aba ldquoResistance-based self-sensing system of active self-melting bolttowards autonomous healing structurerdquo in 2018 IEEE InternationalConference on Soft Robotics (RoboSoft) 2018 pp 88ndash93

[9] A Ono and S Fukumoto ldquoTightening systemrdquo Mar 28 2017 uSPatent 9604329

[10] A Cherubini R Passama P Fraisse and A Crosnier ldquoA unifiedmultimodal control framework for humanndashrobot interactionrdquo Roboticsand Autonomous Systems vol 70 pp 106ndash115 2015

[11] C H Kim and J Seo ldquoShallow-depth insertion Peg in shallow holethrough robotic in-hand manipulationrdquo IEEE Robotics and AutomationLetters vol 4 no 2 pp 383ndash390 2019

[12] B Lara K Althoefer and L D Seneviratne ldquoAutomated robot-based screw insertion systemrdquo in IECONrsquo98 Proceedings of the 24thAnnual Conference of the IEEE Industrial Electronics Society (CatNo 98CH36200) vol 4 1998 pp 2440ndash2445

[13] H H Chen ldquoA screw motion approach to uniqueness analysis ofhead-eye geometryrdquo in Proceedings 1991 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition 1991 pp145ndash151

[14] M Yokomae Y Itsuzaki K Horikami K Okumura et al ldquoMethodof recognizing a screw hole and screwing method based on therecognitionrdquo Sept 19 2000 uS Patent 6122398

[15] Z Cao T Simon S-E Wei and Y Sheikh ldquoRealtime multi-person2d pose estimation using part affinity fieldsrdquo in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition 2017pp 7291ndash7299