[25-30 Sep 2011] Surgical Tools Pose Estimation for a Multimodal HMI of a Surgical Robotic Assistant
15
System Engineering and Automation Department María Belén Estebanez Campos [email protected] System Engineering and Automation Department http://www.isa.uma.es University of Malaga (Spain) INTELLIGENT ROBOTICS AND SYSTEMS (IROS 2011) SURGICAL TOOLS POSE ESTIMATION FOR A MULTIMODAL HMI OF A SURGICAL ROBOTIC ASSISTANT Belén Estebanez Enrique Bauzano, Víctor Muñoz-Martínez
-
Upload
robotica-medica-uma -
Category
Health & Medicine
-
view
105 -
download
0
description
The main objective of this paper is to minimize the occluded areas in order to recognize the navigation of the surgeon’s tools for a two-arm autonomous robotic system for laparoscopic procedures. This robotic assistant needs the tracking of the surgeon’s surgical gestures in order to recognize the current maneuver and to execute the automated tasks of the robot. The surgical tools pose estimation is carried out by a Multiple Extended Kalman Filter (MEKF), where the movement models of the surgical tools depend on the maneuver which is being developed. This information is obtained by a maneuvers recognition system which is a part of the multimodal human machine interface (HMI) of the robot. The method proposed for reducing shadows has been applied to three in-vitro maneuvers which appear in the majority of the surgical protocols. The experiments show the behavior of this method for different time intervals of the occlusions.
Transcript of [25-30 Sep 2011] Surgical Tools Pose Estimation for a Multimodal HMI of a Surgical Robotic Assistant
- 1. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] System Engineering and Automation Department http://www.isa.uma.es University of Malaga (Spain) INTELLIGENT ROBOTICS AND SYSTEMS (IROS 2011) SURGICAL TOOLS POSE ESTIMATION FOR A MULTIMODAL HMI OF A SURGICAL ROBOTIC ASSISTANT Beln Estebanez Enrique Bauzano, Vctor Muoz-Martnez
2. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] I. General Overview ROBOTIC ASSISTANTS AND THEIR HUMAN MACHINE INTERFACES INDEX I.Overview II.Pose Estimation Model III.Experiments IV.Conclusions DA VINCI (Niemeyer , 2000). ERM Robot (Muoz, 2006) FAce MOUSe (Nishikawa, 2003). LAPMAN de Medsys (Polet and Donnez, 2004). 3. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] I. General Overview ROBOTIC ASSISTANTS AND THEIR HUMAN MACHINE INTERFACES INDEX I.Overview II.Pose Estimation Model III.Experiments IV.Conclusions DA VINCI (Niemeyer , 2000). ERM Robot (Muoz, 2006) FAce MOUSe (Nishikawa, 2003). LAPMAN de Medsys (Polet and Donnez, 2004). KaLAR system (Ko, 2007). 4. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] CISOBOT ROBOTIC ASSISTANT INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions Microphone Markers in surgical tools E. Bauzano (IROS 2010) I. General Overview 5. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] I. General Overview PROBLEMS OF SURGICAL TOOLS TRACKING INDEX I.Overview II.Pose Estimation Model III.Experiments IV.Conclusions Disadvantages Existence of devices and surgical personnel in the operation room Difficult location of the 3D sensors in the operation room Specific tools movements during the surgical task where the markers are hidden to the sensors 6. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] HMI ARCHITECTURE II. Pose Estimation Model INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions ROBOTIC ASSISTANT Movements Control SURGEON PATIENT SURGEON MODEL POSE ESTIMATION MODEL TRACKING 3D VOICE RECOGNITION Position and Orientation Interpreted Voice Maneuver Voice Command Maneuver Patient-Tool InteractionPatient-Endoscope Interaction Surgical Tools Movements Position and Orientation Estimated 7. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] SURGEON MODEL II. Pose Estimation Model INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions Insert Needle Hold Tissue Grab Needle Maneuver: Specific sequence of basic actions, modeled by HMM. Assist Knot Tie knot Basic Action: Interaction between the surgical tools managed by the surgeon. Angle Distance Left Tool Velocity Right Tool Velocity Insert Needle B. Estebanez (RAAD 2010) 8. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] SURGEON MODEL II. Pose Estimation Model INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions Maneuvers Library Intervention Model Data Processing RECOGNITION SYSTEM Position and Orientation of the Surgical Tool Maneuvers HMMs Last maneuver Developed Voice Command Current Maneuver with Gestures Command CURRENT MANEUVER B. Estebanez (RAAD 2010) 9. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] POSE ESTIMATION MODEL II. Pose Estimation Model INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions Insert Needle Hold Tissue Grab Needle Extract Tool e2 eNe1 Maneuver is divided into characteristic movements. A Characteristic Movement (ei) is a well defined gesture inside the maneuver. CHARACTERISTIC MOVEMENTS Selection (ei) ESTIMATION FUNCTIONS Selection (fj) Position and Orientation of the Surgical Tool Current maneuver Pose Estimation Model is a Multiple Extended Kalman Filter where their Estimation Functions are selected according to the Current Maneuver and its last Characteristic Movement. Position and Orientation Estimated e3 ei 10. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] III. Implantation and Experiments IMPLANTATION: CUTTING MANEUVER 2 4 6 8 10 12 80 100 120 Y(mm) 2 4 6 8 10 12 -1350 -1300 -1250 time (sec) Z(mm) 2 4 6 8 10 12 0 20 40 60 Y(mm) 2 4 6 8 10 12 -1400 -1350 -1300 -1250 time (sec) Z(mm) 2 4 6 8 10 12 80 100 120 Y(mm) 2 4 6 8 10 12 -1350 -1300 -1250 time (sec) Z(mm) 2 4 6 8 10 12 0 20 40 60 Y(mm) 2 4 6 8 10 12 -1400 -1350 -1300 -1250 time (sec) Z(mm) INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions e1 e2 e3 e4 e3 e3 e5 e1 e2 e6 e1 e6 e1 e5 Surgeons Left Hand Surgeons Right Hand 11. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] III. Implantation and Experiments EXPERIMENTS: CUTTING MANEUVER INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions 2 4 6 8 10 12 14 250 300 350 400 450 Surgeon's Left Hand X(mm) 2 4 6 8 10 12 14 60 80 100 120 140 Y(mm) 2 4 6 8 10 12 14 -1400 -1350 -1300 -1250 time (sec) Z(mm) 2 4 6 8 10 12 14 250 300 350 400 450 Surgeon's Right Hand X(mm) 2 4 6 8 10 12 14 -20 0 20 40 60 80 Y(mm) 2 4 6 8 10 12 14 -1400 -1350 -1300 -1250 time (sec) Z(mm) 2 4 6 8 10 12 14 250 300 350 400 450 Surgeon's Left Hand X(mm) 2 4 6 8 10 12 14 60 80 100 120 140 Y(mm) 2 4 6 8 10 12 14 -1400 -1350 -1300 -1250 time (sec) Z(mm) 2 4 6 8 10 12 14 250 300 350 400 450 Surgeon's Right Hand X(mm) 2 4 6 8 10 12 14 -20 0 20 40 60 80 Y(mm) 2 4 6 8 10 12 14 -1400 -1350 -1300 -1250 time (sec) Z(mm) Shadow (1 s) Characteristic Movement -> Holding Estimation Function -> f1 Estimated Data Real Data Shadow (1 s) Characteristic Movement -> Cut Estimation Function -> f2 2 4 6 8 10 12 14 250 300 350 400 450 Surgeon's Left Hand X(mm) 80 100 120 140 Y(mm) 25 30 35 40 45 X(mm) 2 4 6 8 Y(mm) 2 4 6 8 10 12 14 250 2 4 6 8 10 12 14 60 80 100 120 140 Y(mm) 2 4 6 8 10 12 14 -1400 -1350 -1300 -1250 time (sec) Z(mm) 25 -2 2 4 6 8 Y(mm) -140 -135 -130 -125 Z(mm) 2 4 6 8 10 12 14 250 300 350 X( 2 4 6 8 10 12 14 60 80 100 120 140 Y(mm) -1350 -1300 -1250 Z(mm) X(Y(mm) -1 -1 -1 -1 Z(mm) 12. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] III. Implantation and Experiments IMPLANTATION: SUTURE MANEUVER INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions e2 Surgeons Left Hand Surgeons Right Hand 5 10 15 20 25 30 35 40 45 30 40 50 60 70 80 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) Z(mm) 5 10 15 20 25 30 35 40 45 5 10 15 20 25 30 35 40 45 time (sec) 5 10 15 20 25 30 35 4 -60 -40 -20 0 20 40 Y(mm) 5 10 15 20 25 30 35 4 -1400 -1350 -1300 -1250 time (sec) Z(mm) e1 e2 e3 e2 e4 e1 e4 e8 e7 e4e1 13. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] III. Implantation and Experiments EXPERIMENTS: SUTURE MANEUVER INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions Shadow (1 sec) Characteristic Movement -> Holding Estimation Function -> f1 5 10 15 20 25 30 35 40 45 300 350 400 450 Surgeon's Left Hand X(mm) 5 10 15 20 25 30 35 40 45 30 40 50 60 70 80 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 40 45 250 300 350 400 450 Surgeon's Right Hand X(mm) 5 10 15 20 25 30 35 40 45 -60 -40 -20 0 20 40 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 40 45 300 350 400 450 Surgeon's Left Hand X(mm) 5 10 15 20 25 30 35 40 45 30 40 50 60 70 80 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 40 45 250 300 350 400 450 Surgeon's Right Hand X(mm) 5 10 15 20 25 30 35 40 45 -60 -40 -20 0 20 40 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 40 45 300 350 400 450 Surgeon's Left Hand X(mm) 5 10 15 20 25 30 35 40 45 30 40 50 60 70 80 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 40 45 250 300 350 400 450 Surgeon's Right Hand X(mm) 5 10 15 20 25 30 35 40 45 -60 -40 -20 0 20 40 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 40 45 300 350 400 450 Surgeon's Left Hand X(mm) 5 10 15 20 25 30 35 40 45 30 40 50 60 70 80 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 40 45 250 300 350 400 450 Surgeon's Right Hand X(mm) 5 10 15 20 25 30 35 40 45 -60 -40 -20 0 20 40 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) Estimated Data Real Data Shadow (1 sec) Characteristic Movement -> Turn around Estimation Function -> f3 20 25 30 35 40 45 eon's Left Hand 20 25 30 35 40 45 5 10 15 20 25 250 300 350 400 450 Surgeon's Rig X(mm) 5 10 15 20 25 -60 -40 -20 0 20 40 Y(mm) 20 25 30 35 40 45 20 25 30 35 40 45 5 10 15 20 25 250 300 350 400 X(mm) 5 10 15 20 25 -60 -40 -20 0 20 40 Y(mm) -1300 -1250 mm) 0 25 30 35 40 45 0 25 30 35 40 45 5 10 15 20 25 -60 -40 -20 0 20 Y(mm) 5 10 15 20 25 -1400 -1350 -1300 -1250 Z(mm) 5 10 15 20 25 30 35 40 45 300 350 400 450 Surgeon's Left Hand X(mm) 5 10 15 20 25 30 35 40 45 30 40 50 60 70 80 Y(mm) -1250 250 300 350 400 450 X(mm) -60 -40 -20 0 20 40 Y(mm) 5 10 15 20 25 30 35 40 45 300 350 400 450 Surgeon's Left Hand X(mm) 5 10 15 20 25 30 35 40 45 30 40 50 60 70 80 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 250 300 350 400 450 Surgeon's Right Hand X(mm) 5 10 15 20 25 30 35 -60 -40 -20 0 20 40 Y(mm) 5 10 15 20 25 30 35 -1400 -1350 -1300 -1250 time (sec) Z(mm) 5 10 15 20 25 30 35 40 45 300 350 400 X(mm 5 10 15 20 25 30 35 40 45 30 40 50 60 70 80 Y(mm) 5 10 15 20 25 30 35 40 45 -1400 -1350 -1300 -1250 time (sec) Z(mm) 250 300 350 400 X(mm -60 -40 -20 0 20 40 Y(mm) -1400 -1350 -1300 -1250 Z(mm) 14. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] CONCLUSIONS To add a learning system so the MEKF may change its set of characteristic movements depending on the surgeon Improve the functions that model the characteristic movements or automate their identification (splines, fourier series) FUTURE WORKS A methodology for estimating the location of the surgeons tool has been developed when there are occluded areas. This work is valid when the occluded areas are produced in the short term during the characteristic movements of the surgical tools or in presence of obstacles. The correct estimation can be made only when the maneuver is predicted accordingly. IV. Conclusions and Future Works INDEX I. Overview II.Pose Estimation Model III.Experiments IV.Conclusions 15. SystemEngineeringand AutomationDepartment Mara Beln Estebanez Campos [email protected] Thanks for your attention Surgical Tools Pose Estimation for a Multimodal HMI
