AUTOMATED VIEW SELECTION USING A ROBOTIC TRANS-OESOPHAGEAL ECHOCARDIOGRAM (TOE) SYSTEM FOR CARDIAC...
-
Upload
dev-ebenezer-johnson -
Category
Documents
-
view
24 -
download
3
description
Transcript of AUTOMATED VIEW SELECTION USING A ROBOTIC TRANS-OESOPHAGEAL ECHOCARDIOGRAM (TOE) SYSTEM FOR CARDIAC...
-
DEVAPRIYAN JOHNSON
SUPERVISOR: JAMES HOUSDEN
Project Report
Submitted in partial fulfilment of the Intercalated Bachelor of Science
degree in Imaging Sciences April 2015
AUTOMATED VIEW SELECTION USING A ROBOTIC
TRANS-OESOPHAGEAL ECHOCARDIOGRAM (TOE)
SYSTEM FOR CARDIAC INTERVENTIONS
-
1
1 Contribution
I worked on this project over a course of seven months from September 2014 to April
2015. I contributed towards the development of a view planning software that has a
high likelihood of eventual future clinical use. My contributions include manual
segmentation of the oesophagus, manual definition of the standard TOE view planes
and evaluation of the accuracy of registration and view plane positioning. In addition
to developing the view planning software, I carried out probe force measurements to
test the maximum force applied by the tip of the probe from manual control of the
TOE handle.
I would like to show my appreciation and give my thanks to my supervisor Dr.
Richard James Housden for the prompt help and guidance. Dr. Housden provided
me with the software to develop the view-planning platform. I would also like to thank
Mr. Shuangyi Wang for providing the remote control to carry out force sensor
measurements.
-
2
Table of Contents
1 CONTRIBUTION ................................................................................................. 1
2 ABSTRACT ........................................................................................................ 3
3 INTRODUCTION ................................................................................................. 4
TRANS-OESOPHAGEAL ECHOCARDIOGRAPHY ........................................................ 4
CLINICAL USES ................................................................................................... 7
SCANNING PROTOCOL USING STANDARD VIEWS .................................................. 10
PATIENT SAFETY .............................................................................................. 13
REMOTE CONTROL ........................................................................................... 14
ANATOMY OF THE OESOPHAGUS ........................................................................ 17
OVERALL OBJECTIVES OF THE PROJECT ............................................................. 20
4 METHOD .......................................................................................................... 21
VIEW PLANNING SOFTWARE AND VISUALISATION PLATFORMS ................................ 21
4.1.1 3D visualisation in the Unity platform .................................................... 21
SETTING UP STANDARD VIEW WITH REFERENCE HEART MODEL ............................ 23
SEGMENTATION OF THE HEART AND OESOPHAGUS .............................................. 26
4.3.1 Automatic segmentation of the heart .................................................... 26
4.3.2 Tracking the course of the oesophagus ................................................ 28
REGISTRATION ................................................................................................. 29
EVALUATION OF THE VIEW-PLANNING SOFTWARE ................................................ 31
4.5.1 Imaging Datasets .................................................................................. 31
4.5.2 Evaluation ............................................................................................. 31
PROBE FORCE EXPERIMENT .............................................................................. 33
5 RESULTS ......................................................................................................... 36
ACCURACY OF THE REGISTRATION AND THE PROBE POSITION ............................... 36
FORCE SENSOR MEASUREMENT......................................................................... 45
6 DISCUSSION .................................................................................................... 46
ADVANTAGES TO CLINICAL WORKFLOW ............................................................... 46
VIEW-PLANNING SOFTWARE .............................................................................. 47
LIMITING THE CURRENT IN THE REMOTE CONTROL ............................................... 52
7 CONCLUSION .................................................................................................. 55
FUTURE WORK ................................................................................................. 55
8 REFERENCES.................................................................................................. 57
-
3
2 Abstract
Ultrasonography provides a non-ionising real-time imaging modality to visualise
cardiac interventional procedures and to evaluate the outcome afterwards. In
comparison to Trans-thoracic echocardiography (TTE), the advantage of using
Trans-oesophageal echocardiography (TOE) is that the ultrasound beam does not
have to go through several thoracic structures. The X-ray modality is another gold
standard for cardiac interventional procedures. In several procedures, a combined
approach with TOE is required. The operator of the TOE probe handle is then at risk
of receiving a dose of ionising radiation. Therefore, it would be beneficial to use a
remote control that can control the probe handle and manipulate the knobs to adjust
the position of the probe tip in all the available degrees of freedom. Using remote
control motors rather than manual handling to manipulate the probe increases the
risk of damage to the oesophagus. A view planning software has been developed to
adjust position in the oesophagus and the degree of adjustment of the knobs for
automatic, robust and quick access to precise TOE view planes. This project focuses
on experiments to test the accuracy of the registration of standard views in a
reference model of the heart adjusted to a patient specific model, and the accuracy
of imaging these planes from a probe constrained in the oesophagus. Thirty-one
percent of the available planes accurately provide the required features in the new
model of the heart. The remaining views are only slightly inaccurate and can be
corrected with appropriate adjustment of probe position. Further experiments use a
force sensor to measure the force applied by the probe tip on the oesophagus.
These show that the rear end of the probe exerts six-fold more force than the distal
end. To conclude, results from testing the view planning software developed in this
project are encouraging for eventual clinical use. In addition, the maximum force
exerted from manual manipulation of the knob is under low risk of damaging the
oesophagus: 5N.
-
4
3 Introduction
Trans-oesophageal echocardiography
The trans-oesophageal echocardiogram (TOE) is a diagnostic ultrasound based
modality used to access anatomy and function of the heart [1]. TOE is commonly
used during cardiac surgeries depending on the complexity of the procedure and
pathology of the patient. The TOE instrument was first introduced into the operating
theatre usage of TOE has increased in widespread
areas such as during cardiac transplantations and major vascular surgeries, as well
as in intensive care and emergency medicine. Intraoperative echocardiographer
receives evolving training to become competent to this form of cardiac imaging.
Echocardiographer for TOE are often an anaesthesiologist, they provide accurate
and timely information to the surgeon, and provide peri-operative management for
patients [2].
TOE is a long flexible gastric probe tube that has a miniature ultrasound transducer
at the tip (Figure 1). The probe that was used in this project is the Philips X7-2t.
Currently in clinical practice, a cardiologist controls the position and orientation of the
TOE probe via the handle. The gastroscope tube also has markings to indicate the
depth of the inserted probe. There are two knobs on the probe handle one knob is
larger than the other one. The larger knob controls the anteflexion and retroflection of
the probe. On the other hand, the smaller knob steers to the left or to the right.
Moreover, the ultrasound beam can also be steered electronically.
-
5
Figure 1: Labelled image of the structure of the Philips X7-2t (left hand side). Image
on the right hand side demonstrates the probe handle operated by the TOE
echocardiographer [29].
Five degrees of freedom (DOFs) are available by manual control of the probe (Figure
2). Two of these DOFs include rotation of the probe handle and longitudinal control of
the probe. This controls the rotation of the probe head and the depth of probe
insertion into the oesophagus, i.e. rotating the handle rotates the entire probe
including the tip of the probe. The second DOF is the linear transaction of the probe
either forwards or backwards. Gripping the shaft handle, pushing or pulling to
advance or withdraw the probe from the oesophagus respectively [3].The two knobs
on the probe handle control the other two DOFs, which are the steering of the probe
tip. Finally, the fifth DOF is the omniplane rotation (Figure 3). This DOF is controlled
by the operating system rather than the physical displacement of the probe tip.
-
6
Figure 2: a-d shows the different degrees of freedom of the probe tip. (a) the depth in
the oesophagus, (b) the rotation about the shaft, (c) left-right steering of the tip and
(d) anteflexion -retroflexion of the tip.
Figure 3: Transoesophageal echocardiography (TOE) probes. A, a standard
multiplane TOE probe uses a linear phased array to rotate a 2D plane through 180.
B, a matrix array TOE probe contains piezoelectric elements to scan a 3D pyramidal
volume. The transducer used in this project is B, the matric array TOE probe [3].
-
7
The probe is popular for cardiac interventional diagnosis and surgical procedures.
The probe is inserted into the oesophagus, which runs immediately posterior to the
heart. Therefore, the ultrasound beam does not have to travel a long distance (only a
few millimetres) to reflect back from the heart to the transducer. This reduction in
distance between the transducer and the heart results in better spatial resolution for
medical diagnosis [4]. The ultrasound waves in TTE have to get through many
structures such as the skin, ribs, fat and lungs. All these structures make the received
ultrasound waves weaker [5]. Therefore, in comparison with other similar minimally
invasive imaging of the heart, such as the traditional TTE, a higher quality image is a
significant advantage to TOE.
TOE used for measuring the size of internal organs, chambers and vessels. However
since the majority of the patients are expected to have undergone TTE before TOE,
measurements from TTE are used in these cases. This is because the evidence base
is higher in TTE in comparison with TOE [5]. In addition, some measurements are
difficult to achieve in TOE. This is because of the proximity of the transducer, e.g. LA
dimensions. In addition, some measurements are also more prone to error in case of
images obtained off axis, e.g. LV dimensions. Despite the inappropriateness for
measurements, some measurement are usually more precise in TOE, such as the
aortic root size and the annular dimensions.
Clinical uses
Along with the diagnostic uses of TOE, including detection of prosthetic valve
dysfunction and endocarditis, during surgical procedures TOE guides the operation
by providing US imaging of the heart and the structures of the heart, such as the
leaflets of the heart valves and the subvalvular apparatus [6]. TOE is commonly used
-
8
to evaluate the life-threatening hemodynamic disturbances that may occur during a
surgical or non-surgical procedure. Intraoperative use of TOE included monitoring
the valves and congenital heart disease repairs.
TOE use also includes monitoring left ventricular function during coronary artery
bypass graft (CABG). In addition to confirming a suspected diagnosis, the benefit of
TOE scanning during the procedure involves assistance in positioning of
intravascular devices, early identification of ischemia and altering surgical
management or medical therapy. In patients from Category 1 (28%) (Table 1) use of
TOE is more beneficial and therefore results in alteration of the therapy in
comparison with Category 2 (14%).
One of the strongest clinical benefits is that Intraoperative TOE aids in the evaluation
of the mitral valve during repair. In an assessment of mitral valve morphology using a
systemic approach, there is good agreement between the surgical finding and TOE
(92% agreement in comparison with patients not systemically studied) [6]. Routinely
using TOE during valve repairs and replacements is certainly an advantage in
adjusting the management and providing an effective postoperative care.
-
9
Table 1: Lists the peri-operative uses of TOE [29].
Patients receiving left ventricular assist device (LVAD) implantation treatment
undergo TOE as it plays a vital role in the selection and surgical management. An
intact aortic valve is essential for proper LVAD function. For this purpose, TOE is
considered a beneficial tool to assess the function of the aortic valve. This is
because an incompetent aortic valve would result in significant retrograde flow.
Shunting of right-to left is produced across an atrial septal defect (ASD) and Patent
Foramen Ovale (PFO) because of activation of the LVAD implantation. For this
-
10
reason, it is important to correct the intra-cardiac shunts, PFO and aortic
regurgitation before implanting and activating the LVAD. The mechanism of LVAD is
detected by TOE. This includes obstruction by the cannula or valve regurgitation
through the inlet or the outlet. TOE also aids in the optimisation of the device
performance and guides the operator if the LVAD is deterring [12].
The success and the safety of many catheter based approaches for treating
congenital heart malfunctions is improved by using TOE images. Catheter based
approaches are used to treat 40% of congenital heart malfunctions [13, 14]. It is
appropriate to use TOE for determining the relation of the veins and a septal defect
to the adjacent valves. TOE is used to assess the morphology of the valves and
hence it is an appropriate test to select patients with PFO closure and percutaneous
ASD [15]. Guiding the device during the positioning and placing is aided by the
assistance of TOE during such procedures. After resolving the congenital heart
malfunctions, TOE is used to detect and assess the severity of residual effects, such
as interference with valve function and obstruction of pulmonary venous pathway.
Scanning protocol using standard views
Recent imaging technologies have provided the opportunity to obtain real-time (RT)
3-dimensional (3D) echocardiography of the heart using the transoesophageal or
transthoracic (TTE) matrix array probe that provides online 3D images [3-5]. There is
analytical software providing quick offline reconstruction of these 3D dataset. This is
beneficial in further assessing structures such as the mitral valve and quantifying the
function, e.g. left ventricle.
3D TOE depends on volume datasets in comparison to 2D TOE that depends on
standard imaging planes [7]. The TOE trained echocardiographer must manipulate
-
11
the view to orient the 2D or 3D images to ensure to obtain the basic standard views.
This indicates that that heart can be viewed in several normal and pathological
perspectives [8]. This is an essential visual aid to understand the anatomy of a heart.
In order to minimise the dataset and to provide a framework for performing TOE
imaging in different clinical settings, the British Society of Echocardiography has
come up with 20 standard views of TOE. This also recommends a sequence to
perform a complete TOE imaging to specify areas of interest in the heart (Figure 4)
[2]. Despite providing a systematic approach to acquired TOE imaging there are
specific issues at each views. In addition, it has been recognised that not all views
can be imaged in all patients, especially some poorly tolerated views in which the
probe can cause discomfort to some patients, even in the post recovery phase from
general anaesthesia (GA), e.g. deep gastric.
Therefore, the operator acquiring the TOE images can decide to omit a view taking
into account the balance between risk of insufficient data versus patient comfort and
safety. In recent days, cardiologists, cardiac physiologists or cardiothoracic
anaesthetists operate the TOE probe to obtain the required views for cardiac
interventional procedures. In addition to that, there is medico legal justification to
ensure the format of the standard views has been followed precisely in research
studies.
-
12
Figure 4: Shows the 20 standard views shortlisted to examine all aspects of TOE
comprehensively. The icon adjacent to each view indicates the approximate
omniplane angle indicators. ME Mid-oesophageal, LAX longitudinal axis, TG -
transgastric, SAX short axis, AV aortic valve, RV right ventricle, asc -
ascending, desc descending and UE upper oesophageal [2].
-
13
This approach has been appropriate to avoid missing important diagnostic
information that may not appear in preoperative TTE. This means the cardiac
interventional TOE has to be well coordinated in order to complete the entire format
of the study. For interventional procedures that require the chest to be open it is
recommended to obtain most of the data before opening the chest as the images
may be affected afterwards, e.g. dimensions of the tricuspid annulus.
In addition, all TOE procedures are carried out on the patient under GA. Therefore,
the clinician must take into account that patients physiology might vary due to general
anaesthesia, vasoactive drugs and fluid status. It is important to consider these
principles in deciding to acquire TOE images before listing the patient for surgery. For
example, the severity of the mitral regurgitation varies according to the physiology at
the time of the study.
Patient safety
The duration of the TOE is usually 45-60 minutes, which includes preparing the
patient, e.g. both oral and written consent, cannulations, and could involve a
preliminary TTE [9]. Some clinical circumstances may require more focus on image
acquisition and these procedures may take longer based on clinical judgement.
The pressure exerted by the probe can damage the oesophagus. In comparison with
other cardiac interventional modalities, TOE is relatively safe [9, 11]. However,
inserting and manipulating the probe inside the oesophagus can potentially result in
oropharyngeal, oesophageal, or gastric trauma.
This indicates that TOE is a semi-invasive procedure with potential for serious
complications [10]. Therefore, since patient safety is the crucial first priority, in order to
-
14
ensure these complications are avoided there are mandatory routine checks, e.g.
oesophageal stricture, loose teeth, previous gastro-oesophageal surgery, etc.
plan is required. In this way, identifying the risk factors and manoeuvring gently and
cautiously prevents severe and life threatening compilations.
There are also safety aspects from general anaesthesia that might affect the TOE
acquiring procedure. Firstly, only a trained anaesthetist provides sedation for the
patients. Monitoring the blood oxygen saturation before and after the procedure and
continuously checking the saturation levels during the entire length of the TOE
procedure is also mandatory [11]. The equipment to resuscitate must be fully available
if saturation levels seem to decrease. The echocardiography lab must also provide
protocols on decontamination of the probe after each procedure and sterilisation of
all the equipment that goes along with the TOE probe. These documents must also
agree with protocols from the local infection control department. This ensures the
safety of the patient and follows the code of ethics that the treatment received is in
the best interest of the patient.
Remote control
Interventional procedures usually include a combination of x-ray fluoroscopy. Certain
interventional procedures require patients undergo x-ray tests alongside acquiring
TOE ultrasound images. This often involves a skilled operator in the path of the x-ray
to receive a dose of ionising radiation [10]. This is undesirable for long cardiac
interventional procedures. This implies that the longer the procedures the more
significant dosage is received by the TOE operator. Wearing heavy lead clothing is
also a burden for the operator to protect against x-rays.
-
15
Another disadvantage of manual operating of the TOE probe handle is the demand
of highly trained echocardiographers. This issue inflict financial burden upon the
firms to organise regular training for the staff operating the TOE device. These are
the disadvantages of using TOE imaging for cardiac interventional procedures.
A remote control to operate the TOE probe would be an ideal solution. This is
because the remote control can be operated from a relative distance or even behind
lead protection. The remote control operated robotic system is an advantage as it
allows longer use of the probe (as long as the interventional procedure takes)
without a significant radiation exposure to the operator.
In addition, using a view-planning program that automatically navigates to the
requested plane is a possible solution to address the demand for highly trained
echocardiographers. A semi-automatic control of the probe reduces the need for
skilled radiographers. This as this eliminates the cost
of arranging termly training for the operator to remain up-to-date with the imaging
techniques. These are potential solutions for addressing both these major problems
associated with TOE imaging.
This project focuses on developing an automatic view planning approach in which
the probe is automatically positioned to the appropriate locations in response to a
requested specific view of the heart. The results from the view planning experiments
will form the basis of using 3D ultrasound data to register with the robotic coordinate
system. Furthermore, incorporating these functions to go with the pre-planned views
the remote control software will be developed.
-
16
Figure 5: Shows the TOE probe handle mounted to the remote control. This image
also explicitly show the railing to bring about the longitudinal protraction and
retraction of the probe tip as the motors slide on the railing.
Existing robotic system - The remote control robot manipulates the probe handle with
four out of the five degrees of freedom (DOFs) available using the manual
manipulation. These include, the rotation of the two knobs controlling the anteflexion
retroflexion and left-right steering of the probe tip. The protraction retraction and
rotation of the probe is controlled by the railing in the built in remote control (Figure
5). The software that runs on the PC controls the probe. The software synchronized
to a joystick or a gamepad connected with the computer allows an opportunity for
wireless control of the remote control. The remote control has been tested to find out
the correct working of the control and the mechanisms. A gamepad or joystick is an
-
17
alternative ideal choice for controlling the robotic remote control. The robot can be
manually controlled remotely, either on a PC or via a joystick (Figure 6, 7).
Figure 6: Shows flowchart of the overall interfacing method.
Figure 7: Diagram of the gamepad input for TOE robot control [29].
Anatomy of the oesophagus
The heart and the lower part of the mediastinal oesophagus is in close anatomical
relation to each other. The oesophagus is posterior to the heart and is separated
from the left atrium by the pericardium [16, 17]. In this way, this aspect is very useful in
obtaining an echocardiogram by inserting a transoesophageal probe.
The oesophagus is a tube-like fibromuscular organ through which food passes to get
from the pharynx to the stomach, with the assistance of peristaltic contractions. The
Gamepad/Joystick Software in PCRemote control
motor
-
18
lumen of the oesophagus is quite flexible such that it changes shape to give way for
the food that is passing though [18]. From the lumen to the outside, the oesophagus
consists of mucosa, sub-mucosa, and then the layers of muscle fibres within the
layers of fibrous tissue then finally connective tissue at the outermost layer.
Oesophagus is a thoracic organ, whereas the stomach is an abdominal organ. The
structure of mucosa in the oesophagus is different from that of the stomach.
Oesophageal mucosa consist of stratified squamous epithelium. On the other hand,
the stomach mucosa consist of simple columnar epithelium. This histological
characteristic of the organ is an important consideration to understand the safety of
the patient during the interventional procedure.
Oesophagus has two subtle curves at the commencement at the superior thoracic
aperture and near the descending thoracic aorta. Oesophagus has three
constrictions along its course at the level of cricophargeoal sphincter, between the
aortic arch and left main bronchus and finally as it pieces through the diaphragm [19].
Other than the two area of attachment at the top at the cricophargeoal level and at
the diaphragm, the organ is mobile [21]. This applies even in trunk changes in the
patients from forward to backward, lateral shifts and supine to prone positions [22].
These are the main reason for considering oesophagus rather than any other organ
in the mediastinum for TOE imaging.
However, the relations and common path for pain fibres from both organs via the
sympathetic trunk leads to difficulty in determining the origin of pain. This becomes
an issue as the safety aspects of inserting the semi-invasive TOE probe and the
remote control of the inserted probe has a potential to damage the oesophagus and
the surrounding mediastinal structures [20].
-
19
Oesophagus takes a variable course within each individual. Most commonly
oesophagus is observed to be located rightward of the spine, compressed between
left atrium and the spine. A groove behind the atrium bound by the spine and the
aorta is commonly observed. Oesophagus is also compressed between the left
atrium and aorta. This compression is usually observed leftward of the aorta.
Furthermore, the position of the oesophagus to the LA is most commonly adjacent to
right PV antra and ostia, between the right and left PV, or the left PV antra or ostia
[23, 24, 25, 26]. This evidence shows that the course of the oesophagus has subtle
variation between individuals
In addition to the variation in the course, the oesophagus is also very mobile within
the individual. Relative to the position of the LA the position of the oesophagus can
be dynamic. It shifts laterally several centimetres in relationship to the LA because of
changes in the patient position from right to the left patient recumbent position [27] or
deglutition and peristalsis in the awake or sedated patient [28]. However, the
oesophagus position is constant in a resting position, i.e. in a prone patient lying
down in a bed the position of the oesophagus remains consistent.
The mobility of the oesophagus can affect the position of the probe, and hence the
view planes, compared to the position assumed by the model. In this way, the
physiological or pathological variation in the oesophagus and lateral shifting of the
highly mobile oesophagus has implication for the view planning software developed
in this project. Along with the view planning aspects of the project, the TOE safety
profile was reviewed by experiments to determine the maximum force exerted by the
tip of the probe.
-
20
Overall objectives of the project
I. Develop a clinically practical semi-automatic view-planning platform.
II. Determine the safety of the robotic system by measuring the forces applied.
-
21
4 Method
View planning software and visualisation platforms
The plan for visualising the standard views using the software is to segment a heart
model from pre-acquired MRI data, and manually segment the course of the
oesophagus. These two organs will be combined in a view platform software in
which the standard view slices can be selected and visualised. Then compare the
standard views in the Reference heart model with Experimental heart MRI models.
This aspect is beneficial as it provides an opportunity to develop algorithms that
determine probe position automatically by selecting a view.
4.1.1 3D visualisation in the Unity platform
The automatically segmented heart and the manually segmented oesophagus are
visualised using a gaming platform, known as Unity, developed by Unity
technologies. Unity is a 3D visualisation software that allows us to develop 3D video
games for websites, desktop platforms, mobile devices and consoles (Figure 8).
These features not only enabled us to visualise the segmented heart and the
oesophagus but also to slice the segmented heart to view the standard views.
-
22
Figure 8: shows a screenshot of the unity software visualising a 3D heart model.
The heart data is loaded onto the Unity platform using the five-step process. The first
step involves loading the heart surface file for the dataset under trial. We have two
datasets to trial in the experiments. The next step is to load the corresponding
oesophagus using the similar file type. After the heart and the oesophagus
segmentation has been loaded, the probe direction has to be adjusted. This is
because the software does not identify whether the probe is facing the right direction
along the oesophageal course. Therefore, the direction of the probe is adjusted
(Figure 9). This step is to ensure the
digital segmented probe is aligned in the direction that simulates the direction of the
actual TOE probe, i.e. from superior to inferior.
-
23
Figure 9: Shows the screen shot of the Unity platform after loading the model of the
heart, the oesophagus and reversing the probe appropriately to align the direction of
the probe tip. The panel in the left hand side also shows the probe control
parameters that simulate the actual control of the TOE probe tip
Setting up standard view with Reference Heart model
The Department of Anaesthesia at the Toronto General Hospital has developed a
Virtual TOE interactive educational tool. It is an online website designed to facilitate
learning of TOE. This educational tool provide details on brief descriptions of each
view including bullet point listing of the features and structures expected from each
view (Figure 10). The website also states detailed information to obtain each
standard views. This information was used as a guide to set up the standard views.
-
24
Figure 10: Shows the perioperative interactive educational website that provides 3D
visualisation on the standard views. The website provides a brief description of the
expected structures on each standard view and the steps to obtain the views.
URL - http://pie.med.utoronto.ca/TEE/
A line drawn along the course of the oesophagus in the Rview software. The
landmarks are manually located in the software and the traced oesophagus is placed
next to the automatically segmented heart model of the same MRI data. Then find
the standard view slices using the Unity 3D viewer. In this way, the standard views
are set up using a heart model from the first MRI data. The functions in the software
to control the probe along the course of the oesophagus are very similar to the DOFs
available for the actual probe. This is to simulate obtaining the TOE image using
http://pie.med.utoronto.ca/TEE/ -
25
manual manipulation of the probe. These functions include depth as a percentage of
the segmented length of the oesophagus, axial rotation, left/right steering,
anteflex/retroflex steering and the image plane steering in degrees.
Using the software developed in Unity, the standard views are selected from the
Reference heart model of the automatically segmented heart. In order to set up
standard views, the Reference model of the automatically segmented MRI heart data
and manually segmented oesophagus is loaded. This is to take into account the
position of the TOE probe to image the specific view. For example, the mid-
oesophageal four-chamber view would involve the probe to be in the middle of the
oesophagus, whereas the transgastric two-chamber view would involve the probe to
enter the fundus of the stomach. In this way, adjusting the probe tip position has
enabled to select the thirteen standard views in the heart model.
Only thirteen standard views were available for the Reference heart model. This is
because the other seven view planes require additional features extending further
from the heart. The aorta is the core requirement for imaging six out of these seven
planes. View planes that include the ascending aorta (AA) are the upper-
oesophageal AA LAX, SAX, mid-oesophageal AA LAX and SAX. Moreover, the view
planes involving the descending aorta are the mid-oesophageal descending aortic
SAX and LAX. The automatic segmentation of the heart limited the full extension of
the aorta. Finally, shortened length of the oesophagus post segmentation limited the
unique deep gastric LAX plane. The oesophageal segmentation could not be
extended all the way to the stomach and the fundus of the stomach. Therefore, the
deep gastric LAX view plane was omitted as well. Despite these limitations, thirteen
views were reasonably suitable views for the Reference heart model.
-
26
Segmentation of the heart and oesophagus
4.3.1 Automatic segmentation of the heart
The algorithm to automatically segment the heart to make the 3D visualisation was
first described by Peters et al [30]. 3D MR images are segmented and converted into
a 3D visualization of the heart. The main advantage of using automatic segmentation
is faster and easier than manual segmentation. Manual segmentation of the heart
could take several hours. It is clinically not feasible, as it consumes considerable
amount of staff time. However, automatic segmentations have several
disadvantages that include increased image noise, patient variability, low contrast
between the surrounding tissues and myocardium, spatial magnetic field
inhomogeneities and lack of grey level calibration.
There are several techniques potentially applied to segment the cardiac MR images.
Including active shape models, deformable models, active appearance models, level
sets, active contours and atlas-based methods. Out of all these techniques, shape
constraint deformable models formed the basis of the automatic segmentation
algorithm. The algorithm not only describe the automatic segmentation of the
ventricles but also the other two upper chambers of the heart, ventricular
myocardium, the pulmonary vessels and trunks of the aorta. This developed
algorithm is applied on MR images acquired in a stack of slices in the long axis
and/or short axis view. The dataset was nearly isotropic voxel resolution with static
cardiac (3D) image volume acquired with steady state free precession MRI. One of
the advantages of this automatic segmentation algorithm is that the position of the
heart is unknown prior to the processing. This is a beneficial aspect as the algorithm
-
27
can work out the borders by itself. In addition to that, the surrounding tissues are also
visible since the field of view is not restricted to just the heart alone.
Figure 11: Shows two different views of the heart model using the standard 3D heart
MRI after applying the automatic segmentation algorithms [30].
The anatomy of the automatically segmented heart extracted from the MR image
volume includes the four-chambers, the trunks of the aorta, the myocardium, the
pulmonary artery and the pulmonary veins (Figure 11). The segmented structures
are combined together using deformable mesh features to make the heart model.
The mesh is composed of a combination of numerous triangles combined to make
vertices. This involves a complex function to combine surfaces with three of more
junctions [21]. Prior knowledge of the shape is useful in estimating and closing up the
missing boundaries. In this way, using the shape-constraint deformable adaptation
the segmentations are stabilised and made into a regular shape [22].
-
28
4.3.2 Tracking the course of the oesophagus
The oesophagus was segmented using the viewing software Rview. Rview is a
software that integrates a number of 3D/4D data displays. Using the normalised
mutual information, it comprises fusion routines integrated with 3D rigid volume
registration. It also includes several features to carryout interactive volume
segmentation. Rview also includes painting functions to analyse the structural data.
Out of these features, the most relevant function to this project is placing landmarks
in the 3D MRI data that is used for the segmentation of the oesophagus (Figure 12).
Figure 12: Shows the screenshot of the Heart MRI and the loaded completed
landmarks of the course of oesophagus in the Rview visualisation software.
-
29
The oesophageal course implies the course of the inserted TOE probe. Firstly, load
the heart MRI data on the Rview platform. Then using the segmentation features
with the aid of the cursor, landmarks are selected along the course of the
oesophagus. The landmarks connected in one path by manual efforts to click on the
middle of the oesophageal diameter when selecting the landmarks using the cursor.
These are the steps to perform the manual segmentation of the oesophagus for all
three MRI data sets in the Rview platform. The landmarks of the oesophageal course
is saved for each of the heart MRI (Figure 12). This is accessed later on for the 3D
visualisation on the Unity platform.
Tracking the course by clicking along the centre of the oesophagus is an extremely
simple task. A minimum knowledge and understanding to identify the oesophagus in
the mediastinum is more than adequate to perform the task. This aspect is a major
clinical advantage, as the task does not demand a highly qualified clinician or a great
deal of time to carry out the task.
Registration
After selecting the standard views, experiments were performed to compare the
reliability of the view planning software in producing a similar view plane to that of the
required standard view. The view planning software was used to make this
comparison, as it provided all the relevant features and appropriate functions that
enabled the code to set up the programme to be written. The first step of the
comparison involved loading up the model of the heart segmented from the second
MRI data set. The aim of the experimental MRI data is to simulate the clinically
obtained patient heart model. The oesophagus of the experimental MRI dataset and
the standard view of the Reference dataset that was previously saved. In this
-
30
instance the standard views of the first heart acts as the reference to the standard
views registered onto the Experimental heart models.
Iterative closest point (ICP) registration was used to combine the Reference heart
model with the Experimental heart data. ICP is a point feature based registration in
which the difference between the two clouds of points is minimised using the
algorithm. The registration was affine (12 degrees of freedom), involving translation,
rotation, scale and shear of the model. Affine transformations have the property that
a 3D plane stays as a plane after deformation, which is useful for not distorting the
target image planes.
Root mean squared error (RMSE) is a quantitative measure of displacement between
all the points that occurs even after applying ICP (Figure 13). RMSE is frequently
used to compare the accuracy of registration. RMSE is a reliable measure that can
be used to quantify registration errors between models. The error in this case is the
distance between points on the Reference model to the closest point on the
Experimental heart model.
In this process, the points from the Reference heart model deforms and compute to
match the nearest point in the point set of the Experimental heart MRI data. The
model is divided up into sections (LA, LV, RA, RV, Aorta, Pulmonary artery,
Myocardium) as provided by the automatic segmentation. For increased robustness
in the registration, points in a particular section of the Reference model are matched
only to points in the corresponding section of the new model.
Figure 13: shows the formula to calculate the rooted mean standard error (RMSE).
-
31
Analytically the transformation based Reference model to the patient-specific Heart
models. In this way, best match to the Experimental heart model obtained by the
transformation of the points in the Reference heart model. The ICP algorithm
includes iterative revisions of the transformation to minimise the distance of the
reference point cloud of the Reference model from the source, first MRI out of the
datasets. The process is repeated until maximum possible convergence.
Evaluation of the view-planning software
4.5.1 Imaging Datasets
The MRI datasets that were used in this project were acquired from pre-procedural
scans. All datasets are standard patient MRIs acquired with a Philips Achieva 1.5T
MRI scanner using an ECG gated sequence and balanced SSFP. The sequence
details of the Reference heart are: TR/TE: 5.22s/2.61ms, with 90 flip angle, slice
thickness of 2.74mm, image size of 256 x 256 x 120 and voxel size of 1.34mm x
1.34mm x 1.37mm. The sequence details of the Experimental heart one are: TR/TE:
4.34ms/ 2.17ms, with 90 flip angle, slice thickness of 3.00mm, image size of 224 x
224 x 119 voxels and voxel size of 1.34mm x 1.34mm x 1.5mm. Finally, the data
sequence details of the Experimental heart two are: TR/TE: 4.63s/2.32ms, with 90
flip angle, slice thickness of 1.56mm, image size of 432 x 432 x 180 voxels and voxel
size of 0.79mm x 0.79mm x 0.78mm.
4.5.2 Evaluation
The view planning experiments are in two stages in viewing the new heart under trial.
The first step evaluates the target plane, i.e. the adjusted standard view in the new
model of the heart. This target view is derived from the standard view setup in the
-
32
initial MRI dataset (Reference Model) adjusted to the Experimental heart model one
and two using the ICP registration. This permits the experiments to test the view
planes.
First experiment is to check the accuracy of the registration. The first experiment
involves using the registration and visualising the actual view plane (Figure 9). The
data collection step involves visually analysing the degree of matching of the actual
view plane to the requirement of the target plane from standard views. The
information on the requirements of the each standard views is from the electronic
interactive TOE educator tool designed by the Toronto hospital (Figure 10). The
qualitative approach involves categorising each standard view into four groups (table
2). The reason for using a qualitative approach through visual grading is that the
required proximity in mm for an adequate match is unknown. In addition, the required
closeness may vary between different standard views. For this reason, the qualitative
approach gives information that is more useful.
Table 2: Shows the classification criteria based on the degree of match to the
standard view requirements. These guidelines are used when visually observing and
categorising each view plane
Category Degree of match to the standard view requirements
A All required features are present in the image
B Close but can be corrected by adjusting probe parameters. For
example, in the Mid-oesophageal four-chamber view, all four
chambers present but the view missed the apex.
C Same as Category B, but cannot be easily corrected by
adjusting the probe positions.
D View is completely wrong
-
33
View planes that fall in the Category B are the planes that are easily fixed.
Categorising to group B planes also requires suggestions for suitable adjustments to
make these changes.
The target plane for a standard view is already defined in the new heart by the
registration step. The aim is to position the probe in order to view this plane. The
iterative optimisation adjusts the five parameters, which adjusts the probe position
and hence the view plane position. The aim is to find a set of parameters that position
the actual view plane (which depends on the probe position) as close as possible to
the target plane. The iterative optimisation measures distances between a grid of
points defined in the target plane and the actual plane. The set of parameters that
minimise these distances is taken as giving the best probe position.
In the way, the probe position is taken into account for aligning the image plane in the
second experiment. The second experiment is also graded according to the
categories illustrated in table 2.
The conclusions drawn from both experiments are beneficial as they can be used to
programme the remote control to position the probe tip in the correct place to image a
plane that is as close as possible to the target plane.
Probe force experiment
The robot comprises current sensors that monitor the current applied to control the
steering of the probe tip. This provides feedback for force applied at the tip of the
probe and therefore helps prevent oesophageal damage. As part of the study, the
aim of the experiments carried out is to determine the maximum forced exerted by
the probe.
-
34
Figure 14: Showing the setup of the initial force sensing experiment using the
silicone tube simulating the features of the oesophagus.
The experiments to determine the force exerted on the oesophagus were performed
using a silicone tube with outside dimension (OD) of 26mm, inside dimension (ID) of
20mm and length of 250mm to mimic the basic characteristics of the oesophagus,
such as a mobile and flexible tube (Figure 14). The force was measured using a six-
axis Nano-17 sensor.
Figure 15: Shows the Nano-17 measuring the force generated by the rear end
(above) and the distal (below) end of the tip of the TOE probe.
-
35
The probe is interested into the silicone tube (Figure 15) to measure two aspects of
the generated force. The force generated on the Nano-17 resistance sensor by the
distal end of the probe tip was measured first. Then the force generated by the rear
(proximal) end of the probe tip was also measured. Both the distal and proximal end
measurements were carried out using three individuals to acquire data on the
variation in the force exerted by manual manipulation of probe handle and the
maximum force was noted. Each individual applied the maximum force achievable.
The results from both parts of the probe provide insight into adjusting the maximum
current limit to set on the robot motors to manipulate the probe safely.
-
36
5 Results
Thirteen out of the twenty view planes were appropriately marked in the model of the
Reference heart obtained from the automatic segmentation algorithm. These thirteen
views are listed in table 3.
Mid-oesophageal four-chamber Mid-oesophageal Bicaval
Mid-oesophageal two-chamber Transgastric Mid-SAX
Mid-oesophageal mitral commissural Transgastric two chamber
Mid-oesophageal LAX Transgastric basal SAX
Mid-oesophageal aortic valve SAX Transgastric LAX
Mid-oesophageal aortic valve LAX Transgastric right ventricular inflow
Mid-oesophageal right ventricular inflow-outflow
Table 3: Shows the thirteen view planes that were the heart models.
Accuracy of the registration and the probe position
All the mid-oesophageal view planes were matched well in terms of imaging the
required features with a few exceptions where features were missing or limited from
visualisation by the segmentation and processing of the heart model from the MRI
data. The registration of the Reference heart model to the Experimental heart model
one has a RMSE of 2.414mm2. The registration for the Reference heart model to the
Experimental heart model two has a RMSE of 3.644mm2.
-
37
Figure 16: Visualization of the Reference heart model (red) registered with the
experimental models (blue) - Heart model one (left) and Heart model two (right).
The registration between the Reference heart model and the experimental models
has been successful. The RMSE is smaller in the registration to Experimental heart
model one in comparison to Heart model two. A MSE score of zero indicates perfect
flawless registration. Moreover, a high MSE value indicates poor accuracy in
registration. This indicates that the registration is more accurate in the Experimental
heart model one in comparison with Heart model two.
Ideally, the red mesh should be perfectly aligned with the blue mesh to show an
accurate registration. Inaccuracies in the registration of the Reference heart model to
the Experimental heart model one are relatively minor (RMSE= 2.414mm2). The
misalignment is most noticeable lateral to the RV and the ascending aorta.
Registration of the Reference model with the experiential Heart model two has a
RMSE score of 3.644mm2. This value is higher and the misalignment is
demonstrated in Figure 16. A significant misalignment is noticeable at the level of