Patrick Sim University of Leeds MSc DMS Project 1999/00 ... · PDF file3.0.3 How can Virtual...

Patrick Sim University of Leeds MSc DMS Project 1999/00

Web based Craniotomy Simulation

Table of Contents 1. Resource Locations.........................................................................................................................3 2. Introduction....................................................................................................................................3

2.1 Aim ..........................................................................................................................................3 2.2 Scope of development ...............................................................................................................4 2.3 Stakeholders..............................................................................................................................4

3. Background Research......................................................................................................................5 3.0.1 The development of a surgeon.............................................................................................5 3.0.2 Recent effects on surgical practise.......................................................................................5 3.0.3 How can Virtual Reality benefit Surgical Medicine?............................................................6 3.0.4 The Current Status of VR Medicine....................................................................................8 3.0.5 The Acceptance of VR Medicine.........................................................................................9 3.0.6 VR surgical training and the World Wide Web....................................................................9 3.0.7 Craniotomy Operation ......................................................................................................10

3.1 Evaluation of initial simulator..................................................................................................10 3.1.1 Summary of shortcomings (from ISO 9126 evaluation) .....................................................13

4 Development Approach..................................................................................................................14 4.1 Tools and Resources................................................................................................................15 4.2 Data Integrity/Conformity .......................................................................................................15

5 Simulation Design..........................................................................................................................16 5.1 Requirements Analysis............................................................................................................16 5.2 Task Analysis..........................................................................................................................17 5.3 Cognitive Requirements..........................................................................................................17 5.4 Design Analysis......................................................................................................................18 5.5 Interaction Model ....................................................................................................................18 5.6 Simulator environment design .................................................................................................20 5.7 Design Cycle...........................................................................................................................21

6 Application & Architecture Development.......................................................................................23 6.1 Architecture............................................................................................................................23

6.1.1 VRML structure................................................................................................................24 6.1.2 Data Structure...................................................................................................................24

7 Methods & Algorithms...................................................................................................................25 7.1 Drivers and requirements.........................................................................................................25 7.2 Modelling ...............................................................................................................................25

7.2.1 Skull partitioning..............................................................................................................25 7.2.2 Spherical Mapping............................................................................................................26

7.3 Program Operation..................................................................................................................28 7.4 Discussion of program methods...............................................................................................29

7.4.1 Callback method...............................................................................................................29 7.4.2 Normal Evaluator .............................................................................................................29 7.4.3 Cartesian to Polar Conversion (Spherical Mapping)...........................................................30 7.4.4 Centroid Calculation.........................................................................................................30 7.4.5 Cut region test ..................................................................................................................30 7.4.6 Polygon check ..................................................................................................................31 7.4.7 Cut Edge..........................................................................................................................31 7.4.8 Cut region removal ...........................................................................................................32 7.4.9 Cut point sorter.................................................................................................................32 7.4.10 Cut region modeller ........................................................................................................32 7.4.11 Task evaluation...............................................................................................................32

8 Development History .....................................................................................................................33 8.1 Project Changes.......................................................................................................................33 8.2 Project Time Scales.................................................................................................................33

9 Evaluation of Project......................................................................................................................34 9.1 Simulation Evaluation .............................................................................................................34 9.2 Scope for improvement ...........................................................................................................35

10 Conclusion...................................................................................................................................36 Bibliography.....................................................................................................................................37 Appendix A: Personal Experience.....................................................................................................40



Appendix B: Copy of the Objectives & Deliverables form.................................................................41 Appendix C: Marking Scheme..........................................................................................................42 Appendix D: External Data File Structure..........................................................................................43 Appendix E: List of application files.................................................................................................45 Appendix F Program Code................................................................................................................46



1. Resource Locations

Simulator URL: http://www.csdb.leeds.ac.uk/mscyps/craniotomy.htm Project Log URL: http://www.csdb.leeds.ac.uk/mscyps/projlog.htm

2. Introduction

Figure 1. Simulator introduction screen

This project is concerned with the analysis, design and development of a novel surgical simulator on a web based platform. The final simulation is shown in figure 1. The application aim of the simulation is concerned with the cognitive development of skills and aptitudes relevant for pterional craniotomy surgery. This operation involves the temporary surgical removal of a region of the cranium to allow access to the brain. This multimedia VR application has required a parallel design and application development driven by user and domain requirements. The development has been multi-stranded and involved: • Surgical domain research and user requirements analysis • Web based architecture (incorporating Java 1.2 and VRML 2) • Construction of fast, smart algorithms and data structures • Graphical methods • Cognitive and task driven VR and HCI design

2.1 Aim To further develop the web-based craniotomy training simulation for neurosurgery originated in a final year undergraduate project [Sam Underhill (Computer Studies 98-99)].



The aim will be realised by the implementation of these broad requirements: • More realistic operation modelling • Improved virtual environment and interactivity • Inclusion of CT scan images • Optimised functionality and usability, whilst retaining portability over the web • Increased cognitive value

2.2 Scope of development The project has a wide scope in terms of what it will demonstrate. Web based applications have strengths (remote resourcing, portable code, global accessibility) and also weaknesses (modest client computing power, relatively small code, browser limitations/incompatibility). The simulation development has drivers identified by the challenge in producing a web based simulation application: to optimise its functionality and usability whilst retaining portability. From this challenge, the following innovations have been focussed on: • Operation across the whole dome of the upper cranium • Spherical modelling algorithm (with improved geometric and analytical accuracy) • Inclusion of task related tools in the virtual environment • Task evaluation feedback • Interactive correlation between the simulated operation on the 3D skull model

and 2D CT scan images • Improved cut modelling through an improved polygon rendering routine • A VRML-applet architecture that optimises functionality and usability, whilst

retaining portability over the web.

2.3 Stakeholders The project has a group of stakeholders representing different interests, shown in table 1. All had a positive involvement in the development of the simulator.

Stakeholder Role Organisation Dr Ken. Brodlie Project Supervisor Leeds University School of Computing Dr Nick Phillips User Group Expert Leeds General Infirmary Peter Coltman EPSRC agent Leeds University Media Services

Nigel John Academic Collaborator Manchester Visualisation Centre Table 1. Project stakeholders

Dr Brodlie and Nigel John provided guidance and examples of current simulator practise. Dr Phillips had a strong influence on the user requirements and simulation design because of his direct experience of surgery. Peter Coltman was interested in the conceptual basis of the simulator, as this would form part of the narrative of an EPSRC funded TV production concerning medical visualisation. Dr Phillips and Peter Coltman participated in valuable simulation trials. With Dr Phillips being an 'expert' user and Peter Coltman a 'layman', this enabled attributes aimed at either user community to be evaluated.



3. Background Research

3.0.1 The development of a surgeon Traditionally, surgeon training has been carried out by means of 'mentoring', where a senior surgeon facilitated a trainee surgeon's development. Under this regime the effectiveness of the training is determined by the senior surgeon's ability to: demonstrate, role model, allow hands on experience, provide guidance, set objectives and targets, and accept the trainee as part of a professional obligation [Hargreaves et al, 1997]. This traditional model of training has, in some cases, been equated to an 'apprenticeship' where a surgeon's education occurred during a process of 'osmosis' [Hargreaves et al, 1997]. In other words the development of the 'necessary' skills and aptitudes were gained through observation and practise. This model has two serious shortcomings. 1. The quality and means of training is to a large degree dependent on the

interpretation and inclination of the senior surgeon. This can lead to inconsistency, educational mal-practise and trainee neglect. In part these are symptoms of the informal, peer group culture of surgical practise [Higgins et al, 1997]

2. Pedagogically, it is important to distinguish the difference between training and

education. A key attribute of a competent surgeon is that of clinical judgement. This is acknowledged to be more subtle, reflective and requiring a longer term of development than pure skills and knowledge training [Hargreaves et al, 1997]. It is only through a surgical education that such judgement can be most effectively developed.

3.0.2 Recent effects on surgical practise The requirements of a surgeon have changed along with their training, procedures and patient expectations. These have all had an effect on surgeon training syllabi. Major circumstances that have affected contemporary surgery include: A nation-wide (UK) re-evaluation of surgical training during the 1990's has reduced the length of the training period but also emphasised explicit teaching and coaching over traditional observation based training [Hargreaves et al, 1997]. This addresses the issue of consistent training that is understood at a cognitive level (i.e. though training is undertaken by senior surgeons they are not generally trained educators). It has, probably, influenced the culture of surgery; surgeons must consider education an implicit part of their role. The introduction of laparoscopic surgical techniques ('key hole' or 'minimally invasive' surgery) requires the teaching of a new surgical methodology and the training of new tools and techniques. This new surgery holds great promise for the practise of medicine. Laparoscopic techniques enable smaller and less traumatic surgical intervention. This surgery



requires the acquisition of new techniques, and technology, and the ability to operate remotely (to view the surgery through a mini-camera) [Higgins et al, 1997]. This requires an education in this new surgical understanding. The patient-surgeon relationship has changed. Increasing patient expectations and the developing culture of litigation following 'unsatisfactory' surgical outcomes have affected surgical practise in a paradoxical manner. There is reason to have confidence in modern surgical techniques. Unfortunately because of the nature of surgical development a potentially improved surgical technique can only be effectively trialled on a genuine patient. Patients are increasingly likely to take litigative action over an unsuccessful 'novel' surgical technique even though it presented the best chance at the time. The threat of litigation can force surgeons to restrict their practise to 'tried-and-tested', traditional methods. Paradoxically, it is this lack of application and demonstration that prevents such methods from being proven and becoming the 'norm'. A case in point is that of laparoscopic surgery: "It is probably safe to say that one of the main reasons more doctors do not perform more minimally invasive procedures is that they cannot easily gain 'experience' in the context of a live patient" [Hon, 1996]. However, when practised properly, laparoscopic surgery promises faster, safer and cheaper operations.

3.0.3 How can Virtual Reality benefit Surgical Medicine? VR simulation has a great potential in being a training aid and medical tool for surgery. Appropriate multi-media, graphics, and HCI enable meaningful domain worlds to be constructed representing virtual anatomical models and surgical instruments. Interactive interfaces can also model the mechanical 'feel' of instrument interaction during surgery [Delp et al, 1997]. VR surgical simulation presents opportunities and benefits in every area of consideration, outlined below.

• Cognitive Benefits Educational benefits can be realised by good simulator design. These are maximised when the multi-media and interactive potential is appreciated at the analysis and design stage. As a measure of what VR simulation can deliver; in some applications simulators can increase retention by 50% and reduce training times by up to 80% [Higgins et al, 1997]. These cognitive benefits are induced by the immersive and exploratory nature of interactive environments. Instructive examples of the potential of VR simulation can be taken from flight simulation. Even the basic mechanical flight simulator pioneered by Edward Link is reported to have reduced bad weather crashes by 95% [Satava, 1996]. Modern flight simulators can achieve a training equivalence that allows one hour in simulation to reduce in-flight training by 30 minutes. Taking into consideration the maintenance and operational cost of even a single-engine aircraft, the removal of accident risk, and the flexibility to select simulated training conditions this is an advantageous training proposition.



The proven all-round effectiveness of flight simulation has made it a standard training resource in the international flying world.

• Economics As with aviation simulated surgery can realise significant economic advantages. In a study of the transfer of minimally invasive surgery skills [McGovern & Johnston, 1996] approximately 45 hours of practise were required before a surgeon can be statistically expected to demonstrate significantly faster surgical abilities. With an estimated cost of $750 USD per 1.5 hour session (including theatre, instructor, equipment and live animal) this 45 hour training costs $33,750 for each student. Even taking into account the initial outlay for computer facilities, VR simulated surgery demonstrates a cost-benefit advantage. In conjunction with the training cost-benefit there is the continued decrease in cost of computing power. Moore's Law is an example of this but the phenomenon acts across a wide range of IT. VR related capabilities such as graphics, memory, processor speed, storage capacity and interfaces will continue to reduce in cost in real terms. The immersive and realistic nature of simulators is largely constrained by their hardware capabilities. The reducing cost of computing power will enable more effective and realistic simulations to become affordable.

• Ethicality In attempts to increase the realism of surgical training various options have been used in the past. Each of which have involved either animal or human tissue and thus raised related ethical considerations: 1. The mentor based supervised operation, though inevitable at some stage in a

surgeons training, has the attendant risk to the patient. This risk increases with trainee inexperience.

2. Surgical practise on cadavers has the advantage of anatomical accuracy but the organ states (e.g. colour, feel) and lack of physiology can make the experience unrepresentative of a patient. Cadavers are also becoming more expensive and harder to obtain [Hon, 1996].

3. Live animal surgery, such as pigs, provide realism to the surgical training. There is a cost (see "Economics", above) to such training. Anatomic differences restrict the accuracy and scope of applicability of this training [Higgins et al, 1997]. Animal rights concerns are widely held (including surgical staff) especially in cases where the animal is killed after the operation.

• Realism It is unlikely that a highly realistic surgical simulator will become available in the near future. What is clear is that the simulation experience is beneficial when appropriate task design, interactivity and feedback is provided. These attributes, to a major extent, are not dependent on high graphics realism. Simulation realism is expressed by various qualities, not just graphic representation. Task realism is an important aspect of simulation. Surgery is a procedure that is taken as one of a series of critical stages. The synthesis of pre-operational, inter-operational and post-operational tasks [Satava & Jones, 1997] provide for a more holistic and relevant training experience.



Physical realism is computationally more demanding than graphic realism. This will allow models of the various tissues and organs of the human body to have realistic properties and physiological behaviour. Short term improvements to surgical simulations have been proposed [Reinig et al, 1996], these include realistic fluid models (e.g. blood), separate polygonal surfaces for different tissues, the ability to cut, deform and remove pieces of anatomy. A physiological model of the patient would also model the response of the patients condition to the surgery. • Occupational Considerations The wide spread acceptance of VR surgical simulation will cause a change in paradigm in surgical training. It will also affect the surgical occupational culture. VR simulation can reduce instructor time in a region of 30% [McGovern & Johnston, 1996]. In conjunction with distributed computer learning facilities this can fundamentally alter the surgeon-trainee relationship, enabling independent learning with reduced mentor dependence. Aviation has been used as an example of a simulation becoming universally accepted as a curricula aid. However there are significant organisational differences between these two domains [Higgins et al, 1997]. Civil flight certification is highly regulated and administered to pilots in large airline organisations. Surgery has till recently been characterised by mentor tuition and peer recognition/regulation. The relative lack of universal regulation is compounded by the informality of organisation and the diverse specialisation's in surgery.

3.0.4 The Current Status of VR Medicine There are a number of operational applications of VR simulation in current medical use. Research [Satava & Jones, 1997] has identified the three main areas of VR application to date have been: • Anatomic education • Medical crisis planning and training • Medical virtual prototyping (of tools and procedures) The most challenging technical limitation on the development of highly realistic surgical simulators is the problem of real time organ modelling. Though parallels have been drawn with the application of VR to aviation training the nature of this simulation is very different. Flying simulation can be characterised, graphically, by the modelling of relatively low detail objects with simple behaviour. Surgical simulation, ideally, requires the modelling of relatively detailed objects and their physical properties. This requires computationally expensive, and sophisticated, techniques. The three basic anatomical behaviours that a simulator should model are tissue deformation, tissue cutting, and bleeding [Delp et al, 1997]. It should be noted that there is a complex physiological inter-dependence between organs, such that a change in one affects the physiological state of the others - this is a major modelling challenge.



The hierarchy of surgical simulation requirements [Satava, 1996], organised in descending order of implementation difficulty are: 1. Visual fidelity 2. Organ interactivity 3. Physical properties 4. Physiological properties 5. Sensory input The first three are generally considered as achieved to a satisfactory extent. The current trade of is between physiological and time fidelity, to which the solution (albeit simplistic) appears to be increasing computing power. However, useful systems have been developed. Three hundred Urologists evaluated an Endoscopy simulation, 98% of whom voted that it represented a valuable training aid [Preminger et al, 1996]. VIVIAN, a neurosurgical diagnosis and planning tool "substantially contributed to surgical planning…" [Kockro et al, 2000].

3.0.5 The Acceptance of VR Medicine Apart from the limitations of current technology there are also organisational and cultural barriers that may prevent VR from being readily adopted. In a study of the acceptance of IT based educational technology within a medical college [Gupta et al, 1996] resistance was identified in both the administration and student body. The administration displayed defensive resistance citing litigation concerns, adverse publicity and reputability (if the education delivered was inferior). Student's objections concerned the trade off between learning and the time involvement trialling a new technology and 'technophobia'. Of these the administration/organisational resistance is the more critical and perhaps symptomatic of the surgical profession. VR simulation offers clear advantages to surgical training, namely: 1. Cost-effectiveness 2. Training flexibility and repeatability 3. Reduced patient risk The universal acceptance of this technology will probably follow the demonstration of the following criteria: 1. VR indisputably produces surgeons more cheaply/effectively/faster.

and/or 2. Surgeons from a VR training background improve surgical success rates

3.0.6 VR surgical training and the World Wide Web The growth of the web during the last decade of the 20th century has produced a new paradigm for computer use. Using web technology (such as Java applets, HTML and VRML) allows interactive graphics based training simulators to be downloaded across the web run in a client-side browser. Web based simulators benefit from all of the advantages of Internet media [Brodlie et al, 2000]: • Accessibility • Low-cost • Distributed (leveraging remote computing power) • Class Size (resource flexibility enables teaching of large numbers or

asynchronously) • Generality (a variety training resources can be accessed from one site)



Inherent issues of web based simulations which work against their wide-spread acceptance are: • Authenticity and standardisation of web based resources • Modest client side computing power • Compatibility of software between different browsers The Universities of Leeds and Manchester have developed web-based surgical simulators with medical collaboration. These have centred on critical surgical operations that typically have few training opportunities/facilities and significant operational risk. Examples include: Aortic aneurysm [Brodlie et al, 2000] Percutaneous rhizotymy [Brodlie et al, 2000] Ventricular catheterisation [MVC, 2000] Lumbar puncture [MVC, 2000] Proton Eye Radiotherapy [MVC,2000]

3.0.7 Craniotomy Operation The simulation will focus on tasks and operational criteria of the craniotomy studied [Dudley, 1981, p691]. The object of the craniotomy is to allow access to a tumour on the outer surface of the brain. This simulation begins at the stage where the cranium has been exposed. The task is to perform a craniotomy of the appropriate shape, size and location. This is accomplished in three simulated stages (see figure 2): Stage 1.Two burr holes are drilled into the skull, these act as access holes for a drill (one for entry the other to exit). Stage 2. A drill starts from one of the burr holes and cuts the shape of the craniotomy stopping at the other burr hole. Stage 3. The craniotomy piece is then removed by cutting between the cranium between the burr holes.

Figure 2: Craniotomy operation

3.1 Evaluation of initial simulator The craniotomy simulator[Underhill, 1999] will be evaluated using the ISO9126 template. ISO9126 is an international standard for software evaluation; it sets out six quality characteristics (broken down into quality attributes). The characteristics are functionality, usability, maintainability, reliability, efficiency and portability. This evaluation intends to gauge the initial craniotomy simulator's quality as a learning/training application. From this perspective the first two quality characteristics, Functionality and Usability, are the most important criteria. The

Burr holes

Craniotomy shape



results of this evaluation are qualitative as they are based on the experience of this author. The results of this evaluation will feed into the requirements analysis and design stage. Functionality Accuracy Skull model The skull model is made up of three separate pieces (the right and left skull domes and jaw) which exist as 3D data sets rendered by VRML. The left skull dome is the most crucial model as it is this side which virtual surgery can be practised on. It is made up of approximately 4000 data points in 3D space. The rendering of the skull (without surgical effects) on a VRML enabled browser is qualitatively effective, illuminated by a local light model. Reducing the interpolation across the model (by setting creaseAngle to zero) allows the polygon lattice to be noticed. However, when surgery is performed, the polygon data structure of the model becomes a dominant feature, as cranium removal is rendered as a removal of a group of polygons. Removal routine observations

1. The cranium removal routine is fast but noticeable. A problem with the dataset is that cranium removal is approximate. That is to say a cut between burr holes in the skull is modelled and rendered as that removal of polygons along the cut line. Another problem with the routine is that it is not complete. Some cranium removals leave skull polygons still inside the removal area (this occurs at the edge and also the interior of the removal area).

2. The cut pattern is determined by the order in which the burr holes are made, instead of taking the simplest topographic shape. Because of this odd cut shapes are made.

Suitability "For whom?" The simulator can be used in its current form as an introduction to the principles of pterional craniotomy. And as such could be used at all levels up to the stage of trainee surgeon. As a training aid for surgery the simulator requires enhancement (see below).

"For what task?" As an anatomical model it is visually accurate to an approximate scale of 0.3cm - (based on the 0.1cm2 estimated area of each rendering polygon). There is a lack of physical scale in the environment. There is no guide to the procedure for cranium removal (e.g. how big the removal should be in relation to the tumour size and location). As a co-ordination and motor skills training aid it is hampered by the absence of modelled surgical tools in the environment.



Usability Understandability Are the task objectives and instructions clear? The simulation as it stands is easy to understand, though there is no feedback as to how well the task was carried out. Learnability "How does the simulator facilitate learning?" The visual nature of the simulator lends itself as an ideal surgical training environment. At the moment the emphasis of the simulator is on solely operative techniques. Pre- and post-operative aspects are not present. "What type of learning is supported by the simulation?" Ideally the simulator will enable learning in several related surgical skills; for example anatomy, surgical procedure training, hand-eye co-ordination. Operability Navigation in VE using Explorer VRML plug in: Movement of, and around, the skull model is slow, this is most probably a symptom of the browser having to manipulate a very large data model. Response to actions The placement of the burr hole markers on the skull suffers from a delay in the order of 2 seconds, this has two implications for the HCI.

1. Movement of the mouse arrow during the first second will cause the burr hole to be deposited at a different point on the skull.

2. The delay may lead a user to think that the mouse button has not been clicked or registered and thus cause the user to make another click action.

It should be noted that the Cosmoplayer VRML plug-in, inside an Explorer 5 browser does not suffer from the problems above. Maintainability Changeability Separation of high level Java applet code and VRML modelling script enables straightforward modification (though care must be taken of links and dependencies). Stability Infrequent crashes, caused by shortage of virtual memory whilst manipulating VRML skull model. Reliability Maturity Not very mature. What exists is a demo of concept Fault Tolerance Non-catastrophic response to errors. Efficiency Resource Utilisation Makes a large demand of system virtual memory. This can cause application 'crashes'.



Portability Adaptability Requires Internet Explorer 4+ as a minimum to enable the VRML2 model (see Operability, above). Netscape Navigator will not run the applet at present because of its size (the JVM rejects the applets on the grounds that it is larger than 64K bytes). Installability Java 1.1 and VRML2, portable across web, though client is dependent on having a VRML2 enabled browser (see Adaptability, above)

3.1.1 Summary of shortcomings (from ISO 9126 evaluation) The original simulation is an achievement that demonstrates that a craniotomy simulation can be built on a portable platform with a VRML based world and Java providing the necessary functionality. This simulator also has clear shortcomings that provide opportunities for substantial improvement and development during this project. Cognitive 1. There is no case history/diagnosis and only a brief task description. 2. There is no indication of how well the craniotomy was performed. 3. A scale indicator is not present in the environment. Usability 1. Craniotomy can only be performed on the left side of the skull. 2. The cut region is performed in the sequence in which cut points were made. 3. Simplistic GUI with restricted interaction and functionality. Modelling The simulator employs a simple projection onto a flat plane to perform the cut routine. This is the fundamental reason why a cut can only be performed on the skull's left side. Graphics The VRML skull model is qualitatively representative. There is a key weakness in this area: The surgical cut model is rudimentary and the cut is not explicitly modelled, what is rendered is the skull with the polygons in the cut region removed. This produces an unrealistic jagged cut region. Program Design The Java program includes the entire set of VRML index-faceset points for the left side skull. Though this data is necessary storing it in the applet makes it cumbersome. This data represents the majority of the listing. The size of the data is a secondary reason why the right side skull is not operable on. Operability Will not run on Netscape navigator (because applet > 64KB)



4 Development Approach The project involved the parallel development of two related development strands. The two major strands and constituent sub-strands are shown in table 2:

MAJOR Strand MINOR Strands

Simulation Design • Cognitive and User Requirements • Task Analysis • Environment Design

Application & Architecture Development

• Algorithm design • OO development • Web based architecture • Data structure

Table 2: Project development strands The development of the simulation environment and the application algorithms/architecture were conducted as two distinct but parallel disciplines. These two activities are fundamentally inter-related and have the same development drivers (usability, portability, and functionality). The serial nature of the application development dictated that each module of functionality was built and tested sequentially. In contrast, the development of the simulation environment GUI and HCI was characterised by a rapid prototyping approach. Because of the qualitative nature of GUI evaluation each design layout had to be trialled (e.g. colour, visual arrangement) to identify the most appropriate or effective scheme. The analysis and design stage was carried out in manner that considered the People - Systems - Architecture requirements discussed by Prof. Dew in the Virtual Working Environments module, 2000 (see figure 3).

Figure 3: Development considerations The approach is 'holistic' and attempts to balance all the requirements of the application. This is suited to the development of modern multi-media applications, which have sophisticated HCI and functional attributes with high performance requirements. Moreover, this approach tacitly recognises the interdependence of design and application requirements.

System Architecture

People



4.1 Tools and Resources The main software tools used during the development were the following: • Java 1.2 • Microsoft Development Environment 6.0 • Microsoft Internet Explorer 5.0 and Netscape Navigator 4.0 • Cosmoworlds • VRML skull model • VRML Brain model • CT scan images supplied by Dr Nick Phillips (LGI)

4.2 Data Integrity/Conformity In all medical applications the quality of data is a critical factor. The simulator that has been developed provides a demonstration of what is possible on a web platform. The brain, skull and CT images do not have any clinical conformity as they each come from different sources (and people). The quality and conformity of the data is a key area of future improvement. For this to become a medical training resource the simulator data must either, come from a common source, or be integrated in a manner which is acceptable from a clinical point of view.



5 Simulation Design

Figure 4: Design stages The design of the simulation environment is driven by a number of domain related studies (i.e. surgical research, knowledge of craniotomy operation procedures, and weaknesses of the original simulator). From these it is necessary to identify what are the necessary learning and procedural objectives that the simulator should emphasis (and also aspects that should be avoided); this is provided by cognitive requirements analysis and task analysis. The cognitive requirements and task analysis enable tools, objects and functions to be designed into the simulation environment to maximise the cognitive benefit and usability. The Interaction model dictates the sequence and relation of events that will be allowed in the world, this is to maximise the consistency, realism and cognitive 'sense' of the simulation. The environmental features and interaction model are synthesised into the initial simulation design, which is refined by user feedback in a 'rapid prototyping' development. The whole process is shown in figure 4.

5.1 Requirements Analysis The requirements for the simulator are obtained from a number sources. A web based multimedia application has a great potential to allow exploratory and interactive learning experiences. To make full use of its capabilities thorough requirements analysis is required. The main aims of the requirements gathering stage were to identify: • Aims & methodology of surgical training from background research • Task Analysis from the surgical procedure knowledge • Cognitive requirements from task analysis and discussion with Dr Phillips

Surgical Research Evaluation of

original simulator Craniotomy Operation

Task Analysis Cognitive Requirements

Interaction Model

Environmental Features

Simulator Design (Rapid Prototyping Implementation)



Input from Dr Phillips During the design stage Dr Phillips suggested that it would be useful to incorporate CT scan images into the simulation. CT scan interpretation and diagnosis is a fundamental pre-operational task. As an 'expert' representative of the user community Dr Phillip's suggestions play an important role in developing a useful and relevant simulator. CT images are presented in a grid. One of the key skills involved in CT scan interpretation is the ability to visualise the left-right orientation of the image.

5.2 Task Analysis The simulation is concerned with replicating the key cognitive and skills based tasks in the problem domain. An analysis model allows the tasks to be abstracted. Hierarchical task analysis produces a structured task orientated model. A hierarchical task model (see figure 3) resolves the problem domain into a sequence of discrete events. Each event is resolved into the key actions and cognitive steps that constitute its successful completion. TASK EVENT Constituent Actions & Requirements Interpret CT scans

• Determine orientation of scan images (e.g. left - right, up - down) • Estimate location of tumour • Calculate size of tumour

Pre-operational Planning

• Size and location of craniotomy • Placing of initial Burr holes

Position Patient

• Visualise tumour location with respect to patient's skull • Orientate head to optimise surgical access

Drill initial Burr holes • Spatial judgement of burr hole position Cut bone using drill • Spatial judgement of cut track (size and shape)

• Hand-eye co-ordination of drill Table 3. Task analysis model

Implications: 1. Selected craniotomy operational criteria from the above task analysis can be

measured and used as task evaluation feedback. These are craniotomy size, vicinity and shape.

2. Time to complete task was not adopted as a criteria because the simulation is expected to be accomplished in an unrealistically short timescale. It's use as a metric is potentially misleading.

3. Cursor movement was initially a good candidate for a measurement criteria however the lack of a continuous 'drag sensor' in VRML2 ruled this out.

5.3 Cognitive Requirements Cognitive requirements indicate what the learning objectives of the simulator should be to optimise the educational benefit of the experience. From analysis of the craniotomy operation and discussion with a domain expert (Dr Phillips) table 4 indicates the demands that a simulator should make on the user. These cognitive demands inform the simulation design what processes should be stressed.



Task Related Other Cognitive • Anatomical awareness

• Physiological spatial relationships • CT interpretation • Operation evaluation

• Awareness of how VR can aid surgery

• Appreciation of pre-op/op interface

Meta-cognitive • Pre-operational planning skills • Spatial visualisation

• Familiarity with GUI Interaction

Motor • Hand-eye co-ordination • Interactive application navigation Table 4. Cognitive requirements

An important distinction should be made between cognitive and meta-cognitive skills. Cognitive attributes are those learned abilities that constitute the task (e.g. knowledge of the cranial structure). Meta-cognitive skills are abilities required to enable or promote the acquisition of cognitive skills (e.g. spatial visualization, planning skills etc.). For optimised learning both sets of skills must be identified and supported by the environment.

5.4 Design Analysis The simulator development has three drivers (usability, functionality and portability). These translate well into application performance and functional requirements. However the design requirements are more subtle and only share some of the same metrics. More fundamentally the design requirements are fed directly by analysis of the problem domain. For the design stage it is important to clarify what the perceived requirements of the problem domain are, in this case:

The simulation must provide meaningful exercises in: 1. Pre-operational planning from CT scans 2. Operational activities of burr hole placement and craniotomy region

cutting 3. Craniotomy cut region modelling

5.5 Interaction Model Interaction between the simulator and the user provides a powerful mechanism for reinforcing the learning experience. The 'Norman Theory of Interaction' is a seven-stage model that represents the key events that should occur for effective user interaction. Using the Norman model as a template it is possible to construct an interaction sequence for the tasks and cognitive aims (table 5).



Norman Interaction Stage Interaction Character isation Simulator Stage/

Requirements 1.Establishing the Goal

Users initial idea of what is to be undertaken

INTRODUCTION

2.Forming the Intention User associates objectives and results that signify the accomplishment of the Goal.

INTERPRETATION OF CT SCANS

3.Specifying the Action Sequence

Scope of user action is clear. MEDIATED BY SIMULATOR

4.Executing the Action

Means of implementing actions are intuitive and/or clear.

PERMISSIBLE ACTIONS CONTROLLED BY SIMULATOR

5.Perceiving the System State

User observes the new System State CLEAR VISUAL RESPONSE TO ACTIONS

6.Interpreting the System State

User compares System State against mental model and expectations

WELL DESIGNED INDICATORS AND METRICS

7.Evaluating the System State

Reflection and examination of outcome (System State) and how this has been achieved through evaluation and execution stages

TASK EVALUATION

Table 5. Task interaction model Implications: The interaction model stresses the stages of communication that a human-computer interface must facilitate. The important lessons from the analysis are: 1. The scope of actions must be clearly expressed in the environment 2. Environmental indicators and feedback must be unambiguous and intuitive 3. The user must receive feedback on the state/situation after action

Interactivity is such a powerful effect that its deployment has to be carefully designed. One warning [Johnson et al, 1998] 'One must…consider how the environment supports the delivery of instruction and facilitates effective learning experiences.' , furthermore '-unguided interactions must be avoided', and '-opportunities for incorrect learning must be avoided'. interactivity is capable of swamping or corrupting the central learning process if it is not appropriately designed. Interactivity is a function of the HCI. Good HCI provision will reduce the 'gulf of execution' and 'gulf of evaluation' during a simulation experience. The 'gulf of execution' is caused by poor input devices having inadequate correlation to the system which they model (e.g. controlling a cursor through a voice interface). The 'gulf of evaluation' concerns the output format chosen to represent a system. If this is overly abstractive, misrepresented or a poor visual metaphor then its interpretation by the user will require more effort and be liable to misinterpretation. These 'gulfs' are diminished by intuitive interfaces; design has a crucial role to play in developing inputs and output that mirror the actions and learning processes of the original problem domain.



5.6 Simulator environment design Using the environmental and interactive requirements the simulation environment can be prototyped.

Figure 5. Final application This final environment layout (figure 5) has been achieved through cycles of Rapid Prototyping (see 5.7 Design Cycle). Rapid Prototyping is suited to the development of interactive multi-media applications/HCI. The numerous design factors benefit from the flexibility and user-centred focus it stresses [Hoffman et al, 1996]

HUD Viewer Cut region modelling

Applet CT Viewer

Selectable Interactivity

Task tool

CT scans arrangement

similar to reality

VRML Brain model includes tumour

Task evaluation feedback

Mouse based HCI



5.7 Design Cycle During the development cycle of the project Dr. Phillips and Peter Coltman provided user feedback. This feedback is valuable for assessing and modifying features of the simulator. A record of their observations follows. Friday 25th August: Prototype 1 Peter Coltman had a first trial of the simulator (at this stage it allowed CT scan selection but showed only the upper skull dome). His feedback suggested: 1.Ability to inspect full screen CT scan images would be helpful 2.Problems with double mouse click during operation stage 3.CT scans being viewed should be highlighted on the applet selection 4.Indicators for the location of CT scans from the scout image(e.g. where is 5/13?) Remarks to feedback. Point 1 is a low priority development possibility, the applet already has a viewing facility (but not full screen because of screen size restrictions). Point 2 requires attention as the HCI is key to the simulator's usability. Point 3 is an original requirement and in development. Point 4 is a low priority possibility, though desirable the indexing of the CT images may not be critical or crucial to the learning experience. The priority of points 1 & 4 will remain low unless the expert user (Dr Phillips) makes the same suggestions. Friday 1st September: Release 1 Peter Coltman provided the second user trial feedback: 1. Reiterated point 1(from Friday 25th August) about ability to inspect full size CT

scans 2. Reiterated point 4(from Friday 25th August) about locating the positions of the CT

scans in the skull 3. Identified an error that occurred during the 'instrument led' mode causing the CT

viewer to lock. Remarks to feedback Point 1 will remain a low priority development possibility Point 2, time-scales allow this to be pursued for implementation Point 3 Rectified



Monday 4th September: Release 2 Dr Nick Phillips provided the third trial, his comments are quoted (from e-mail): 1. "The remove button seems to have variable results on my

pc - usually removing half the head then redrawing the appropriate craniotomy. This is OK as long as you warn the trainee."

2. "The instructions on what to do could be clearer." 3. "You need to be able to see the slice numbers better

and this is my fault - perhaps we could relabel them." 4. "Otherwise an excellent piece. BTW you have got

everything the right way round." Remarks to feedback Point 1 This maybe an error Point 2 This relates to instructions on the HTML web page (to be amended) Point 3 The resolution of the images have been reduced to reduce bandwidth requirements Point 4 An improved version will be released on Tuesday 5th September

Tuesday 5th September: Release 2 Final user evaluation by Dr Nick Phillips, his e-mail comments: "Dear Patrick, Its very good. The functionality is excellent. I particularly like the results of the craniotomy printed at the end - the circularity is interesting . It is not an aim of the operation that the craniotomy is circular but it gives interesting information.\ On my NT machine it runs ok in netscape but if I want to reload it can only get the applet craniotomy to run if I shut netscape down and start again. The instrument led option where the ct [CT image] comes up in its own right is very good for training as it trains the surgeon what to expect in the brain at a particular level. This hasnt been possible before to demonstrate in such a clear way. As far as teaching goes it is a good trainer for early surgeons - possibly before they try a craniotomy for the first time and to teach them to order their thoughts about how they are going to tackle the procedure. Eventually it would be nice to get the numbering sorted. Regards Nick" This feedback will be fed into the evaluation stage.



6 Application & Architecture Development The three drivers of the project (functionality, usability and portability) have implications for the development, these are shown in table 6.

Dr iver Desired attr ibutes Implications Portability Efficient

Compact Web generic tools Reusability

Smart algorithms External data Java / VRML OOP / modular design

Usability Fast Smart ('fool proof')

Optimised Routines and Architecture Testing

Functionality Comprehensive Realism/Accuracy

Domain requirements driven Does not compromise design

Table 6. Driver implications

6.1 Architecture The architecture that has been adopted takes full advantage of distributed resources to enable web portability. The applet contains very little internal data, accessing relevant images and data (from the web-server) when required (see figure 6). The modularity of data also has maintenance advantages.

Figure 6. Application architecture See Appendix E for listing of simulator files.

VRML Skull Model

EAI 'Craniotomy' Java Applet

data_car tcoord.txt Cartesian 3D coordinates

of skull model [345 elements]

data_polarcoord.txt Polar 3D coordinates

for skull model [345 elements]

data_polygonindex.txt Polygon Index for skull

model (this is the indexfaceset structure for the

points in textfile 1) [646 elements, indexed]

data_ifsetcomb.txt Search efficient indexfaceset combinations for the skull

model [1938 elements]

Image data VRML

files Data files



6.1.1 VRML structure The VRML skull structure is made up of separate VRML files which are 'inlined' to form the completed skull. The elements are shown in figure 7:

Figure 7. Skull VRML elements

The applet can communicate with the VRML world by using the Java External Authoring Interface (EAI) methods. This allows the VRML world to feed data to the applet and for the applet to produce events and create new VRML nodes [McCarthy & Descartes, 1998].

6.1.2 Data Structure In conjunction with an appropriate architecture, a well designed data structure can improve the usability by increasing performance. The data structure also facilitates the functionality of the application, achieved through well-designed data fields and routines. Having externally held data frees the application from having to do computationally expensive processing. With the demanding trigonometric operations it was decided that the following would deliver performance benefits: 1. Pre-calculate results that would be repeatedly computed 2. Place data in external text files, see table 7 (placing data in the applet makes it

cumbersome, and in the worst case affects portability) 3. Design algorithms that would only have to read the file once (i.e. data files are not

repeatedly open, read and closed) Data file Description Used by Method data_car tcoord.txt Cartesian co-ordinates of skull points In_check(), cut_edge() data_polygonindex.txt VRML Polygon index numbers Poly_removal(), data_polarcoord.txt Polar co-ordinates of skull points In_check(), data_ifsetcomb.txt Search efficient indexfaceset polygon data Polygon_check(),

Table 7. Data file descriptions See Appendix D for data file descriptions.

CRANIUM Has an attached TouchSensor

INSTRUMENT The cranium TouchSensor is rooted to it so that the tip always follows the

cursor on the skull

BRAIN

LOWERSKULL



7 Methods & Algorithms

7.1 Drivers and requirements The ability to operate across the whole of the upper skull requires a more flexible and accurate mathematical modelling method. To facilitate this polar co-ordinate modelling has been adopted, carried out in radians. Radians have the advantage of being the default computational quantities for trigonometry. Also radians are geometrically fundamental quantities (rather than degrees which are derived).

7.2 Modelling

7.2.1 Skull partitioning

Figure 8. Upper skull model (front facing out of page) The VRML skull model was partitioned into two functionally different data sets. The first and most important data set is the upper skull (cranium) data set (shown in figure 8). The upper skull is the operative region (i.e. craniotomy can only take place within this data set - which reflects operational reality). For operation the upper skull requires the following data sets: 1. VRML cartesian data set and indexfaceset required to render the cranium 2. Polar co-ordinate data set (mirroring the VRML Cartesian data set) for each point,

this is used in the cut-region algorithm 3. An indexedfaceset list showing every permutation of the VRML indexedfaceset,

this is to enable efficient searching of cut region points. The origin is defined as the centroid of the all the points that make up the upper skull model. The origin thus lies in the interior of the skull dome. The advantage of this is that the origin can be defined uniquely for this data set and remains unchanged if the upper skull is scaled. The rest of the skull (e.g. lower cranium, jaw and teeth) has it's own VRML data set. These features are added for anatomical reference and cannot be operated on.

x

y

z



7.2.2 Spherical Mapping Spherical mapping involves mapping the points on the upper skull onto the surface of a sphere and then carrying out operations in polar co-ordinates (r, Θ, Φ). Spherical mapping has a number of functional and analytical advantages: 1. The sphere is a geometrically more accurate approximation to the skull than a

planar surface. 2. As in reality, it allows the whole of the upper skull dome to be operated on. 3. The mapping and processing of points in spherical space reduces numerical error

and spatial distortion (e.g. that would occur if mapped to and from a planar surface).

4. Enables a skull model (with a smaller data content) to be used with no loss of qualitative effect

5. Polar co-ordinate analysis has descriptive qualities that are easier to interpret at a raw data level (e.g. Φ = 0 indicates top of cranium, Θ = 0 indicates front radial of cranium).

Figure 9. Points on cranium Figure 10. Points mapped onto a sphere The analytic challenge is to model a cut across the curved surface of the skull, the derivation of the formula is shown below. If A and B are two points on the cranium (see figure 9). The problem is how to model the curved line on the skull's surface that connects them. These can be expressed as position vectors a and b from the Origin O. Vectors a and b are converted into polar unit vectors. These vectors now represent points on a sphere of unit radius (see figure 10). The vectors a and b and the origin O lie on a plane which has a normal n = a ^ b Points a and b on the surface of the sphere lie on a circle of circumference c defined by the locus r.n = 0, where r is a unit vector originating from the origin. r can be expressed in polar co-ordinates: r = iSinΘ SinΦ + jCosΘ SinΦ + kCosΦ

x

n z

y

AB

b a

O

Φ = 0

Θ

c



n can be evaluated numerically from the two vectors a and b i.e. n = a ^ b = inx + jny +knz Evaluating r.n = 0 yields: r.n = ( iSinΘ SinΦ + jCosΘ SinΦ + kCosΦ) . ( inx + jny +knz ) = 0 thus: ( nx SinΘ SinΦ ) + ( ny CosΘ SinΦ ) + ( nz CosΦ ) = 0 [Equation A] The solution requires rearranging equation A such that it yields Φ as a function of Θ:

Step 1: Cos Φ coefficient to the LHS of equation

- nz Cos Φ = ( nx SinΘ Sin Φ ) + ( ny CosΘ Sin Φ )

Step 2:factorise RHS with SinΦ - nz Cos Φ = SinΦ ( nx SinΘ + ny CosΘ )

Step 3:divide through by - SinΦ [NB Sec Φ = Cos Φ / Sin Φ ]

nz Sec Φ = - ( nx SinΘ + ny CosΘ )

Step 4:divide through by nz Sec Φ = - ( nx SinΘ + ny CosΘ ) / nz

Step 5:invert both sides [NB Tan Φ = (Sec Φ)-1 ]

Tan Φ = - nz / ( nx SinΘ + ny CosΘ )

Step 6 :take arctan of both sides of equation

ΦΦΦΦ = arctan [ - nz / ( nx SinΘΘΘΘ + ny CosΘΘΘΘ ) ] [Formula 1]

Formula 1 enables the calculation of Φ [angle of declination] from the angle of longditude, Θ, for the radius of a circle which intersects points a and b.

Application: On the surface of a sphere (of unit radius) the shortest distance between two points, described by vectors A and B, is described by the formula:

ΦAB = - arctan ( nz / [ nx Sin Θ + ny Cos Θ ] ) [Formula 1]

(in the interval ΘA,ΘB)

where: Φ is the angle of declination Θ is the longditudinal angle (nx ,ny , nz) is the vector normal to the surface intersecting OAB (see Normal Evaluator)



7.3 Program Operation The description below outlines the processes that occur during the simulation operation. For more information refer to the bracketed sections in the following 'Discussion of program methods'. 1. Hit-point (from cut point events) received from VRML skull model. The hit-point

is a Cartesian co-ordinate. (See 7.4.1 Callback) 2. Hit-point converted from Cartesian to polar vector from the origin (the centroid of

the skull is defined as the origin). The polar vector has components r,Θ and Φ. (See 7.4.3 Cartesian to Polar Conversion)

3. A check is made of all points on the cranium to see if they lie within the cut

region. This is performed in polar co-ordinates (using Φ and Θ values across the surface of a unit sphere) and employs an 'odd-even rule' [Underhill, 1999]. The routine uses formula 1 (see 7.2.2 Spherical Modelling) to check each skull point. The reference number of each point within the cut region is recorded for use in step 4. (see 7.4.5 Cut region test)

4. The polygons in the VRML skull model affected by the cut region are identified.

(See 7.4.6 Polygon check). Similarly the vertices connected to the vertices to be removed are identified (see 7.4.7 Cut Edge).

5. The reference numbers of each skull point within the cut region (established in the

previous step) are used to alter the VRML skull model indexedfaceset arrangements (See 7.4.8 Cut region removal).

6. A cranial cut modelling routine renders the cut. (See 7.4.9 Cut point sorter &

7.4.10 Cut region modeller) 7. The simulated task is evaluated (See 7.4.11 Task evaluation).



7.4 Discussion of program methods The major methods of the application are shown in table 8. Following the table is a description of the algorithms and significant variables used by the methods to efficiently find solutions.

Method Key Method name Short Description Callback method Callback Initial processing of cut points Normal evaluator Normal Evaluates Cartesian vector product (A^B) Cartesian to Polar Conversion

Polar Produces unit vector V(Θ,Φ)

Centroid Centroid Calculates position of cut region centroid Cut region test In_check Test to see which skull points are within

cut region Polygon check Polygon_check Identifies polygon in cut region Cut Edge Cut_edge Identifies vertices on edge of cut region Cut region removal Poly_removal Remove skull points within cut region Cut point sorter Rotation Cut points arranged into rotational cut

sequence Cut region modeller

Cut_model Smooth cut region production

Task Evaluator Evaluate Calculates task evaluation metrics Table 8. Major application methods

7.4.1 Callback method The callback method records the cut points made on the VRML skull model using a getValue method to record touchSensor events. The cut point cartesian positions are recorded in the array points[][]. This method also contains the cut axis algorithm, to identify the major axis of the operation (this is recorded in the boolean variables: is_y, x_or_z).

7.4.2 Normal Evaluator A quantity required by the cut model is the normal vector between each adjacent cut point vector. This is found by calculating the vector product. This calculation is done after converting the cut points A and B (on the skull surface) into respective Cartesian vectors A and B. The normal is evaluated in Cartesian form using the formula: i j k Normal N = A ^ B = Determinant Ax Ay Az Bx By Bz

Thus: N = i (Ay.Bz - By.Az) - j (Ax.Bz - Bx.Az) + k (Ax.By - Bx.Ay)

∴∴∴∴ N = nx i + ny j + nz k In the code this is accomplished by the method normal().



This method calculates the normal vector components (nx, ny, nz ) and stores them in array normal[][1 - 3]. Additionally the longditudinal region (ΘAΘB) between the cut points A and B are stored in array normal[][4 - 5] by this method.

7.4.3 Cartesian to Polar Conversion (Spherical Mapping) Each cut point is converted into a polar vector (expressing a position on the surface of a sphere) which allows spherical analysis to be performed. A cut point is initially sampled as a Cartesian position vector A on the skull surface, this has components (Ax Ay Az ). A is converted in to a polar vector, with components (r,Θ,Φ) by the following: Magnitude r = �(Ax

2 + Ay2 + Az

2 ) Declination Φ = arccos ( Ay / r ) Longditude Θ = arcsin (Ax / (r * sin Φ )) Method polar() accomplishes this and stores the value for each cutpoint in array polar_points[][0 - 2].

7.4.4 Centroid Calculation Method centroid() calculates the three dimensional vector position of the centroid defined by the Cartesian cut points. This is accomplished by the finding the average value of each Cartesian component. For n cut points AI:

Centoid C = Σ Ai n

Method centroid() stores the centroid coordinates in variables xc,yc,zc.

7.4.5 Cut region test This computation is accomplished by the in_check() method. This method yields the set of skull data points that lie within the cut region. 1. Assume that point X, on the surface of a sphere, is being checked to see if it lies

within a cut region polygon defined on the sphere's surface. 2. The sphere has a constant radius, thus point X on the sphere is uniquely described

by its polar position by X(ΘX,ΦX). Θ Being the longitudinal angle [ interval 0, 2Pi] , Φ is the vertical angle of declination [interval 0 (vertically up), pi (vertically down)] .

3. A hypothetical line is extended from X to the downward pole (Φ = Pi).



Points A and B, given by polar position vectors A = A(ΘA,ΦA) and B = B(ΘB,ΦB), define the two endpoints of a vertex making up a polygon cut region.

IF { ΘA <= ΘX < ΘB OR ΘB < ΘX <= ΘA}

AND

ΦX < Φ(ΘX) [Φ(ΘX) given by analytical formula 1 from 7.2.2 Spherical Mapping]

THEN add 1 to Counter

4. This is repeated for every vertex of the cut region. A count is made of the number

of times the 'line to the pole' from X bisects the vertices of the polygon. 5. After every vertex of the polygon is checked for bisection with the point X, the

counter is checked: If the Counter is even, Point X lies outside the polygon If the Counter is odd, Point X is within the polygon (and hence the Cut region) This uses an odd-even check [Underhill, 1999] to find out if a point on the VRML indexfaceset set is within the polygon that describes the cut region. The index of the indexfaceset points that lie within the cut region are stored in the array points_to_remove[].

7.4.6 Polygon check Method polygon_check() uses the index of points in the array points_to_remove[] (from method in_check() ) for two purposes: 1. Identify the VRML indexfaceset polygon that need to be 'culled' from the skull.

(See 7.4.8 Cut region removal). 2. Identify all VRML skull vertices on the edge of the cut region (i.e. directly

connected to cut region vertices). The indices of the identified polygons and vertices are stored in the arrays polygons[] and vertices[] respectively.

7.4.7 Cut Edge The method cut_edge() uses the set of vertices (in array vertices[]) and the set of cut region vertices (in array points_to_remove[]). The first task of this method is to identify the vertices that are on the edge of the cut region. The cut edge vertices are then stored in rotational order around the centroid (see Centroid, above) in the array cutedgevertices[][].



7.4.8 Cut region removal The polygons identified as lying in the cut region are removed (culled) by the method poly_removal(). This removal is the first step in rendering the craniotomy cut. The indices of the polygons to be removed are stored in array polygons[] (see 7.4.6 Polygon check). The polygon indices refer to the polygon referencing in the VRML indexfaceset. The method poly_removal() reads the data file 'data_polygonindex.txt' and amends the polygons in the rendered VRML skull model using the set1Value method.

7.4.9 Cut point sorter The method rotation() calculates the angle of each cut point around the centroid (see Centroid, above). These angle values are stored in array points_angle[]. The angles specify the cut sequence between the cut points. In order to accomplish the task the rotation() method also depends on the cut axis plane, specified in the callback() method (using boolean variables: is_y, x_or_z).

7.4.10 Cut region modeller The task of graphically rendering the cut is handled by the cut_model() method. The cut is modelled from two sets of data points, the cut points vectors (stored in array points[][], see 7.4.1 Callback) and the cut edge vertices (stored in array cutedgevertices[][], see 7.4.7 Cut Edge). The cut is modelled by a series of triangles using a 'greedy triangle' type algorithm.

Figure 11. Cut modelling result A modelled cut region result is shown in figure 11, the cut region is the dark brown area between the cut points and the cut edge (left by the culled polygons).

7.4.11 Task evaluation Three criteria selected from the cognitive and task analysis have been implemented for evaluation. These are shown in table 9. Task Evaluation Method of calculation Distance from tumour Distance from centroid of cut region to centre of tumour on skull

surface (using centroid variables xc,yc,zc) Cut area Used Hero's formula to calculate area of as sum of triangles

connecting the cut points to the cut region centroid* (using array points[][] and centroid variables xc,yc,zc)

Circularity Calculated as area of cut divided by area of a circle with radius equal to average distance between the cut points and the cut region centroid. (Answer >=1)

Table 9. Task evaluation criteria

• Area of triangle with side lengths a,b,c is given by Area = �[s(s-a)(s-b)(s-c)], where s = (a+b+c)/2 [Loy, 1998]



8 Development History

8.1 Project Changes There was only one major change during the project, this was the discarding of one of the original objectives: " Investigating a collaborative teaching mechanism. A collaboration facility will require an architecture to be developed in conjunction with HCI design considerations."

This followed a discussion with Nigel John (10th July 2000) regarding the use of Deepmatrix (a VRML browser that allows collaborative interaction). Unfortunately, Deepmatrix requires modification if the collaborative worlds are run in conjunction with Java applets. This modification was deemed out of scope and not realistically feasible during the project. In compensation the project accepted Dr Phillips's suggestion that CT scan image should be built into the interactive environment.

8.2 Project Time Scales The major milestones and phases of the project are shown in table 10. The development involved overlapping stages of analysis, design and development. The time scale of the project and availability of advice impacted upon the development time with the outcome that the application development was only completed near to the project deadline (7th September).

Phase Start Completion Project Week (Original Simulator research) 24/4/00 8/5/00 Evaluation of original simulator 5/6/00 9/6/00 Surgical research 5/6/00 ~1/8/00 Virtual Environment Research 5/6/00 ~1/8/00 Requirements Collection and Analysis 5/6/00 1/7/00 Method and procedures modelling 19/6/00 14/6/00 VRML data-set partitioning and tuning 7/7/00 20/7/00 Coding of application 19/6/00 5/9/00 Architecture 26/6/00 25/8/00 Simulation Design 28/6/00 24/8/00 Build Simulation environment (rapid prototyping) 16/8/00 5/9/00 Project Report 15/7/00 6/9/00

Table 10. Major project phases [dates approximated from project log]

The project log is viewable: http://www.csdb.leeds.ac.uk/mscyps/proj log.htm



9 Evaluation of Project

9.1 Simulation Evaluation With the evaluation feedback from Dr Phillips (see 5.7 Design Cycle) it is possible to carry out an evaluation of the simulator commenting on its cognitive value as a training aid. The ISO 9126 evaluation scheme has been used which is the same as for the evaluation of the original simulator. The evaluation makes its comparison with the original craniotomy simulator [Underhill, 1999]. The simulator can be viewed at: http://www.csdb.leeds.ac.uk/mscyps/craniotomy.htm Functionality Accuracy Skull model The simulator produces a qualitatively accurate model of the skull. The cut model represents the most substantial improvement in graphic accuracy and the cut is modelled to a user satisfied extent. Suitability "For whom? And what task" Dr Phillips believes that the simulation in its current state can be used for the early training of surgeons, in particular to help visualise the relationship of CT scans to the skull and brain. Moreover, its ease of use and graphical interface enable it to be used by a wide variety of users (e.g. school pupils, patients, medical visualisers). Usability Understandability The graphics based visualisation of data makes the context and interpretation of image/information easier to ascertain and assimilate. Learnability Three levels of selectable interactivity differentiates the challenge of the task (i.e. allows the level of cognitive challenge to be varied by selecting the level of image interaction they would like to learn/practise with. Task feedback gives metrics as to how accurately the task craniotomy was performed (against the criteria defined during development). [Dr Phillips suggests that these can be modified] Operability The mouse interface and graphical interfaces (e.g. radio buttons, clickable images) reduces the 'gulf of execution' (see 5.5 Interaction Model). The scope of actions permitted by the environment are represented in an intuitive and visual manner. Maintainability Changeability Separation of high level Java applet code and VRML modelling script enables straightforward modification (though care must be taken of links and dependencies). Stability Stability acceptable for user evaluation.



Reliability Maturity Lack of timescale has prevented the simulator from being fully user tested. Fault Tolerance Java applet has try-catch mechanisms to handle errors when they occur, though application appears durable. Efficiency Resource Utilisation The demands on system virtual memory seem to have reduced as application 'crashes' have not occurred. [This is probably a benefit of using files of pre-calculated data] Portability Adaptability The application now runs on both Internet Explorer 4+ and Netscape Navigator with VRML plug-ins. From anecdotal evidence it runs better on Explorer (maybe because of compatibility problems inheritent from development on a Microsoft product) Installability Java 1.2 and VRML2, portable across web, though client is dependent on having a VRML2 enabled browser.

9.2 Scope for improvement There is substantial scope for improving the quality of the simulation. Future challenges that have been identified during this development are: 1. Implementing a more rigorous object orientated design (OO) will provide the

simulator with a more stable organisation for on-going modification. 2. The polygon removal can be enhanced such that it removes a larger area and

produces more reliable cut models 3. Improving the conformity between head, brain and CT images. 4. An architecture that enables patient specific data to be diagnosed/practised on 5. Collaborative training both synchronously and asynchronously. This may involve

browser development to enable applets to interact with VRML worlds. 6. An investigation into the extent of cognitive optimisation in the simulation

environment (i.e. which features are most effective? to whom? what principles and conclusions can be drawn from it?)

7. Extend the multimedia and pedagogical scope of the simulation (e.g. implement intelligent agents, context based assistance, further multimedia).



10 Conclusion The project has successfully addressed its objectives. This has been demonstrated by its successful development and evaluation feedback from stakeholder-users. In comparison to the demonstration simulation developed by Sam Underhill this simulation has a much improved environment (with tools, interactivity and additional multimedia), cut-modelling, operability over the whole skull and better portability. The simulator also provides task feedback to the user. The design has been driven by cognitive and task requirements to ensure that the interactivity and functionality is appropriate to the learning task. Domain research and collected user requirements are integral elements of this project's success. Usability has improved, with the simulator having an intuitive HCI and fast operation. The simulator has error handling capability but the lack of time prevented rigorous usability testing. A more thorough evaluation of the simulator's cognitive value would have been desirable. Lack of time prevented this. However, anecdotal and qualitative evidence overwhelmingly supports the conclusion that the simulator is a vastly improved training resource. This project approached the simulation environment design and application / architecture development as two parallel, inter-related activities. Thus both were given equal emphasis which is critical for addressing the primary drivers of usability, functionality and portability. With short time scales it was important to overlap stages, such as design and development. This was facilitated by a rapid prototyping approach applied to the simulation design and a modular approach applied to the development of the program methods. An earlier indication of what was achievable technically during the project and a more rapid selection of the project's objectives would have been the most desirable 'hindsight' improvements. Crucially, the simulation has been judged by the expert user (Dr Nick Phillips) as being a tool that can be used for initial surgeon training, moreover it is also appropriate as a teaching tool for non-specialists. The satisfaction of the user community is a major achievement of this project.



Bibliography Bell D, Parr M, "Java for Students", 2nd Edition, Prentice-Hall, Hemel Hempstead, 1999 Brodlie K, El-Khalili N, Li Y, " Using Web-based Computer Graphics to Teach Surgery" , Pergamon, Computer & Graphics 24 (2000), 2000 Bystom K, Barfield W, Hendrix C, "A Conceptual Model of the Sense of Presence in Virtual Environments", 'Presence' Journal, Vol. 8, No.2, pp241-244, April 1999 Bystrom K, Barfield W (Virginia Polytechnic Institute and State University), "Collaborative Task Performance for learning using a Virtual Environment", 'Presence' Journal, Vol 8, No.4, pp435-448, August 1999 Delp S, Loan P, Basdogan C, Rosen J, "Surgical Simulation: An emerging Technology for Training in Emergency Medicine", Presence Journal Vol 6, No.2, pp147-159, April 1997 Dix A, Finlay J, Abowd G, Beale R, "Human-Computer Interaction", 2nd Edition, Prentice-Hall, Hemel Hempstead, 1998 Dudley H (compiler), "Operative Surgery Selections", (Rob & Smith Atlas of General Surgery), pp 685-696, Butterworth, London, 1981 Ehrlich K, "Designing Groupware Applications: A work-centred design approach", Chapter 1, CSCW, M. Beaudouin-Lafen (editor), pp1-24, Wiley, Chichester, 1999 Gupta S, Klein S, Mehl D, Verdi M, Andersen G, "Introduction of New Technologies to the Medical Undergraduate Curriculum", Chapter 25, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care in the Information Age" (Proceedings of the Medicine meets Virtual Reality 4 conference, San Diego, USA, 1996), pp216-217,IOS Press, Amsterdam, 1996 Hargreaves D, Bowditch M, Griffin D, "On the Job Training for Surgeons", Royal School of Medicine Press, London, 1997 Hartman J and Wernecke J, "The VRML 2.0 Handbook", Addison-Wesley, Reading, Mass., 1996 Higgins G, Merrill G, Hettinger L, Kaufmann C, Champion H, Satava R, "New Simulation Technologies for Surgical Training and Certification", Presence Journal Vol 6, No.2, pp160-172, April 1997 Hoffman H, Murray M, Irwin A, McCracken T, (School of Medicine, Univ. of California, San Diego, USA), "Developing a Virtual Reality - Multimedia System for Anatomy Training", Chapter 24, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care in the Information Age" (Proceedings of the Medicine meets Virtual



Reality 4 conference, San Diego, USA, 1996), pp 205-207, IOS Press, Amsterdam, 1996 Hon D. (Ixion Inc., Seattle, USA), "Medical Reality and Virtual Reality", Chapter 41, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care in the Information Age" (Proceedings of the Medicine meets Virtual Reality 4 conference, San Diego, USA, 1996), pp331-333, IOS Press, Amsterdam, 1996 Howe T and Sharkey P, "Identifying Likely Successful Users of Virtual Reality Systems", Presence Journal Vol 7, No.3, pp308-316, June 1998 Johnson W, Rickel J, Stiles R, Munro A, "Integrating Pedagogical Agents into Virtual Environments", Presence Journal Vol 7, No.6, pp523-546, December 1998 Johnston R, Bhoyru S, Way L, Satava R, McGovern K, Fletcher J, Rangel S, Loftin B, "Assessing a Virtual Reality Skills Simulator ", Chapter 66, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care in the Information Age" (Proceedings of the Medicine meets Virtual Reality 4 conference, San Diego, USA, 1996), pp608-615, IOS Press, Amsterdam, 1996 Khanna R, "What is Laparoscopic Surgery", http://personal.vsnl.com/docrajiv/index.html, June 2000 Kockro R et al., "Planning and Simulation of Neurosurgery in a Virtual Reality Environment", Neurosurgery Journal , Vol 46, Number 1, pp118-137, 2000 Loy J, "Jim Loy's mathematics Page", http://www.mcn.net/~jimloy/math.html, September 2000 MacCarthy M and Descartes A, "Reality Architecture", Prentice Hall Europe, Hemel Hempstead, 1998 McGovern K and Johnston R,"The Role of Computer-Based Simulation for Training Surgeons", Chapter 42, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care in the Information Age" (Proceedings of the Medicine meets Virtual Reality 4 conference, San Diego, USA, 1996), pp342-345, IOS Press, Amsterdam, 1996 Petry J MD, "Surgery and Meaning", 'Surgery' Journal, Vol.127, No.4, April 2000 Preminger G, Babyan R, Merrill G, Raju R, Millman A, Merrill J,"Virtual Reality Surgical Simulation in Endoscopic Urologic Sugery", Chapter 19, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care in the Information Age" (Proceedings of the Medicine meets Virtual Reality 4 conference, San Diego, USA, 1996), pp 157-163, IOS Press, Amsterdam, 1996 Reinig K, Rush C, Pelster H, Spitzer V, Heath J, (The Center for Human Simulation, Univ. of Colorado, USA), "Real-Time Visually and Haptically Accurate Surgical Simulation", Chapter 60, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care



in the Information Age" (Proceedings of the Medicine meets Virtual Reality 4 conference, San Diego, USA, 1996), pp542-545, IOS Press, Amsterdam, 1996 Riva G (Instituto Auxologico Italiano, Verbania), "Virtual Reality as a Communication Tool: A Sociocognitive Analysis", 'Presence' Journal, Vol.8, No.4, pp462-468, August 1999 Salzmann M, Dede C, Loftin R, Chen J, "A Model for Understanding how Virtual Reality Aids Complex Conceptual Learning", 'Presence' Journal Vol. 8, No.3, pp293-316, June 1999 Satava R. (Walter Reed Army Medical Center, Washington DC), "Medical Virtual Reality: The Current Status of the Future", Chapter 12, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care in the Information Age" (Proceedings of the Medicine meets Virtual Reality 4 conference, San Diego, USA, 1996), p100, IOS Press, Amsterdam, 1996 Satava R and Jones S, "Virtual Environments for Medical Training and Education", Presence Journal Vol 6, No.2, pp139-146, April 1997 Stredney D, Sessanna D, McDonald J, Hiemenz L, Rosenberg L, "A Virtual Simulation Environment for Learning Epidural Anesthesia", Chapter 20, Sieburg H, Weghorst S, Morgan K (Editors), "Health Care in the Information Age" (Proceedings of the Medicine meets Virtual Reality 4 conference, San Diego, USA, 1996), pp164-166, IOS Press, Amsterdam, 1996 Underhill S, " Interactive Computer Visualisation of a Pterional Craniotomy" , BSc Final year project, School of Computer Studies, Leeds University, 1999 Welch R (NASA - Ames Research Centre), "How can we determine if the sense of presence affects task per formance", 'Presence' Journal, Vol 8, No.5, pp574-577, October 1999 Additional Resources: "Floppy's VRML Guide", http://www.vapourtech.com/vrmlguide/, August 2000 MVC (Manchester Visualisation Centre), http://www.man.ac.uk/MVC/research/visual/medical/medical.shtml#projects, July 2000



Appendix A: Personal Experience This project has been satisfying from a number of perspectives. It has been an opportunity to use web-based technologies to produce a useful application driven by user requirements. The project has provided challenge on a number of levels; the most valuable of which has been the use of Java in a substantial web-based development. At a practical level I feel that I have a grasp of the issues and challenges that exist at each stage of the development cycle of an interactive VR/web application. The scope of the project has provided insight into different aspects of interactive application development. Starting from the collection of user requirements, I then piloted a combined simulation design and application development. This is a relatively immature field in multimedia and VR development. The design stage enabled me to research contemporary practise in cognitive, task and interaction modelling. All of which are crucial for optimising the strengths of interactive multimedia to the learning objectives. The method and algorithm design was a source of many interesting trigonometric and spatial problems, which were satisfying to solve. [A recurrent 'niggle' of using polar co-ordinates in computation is the problem of performing trigonometric operations across the 0/2pi interface in the Θ axis]. The pressure of project time scales prevented me from practising a more object-orientated development. This is a fundamental strength of Java, and it is unfortunate that I didn't feel confident or experienced enough to develop the application in a true OO manner. The rapid prototyping stage of the simulation environment enabled stakeholder involvement and made the cycle very swift. This was a very satisfying experience for both myself, and the stakeholders, who could see their suggestions taking shape. On reflection the project was probably too large in scope, than would have been otherwise ideal, for a masters project. However, it has been successful both from an objectives and (more importantly) from a user/stakeholder perspective. I consider its completion a personal achievement.



Appendix B: Copy of the Objectives & Deliverables form



Appendix C: Marking Scheme



Appendix D: External Data File Structure The external data files have the following structure and sit on the web-server with the other simulation programs. File: " data_cartcoord.txt" Cartesian 3D coordinates of skull model [345 elements]

0.0 0.6978178 0.79836005, 0.0 -0.715878 -0.56739396, 0.0 -0.6759081 -0.660258, 0.0 -0.4428978 -0.748706,

File: " data_polygonindex.txt" Polygon Index for skull model (this is the indexfaceset structure for the points in textfile 1) [646 elements, indexed] This table lists indexedfaceset polygon pattern that makes up the VRML cranium model. The polygons in the VRML model are all triangular. Each triangle is defined by the indices of three vertex points (defined in the VRML script).

275 276 245 -1 0 275 244 216 -1 1 275 216 217 -1 2 326 298 341 -1 3

File: " data_polarcoord.txt" Polar 3D coordinates for skull model [345 elements] This data is used to identify which of the vertex points in the cranium VRML set exist within the cut region. [NB the test is done in polar co-ordinates as opposed to VRML Cartesian; the values in this data field reduce the requirements of system resources as they are pre-calculated]

0.80140394 0.0 1.4836115 -1 0.9134644 3.1415927 2.4713895 -1 0.9448769 3.1415927 2.3679066 -1 0.86989605 3.1415927 2.1049802 -1

The fifth number is the polygon index (in the VRML indexfaceset)

Polar values given by: First number is magnitude (r) Second number is longitude (Θ) Third number is declination (Φ)

Polar values given by: First number is x coordinate Second number is y coordinate Third number is z coordinate



File: " data_ifsetcomb.txt" Search efficient indexfaceset combinations for the skull model [1938 elements] This data field allows the polygons that have a particular constituent vertex to be identified. This is accomplished by listing the three permutations of the polygon sequence:

0 169 337 -1 48 0 226 319 -1 9 0 319 337 -1 47 1 2 110 -1 181 1 2 288 -1 623

First 3 numbers are indices for Cartesian points held in textfile1.txt

The fifth number is the polygon index (in the VRML indexfaceset)



Appendix E: List of application files The simulation is constituted of several different web distributed files. At the time of writing these resided at http://www.csdb.leeds.ac.uk/mscyps/ Java class files: craniotomy_sim.class (Applet)

smallpic.class (Object) largepic.class (Object)

Java file: craniotomy_sim.java (Not necessary to run simulator) Data files: data_cartcoord.txt

data_polygonindex.txt data_polarcoord.txt data_ifsetcomb.txt

Simulator Images: ct_image4.jpg

through ct_image23.jpg (20 CT scan images) scout.jpg (CT scan reference image)

Web page: craniotomy.htm (Simulation is embedded in this) Web page images: modes.jpg

skullview.jpg VRML files: cranium_nodes.wrl (main model with upper skull)

lowerskull.wrl (inlined face and jaw) brain.wrl (inlined brain) instrument.wrl (inlined tool)

Presentation: project_presentation.ppt (Presentation regarding project)



Appendix F Program Code

Patrick Sim University of Leeds MSc DMS Project 1999/00 ... · PDF file3.0.3 How can Virtual...

Documents

Transcript of Patrick Sim University of Leeds MSc DMS Project 1999/00 ... · PDF file3.0.3 How can Virtual...