WG5WG5Steve van Albert, Rajeev Alur, Douglas

Barton, Ralph DePalma, Matthew Dwyer, Steven Getz, LeRoy Jones, Peter Kuzmak,

Peter Lee, Rami Melhem, Jens Palsberg, John Regehr, Victoria Rich, Michael Robkin, Gregg Rothermel, Vish Sankaran, Bow-Yaw Wang

Steve van Albert, Rajeev Alur, Douglas Barton, Ralph DePalma, Matthew Dwyer, Steven Getz, LeRoy Jones, Peter Kuzmak,

Peter Lee, Rami Melhem, Jens Palsberg, John Regehr, Victoria Rich, Michael Robkin, Gregg Rothermel, Vish Sankaran, Bow-Yaw Wang

High-confidence Medical Device High-confidence Medical Device Software Development and Software Development and

Assurance /Assurance /Medical Practice-driven ModelsMedical Practice-driven Models

General observationsGeneral observations

•Medical device software, while potentially safety-critical in nature, appears not to be developed using practices well-accepted by other safety-critical domains, most notably avionics.

•The proprietary nature of medical device products is a barrier to participation by academic researchers.

•Software quality may not be the critical issue; however IT is a key enabler and shows clear benefits in the cost and quality of patient care.

•Medical device software, while potentially safety-critical in nature, appears not to be developed using practices well-accepted by other safety-critical domains, most notably avionics.

•The proprietary nature of medical device products is a barrier to participation by academic researchers.

•Software quality may not be the critical issue; however IT is a key enabler and shows clear benefits in the cost and quality of patient care.

Problem summary, 1Problem summary, 1

•Software quality. Specific IT-based medical systems have clearly identified faults/defects, some of which have led to loss of medical records and adverse events. In some cases, technical solutions exist but other barriers (some non-technical) prevent their application. These include lack of standards, interoperability, and metrics.

•Software quality. Specific IT-based medical systems have clearly identified faults/defects, some of which have led to loss of medical records and adverse events. In some cases, technical solutions exist but other barriers (some non-technical) prevent their application. These include lack of standards, interoperability, and metrics.

ExamplesExamples

•Uniqueness of DICOM / HL7 identifiers is neither standardized in the industry nor is it properly implemented in deployed devices, leading directly to loss of medical records.

•BCMA is unreliable. Barcoding is not standard across blood suppliers. Furthermore, barcode tags are physically unreliable and readers can be faulty. This has led directly to patient death.

•Others: VisiCue software errors; WebVista errors induced by cookies; ...

•Uniqueness of DICOM / HL7 identifiers is neither standardized in the industry nor is it properly implemented in deployed devices, leading directly to loss of medical records.

•BCMA is unreliable. Barcoding is not standard across blood suppliers. Furthermore, barcode tags are physically unreliable and readers can be faulty. This has led directly to patient death.

•Others: VisiCue software errors; WebVista errors induced by cookies; ...

Problem summary, 2Problem summary, 2

•Feature creep. There is a lack of visibility in medical device software and the processes used to create it. Feature creep leads to interactions in the software that are hard to specify and even harder to analyze. The result is that no one, not even the developers, understand how the systems behave, nor do they provide metrics on how well they work.

•Feature creep. There is a lack of visibility in medical device software and the processes used to create it. Feature creep leads to interactions in the software that are hard to specify and even harder to analyze. The result is that no one, not even the developers, understand how the systems behave, nor do they provide metrics on how well they work.

ExamplesExamples

•Feature interactions are a major problem. The value of testing and analysis techniques for validation has not been investigated.

•Developers lack metrics for reliability, usability, etc. This makes it impossible to assess the costs and benefits of validation approaches.

•Feature interactions are a major problem. The value of testing and analysis techniques for validation has not been investigated.

•Developers lack metrics for reliability, usability, etc. This makes it impossible to assess the costs and benefits of validation approaches.

Problem summary, 3Problem summary, 3

•Usability. The proliferation of information technologies has provided clear benefits for the quality and cost of patient care. But it may introduce significant and often unnecessary complications to hospital procedures. This can lead to situations that are confusing to caregivers (e.g., task overload) and unacceptably perilous to patients.

•Usability. The proliferation of information technologies has provided clear benefits for the quality and cost of patient care. But it may introduce significant and often unnecessary complications to hospital procedures. This can lead to situations that are confusing to caregivers (e.g., task overload) and unacceptably perilous to patients.

ExamplesExamples

•Practitioners are on a gadget treadmill. Besides the obvious training and operational challenges, version skew is inevitable and comes at times with severe, even disabling, consequences.

•Unlike aircraft, medical devices often do not have “black boxes,” making audit and postmortem analysis of failures difficult or even impossible.

• Interventional procedures have become extremely complicated, even unsafe, making it hard to know when all systems are operational and ready to perform.

•Practitioners are on a gadget treadmill. Besides the obvious training and operational challenges, version skew is inevitable and comes at times with severe, even disabling, consequences.

•Unlike aircraft, medical devices often do not have “black boxes,” making audit and postmortem analysis of failures difficult or even impossible.

• Interventional procedures have become extremely complicated, even unsafe, making it hard to know when all systems are operational and ready to perform.

Problem summary, 4Problem summary, 4

•Knowledge sharing. Knowledge saves lives. Data can be transformed into information. However, sharing of information is hampered by the lack of interoperability standards and technologies, and by confusion over privacy regulations (e.g., HIPAA).

•Knowledge sharing. Knowledge saves lives. Data can be transformed into information. However, sharing of information is hampered by the lack of interoperability standards and technologies, and by confusion over privacy regulations (e.g., HIPAA).

ExamplesExamples

•The proprietary nature of the devices and privacy requirements means that researchers and industry lack access to platforms and data needed for innovation.

•Lack of standardization in BCMA, DICOM IDs, etc.

•The proprietary nature of the devices and privacy requirements means that researchers and industry lack access to platforms and data needed for innovation.

•Lack of standardization in BCMA, DICOM IDs, etc.

Problem summary, 5Problem summary, 5

•Systems engineering. Integration of technology into the clinical environment (including the practitioners, workflows, and specific devices) often lacks a total system perspective. Specifically, device integration capabilities, interoperability, safety features are not considered during development, acquisition, or deployment. Some safety issues are systems issues. This results in adverse events and lost opportunities.

•Systems engineering. Integration of technology into the clinical environment (including the practitioners, workflows, and specific devices) often lacks a total system perspective. Specifically, device integration capabilities, interoperability, safety features are not considered during development, acquisition, or deployment. Some safety issues are systems issues. This results in adverse events and lost opportunities.

ExamplesExamples

•Maintaining configurations is difficult. Understanding the impact of version and feature upgrades in a large hospital context is difficult.

•Analysis and modeling tools used in other industries (e.g., automotive) are not widely adopted by the medical device industry.

•Patient modeling would help.

•Maintaining configurations is difficult. Understanding the impact of version and feature upgrades in a large hospital context is difficult.

•Analysis and modeling tools used in other industries (e.g., automotive) are not widely adopted by the medical device industry.

•Patient modeling would help.

Summary of state of the art

Summary of state of the art

•algorithms for distributed unique identifiers

•software processes for ensuring “five 9s” safety and availability in avionics

•double-helix spiral development model (e.g., CPoF)

•open research platform development and dissemination (a la motes/TinyOS)

•modeling tools; analysis tools

• lightweight formal methods; government-mandated formal methods (Europe)

•algorithms for distributed unique identifiers

•software processes for ensuring “five 9s” safety and availability in avionics

•double-helix spiral development model (e.g., CPoF)

•open research platform development and dissemination (a la motes/TinyOS)

•modeling tools; analysis tools

• lightweight formal methods; government-mandated formal methods (Europe)

R&D challengesR&D challenges

•Realistic experimental platforms and data for research purposes

•Analysis/validation/verification of feature interactions

•Metrics for reliability, usability, etc.

•Transparency, interoperability, and reliability in the face of market forces that promote features and low cost

• Integration of disparate systems into a coherent whole

•Realistic experimental platforms and data for research purposes

•Analysis/validation/verification of feature interactions

•Metrics for reliability, usability, etc.

•Transparency, interoperability, and reliability in the face of market forces that promote features and low cost

• Integration of disparate systems into a coherent whole

Research needsResearch needs

•Not just IT. This requires an interdisciplinary, whole systems approach involving computer scientists expert in areas such as software engineering and HCI, as well as caregivers, biomedical engineers, and device manufacturers.

•Not just IT. This requires an interdisciplinary, whole systems approach involving computer scientists expert in areas such as software engineering and HCI, as well as caregivers, biomedical engineers, and device manufacturers.

Research needsResearch needs•Rich, open experimental platforms and

testbeds

•Measurable assurance: metrics and standards for reliability, usability, interoperability, etc.

•New methods for high-confidence software and systems, addressing independently and together interoperability, cost, feature interaction, privacy, visibility, and usability

•Methods for evaluating the cost-benefit tradeoffs in various methods for high-confidence systems

•Patient modeling and systems engineering of clinical processes

•Rich, open experimental platforms and testbeds

•Measurable assurance: metrics and standards for reliability, usability, interoperability, etc.

•New methods for high-confidence software and systems, addressing independently and together interoperability, cost, feature interaction, privacy, visibility, and usability

•Methods for evaluating the cost-benefit tradeoffs in various methods for high-confidence systems

•Patient modeling and systems engineering of clinical processes

Roadmap, 1Roadmap, 1

•Develop open experimental platforms and testbeds to foster collaboration between academic, industry, and government

•Regulatory mechanisms should be supportive of the platforms and encourage a systems approach, facilitate integration of components

•Support and encourage the development and use objective measures

•Develop open experimental platforms and testbeds to foster collaboration between academic, industry, and government

•Regulatory mechanisms should be supportive of the platforms and encourage a systems approach, facilitate integration of components

•Support and encourage the development and use objective measures

Roadmap, 2Roadmap, 2

•Develop and perform experimental evaluation of the effectiveness of integrated systems

• Involves extraction of models from clinicians, analysis of change effects, measurement of system performance, etc. to support a systems engineering approach

•NCO should coordinate the federal gov’t, state gov’ts, providers and industry to remove regulatory / legal roadblocks to encourage new IT-facilitated processes.

•Develop and perform experimental evaluation of the effectiveness of integrated systems

• Involves extraction of models from clinicians, analysis of change effects, measurement of system performance, etc. to support a systems engineering approach

•NCO should coordinate the federal gov’t, state gov’ts, providers and industry to remove regulatory / legal roadblocks to encourage new IT-facilitated processes.

Roadmap, 3Roadmap, 3

•Academia, manufacturers, and government should work together to promote standardized barcode technologies in bloodbank applications and administration.

•Similarly, promote universally unique identifiers for medical information objects.

•Academia, manufacturers, and government should work together to promote standardized barcode technologies in bloodbank applications and administration.

•Similarly, promote universally unique identifiers for medical information objects.