Human-Centered Systems Engineering

Design controls, a relatively new name for a nearly century-oldsystems engineering paradigm, describe an engineering man-agement process that serves both producers and consumers.In my engineering practice, I have observed the use, misuse,

and abuse of design controls. Misuse and abuse are not economicallyadvantageous to the producer and create risks for the consumer.

Understanding what are design controls, how and when theyare used, and why they add value may encourage proper use

and mitigate risks for both producers and consumers.

The Use, Misuse, and Abuse of Design ControlsThe U.S. Food and Drug Administration (FDA)issued a final rule called the quality system regula-

tion in the Fall of 1996. Although the primaryemphasis was a recast of good manufacturingpractices regulations for medical devices, itcontained a unique section called designcontrols. In my engineering practice since1996, I have observed three fundamental ap-proaches to design control compliance:1) continuous improvement understandingand following the process, 2) reinterpreta-tion of the terminology to conform toexisting internal product developmentpractices, and 3) a posteriori creation ofthe requisite design documentation. Thelatter two, misuse and abuse of designcontrols, are not economically advanta-geous; they do little to improve theattributes of effectiveness of the product(Table 1) and ultimately result in areduced internal rate of return (IRR) forboth the producer and the consumer. Mypersonal observation is that those whomisuse or abuse design controls duringdevelopment of their medical productsseem to have a much higher incidence of

expensive product recalls and plaintiff liti-gation. Understanding exactly what are

design controls, how and when they areused, and why they add value may mitigate

risks for both producers and consumers.

© STOCKBYTE

Digital Object Identifier 10.1109/MEMB.2010.936551

BY GEORGE M. SAMARAS

12 IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE 0739-5175/10/$26.00©2010IEEE MAY/JUNE 2010

What Are Design ControlsThe origins of engineering design con-trols (though not the name) trace back toclassical systems engineering [1]. Theyare identified by both FDA’s CFR Part820.30 [2] and by international consen-sus standard ISO 13485 x7.3 [3]. Thecorrect application of engineering designcontrols reduces the risks for both pro-ducers and consumers, decreases time tomarket for viable products, and satisficesidentified stakeholders’ needs, wants,and (often) desires (NWDs). (Needs arewhat each stakeholder believes theymust have. Wants are what each stake-holder believes they would like to have.Desires, also called latent needs, are notknown in advance by the stakeholders,but they know it when they see it.) Satisfice, a term coined bySimon [4], means to obtain a good result that is good enough,though not necessarily the best, for each stakeholder. Therationale is that different stakeholder groups have evolving andconflicting NWDs; these stakeholder dissonances must be rec-onciled. Design controls are processes that ensure a bottom-upproduct development approach (satisficing stakeholder NWDs)and not a top-down approach (finding a new intended use for agiven technology).

Design controls may be applied to the development of prod-ucts, processes, or services. For a medical device, Figure 1shows the state space—a three-dimensional microergonomicsystems engineering state space—that incorporates all possibleengineering tasks throughout a tool’s full life cycle from con-cept to disposal [1]. The domain of all these activities (seeTable 2) consists of requirements engineering (what to build),compliance engineering (what not to build), and reliability engi-neering (reducing risks for both producer and consumer). [Com-pliance engineering may be viewed as what not to do: do nothave accessible sharp points near wiring harnesses, do not haveexposed high-voltage conductors or connection points, do nothave product emit nonessential electromagnetic levels beyond acertain wattage, etc.] The range of these design engineeringactivities is hardware design, software design, human factorsdesign, and seller/purchaser (S/P) economics design (the lattertwo being industrial engineering activities); each of these ischaracterized by multiple subdisciplines. Figure 2 shows theprincipal elements of the iterative design control process con-trasted with the four elements of the scientific method.

In Figure 2, the initial task following conceptualization, and ineach subsequent iteration, is identification of all the stakeholders

and their NWDs. After reconciliation, a subset of these NWDsdeemed technologically and economically feasible is chosen forformulation as design inputs. These design inputs are the prob-lems presented to the technical staff; their solution (their workproducts) are the design outputs. In the transformation from rec-onciled stakeholders’ NWDs to design outputs, a number of addi-tional engineering activities take place, including developmentplanning, risk management, five types of verifications, and

Table 1. Nine design attributes of effectiveness.

Functional safety Device helps (intended use)Physical safety Device does not physically hurt (basic safety)Functional security Device prevents data loss or corruption (integrity)Physical security Device cannot be damaged or stolen (denial of service)Usability Device reduces probability of errors in intended use by

intended usersReliability Device operates as intended in intended use environment

for intended lifetimeMaintainability Device repaired in reasonable time at reasonable costAvailability Device accessible when and where it is actually neededAffordability Device manufacturer and end user each obtain accepta-

ble IRR (real cost)

IRR: internal rate of return.

S/P EconomicsDesign

ErgonomicsDesign

SoftwareDesign

HardwareDesign

RequirementsEngineering

ComplianceEngineering

ReliabilityEngineering

DomainConcept

RDDT&E

DeploymentOperation

SalvageDisposal

TimeMicroergonomic

SE Space

Ran

ge

Fig. 1. The systems engineering state space for developingand deploying tools; all possible engineering activities resideon this three-dimensional framework (manufacturing anddistribution are included in deployment and service isincluded in operation). S/P: seller/purchaser; RDDT&E:research, design, development, testing, and evaluation; SE:systems engineering.

Design inputs are the subset of the stakeholders

NWDs that the designing organization believes is

technologically and economically feasible.

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE MAY/JUNE 2010 13

design reviews. They are not depicted here in the actual sequencethat they occur; please refer to Figure 3 for the detailed flowchart.With each iteration, engineering validation experimentally dem-onstrates that design inputs were or were not correctly translatedto the current implementation. On the last iteration, the finaldesign outputs are transferred to manufacturing for mass produc-tion. Postmarket surveillance helps identify missing misunder-stood NWDs that allow future corrective and preventive actions.

The FDA-mandated design controls regulation (21 CFR820.30:1996) changed two key terms (requirements and spec-ifications) that were historically terms of art in classical sys-tems engineering; these are now called design inputs anddesign outputs, respectively. Design inputs are the subset ofthe stakeholders NWDs that the designing organizationbelieves is technologically and economically feasible; theyare testable natural language (e.g., English) statements under-standable by all stakeholders. Design outputs are how engi-neers solve the problem posed by the design inputs; often,there may be multiple solutions, depending upon what

optimization criteria are applied. Design outputs (the designengineers’ work product) tell manufacturers what to buildand how to build it. These modern terms resolve a longstand-ing nomenclature problem that occurs when English-speak-ing engineers and managers interchange systems engineeringterms with lay terms, creating phrases that are technicallyambiguous or nonsensical (e.g., requirements specificationand specification requirements). With the modern terms, it isnow possible to speak unambiguously of a design inputsspecification (a documented compendium of design inputs,also called a design requirements document) and design out-put requirements (the required elements of a properly formu-lated design outputs document). Often, it is confusing tryingto identify whether something is a requirement or a specifica-tion. A useful rule of thumb is if it has a value, unit, and tol-erance, it’s a spec; if it doesn’t, it’s probably a requirement.

Design controls are nothing more than the fundamentalelements of classical systems engineering. The classical systemsengineering process is a very powerful risk-reduction mecha-

nism that serves both consum-ers and producers, but whosevalue is diluted or negated ifnot followed rigorously. Threekey process attributes for cor-rectly applying design con-trols are: 1) multiple iteration,2) comprehensive contempo-raneous documentation, and3) flexible decision making.The human focus, iterativelyreassessing stakeholders andreconciling their often con-flicting and evolving NWDs,moves the innovation processtoward human-centered sys-tems engineering [5].

Multiple IterationDeveloping new/improvedproducts is always a learningprocess; iteration allows theengineering team to engage instructured learning, whiletime-constrained iterations maymake the process agile. It hasalways been unrealistic tobelieve that first you establish

ConceptStakeholder

NWDs

Risks,Hazards,and Harm

Design Inputs(Requirements)

Design Outputs(Specifications)

Verification,Validation, andDesign Transfer

PostmarketSurveillance

Planning and Reviews

Val

idat

ion

Synthesis

Analysis

Hyp

othe

sis

ConceptStakeholder

NWDs

RiHazand

Design (Require

Design Outputs

erification,idation, andsign Transfer

PostmarketSurveillance

Planning and Reviews

Val

idat

ion

Synthesis

Analysis

Hyp

othe

sis

Fig. 2. The major elements of design controls are mapped on the four elements of the scien-tific method.

Table 2. Activities in the systems engineering state space domains.

Requirements Engineering Compliance Engineering Reliability Engineering

Stakeholder identification, NWD assessmentand reconciliation

Identification of laws, regulations,and standards

Defining minimum nec-essary reliability

Risk management Applicability assessment Fault preventionDesign input formulation and five verifications Design impact assessment Fault removalVersion validation Test design Fault toleranceVersion postmarket surveillance Operational considerations Fault/failure forecastingCAPA-driven design input changes Salvage and/or disposal

considerationsTest design

NWD: needs, wants, and desires of stakeholders; CAPA, corrective and preventive action.

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE MAY/JUNE 201014

the requirements, then you develop the specifications, test theimplementation, and field the product. The waterfall model [6],[7], except for the most trivial innovations, is never strictly imple-mented in the real world. Iteration always occurs, even though itmay be disguised or denied. The human learning process is incre-mental and builds upon experience and repetition, primarilygained from testing—some of which is formal, but much ofwhich is informal, accidental, and experiential. Figure 3 illus-trates the iterative process in the form of a flowchart. Each itera-tion during product development yields an internal release, untilthe final version yields the external release. The multiple feed-back loops formed by verification testing (of design inputs for-mulation, design outputs development, and design outputsimplementation) provide the opportunity to correct internal mis-understandings and technical errors. Two additional types of

verification are not illustrated here: verification that risk mitiga-tion was properly applied and verification that properly appliedmitigation actually reduced risk [8, paragraph 6.3]. The validationloop (from implementation back to design inputs) permits identi-fication and correction of a mismatch between what was agreedwould be built and what was actually built; this is invariably aclinical (clinical refers to dealing with humans, patients in thecase of medicine and psychology, and users in the case of humanfactors engineering) trial involving users in their expected useenvironment [9]. A corrective and preventive action loop is thepath to the next iteration; it provides the opportunity to correctmajor and minor flaws, most often related to internal or externalcomplaints about the identified stakeholders’ evolving NWDs.Iteration alone, without rigorously following the process in Fig-ure 3, is inadequate. Understanding what will satisfice all the

Prior Version FromIteration n – 1

Stakeholder NWDsAssessment

Update NWDsUpdate HA

Document DR

Update DIsUpdate HA

Update VerificationDocument DR

Update DOsUpdate HA

Update VerificationDocument DR

Update VersionUpdate HA

Update Verificationand ValidationDocument DR

HA

HA

HA

HA

NoNWDsReview

Design InputsFormulation

Design OutputsDevelopment

DI Review

DO Review

VersionReview

Implementation

Internal Release

ToIteration n + 1

Next Version

ArchiveDHF

NWDs: Needs, Wants, and DesiresDI: Design InputDO: Design OutputDR: Design ReviewHA: Hazard AnalysisV&V: Verification and ValidationDHF: Design History FileSOP: Standard Operating Procedure

No

No

No

NWDs Review1. Identified Stakeholders2. NWDs3. Hazard Analysis

(No V&V Possible)

DI Review1. Design Inputs2. Hazard Analysis3. DI Verification

DO Review1. Design Outputs2. Hazard Analysis3. DO Verification

Version Review1. Version Implementation2. Hazard Analysis3. Version Verification4. Version Validation

Des

ign

Con

trol

Doc

umen

ts (

in D

HF

)

Impl

emen

tatio

n V

alid

atio

n

Design InputsV

erificationD

esign Outputs

Verification

Version

Verification

Design R

eviews (per S

OP

)

(a) (b)

Fig. 3. The flowchart depicts some arbitrary iteration of the design control process and identifies (a) requisite documentationactivities and (b) design review targets. Three of the five verification activities are identified; the two risk management verifi-cation activities (ISO 14971 x6.3) are omitted for clarity.

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE MAY/JUNE 2010 15

stakeholders improves with each iteration, if there are multipleiterations and real opportunities for the development team tolearn [10].

Comprehensive DocumentationStandard operating procedures (SOPs) and their requisite docu-mentation work products have become the bane of engineers’existence. Intended to be a powerful management technique thatstandardizes the process, character, and quality of work, it hasinstead become an annoyance to engineers, a frustration for man-agement, and a regulatory vulnerability. From the engineers’perspective, SOPs are fine guidelines, but everyone knows thework changes, and no one is keeping up with the SOPs. Frommanagements’ perspective, SOPs make excellent training materi-als in addition to establishing the manner in which the work (e.g.,design controls) should be accomplished and the achievementsshould be documented. From a regulatory perspective, as a prac-tical matter, it is better to have no SOP than to have an SOP thatis being violated. Yet, it is important to realize that creating andmaintaining comprehensive documentation has enormous techni-cal, financial, and intellectual property value.

Comprehensive is not synonymous with elaborate. Thereexist many sophisticated electronic document managementsystems, both commercially available and internally devel-oped. However, a well-maintained, hand-written laboratorynotebook can be a perfect example of comprehensive contem-poraneous documentation. A well-maintained laboratorynotebook often may be far more valuable than terabytes orcabinet drawers of documentation. Not only does it complywith regulatory requirements, it traces the logical history ofall the engineering activities across the product life cycle andsupports intellectual property claims. Structured reviews ofnotebooks by the established company procedure furtherincreases their value [Figure 3(b)]. The reviewed notebookshelp to identify inventors, witnesses, and critical dates neces-sary for successful intellectual property claims.

The basis for engineering validation is the design inputsand not the stakeholder NWDs. Since it is not possible toindependently verify or validate the list of stakeholders andtheir NWDs, this is the soft underbelly of any systems engi-neering effort. System failures in the field are very often theresult of overlooking stakeholders and/or NWDs. This leadsto essential systems that hinder rather than help; it is a majorconcern in high-confidence systems. Emphasis on repeatedlyidentifying all the stakeholders and their dissonances in eachiteration (human-centered systems engineering) reduces theprobability of these failures.

Product development, like clinical practice and biomedicalresearch, is inherently a set of critical information-manage-ment tasks [11], [12]. Documentation [Figure 3(a)], whilecritical to the process of information management, shouldnever distract from engineering problem-solving activities orconsume unnecessary resources. There is no regulation thatstates how you must present your documentation. In my expe-rience, the source of most documentation problems stemsfrom what students are taught regarding the structure, content,and maintenance of laboratory notebooks. As I recall, keepinga notebook seemed just another annoyance and distractionfrom the joy of being creative; only much later in my careerdid I appreciate the critical value of capturing informationand experience in my notebook, especially when I needed thatinformation after having forgotten it.

Flexible Decision MakingToo much time spent on any particular step in a process during asingle iteration is usually counterproductive. The optimal designinvariably changes with each iteration. Trying to finalize any sub-system usually constrains future decision-making options, result-ing in suboptimal results. This is initially anathema to mostengineers and managers, who wish to believe that a knownpercentage of the project is complete. Once they experience thisas an agile process, the majority become far less skeptical andinsecure about not constantly computing percentage complete.(All agile processes are iterative, but not all iterative processesare agile. A key attribute of an agile process is that each iterationis time constrained.) Instead, they focus on estimating the remain-ing risk with each ensuing iteration [13], fully aware that in thisprocess the resource costs of change are no longer onerous. It hasbeen my observation that high-performing individuals usuallystrive to keep their options open; this flexible approach to deci-sion making and high tolerance for ambiguity/uncertainty is akey element of their successful performance [14].

How Design Controls Are ImplementedImplementing design controls is less about engineering andmore about project and quality management. Yes, engineersneed to understand the process and understand that they areresponsible for both problem solving and maintaining compre-hensive contemporaneous documentation. The information theycreate is the primary basis for iteration control. Iteration pro-motes problem-based learning and inquiry learning (structuredlearning driven by project objectives) and is an enabler of flexi-ble decision making. More importantly, engineers and manag-ers need to understand that they should not eschew changeduring the development project; quality is ultimately about sat-isficing the reconciled NWDs of the identified stakeholders andnot just extending component reliability. Identifying new stake-holders or new stakeholder NWDs within each iteration resultsin human-centered systems engineering (stakeholders are eitherhuman individuals or human organizations). This human focuscontinuously refines what should be built, tends to eliminateextraneous features and costs, and increases the probability ofacceptance; the five verifications identify errors, while valida-tion activities identify the mismatches between what was agreedwould be built and what was actually built. When this deviationis sufficiently small, senior management may decide that thecurrent development iteration will be the final iteration.

When Design Controls Are ImplementedFigure 4 is an example of an innovation standard operatingprocedure (ISOP) for regulated medical devices. When thedesign controls (in the regulatory sense) are instantiated is acritical element of the procedure. In this ISOP, they begin aftermanagement approves the project (Gate 1) and before anycommercial design begins. This absolutely includes feasibilityexperiments and proofs of concept, whose uncontrolleddesigns too often are the basis for future liabilities in commer-cial products. In this way, no design survives in the commer-cial product that was not subject to design controls and riskmanagement. True basic research activities, prior to projectapproval, are excluded from design controls, and this is con-sistent with the regulators’ expectations. However, once anorganization has implemented design controls in one or twoprojects, I have found that the members of the project teamswill often just do it as a matter of practice, probably because

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE MAY/JUNE 201016

they learned that this agile process is more forgiving of errorsand more prone to success.

Why Design Controls Have ValueThe value of implementing engineering design controls duringdevelopment far exceeds mere regulatory compliance.Design controls, properly implemented as human-centeredsystems engineering, have tangible value to stockholders; man-agement; the development team; manufacturing, distribution,and service (MD&S) personnel; consumers (both purchasersand end users); and regulators. The central value element isreduction of risks, including economic risks, technical risks, andMD&S risks.

Economic risks are reduced by the enormous emphasis oniterative reassessment of stakeholders, their NWDs, and the dis-cipline of repeated cumulative hazard analyses throughout theprocess. These activities clarify and refine the understanding ofthe intended uses, users, use environment, and lifetime of thedevice. This increases the probability of acceptance of theproduct, process, or service by all stakeholders.

Technical risks reduce through a combination of thefollowing:� time-constrained iterations permitting structured learning� comprehensive contemporaneous documentation provid-

ing efficient traceability and supporting flexible decisionmaking

� realization that the cost of change in this process isnearly flat from beginning to end of the developmentcycle

� repetition of validation studies in each iteration reduc-ing the incidence of latent failures as described byReason [15].

MD&S risks are reduced primarily by overt recognitionof these critical stakeholders early in development (inessence, concurrent engineering). Secondarily, manage-ment of information that permits tracing and appreciatingdecisions made during development, long after develop-ment ended, further reduces MD&S risks. These two activ-ities reduce the potential for postdevelopment failuresdescribed by Dekker as drift [16]. Figure 5 illustrates howhazards due to latent failures and drift avoid discovery inthe absence of design controls. Only Hazard 2 in the figureis reliably detectable by premarket validation. Hazard 1cannot be reliably detected by premarket validation becauseof missing design inputs combined with unverified designoutputs. Hazard 3, the result of unanticipated variations inmanufacturing or maintenance, cannot be reliably detectedby premarket validation or even by periodic postmarketrevalidations. The principle of correct by design, funda-mental to the development of high-confidence systems, pre-supposes effective design controls throughout the full lifecycle (from lust to dust).

StartISOP

StopISOP

KnowledgeAcquisition

No

No

No

No CAPA Plan

Device MasterRecord

DeviceHistory(Batch)Record

PreproductionPrototype(s)

ProductionPrototype(s)

Final DesignOutputs

Final DesignHistory File

OperationalPlan

SupplierAudits

RegulatorySubmissions

FinalClinical andUser Studies

DesignInputs?

DesignOutputs?

PrototypeBuilding

Analysis andTest Results;ExploratoryClinical andUser Trials

No

StrategyDevelopment

BusinessPlan

High-LevelProject Plan

G#0:PromisingNew Idea?

G#1:Strategy

Approval?

G#3:PrototypeApproval?

G#4:Launch

Approval?

G#2:FeasibilityApproval?

EngineeringPrototype(s)

DesignInputs?

FeasibilityEvaluations

Basic Design

Start DesignControlsand Risk

Management

PilotProduction

Launch

Process Decision Document

Fig. 4. An ISOP flow diagram for regulated medical devices complying with 21 CFR 820.30 and ISO 13485:2003. It is importantto note that design controls and risk management are instantiated prior to feasibility or proof-of-concept studies. This elimi-nates the possibility of uncontrolled designs and their associated risks ending up in the final production units.

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE MAY/JUNE 2010 17

ConclusionsFDA-mandated design controls for medical devices providea structured, systematic engineering paradigm that supportshuman-centered systems engineering. Engineering designcontrols are nothing more than the fundamental elements ofclassical systems engineering. The human focus is enabledthrough the iterative reidentification of stakeholders, reas-sessment of their NWDs, and reconciliation of their evolv-ing/conflicting NWDs. The implementation of designcontrols and embedded risk management must begin prior tocommercial development to reap the full benefit of theapproach; partial approaches dilute or negate the effective-ness and efficiency of this nearly century-old systems engi-neering paradigm. Properly employing engineering designcontrols is a strategic business decision. The central value ofthis proposition is the reduction of economic, technical, andoperational risks for both producers and consumers; regula-tory compliance is a secondary benefit. Misuse or abuse ofdesign controls only undermines long-term profitability andincreases the risks to the consumer.

George M. Samaras received hisB.S.E.E. degree in electrical engineeringin 1972 and M.S. and Ph.D. degrees inphysiology from the University of Mary-land in 1974 and 1976, respectively. Hereceived a D.Sc. degree in 1992 from theGeorge Washington University, Depart-ment of Engineering Management and

Systems Engineering. He has been a professional engineer,licensed in Maryland since 1980. He worked as a professor atthe University of Maryland School of Medicine and George

Washington University Grad-uate School of Engineering.He founded and managed acontract biomedical engineer-ing firm and has worked as areviewer and manager at theU.S. Food and Drug Adminis-tration Center for Devices andRadiological Health. He hasbeen in private practice since1996. His firm providestechnical, regulatory, and man-agement consulting services,primarily to medical deviceand pharmaceutical manufac-turers. He has a number of bio-medical engineering patentsand numerous peer-reviewedpublications in physiology andhardware, software, humanfactors, and quality engineer-ing. He is a board-certifiedhuman factors engineer (Certi-fied Professional Ergonomist,1998) and an American Soci-ety for quality-certified qualityengineer (2005).

Address for Correspondence: George M. Samaras, Samaras& Associates, Inc., 7755 Soda Creek Road, Pueblo, CO 81005USA. E-mail: george@samaras.eng.pro.

References[1] G. M. Samaras and R. L. Horst, ‘‘A systems engineering perspective on thehuman-centered design of health information systems,’’ J Biomed. Inform.,vol. 38, no. 1, pp. 61–74, 2005.[2] United States Food and Drug Administration Code of Federal Regulations,Part 820 (Quality System Regulation), Final Rule, Oct. 1996.[3] Medical Devices—Quality Management Systems—Requirements for Regula-tory Purposes. 2nd ed., International Standard ISO 13485:2003(E).[4] H. A. Simon, Models of Man: Social and Rational. New York: Wiley,1957.[5] G. M. Samaras, ‘‘Human-centered systems engineering: Building products,processes, and services,’’ in Proc. 2010 SHS/ASQ Joint Conf., Atlanta, GA (CD-ROM), Feb. 26, 2010.[6] W. W. Royce. (2008, Oct. 24.). Managing the development of large softwaresystems, Proc. IEEE WESCON, Aug. 1970, pp. 1–9 [Online]. Available: http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf[7] D. Parnas and P. C. Clements. (2008, Oct. 24). A rational design process: Howand why to fake it. IEEE Trans Softw. Eng., vol. 12, no. 2, pp. 251–257. [Online].Available: http://users.ece.utexas.edu/~perry/education/SE-Intro/fakeit.pdf[8] Medical Devices—Application of Risk Management to Medical Devices. 2nded., International Standard ISO 14971:2007(E).[9] G. M. Samaras, ‘‘An approach to human factors validation,’’ J Valid. Tech-nol., vol. 12, no. 3, pp. 190–201, 2006.[10] J. G. March, ‘‘Exploitation and exploration in organizational learning,’’Organization. Sci., vol. 2, no. 1, pp. 71–87, 1991.[11] V. L. Patel, E. H. Shortliffe, M. Stefanelli, P. Szolovits, M. R. Berthold,R. Bellazzi, and A. Abu-Hanna, ‘‘The coming of age of artificial intelligence inmedicine,’’ Artif. Intell. Med., vol. 46, no. 1, pp. 5–17, 2008.[12] J. H. Poore, ‘‘A tale of three disciplines . . . and a revolution,’’ IEEEComputer, vol. 37, no. 1, pp. 30–36, Jan. 2004.[13] C. Larman, Agile and Iterative Development: A Manager’s Guide. Boston:Addison-Wesley, 2004.[14] P. C. Nutt, ‘‘Flexible decision styles and the choices of top executives,’’ J.Manage. Stud., vol. 30, no. 5, pp. 695–721, 1993.[15] J. Reason, Human Error. Cambridge: Cambridge Univ. Press, 1990.[16] S. W. A. Dekker, Ten Questions About Human Error: A New View of HumanFactors and System Safety. NJ: Lawrence Erlbaum Associates, 2005.

TriggerHazards 1 and 3 Not Accessible by Testing Implementation

Hazard 3 Appears Only AfterVariations in Manufacturing,

Maintenance, etc.

Hazard 1 Appears Only Aftera Change in the Specifications

Hazard 2

Hazard 3

Hazard 1

Implementation(Version Realization)

Specifications(Design Outputs)

Requirements(Design Inputs)

StakeholderNWDs

Latent Failures(After [15])

Drift(After [16])

Fig. 5. Three different types of hazards are shown. Hazard 2 may be detected with a well-designed, premarket engineering validation study. Hazards 1 and 2 cannot be detected reli-ably with premarket validation. Rigorous adherence to design controls will reduce the proba-bility of occurrence of latent hazards [15] and drift [16].

IEEE ENGINEERING IN MEDICINE AND BIOLOGY MAGAZINE MAY/JUNE 201018