A Framework for Space Systems Architecture under...

Research Article

A Framework for Space Systems Architectureunder Stakeholder Objectives AmbiguityAlessandro Golkar1,2,∗ and Edward F. Crawley1,2

1Skolkovo Institute of Science and Technology, Skolkovo, Russia2Massachusetts Institute of Technology, Cambridge, MA 02139

Received 13 November 2013; Revised 26 March 2014; Accepted 12 June 2014, after one or more revisionsPublished online in Wiley Online Library (wileyonlinelibrary.com)DOI 10.1111/sys.21286

ABSTRACT

Matching high ambitions with scarce resources is one of the primary challenges of aerospace and otherindustries concerned with the development of unprecedented infrastructures, on par with the technicalchallenges associated with developing new technology. Stakeholder objectives are often unclear due tohighly exploratory business cases. Further ambiguity emerges from disagreement between stakeholdersand decision makers called to formulate scientific, technological and policy requirements for new systems.

This paper develops a structured approach, called the Delphi-based Systems Architecting Frame-work (DB-SAF), which has been conceived to develop recommendations to system architects concernedwith the design of unprecedented large infrastructures for which objectives are ambiguous or unclear.

The objectives of DB-SAF are to identify sources of ambiguity in the value proposition of asystem architecture and the associated trade-space exploration, characterize and model sourcesof ambiguity, and assess the impact of requirement ambiguities on the architectural trade space.

The proposed systems architecting approach is demonstrated in this paper through the as-sessment of a robotic Mars Sample Return Campaign, which serves as a test bed case study todescribe the proposed methodology and to discuss its extension to other fields of engineering.

The proposed framework integrates methods from systems engineering, computational systems archi-tecting, multidisciplinary systems design and optimization, uncertainty modeling, utility theory, and socialscience research. It allows decision makers to visualize an architectural synthesis of aerospace systems,understanding adverse impacts of ambiguity, and supporting negotiations among stakeholders for efficientcompromise in systems architecting. C⃝ 2014 Wiley Periodicals, Inc. Syst Eng 00: 1–24, 2014

Key words: systems architecture; stakeholders ambiguity; requirements elicitation and management;project formulation; modeling and simulation

∗Author to whom all correspondence should be addressed (e-mail:[email protected]).

Systems Engineering Vol. 00, No. 0, 2014C⃝ 2014 Wiley Periodicals, Inc.

1. INTRODUCTION

Matching high ambitions with scarce resources is one ofthe primary challenges of the aerospace and other industriesconcerned with the development of unprecedented infrastruc-tures, on par with the technical challenges associated withdeveloping new technology. In this kind of developments,

1

2 SYSTEMS ARCHITECTURE UNDER STAKEHOLDERS AMBIGUITY

objectives are often unclear due to the ambiguity surroundingthe subject of the investigation, which is highly exploratoryin its own nature. Ambiguity further arises from disagree-ment that is often found between experts called to specifyscience and engineering requirements for new projects. Over-designing requirements without a well-agreed rationale, suchas setting a “too high” mass amount of samples to be returnedto Earth on aMars Sample Return (MSR)mission, can be fatalto mission success due to high mission costs. On the contrary,underdesigned requirements (a “too low” mass amount ofsamples in the MSR example) preclude scientific discoveriesand overall value delivery of the mission to stakeholders. Inthe worst case, poorly specified requirements can lead to notanswering any scientific question at all. It is therefore crucialto identify architectures with the highest likelihood of satisfy-ing goals, while finding consensus among stakeholders, andmeeting engineering and programmatic constraints.This paper presents a Delphi-based Systems Architecting

Framework (DB-SAF) aimed to define, identify, characterize,mitigate, and analyze ambiguity in the systems architectingprocess. The framework identifies areas of opportunity ofambiguity mitigation, and supports the formulation of rec-ommendations to support systems engineers and decisionmakers in reducing ambiguity in their objectives, identifyingarchitectures with effective and robust programmatic trade-offs, engineering performance, and meet desired science andpolicy objectives.The framework presented in this paper integrates methods

from systems engineering, computational systems architect-ing, multidisciplinary system design and optimization, uncer-tainty modeling, utility theory, and social science research. Itallows decision makers to visualize an architectural synthesisof their engineering systems, understand the impact of am-biguity in the definition of requirements, and consequentlysupport negotiations in reaching consensus towards “glob-ally best” system requirements and associated Pareto-efficientsystem architectures.The remainder of this paper is structured as follows. Sec-

tion 2. provides context to the framework with a review ofthe scientific literature. Section 3. describes the DB-SAF, withall the steps involved in the analysis. Section 4. demonstratesthe framework on a MSR case study, with emphasis on thedescription of the application of the proposed methods andinherent limitations and trade-offs that are encountered. Sec-tion 5. draws conclusions from the research, deriving avenuesfor future work and discussing the extension of DB-SAF toother fields in engineering systems.

2. LITERATURE REVIEW

Management of ambiguities and systems architecture arebroad topics which contributions have been covered by dif-ferent disciplines. This section presents a review of literatureof interest to this paper, ranging from social sciences to engi-neering design research. Ambiguity is a theme that has beentraditionally covered in other disciplines than engineering.In particular, this survey covers management of ambiguitiesas treated in political science (Section 2.1) and managementscience (Section 2.2). Furthermore, this section surveys rele-

vant research in systems engineering (Section 2.3) and expertelicitation (Section 2.4), providing comprehensive context forthe framework proposed in this paper. A distinction mustbe made between uncertainties and ambiguities. This paperadopts Murray’s [1961] definition of uncertainty, as “some-thing not definitely known or knowable”, and Camerer andWeber’s [1992] definition of ambiguity, as “missing informa-tion that is relevant and could be known”. Browning, Fricke,and Negele [2006] discuss how uncertainty and ambiguityhave different impact in the product development process ofa complex system.

2.1 Political Science

Academic research in political science and problems encoun-tered in policymaking are often faced with the question ofmanagement of risk and uncertainties. In the policy domain,ambiguity management is seen in its downside (risk) aspect,and discussed in the field of risk shielding [Oye, 2010]. Twodifferent approaches are found: a libertarian (or laissez-faire)approach, as advocated by [Sapolsky, 1990] and [Viscusi,2005], and a regulatory approach, based on the use of theprecautionary principle [Harremoes, 2001]. Morgan [1993]provides an intermediate approach to ambiguity (risk) inpolicymaking, advocating different management approachesbased on the nature of the risk being considered—making adistinction between known and unknown risks—and based onwhether the exposure to risk is on a voluntary or involuntarybasis.

2.2 Management Science

Literature in management science has explored extensively,in a qualitative way, how to manage ambiguities in projects.The focus of this literature is in the development of busi-ness strategies under uncertainty. Courtney, Kirkland, andViguerie [1997] classify uncertainty in strategy planning (i.e.,ambiguity, in this paper’s nomenclature) in four levels, ac-cording to the degree of uncertainty to be faced. From the low-est (level 1) to the highest (level 4) degree of uncertainty, theydistinguish between clear-enough futures, alternate futures,a range of futures, and true ambiguity. They note that level4 uncertainty (true ambiguity) is often encountered in earlystages of strategy planning, and it is often reduced to lowerlevels of uncertainty. They identify three strategic posturesin management under uncertainty: shape the future, adapt tothe future, and reserve the right to play, implemented usingthree different types of management actions (no-regret moves,options, and big bets) and provide guidelines on their useaccording to the level of uncertainty being faced [Courtneyet al., 1997]. Brandenburger and Nalebuff apply game theoryto strategy planning under uncertainty, discussing the op-tions available to managers to shape strategies by “changingthe game,” identifying lose–lose situations and transform-ing them into win-win strategies [Nash, 1950], based on avalue-net framework [Brandenburger and Nalebuff, 1995].McGrath andMacMillan propose the idea of discovery-drivenplanning, where strategies are phased over time to allow

Systems Engineering DOI 10.1111/sys

GOLKAR AND CRAWLEY 3

uncertainty to unfold and adapt decisions accordingly [Mc-Grath and MacMillan, 1995]—analogously of what is dis-cussed in the engineering section of this literature review inreal option analysis and phased development strategies [deNeufville et al., 2004; De Weck, De Neufville, and Chaize,2004].

2.3 Systems Engineering Theory

Management of ambiguities in systems engineering theory isan emergent body of literature, dealing with approaches tocope with uncertainty in engineering systems. Two main ap-proaches have been proposed: the implementation of robust-ness [Taguchi, 1986; Phadke, 1995] and the implementationof flexibility options in the system of interest [de Neufvilleet al., 2004]. Changeability and adaptability have also beendiscussed as a critical issue to consider to ensure that systemsdeliver value to stakeholders over time [Fricke and Schulz,2005; Engel and Browning, 2008; Ross, Rhodes, and Hast-ings, 2008]. De Weck et al. [2004] demonstrated the valueof implementing flexibility options in a satellite constellationarchitecture using lattice analysis to describe propagation ofuncertainty over the lifecycle of the architecture and advocat-ing for a phased development approach for large-scale sys-tems to hedge endogenous and exogenous uncertainties andcapture upside opportunities due to uncertainty. Furthermore,Silver and de Weck [2007] proposed a network-based ap-proach to analyze flexibility for complex evolutionary large-scale systems, the Time-Expanded Decision Networks. Morerecently, screening models based on aMonte Carlo simulationframework have been proposed for the evaluation of flexi-bility options in engineering systems [Lin, 2008]. Screeningmodels have the advantage of having less constraints on theformulation of the problem than other methods (for example,they do not require the system representation to be path in-dependent), assuming the problem is formulated and decom-posed properly to be resolved in reasonable computationaltimes, but they do not include an assessment of the impactof ambiguity of stakeholder objectives in the evaluation ofsystem architectures.Architecting methods allow the identification of Pareto-

efficient architectures, as described previously; they canbe complemented with quantitative tools from decision-making theory [Edwards, 1954; Bellman and Zadeh, 1970;Keeney and Raiffa, 1976; Kahneman and Tversky, 1979;Zeleny, 1982; Howard, 1988; Dyer et al., 1992; Roy andMcCord, 1996]. A tool of practical use in this context is thedevelopment of decision trees [Howard, 1988], which requirea formal choice-decision to enumerate possible scenarios,associated expected outcomes and subjective probabilities ofoccurrence.Recommendations based on decision-tree analysis are

based on the maximization of expected value criteria [Meyer,1987]. Decision tree analysis, however, has several limita-tions. In fact, it requires a “discretization” of the range ofpossible outcomes in a set of discrete occurrences. Further-more, full enumeration of full scenarios is prohibitive if thenumber of possible scenarios is too large; this issue is oftenovercome with dynamic programming approaches [Bellman,

1957], which however require the decision-tree representationto have specified characteristics such as path independence.Most importantly, for the purposes of modeling ambiguity inthis paper, the main limitation of traditional decision treesis that they require a deterministic knowledge of subjectiveprobabilities of occurrence for each scenario.Decision-making support often requires the synthesis of

subjective opinions, for instance, in the definition of valuemetrics in systems architecting. Multi Attribute Utility Anal-ysis (MAUA) [Keeney and Raiffa, 1976] and the AnalyticalHierarchy Process (AHP) [Forman and Gass, 2001] are toolsthat have been applied for decision-making purposes in sev-eral disciplinary fields.While MAUA has been applied to systems architecting

previously [Ross and Hastings, 2005], little research hasbeen done on how to define utility under ambiguous stake-holder objectives—where the definition of value-based utilityfunctions is unclear. Both methods have known limitations.MAUA encounters challenges in preference elicitation whenapplied to the evaluation of attributes with nomonetary equiv-alents. AHP on the other hand is prone to rank reversal is-sues when new attributes are considered in the trade space[Schenkerman, 2003]. None of these two methods can beapplied effectively when ambiguity is introduced in groupdecision making.

2.4 Expert Elicitation

Expert elicitation, among other applications, is the disciplineconcerned with the synthesis of expert knowledge for require-ments formulation. Expert elicitation has been historicallyused for the elicitation of probabilities of occurrence in safetyanalysis, such as the famous Rasmussen report on nuclear re-actor safety [Rasmussen, 1975]. Other historical applicationsof expert elicitation include expert assessments synthesis forcomplex systems analysis [NAS, 1975].Expert elicitation techniques are of particular interest for

engineering systems, as they allow the quantification of sub-jective metrics that are of paramount importance to the designprocess during formulation of requirements. Elicitation tech-niques has been used to estimate experts’ preference struc-tures for multi attribute analysis, such as the ratio method[Edwards, 1977], the swing method [von Winterfeldt andEdwards, 1986] and the trade-off method [Keeney and Raiffa,1976]. Focus group methods [Terpstra, Lindell, and Guttel-ing, 2009] are routinely used for elicitation of expert knowl-edge. While focus group and conventional group-decisionmaking processes are effective in improving convergence to-wards consensus, they suffer adverse behavioral effects orig-inating from peer pressure and hidden agendas. Focus groupsare also ineffective in presence of the highest degree of am-biguity originated by the unknown (such as forecasting offuture events). The Delphi method is a qualitative tool thathas been originally developed to this end, to improve forecast-ing in expert policy-making [Adler and Ziglio, 1996; Roweand Wright, 1999; Rowe, Wright, and Bolger, 1991]. Delphisynthesizes expert knowledge by anonymous elicitation ofexperts, while reducing adverse peer-pressure effects throughanonymity in expert elicitation. While Delphi has been used



Literature Review Systems-specificexpertise

ProblemFormulation

Expert PanelFormation

ProblemFormulationReview withExpert Panel

Design ofInterview

Elicitation ofExpert ValueJudgement

Results AnalysisAggregate Results

Discussion withIndividual Experts

ConvergenceCriteria is Met

Documentationand Development

ofRecommendations

yes

iterations

no

Step 1

Step 2 Step 3 Step 4

Step 5 Step 6 Step 7 Step 8 Step 9 Step 10

Figure 1. Delphi-based Systems Architecting Framework Overview.

Identification ofquestions of

interest

GoalsIdentification

FunctionalDecomposition

RequirementsEnumeration

Function – FormMapping

Development ofEvaluation Metrics

1

2

3

4

5

6

IterationCycle I

IterationCycle II

Figure 2. Step 2: Problem formulation.

in the context of forecasting and decision making, the frame-work presented in this paper engineers the Delphi process in aquantitative form and implements it in the context of decisionmaking, assessing advantages and disadvantages.

2.5 Systems Architecture

Systems architecture is the discipline that provides “an ab-stract description of the entities of a system and the rela-

tionships between those entities” [Crawley, 2008]. The liter-ature in systems architecture includes formal languages forsystem decomposition, such as Unified Modeling Language(UML) [Booch, Rumbaugh, and Jacobson, 1996] and Ob-ject Process Modeling (OPM) [Dori and Crawley, 2002], andquantitative methods for tradespace exploration [Ross andHastings, 2005]. Koo developed a meta-language for systemsarchitecting based on OPM, called Object Process Networks(OPN) [Koo, 2005]; building on his work, Simmons devel-oped a framework for quantitative, decision-based systemsarchitecting, based on a method called the Architecture Deci-sion Graph (ADG) [Simmons, 2008].Both OPN and ADG are used to perform trade-space ex-

ploration, based on a three step approach: (1) generation of ar-chitectures using a full enumeration approach, (2) evaluationof architectures, and (3) identification of Pareto-efficient ar-chitectures [de Weck, 2009]. Hastings and Weigel proposed asystems architecting methodology accounting for uncertaintyusing portfolio theory [Hastings, Weigel, and Walton, 2003],looking at the impact of traditional uncertainties “in” theproblem on systems architecting, with an application tospace systems developed for commercial purposes (a satel-lite constellation for telecommunications). This method canbe used to determine portfolios of systems robust to un-certainty, but does not include an assessment of the impactof ambiguity in stakeholder objectives, and does not con-sider the value of flexibility in engineering systems, andit is only applied to architectures represented by purelydiscrete design vectors, being formulated as a portfoliooptimization.

3. DB-SAF

This section develops the theory underlying the proposedapproach for systems architecting under stakeholders ambi-guity: the DB-SAF, which overview is shown in Figure 1.



DB-SAF is a structured, iterative tool to inform am-biguity mitigation strategies in systems architecting understakeholder ambiguity. The framework is inspired by theDelphi method in policy-making, and proposes a novel sys-tems architecting implementation in the context of formula-tion of requirements for new, unprecedented systems. DB-SAF consists in a structured process decomposed in tenSteps.

3.1 Step 1: Literature Review andSystems-Specific Expertise

The process starts from a preliminary literature review and thesystem architect’s system-specific expertise (Step 1). The pur-pose of the literature review is to gather existing informationon the architecting problem of interest and inform problemformulation (Step 2).

3.3 Step 2: Problem Formulation

In Problem Formulation (Figure 2) the system architect de-fines the problem he/she wishes to address in support of thecustomer’s project. The step consists in two sequential iter-ative cycles. Iterations are devised to refine each individualsubsteps, which definition benefits from the definition of theother substeps in the cycle.

3.3.1 Iteration Cycle 1In the identification of questions of interest (substep 1),the system architect identifies the needs of the beneficia-ries of the system, formulating questions of interest tobe addressed. Questions are formulated by direct interac-tion with the customer that is commissioning the study.In goals identification (substep 2), system architects iden-tify and characterize stakeholder goals to be fulfilled bythe system. Stakeholder goals derive from beneficiaries’needs as well as additional socio-political considerations. Pri-mary stakeholder goals can be identified with structured ap-proaches such as quantitative stakeholder analysis [Cameron,Catanzaro, and Crawley, 2006]. In addition to beneficia-ries needs, example additional goals that are set by stake-holders are policy robustness, economic sustainability andeducation and outreach. Finally, following the identifica-tion of goals, functional decomposition (substep 3) is per-formed in order to identify the functions that the systemneeds to implement. Functions are typically decomposed instructured hierarchies, and they are formulated as solution-neutral, as they do not depend on specific technologies orarchitectures.

3.3.2 Iteration Cycle 2In requirements enumeration, the system architect enumer-ates all the possible sets of system requirements that the sys-tem could be designed for. Consideration of multiple set ofrequirements is desired for successive evaluation of the over-

all architecture and its value in terms of science performed,overall engineering complexity and cost (Steps 5–9). Systemrequirement sets are enumerated also according to feasibilityconstraints, which are used to prune unfeasible requirementcombinations out of the trade space and leave only feasibleoptions for study. In cases where exhaustive combination isnot feasible, due to curse of dimensionality issues, heuristicenumeration or algorithmic search approaches can be used toconduct a partial enumeration. Functions are then mapped toelements of form in function-form mapping, in order to movefrom the solution-neutral domain to the selection of specifictechnologies that compose the architectures. Architectures arethen evaluated through the development of evaluation metrics,which are required to assess the overall value of enumer-atedarchitectures.Metrics can be classified as objective metrics and sub-

jective metrics. Objective metrics are quantities that can bemeasured or estimated either by direct measurement, first-principle modeling, parametric modeling or analog estimates.Examples of objectives metrics are dry mass, design velocity,time and lifecycle cost. Subjective metrics are quantitativemeasures of subjective judgments. Examples of subjectivemetrics are perceived technical risk, perceived engineeringcomplexity and perceived delivered value to scientists. Sub-jective metrics are estimated either by heuristic rules definedby experts, or by structured methods. An example of heuristicrule is to associate a higher perceived technical risk to archi-tectures that feature a higher number of development projectsand/or a higher number of operations perceived as poten-tially “risky” by the system architect. In addition to heuristicrules, several structured methods exist to measure subjectivemetrics. This paper considers two alternative approaches forsubjective metrics evaluation: score cards and multi-attributeutility theory. Method selection depends on the characteris-tics of the problem at hand and types of property values tobe assessed. Strengths and limitations of these methods asidentified by this paper are discussed. It is useful to makein this context the distinction between ordinal and cardinalmetrics that can be developed for systems architecting. Ordi-nal metrics are used to provide a ranking between competingarchitectures. They are therefore metrics for relative rank-ing. Cardinal metrics, in turn, are absolute metrics for rela-tive ranking. Cardinal metrics are possible when incrementalunits are constant and objective. As this is seldom the casein subjective value metrics, subjective metrics as defined inthis paper must be intended in an ordinal sense. Indifferenceanalysis for the identification of Pareto-efficient architectureis still possible, in fact, with ordinal metrics. Score cards isthe first method here employed for the development of sub-jective value metrics. Score cards consist in eliciting knowl-edge by asking experts to rate property variables of intereston a pre-defined ordinal Likert scale [Likert, 1932]. Expertscan also be asked to provide motivations to motivate elicitedscores, which forms valuable documentation for traceabilityand credibility of the subjective valuemetric being developed.Score card results can then be integrated in a joint valuemetric as a weighted linear combination or other represen-tative functions of interest. Weights can be either elicited byexperts or assumed a priori by the systems architect. Utilitytheory [Fishburn, 1970] is a second approach to subjective



Figure 3. Design Space Example, Identification of Set of Pareto-efficient architectures.

0.0000.1000.2000.3000.4000.5000.6000.7000.8000.9001.000

Surface 1-meter 10-meter Deep Drill

Scie

nce

Ulit

y

Sample Depth Science U lity, Geologist View (ID: SCI004)

0.0000.1000.2000.3000.4000.5000.6000.7000.8000.9001.000

Surface 1-meter 10-meter Deep Drill

Scie

nce

Ulit

y

Sample Depth Science U lity, Astrobiologist View (ID: SCI002)

Figure 4. Science utility associated with different sample depths as seen by different views.

evaluation. Utility theory comprises several approaches usedin finance and econometrics to estimate perceived value—forinstance of value that an investor associates to some capitalinvestment. In recent years, utility theory has been appliedto estimate value in engineering systems analysis problems[Ross and Hastings, 2005].

3.4 Step 3: Expert Panel Formation

Step 3 is concerned with the formation of the panel of expertsto be involved in the study. In this paper, the word “panel”refers to the aggregate of experts being involved in the study.However, it must be noted that experts do not meet duringthe decision-making process, as they interact by means of amoderator. Selecting experts is a critical step in the frameworkas the quality of the results is strongly dependent on thequality of the answers given by the expert panel as a whole.The system architect in charge of the study identifies a first set

of individuals to be involved. The sampling method used inDB-SAF is judgment sampling [Marshall, 1996]. As all non-probabilistic sampling methods, judgment sampling is only apartial sampling of the population of experts, which then doesnot allow to make generalizations on the aggregate opinion ofthe expert community from a strictly scientific perspective.Nevertheless, polling the entire population on experts wouldbe impractical or most likely impossible. The experience ofthe systems architect plays a key role in determining a “good”set of experts of the study. A method to mitigate bias inexpert selection is the use of snowballing sampling methodsin DB-SAF [Mason, 1996], asking participating experts toidentify additional participants among their acquaintances,using their judgment. The process can be repeated until thesystem architect is satisfied with the number of experts avail-able for the study. The panel should be a good representativeof the community of stakeholders interested in the mission,in order to represent all views and biases in an equal and fairmanner.



Table I. Example Stakeholder Goals for the Mars SampleReturn Campaign

Stakeholder Goals

To collect samples of the martian surfaceTo conduct in situ science on MarsTo return collected samples to EarthTo ensure collected samples comply with planetary protection

requirements

Table II. Functional Decomposition for the Mars SampleReturn Campaign

Functions

1 To reach low Earth orbit2 To transit between low Earth orbit and low Mars orbit3 To entry Martian atmosphere4 To descend and land on Martian surface5 To drill Martian surface and prepare sample caches for Fetching6 To fetch sample caches7 To bring sample caches to Earth surface

[Rowe et al., 1991] identifies four key characteristics thatdefine a “good” expert:

1. Knowledge and experience with the issues under inves-tigation

2. Capacity and willingness to participate3. Sufficient time to participate in the study4. Effective communication skills

Once a panel of experts is formed, the architecting teamwill start the first round of interviews by reviewing the initialproblem formulation, as outlined in Step 4.

3.5 Step 4: Problem Formulation Review withExpert Panel

This step ensures the validity of the problem formulationdeveloped in Step 2, improving the alignment of the study tothe issues that are felt relevant by the expert panel and thestakeholders. Iterations are required to review the problemformulation until the architecting team reaches a satisfyingconceptualization of the problem. A typical review list con-sists of the following items:

• Ensure the validity of modeling assumptions adopted inthe framework;

• Ensure the validity and completeness of the list of ques-tions to be addressed by the study;

• Ensure the completeness of the needs perceived by thebeneficiaries;

• Ensure the completeness of the list of stakeholder goalsand their biunivocal mapping to beneficiaries’ needs;

• Validate the minimum and maximum boundaries of in-terest for each requirement option that involves a quan-titative assessment;

• Ensure the completeness of the list of system functionsand their biunivocal mapping to stakeholder goals;

• Ensure the completeness of the list of requirements andtheir biunivocal mapping to system functions;

• Ensure the completeness of the list of options of formand their biunivocal mapping to the list of system func-tions;

• Ensure the validity of the evaluation metrics as devel-oped.

Once the architecting team is satisfied with the revisedproblem formulation, it proceeds performing the elicitationof knowledge from the experts by designing a survey or in-terview (Step 5) and synthesizing their value judgment (Step6).

3.6 Step 5: Design of Interview

The design of the interview is a crucial step of the frame-work, as it involves careful consideration of the questionsto be answered by the study as defined in problem formu-lation (Step 2), as well as considering behavioral aspectson choosing the best method to elicit knowledge from ex-perts. The interview makes use of different analytic toolsfor expert elicitation, depending on the nature of the designvariables and requirements being assessed, such as whetherthey vary on continuous or discrete domains, and whetherthey exhibit mutually excluding attributes or complementaryattributes.

3.6.1 Estimation of Utility Functions for Attributes Varyingon Continuous DomainsClassical utility functions from utility theory are effectivetools to represent value judgment for attributes varying ona continuous domain. Utility theory assumes that value canbe represented by a normalized function of some attribute ofinterest. However, it is often the case that value is functionof multiple attributes of interest, for which more articulatedtheories must be employed for value elicitation. MAUA isa decision analysis tool to represent an expert’s value as-sessment as a function of multiple attributes. MAUA hasbeen extensively surveyed in the literature (see [Abbas, 2010;Wallenius et al., 2008] for an extensive review). Methodsused for estimation of single-attribute utility functions andweights depend on the type of attribute being assessed andon their mathematical expression—whether they are discretevalues or span a continuous range of possible values. Util-ity function estimation methods are reviewed in Keeney andRaiffa [1976]. Two popular methods for utility elicitationare the Certainty Equivalent Probability (CEP) [Keeney andRaiffa, 1976] method and the Lottery Equivalent Probability(LEP) [McCord and de Neufville, 1986] method. CEP and/orLEP are used to estimate utility values at selected require-ment points. The utility curve is then obtained through least-square fitting of known utility models (such as the negative



Table III. Structural Morphological Matrix of Possible Requirement Sets for the Mars Sample Return Campaign

Options

Requirements 1 2 3 4 Number of Options

Drilling system maximum reachable depth Surface (∼2.5cm)

1-m 10-m 3

Total number of samples collected 10 20 30 40 4Sample size Small (0.5cm D

× 1.0 cm H)Medium (1.0 cm

D × 5.0 cmH)

Large (5.0 cm D× 15 cm H)

3

Horizontal diversity (characteristic radius) 5 km 10 km 25 km 50 km 4Collect sedimentary material samples Yes No 2Collect hydrothermally and low temp. altered samples Yes No 2Collect igneous rock samples Yes No 2Collect regolith, dust & atm. gas samples Yes No 2Total No# of Possible Requirement Sets 2304

Table IV. Structural Morphological Matrix of Alternative Forms for the Mars Sample Return Campaign

Options

Forms 1 2 3 4 No# of Options

Number of Elements 1 (Drill + Fetch+ Return)

2 (Drill + (Fetchand Return))

2 ((Drill andFetch) +Return)

3 ((Drill) +(Fetch) +(Return))

4

Mars ascent vehicle (MAV) number of stages 1 2 3 3Earth return vehicle (ERV) number of stages 1 2 2Mars ascent vehicle (MAV) propulsion type Solid Storable

NTO/N2H42

Earth return vehicle (ERV) platform type MAV only MAV + Orbiter 2Earth return vehicle (ERV) propulsion type Storable

NTO/N2H41

Total No# ofArchitectures

96

exponential utility function formulation [Keeney and Raiffa,1976]) or through piece-wise interpolation. The latter methodis found to be more useful in engineering analysis, as standardformulations were developed with specific purposes for dif-ferent types of applications—such representing risk aversionin investment selection in finance and econometrics [Meyer,1987]—and do not account for typical situations encounteredin the real world of engineering design practice. CEP andLEP methods can be used concurrently to obtain redundantmeasurements and therefore estimate the uncertainty withinthe answers provided by the interviewee. The literature pro-vides methods with which calibrate answers depending onthe degree of consistency of experts and evaluate the qualityof the overall assessment [Morgan and Henrion, 1990]. Inboth the additive and multiplicative formulations of multi-attribute utility, weights play the role of representing prioritiesacross attributes as specified by experts. There are severalweight elicitation procedures that have been proposed in theliterature. Three weight elicitation procedures that have beencommonly used in engineering literature, and that can beadopted in the context of the present framework are the ratio

method [Edwards, 1977], the swing method [von Winterfeldtand Edwards, 1986], and the trade-off method [Keeney andRaiffa, 1976].

3.6.2 Estimation of Utility Functions for Attributes Varyingon Discrete DomainsThe CEP and LEP methods discussed previously are not suit-able methods for utility estimation if attributes vary on dis-crete domains. Discrete attributes can be classified betweenmutually excluding attributes and complementary attributes.Mutually excluding attributes are defined as attributes thatcan take only one value among a range of possibilities. Com-plementary attributes are defined as attributes that can takemore values among a range of possible options. By necessity,utility in this case is defined by the interviewee relatively toa reference choice (which utility is normalized to one). Amodified version of the ratio procedure discussed for weightelicitation can be employed for utility estimation of mutuallyexcluding discrete attributes. In the proposed procedure, the



Table V. Evaluation Metrics for the Mars Sample ReturnCampaign Case Study

Objective SubjectiveMetrics Units Metrics Units

Total dry Mass kg Utility to Scientists –Total wet Mass kg Utility to Engineers –Total lifecycle cost FY15 M$

Table VI. Expert Panel Composition

Experts AcademiaAffiliation JPL ESA (EU) TOTAL

“Test pilot” interviews 6 2 – 8Scientists 6 0 2 8Engineers 3 1 0 4

Grand Total 20

experts are asked to rank alternative values of the attribute ofinterest. The first-ranking (best) attribute is given 100 points.The interviewee is then asked to assign decreasing points—multiple of 10—to the other alternatives. As a consistencycheck, a second round of interview can be conducted wherenow the last-ranking (worst) attribute is given 10 points. Thistime, the expert is asked to assign increasing points to theremaining alternatives. Successive normalization to the best-ranking alternative will yield utility values for the attributeof interest. This proposed procedure features high ease ofimplementation and rapid understanding from interviewedexperts; however, it is not as rigorous as what it could beachieved using a tradeoff procedure such as the one discussedin weight elicitation. Selection of a method over another willresult from a balance of available time and complexity in theinterview, and is left to the judgment of the architecting teamimplementing the proposed framework.

3.6.3 Estimation of Utility for Complementary DiscreteAttributesPortfolio compositions are typical applications that can bedescribed using complementary discrete attributes. A com-bined implementation of an additive utility formulation andCEP/LEP interviews for weight elicitation can be used toestimate utility associated with a complementary discrete at-tribute set. Associated utility equals:

U(X)=∑

iki ui (xi ), (1)

where ui (xi ) are binary single-attribute utility functions thatassume value 0 if attribute option i is selected, if selection isdeemed beneficial by the expert and value 1 if attribute optioni is not selected, if selection is deemed non beneficial. Utilityvalues are reversed if selection is deemed as non-beneficialand non-selection as beneficial.

Normalized weights ki represent the relative importance ofeach attribute value selected in the attribute and are estimatedthrough CEP/LEP interviews.

3.7 Step 6: Elicitation of Expert Value Judgment

Elicitation of expert value judgment consists in the admin-istration of the interview designed in Step 5 with each expertvia individual face-to-face meetings, phone meetings or usinga custom designed web tool. The interview is required tocomply the requirements of a Delphi study that is, ensuringanonymity of participants of the study and verifying the expertcomplies with requirements discussed in Step 3 (Expert PanelFormation).Several factors need to be accounted when eliciting knowl-

edge from experts, as in the case in DB-SAF. Issues to beconsidered in expert elicitation include the sampling method-ology, as well as the formulation of interviews and their ad-ministration.

The issue of sampling refers to decisions to be taken inselecting experts and expert groups for the interview, as wellas how to analyze and interpret data collected during inter-views, so that the results can be generalized to the originatingpopulation as a whole [Marshall, 1996]. It is in fact practi-cally impossible to gather the entirety of experts for a study:selection choices have to be made. Flick [Flick, 2009] definesfour occurrences in which sampling plays a role in expertelicitation. First, sampling plays a role at the beginning of theelicitation process, where experts to be interviewed have tobe selected (“case sampling”). Second, sampling decisionsoccur when defining stakeholder groups from which selectexperts are selected (“group sampling”). In the case studyillustrated in this paper, for instance, an implicit decision hasbeen made in choosing the science and engineering commu-nities as representative stakeholder groups for the problemat hand, excluding for instance other societal groups thatwere deemed less relevant—for instance the general public,or other professional communities. Third, sampling occurswhen making decisions on which interviews to consider asrelevant to the study at hand (“material sampling”). Fourthand last, sampling occurs when specific parts of interviewsare selected for more detailed analysis (“sampling within thematerial”). Different sampling techniques can be employedin expert elicitation, hence in DB-SAF. Marshall [Marshall,1996] defines convenience sampling, judgment sampling, andtheoretical sampling strategies. Convenience sampling is theleast rigorous technique available, where experts are chosenpurely on the subjectivity of the researcher and on the avail-ability of experts he or she is acquainted to. As consequence ofthe nature of this sampling technique, convenience samplingtypically results in poor quality data and in a lack of credibilityof the results [Marshall, 1996]. Judgment sampling on theother hand consists in an active selection of experts by theresearcher based on predefined selection criteria. Flick makesa further distinction in judgment sampling, distinguishingbetween statistical sampling [Flick, 2009], when selectioncriteria are formulated independently of the data collectedand analyzed during the expert interview process, and the-oretical sampling (as also defined in Marshall [1996] and



Figure 5. MSR case study results analysis example.

Figure 6. Engineering panel: Round 1.

Glaser and Strauss [1967]), where criteria are formulated andrefined throughout the interview process. Snowball samplingcan also be employed to enlarge sample size, where referralsare elicited from experts that are initially invited to the study.Biernacki and Waldorf analyze the advantages and limita-tions of snowball sampling [Biernacki and Waldorf, 1981],including biases that could be introduced in the composition

of the sample due to the nature of this sampling technique.No sampling technique is right per se [Flick, 2009]; tradeoffshave to be conducted when choosing sampling methods toachieve an optimal balance between width and depth of thestudy [Flick, 2009]. As discussed in Steps 1 and 3 of themethodology, the proposed formulation of DB-SAF employsjudgment sampling to choose experts, based on literature



Figure 7. Engineering panel : Round 2.

Figure 8. Engineering panel: Round 3.

review and interviewer’s experience. Snowball sampling hasbeen then employed to crosscheck expert choices made by theinterviewer and to enlarge sample size based on experts’ in-puts. Nevertheless, the system architect has a choice on whichsampling techniques to employ based on his experimentaldesign.The formulation of interview questionnaires is a critical

step in the definition of a DB-SAF study. The goal of in-

terview formulation is to design research questions in a waythat the researcher is able to collect sufficient data effectively,while meeting schedule and budgetary constraints imposedon the study. Flick recommends questions to be centered insensitizing concepts [Flick, 2009], that is, key concepts thatprovide the researcher a wide access to relevant informationto the study. In the DB-SAF case study presented in thispaper, sensitizing concepts that have been chosen are science



Figure 9. Science panel: Round 1.

value and engineering complexity, around which the inter-view design is made. In DB-SAF, questions are formulated ina clear way so that to maximize information retrieval from theexpert in the most unbiased and unambiguous way possible.Questions are structured in an interview questionnaire, usinga combination of quantitative and qualitative techniques asillustrated in Step 2 of this paper. Test interview with pilotexperts are employed in DB-SAF to refine questionnaire be-fore their use on the field. A typical DB-SAF questionnaireis a combination of open-ended and narrow-scope questions,where open-ended questions are asked at the beginning ofthe interview to facilitate communication with the expert,followed by more precise, narrow questions as the interviewprogresses. In DB-SAF test interviews, open-ended questionsare also used as a test to refine the scoping of questions, andsupport revisions in successive iterations of questionnaires.Ulrich defines guidelines to check interview formulations,looking at formulation criteria, as well as easiness of compre-hension, absence of ambiguity, and structure of the interviewquestionnaire [Ulrich, 1999].When applying the DB-SAF methodology, researchers are

advised to be open to reformulate questions are required, asadditional information on the study at hand emerges duringinterviews. Furthermore, a potential bias in interview formu-lation is introduced by the origin of questions being posed[Flick, 2009]; the researcher, whom is biased by his or hersocial and historical context, as well as his professional back-ground and experience, formulates questions. These biasesare mitigated in DB-SAF in Step 4—Problem FormulationReview with Expert Panel, as previously discussed, where

peer review ensures minimization of biases to the best extentpossible.Interview administration in DB-SAF expert elicitation is as

much as important to control as the preceding sampling andformulation steps. Flick argues that mediation and steeringare two common issues to be considered while administeringinterviews [Flick, 2009]. In mediation, the issue is to balancethe input of the interview guide and the aims of researchquestions with personal style of presentation. Furthermore,the interviewer faces steering choices in whether to follow thequestionnaire strictly, or to rather let experts elicit their judg-ment according to their train of thoughts. Hopf warns againststrict interpretation of interview questionnaires [Hopf, 1978],which may hinder the benefits of openness and contextualinformation of each expert. In DB-SAF, expert questionnairehave been followed as guidelines, however, giving leeway toexperts to provide additional information as deemed appropri-ate, recognizing individual preferences of each interviewee.Guidelines for interviewers on how to prepare and conductinterviews are discussed in Hermanns [2004].

3.8 Step 7: Results Analysis

After collecting data from experts, results are processed andincorporated into a systems architecting analysis model. Theobjectives of the results analysis are:

1. To identify a set of optimal requirements and archi-tectures of interest for further consideration by deci-sion makers by identifying the set of Pareto-efficient




architectures among the feasible architectures withinthe design space and providing a first-order scenarioanalysis;

2. To understand the impact of requirement options in theoverall performance of feasible architectures by per-forming a multiperformance architectural analysis ofthe design space;

3. To understand the impact of architectural options inthe implementation of all possible sets of requirementsfor a mission or campaign by performing a trade-offarchitectural analysis in the design space.

Figure 3 provides an example of identification of Paretoefficient architectures in design space analysis. The figureshows a sample bi-dimensional trade space where each bluedot represents a feasible architecture as evaluated by twoobjective metrics defined by system architects. Assuming thatmaximization of both metrics is desired, it is possible toidentify the utopia point, which is the ideal point where bothobjectives are at their maximum. The set of Pareto-efficientarchitectures (dots joint by the dashed line in the figure) arethe ones that tend towards the utopia point and exhibit effi-cient trade-offs between objectives. In the notional example inthe figure, Pareto-efficient architectures are options of interestto decision makers, featuring optimal solutions at differentlevels of metrics satisfaction.Design space exploration allows further elicitation of in-

sights beyond simple Pareto analysis. It can be used to:

• Analyze the variation in performance/utility on systemarchitectures due to change in requirements (multiper-formance architectural analysis).

• Analyze the variation in performance/utility due to theimplementation of different options to implement an ar-chitectural function of interest (trade-off analysis).

Design space exploration assumes consensus between ho-mogeneous groups of stakeholders. This condition is rarelyverified in initial phases of a complex engineering project.In reality, ambiguity in the definition of objective impliesthat utility functions are surrounded by uncertainty, and thatthe actual value judgments cannot be represented by univo-cal functions. Results analysis provides the means to reachoptimal compromises between heterogeneous stakeholdergroups—for instance, between the engineering and scientificcommunities. However, preliminary reaching of consensus isrequired within homogeneous groups of stakeholders. Thisis achieved through negotiation; the following step providesstructured tools to guide those negotiations and progressivelyguide experts to agree towards an optimal definition of objec-tives.

3.9 Step 8: Aggregate Results Discussion withIndividual Experts

The Delphi method requires iterations to discuss results withindividual experts. The goal is to guide experts in revis-ing their answers towards reaching consensus in the group.Rowe et al. describe the advantages of considering the ag-gregate response of a panel of experts towards reading thetrue answer being sought by the study [Rowe et al., 1991]:“the so called ‘theory of errors’ assumes that the aggregate




of a group will provide a judgment/forecast that is gener-ally superior to that of most of the individuals within thegroup: when the range of individual estimates excludes thetrue answer (T), then the median (M) should be at least asclose to the true answer as one half of the group, but whenthe range of estimates includes T, then M should be moreaccurate than more than half of the group. This indicatesthe advantage of taking a statistical aggregate of individualestimates.”

The Delphi method, out of all surveyed expert elicitationmethodologies, is the most appropriate tool to implementfor a systems architecture study. The reasons for this as-sessment are several: first, the Delphi allows the executionof studies in a distributed setting, with experts residing indifferent parts of the world. The method does not requireexperts to attend design sessions all at the same time, whichalso proves useful when probed experts are senior managerswith limited time to devote to this kind of studies. Sec-ond, Delphi allows having an aggregate picture of collectiveevaluation of metrics, with complete documentation of pro-cess and rationale that led to the final recommendations. Fi-nally, Delphi allows the implementation of anonymous stud-ies, enabling all experts to express judgment with no peerpressure.Round iterations as prescribed by the Delphi have been im-

plemented by showing stakeholders aggregate boxplot charts.In descriptive statistics, boxplots describe groups of datathrough their five-number summary: the lowest observation(sample minimum), lower quartile (Q1), median (Q2), up-per quartile (Q3), and largest observation (sample maximum)[Moore, McCabe, and Craig, 2007].

After each round, experts are asked to review their answerin light of the aggregate answer of the group.Overall, DB-SAF iterations reduce ambiguity that sur-

rounds the definition of objectives for a system’s architecture.This confirms previous empirical observations by researchersin social science [Rowe and Wright, 1999]. Furthermore,the application of DB-SAF to engineering case studies hasfound that there exists an irreducible amount of ambiguitythat cannot be resolved. This irreducible ambiguity can onlybe resolved after the engineering system being studied hasactually been developed and operated.An additional result of interest is to assess the impact of

identified ambiguities on the trade space of feasible systemarchitectures. Ambiguity Impact Analysis can be done withseveral analytic and numeric methods, including:

• Design of Experiments (DoE) and Sensitivity Analysis:performing a full or factorial experiment, with ambigu-ous variables as factors (i.e., the property variables oftheMAUA functions), and a number of percentile valuesas levels (such as the 0 [min]/25/50 [median]/75/100[max] percentiles). Main effects and main interactionscan be measured on a set of variables of interest, such asarchitectural utility values, mass, cost, and so forth. Bythis analysis, one can rank ambiguities by their overallimpact on selected variables. This approach can be ap-plied to compare individual system architectures, or tomeasure overall effect of ambiguities on the trade space.

• Correlation Analysis: In correlation analysis the goal isto identify correlations of the trade space with ambigu-ous variables of interest. This is done by the examination



10 20 30 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

# SamplesAvg

.Mai

nE

f fect

o nC

ampa

ign

Cos

t(pr

op.t

ot o

talm

ass)

Surface 1m drill 10m drillDrilling Depth

5km 10km 25km 50kmHoriz. Mobility

Small Medium0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Sample Size

Avg

.Mai

nE

ffect

onC

amp a

ign

Cos

t(pr

op.t

oto

talm

ass)

2(D)(FR) 2(DF)(R) 3(D)(F)(R) 1(DFR)# Missions

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

SedimentaryAvg

.Mai

nE

ffect

onC

ampa

ign

Cos

t(pr

op.t

oto

talm

ass

N YN YHydrothermal

N YIgneous

N YRegolith

Figure 12. Mars sample return ambiguity impact analysis—Maineffects.

of the trade space with proxy metrics for variability in-duced by ambiguity. For instance, one could choose thestandard deviation of subjective utility metrics associ-ated with each architecture (where the spread is givenby differences in opinions of individual experts in theirrespective panels). This approach is suitable for assess-ment of ambiguity impact on the trade space as it doesnot require the definition of arbitrary weights, while al-lowing the analysis of a large data set.

• Monte Carlo Analysis: Monte Carlo is a popular ap-proach for quantification of effects of multidomain un-certainties, which can also be applied to the analysis

of ambiguities. In this setting, a Monte Carlo analysisimplies the definition of stochastic valuemetrics, definedby assuming distribution shapes to fit ambiguous dataof interest (such as normal or triangular distributions).Stochastic value metrics are used to evaluate the tradespace a large number of times, therefore deriving statis-tics of interest to assess variability induced by ambiguityon output metrics of interest. The assumption of a dis-tribution is arbitrary in this context, as there is no firmrationale on how to choose a distribution over another.Monte Carlo is a valid choice for comparison of selectedarchitectures. However, if the trade space is of large size(>106 architectures), Monte Carlo may not be a suitableapproach due to large computational times.

3.10 Step 9: Convergence Criteria

DB-SAF is terminated when a criterion for convergence ismet. Typical convergence criteria are the achievement of apre-defined number of iterations, the achievement of consen-sus and the stability of results when variations in answersbetween two rounds are less than a pre-specified tolerancecriterion. Themedian ormean results are used at the end of theDB-SAF process. The standard deviation at the final iterationrepresents the variability induced by the estimated irreducibleambiguity.

3.11 Step 10: Documentation and Developmentof Recommendations

Once convergence has been met, the results of the resultingdesign space exploration are documented and used for thedevelopment of recommendations to systems engineers anddecision makers.

4. MSR CASE STUDY

The previous section has presented the methodological ap-proach that has been developed for architecting complex engi-neering systems under ambiguity. This section demonstratesthe application of the framework to a MSR campaign casestudy. While the discussion of specific case study results areoutside of the scope of this paper, the discussion will focus onmethodological aspects of general interest for the applicationof the framework to different complex engineering systemprojects.

4.1 Case Study Description

The goal of MSR missions is to retrieve samples of theMartian surface and bring them back to Earth. For decades,MSR has been the holy grail of planetary science; scientistshave been advocating a sample return mission to Mars forthe past 30 years [NRC, 2011]. Nevertheless, MSR is stillbeing studied in its formulation phase, trying to match theambitious goal of returning samples from the Martian surface



# Missions

# Missions# Missions = 2(D)(FR)# Missions = 2(DF)(R)# Missions = 3(D)(F)(R)# Missions = 1(DFR)

Small MediumSample Size

5km 10km 25km 50kmHoriz. Mobility

Surface 1m drill 10m drillDrilling Depth

10 20 30 40

0.2

0.4

0.6

0.8

# Samples

Util

ityto

Sc i

ent is

ts

0.2

0.4

0.6

0.8

Util

ityto

Scie

ntis

ts

Sample SizeSample Size = SmallSample Size = Medium

0.2

0.4

0.6

0.8

Util

ityto

S cie

nti s

t s

0.2

0.4

0.6

0.8

Util

ityto

Sci

entis

t s

Horiz. Radius

Horiz. Radius = 5kmHoriz. Radius = 10kmHoriz. Radius = 25kmHoriz. Radius = 50km

0.2

0.4

0.6

0.8

Util

ityto

Sc i

ent i s

t s

0.2

0.4

0.6

0.8

Util

ityto

Sci

ent is

ts

Drilling DepthDrilling Depth = SurfaceDrilling Depth = 1m drillDrilling Depth = 10m drill

0.2

0.4

0.6

0.8

Util

ityto

Sci

enti s

ts2(D)(FR) 2(DF)(R)3(D)(F)(R) 1(DFR)

0.2

0.4

0.6

0.8

Util

ityto

Scie

ntis

ts

Small Medium5km 10km 25km 50kmSurface 1m drill 10m drill

# Samples# Samples = 10# Samples = 20# Samples = 30# Samples = 40

Figure 13. Mars sample return ambiguity impact analysis—Main interactions.

with the more terrestrial requirement of meeting available fi-nancial budgets. Ambiguity in the definition of objectives forMSR is related with the difficulty of finding a suitable com-promise with stakeholders between science and engineeringrequirements of the mission. Scientists involved in planninghave different views on how a particular objective should bedefined, such as how many samples collect from Mars toachieve a set of scientific goals. A clear example that has beenidentified in this study is the difference in value judgmentbetween astrobiologists and geologists in the evaluation ofalternative sets of requirements for the campaign. Figure 4shows an example of science utility associated with differentsample depths, as seen by two different stakeholders.In this example, experts were asked for their value judg-

ment as a function of different attributes of interest. Valuejudgments were successively codified in utility functions.Sample depth could vary between surface drilling (2.5-cmdepth), 1-m drilling, 10-m drilling and deep drill (100 m).Results were normalized to the maximum utility value foreach set. The results showed that geologists were mostlyinterested in surface samples: 2.5 cm in depth allows thecollection of samples that have not been altered significantlyby atmospheric processes (unweathered samples [Gooding,Arvidson, and Zolotov, 1992]) while providing relevant in-formation on the geologic processes that shaped the Martiansurface. Deeper sample collections are deemed less valuableas their vertical distribution history is harder to keep duringcoring operations, rendering science hypothesis formulationand testing harder and less reliable. On the other side ofthe spectrum, astrobiologists indicated 10 m as their mostinteresting sample depth, having higher likelihood of findingsigns of life at those depths.

This case study was formulated by the first author be-tween June 2011 and September 2011 at Caltech/NASA’sJet Propulsion Laboratory (JPL), and proceeded through De-cember 2011 at MIT. The study involved expert panels witha total of 12 experts (senior engineers and scientists) fromJPL, the European Space Agency, and international academicinstitutions.

4.2 DB-SAF Application to MSR Architecture

The application of DB-SAF is here demonstrated on the MSRarchitecture case study. The remainder of this section is struc-tured using the 10-step approach that has been describedpreviously.

4.2.1 Step 1: Literature Review and Systems-SpecificExpertisePoint designs and architecting studies that have been con-sidered in this case study are the Mission Concept Studiesdeveloped in support of the 2010 Planetary Science DecadalSurvey, such as theMars 2018MAX-CCaching Rover Assess-ment Study [NASA, 2010], the MSR Lander Mission Study[NASA, 2010], and the MSR Orbiter Mission Study [NASA,2010]. Program-level documents such as the Planetary Sci-ence Decadal Survey [NRC, 2011] were also considered, inaddition to the science-prioritization documents formulatedby the Mars Exploration Program Analysis Group (MEPAG)[NASA, 2010].



Figure 14. Correlation analysis results.

4.2.2 Problem Formulation4.2.2.1. Iteration Cycle 1.

MSR beneficiaries are astrobiologists and geologists thatare wishing to further their science by retrieving data fromMartian samples. Questions of interest as elicited from itera-tions with expert are:

• What kind of sample types should MSR bring back toEarth to maximize scientific value while being imple-mentable from an engineering and programmatic stand-point?

• What is the impact of the maximum drilling depth on theoverall performance/cost/engineering complexity of theMSR campaign architecture?

Stakeholders goals for MSR as elicited from experts arepresented in Table I.Following goals elicitation, a one-level functional de-

composition exercise was performed. The one-level func-tional decomposition for the MSR campaign is shown inTable II.

4.2.2.2. Iteration Cycle 2.

Table III summarizes requirements and requirements op-tions that have been identified in the MSR case study.This type of tabular representation is usually defined asstructural morphological matrix [Crawley, 2008]. A require-ment set is defined by selecting one option per require-ment. A full enumeration leads to 2304 possible requirementsets.

Functions elicited in iteration cycle 1 are then associated tocorresponding elements of form (Table IV). An architectureis defined by selecting one option per element of form. A fullenumeration in the MSR case renders 96 possible architec-tures.Insofar evaluation metrics, Table V summarizes the evalua-

tion metrics that have been developed for the MSR Campaigncase study. Total technical risk is defined heuristically as thetotal number of development projects within architecture def-initions.

4.2.3 Step 3: Expert Panel FormationThe expert elicitation questionnaire has been tested and cal-ibrated with eight test pilot interviews conducted with engi-neers and scientists at the NASA JPL and at the EuropeanSpace Agency. These interviews were not included in theassessment as they were used to refine the expert elicitationprocedures. Two experts panels were composed for this study,representing science and engineering respectively. Interviewswere monitored asking control questions to experts, suchas posing the same question using two different formula-tions, in order to ensure consistency of the answers provided.Experts were selected using judgment sampling, based onthe decision-making roles held in their organizations, fol-lowed by snowballing sampling. Experts were recruited ona volunteer basis. The goal was to obtain a balanced US–EU panel. However, as the study employed senior expertsand decision makers and relied on voluntary contributions,a perfect 50/50 distribution could not be achieved. Morethan 30 experts were contacted worldwide to obtain a total



0 0.05 0.1 0.15 0.2 0.25 0.3 0.350

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Utility to Scientists (Standard Deviation)

Util

ityto

Sci

enti s

ts

Surface1m drill10m drill

Figure 15. Impact of drilling depth ambiguity on engineering complexity value assessment.

of 12 participating representatives. However, low participa-tion was offset by proven experience and role of individualexperts in their respective organizations. Furthermore, thestudy has been cross-validated a posteriori by comparisonwith current or previous Mars exploration architectures, andby presenting final results to larger poll of scientists andengineers. Lack of availability of ESA experts from the sci-ence side has been offset by recruitment of two scientistswith Principal Investigator roles in European missions, af-filiated to European universities. The number of scientistshas been doubled with respect to the number of engineersas the observed variability on science measures was con-siderably higher than that observed for engineering assess-ments. An overview of expert panels composition is shown inTable VI.

4.2.4 Step 4: Problem Formulation Review with ExpertPanelAn initial problem formulation has been outlined using in-formation obtained by surveying the public literature on theMSR campaign and on expertise of the first author in spacesystems modeling. Successively, the formulation of the inter-view has been refined and extended through a “Round 0” in-teractionwith “test pilots” fromNASA JPLMission ConceptsSection and the JPL Mars Office.

4.2.5 Step 5: Design of InterviewThe interview has been designed using amulti-attribute utilityformulation. Two types of interview questionnaires have beenelicited:

• An interview of Scientists, to estimate the scientific valueof a given set of requirements for a MSR architecture.

• An interview of Engineers, to estimate the engineeringdifficulty perceived for the implementation of a given setof requirements (as a proxy of complexity and technicalrisk).

The interviews have been designed with the assumptionthat all interviewees are rational utility-maximizing individ-uals. That is, a utility value of 1 in the scientific interviewtranslates to 100% satisfaction of the given science with aparticular set of requirements. A utility value of 1 in theengineering interview means 100% satisfaction from an en-gineering perspective (100% ease in implementing the pre-scribed set of requirements). A utility value of 0 correspondsto no scientific value and very high-perceived difficulty inengineering implementation, respectively.Both the Scientific and Engineering interviews were based

on MAUA formulations developed on the following at-tributes:

• Sample types• Total number of samples• Sample depth• Sample size• Horizontal traverse distance.

Sample types have been modeled using a complementarydiscrete utility model. Total number of samples and horizontaltraverse distance have been modeled as continuous utilityfunctions using a combination of CEP/LEP interviews. Sam-ple depth and sample size have been modeled using mutuallyexcluding discrete utility models.



0 0.05 0.1 0.15 0.2 0.25 0.3 0.350

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Utility to Scientists (Standard Deviation)

Util

ityto

Sci

enti s

ts

Surface1m drill10m drill

Figure 16. Impact of drilling depth ambiguity on science value assessment.

4.2.6 Step 6: Elicitation of Expert Value JudgmentExpert elicitation has been done through interviews con-ducted by the first author at NASA JPL, and by phone withexperts in other institutions.

4.2.7 Step 7: Results AnalysisDesign space exploration results are analyzed at each stepof the study to prepare the discussion in successive roundswith experts. If convergence is met (as discussed in Step9), results analysis provides the data for documentation anddevelopment of recommendations (Step 10).Figure 5 shows an example of results analysis of the

MSR case study, which generates a trade space composedof 442,368 feasible architectures. Architectures are repre-sented by dots that are color-coded based on their estimatedtotal campaign lifyecycle cost. The x-axis represents utilityprovided to scientists (i.e., science value), the y-axis rep-resents utility provided to engineers (i.e., engineering com-plexity). The larger red dot on the upper-right corner of theplot shows the current MSR baseline architecture as eval-uated by the model. The MSR baseline lies on the Paretofrontier as defined by utility functions elicited by experts.This provides confidence on the coherence of results of thedesign space exploration with existing value judgments. Assome of the experts were directly involved in the definitionof the baseline, they would be expected them to provide apositive evaluation of their own architecture. Once validated,the same value judgment criteria can be applied with in-creased confidence to the other options available in the tradespace.

4.2.8 Step 8: Aggregate Results Discussion with IndividualExpertsBoxplot charts have been used to aggregate results and fa-cilitate negotiations with individual experts during DB-SAFelicitation. Experts have been asked to revise their answers inlight of the aggregate of the answers provided by other ex-perts in the group. In addition to boxplot charts, experts havebeen presented rationale from the peer group in anonymousform, in order to inform the discussion. Figure 6 (Round 1),Figure 7 (Round 2), and Figure 8 (Round 3) show the evolu-tion of answers in the three Rounds for the engineering panel,whereas science panel evolution is shown in Figure 9 (Round1), Figure 10 (Round 2), and Figure 11 (Round 3).The first empirical observation to make is the signifi-

cant difference in convergence between engineering and sci-ence panels. Engineers were able to achieve agreement morerapidly and in more areas between Round 1 and Round 2. Asignificantly larger relative divergence was observed in thetest pilot phase of the science panel. As an initial mitigationto this phenomenon, the science panel was doubled in size,by interviewing additional experts with respect to the engi-neering panel. These experts were given Round 1 interviews,and their answers were included in Round 2 computations.This addition of experts did not increase or decrease ob-served variability significantly, therefore confirming prelim-inary findings. It must be highlighted that both panels werecomposed by senior experts and high-level decision makers,with multiple years of experience in contributing to the sys-tems architecting process of exploration missions to Mars.This is, however, a double-edged sword: while the expertsample is representative of the MSR decision-making com-munity fromAmerican and European perspectives, a potential



Figure 17. Engineering panel: Round progress overview.

bias is introduced in having the majority of panelists beingsenior experts in their respective fields. Biases are introducedby their professional experience in the field, having analyzedsystems trade-offs for MSR in multiple occasions. This as-pect has been partially mitigated by having an internationalcomposition of the panel with both American and Europeanperspectives—as US and EU space agencies are the historicalmajor developers of robotic exploration missions. Neverthe-less, this is an aspect to account for as an inherent feature ofthe proposed framework. Results of the model are no betterthan the inputs that were provided. Nevertheless, the modelthat has been developed is able to evaluate quickly a verylarge amount of system architectures, by using the same logicemployed by the panel of experts. This feature allows deci-sion makers to screen a much higher number of requirementsets and architectures that could be done by conventionalprephase A study. This represents one of the main value-enhancing contributions of the application of the comprehen-sive systems architecting approach to Mars Sample Returndesign.Following iterations, the reducible part of ambiguity is

minimized; it is therefore possible to assess the impact ofirreducible ambiguities that have been identified within thearchitectural trade space using Ambiguity Impact Analysis asdiscussed in Section 3..Figure 12 shows the results of a main effects analysis on

ambiguity impact. The goal of the analysis is to identifydriving design parameters that determine the shape of the

trade space, hence its tradeoffs. The x-axes on the plots showrequirement variables and partition the trade space in sub-sets using possible values such variables can assume. They-axis on the plots is the average effect of requirement vari-ables on campaign cost normalized to the maximum and min-imum cost values observed in the architectural trade space—absolute cost numbers are not relevant in this context as weare interested in relative rather than absolute effects. Cam-paign cost is a proxy variable for architecture performance,as the model employs mass-based cost models. The analysisshows that requirement variables have varying impact on thearchitectural trade space. From highest to lowest impact, theirrank is the following:

1. Sample size2. Drilling depth3. Total number of samples4. Total number of missions5. Sample types6. Horizontal mobility

The result from this analysis shows that main impacts arelargely related to requirements driving the sizing of payloadmasses on theMartian surface. By comparing impact analysisresults with the DB-SAF results shown in Figures 8 and 11,drilling depth emerges as the area in requirements analysiswhere system architects will need to concentrate on.Following main effects analysis, it is then of interest to in-

vestigate the occurrence of interactions between requirement



Figure 18. Science panel: Round progress overview.

variables and their impact in the architectural trade space.The goal of this interaction analysis is to identify pairs ofdesign parameters that need to be considered concurrently (asopposed to individually), as their effect on the trade space ishighly coupled. Interactions occur because of nonlinearitiesin the systems architecture, which are typical of complexsystems engineering problems. Figures 13 and 14 show ex-amples of results of this analysis. The matrix plot in Fig-ure 13 shows interactions between variable pairs. Features ofinterest include crossings—indicating inversions in the effectstructure—and magnitude of effects. Such magnitude is com-parable to what has been found in the main effects analysis,and nomajor interactions are highlighted. This analysis there-fore confirms results found with the main effects analysis ofambiguity impact on the trade space. Results of interest for theanalysis of ambiguities can be found on the 4th and 5th rowsand columns, associated to standard deviations in engineeringand science panels, respectively. Were crossings to be foundin the trade space, system architects would then be urged torun their analyses multiple times, considering all the possi-ble combinations of the interacting requirement parameters.Crossings are in fact of particular relevance, as they indicatehow the effect of a given input design parameter on the metricof interest depends on the value of some other parameters,with which the input parameter of interest interacts.Figure 14 shows the application of correlation analy-

sis, which exemplifies the interaction among metrics, show-

ing whether metrics are mutually supporting each other, orwhether they are in tension, and therefore exhibit Pareto-optima subset. Mutual support is represented by metricsshowing a direct correlation, whereas tensions are found byidentifying inverse correlations. No correlation correspondsto independence, that is, for a specific set of interest, met-rics are not affected by each other and thus their effect canbe considered in a separate way. The diagonal on the Fig-ure 14 shows the density of architectures in the x-axis ofthe trade space, providing a sense to the designer of theirdistribution. A distribution of the y-axis is easily obtainedby performing the same analysis on the transposed data set.Knowledge on the distribution of architectures provides sys-tem architects potential insights on architectural decisionsdriving such density. Conclusions can be derived by assess-ing the impact of the variable of interest on the trade space.Consider for instance the assessment of ambiguity associatedwith maximum drilling depth in engineering (Figure 15) andscientific value (Figure 16) assessments. By comparing thesecharts, it is clear that such ambiguity has a significant ef-fect in engineering complexity with increasing drilling depth.This is confirmed by conversations with engineers involvedin the design of drilling payloads in exploration missions, asdiscussed previously in this chapter. Associated benefits ofdeep drills, as shown in Figure 16, are modest if compared toassociated complexities. Decision makers can use correlationanalysis, for instance, to defer the implementation of new



drilling systems, waiting for ambiguity to unfold. Similaranalysis can be conducted for other architectural variables ofinterest.

4.2.9 Step 9: Convergence CriteriaA set number of three iteration rounds was chosen as conver-gence criteria for the case study. This criteria was selected toadapt to the time available for the study, the availability ofthe panel of experts, and verified by final round interviews.At its third iteration, experts felt no more need to changetheir assessments as they believed to have reached a validrepresentation of their opinions.Figures 17 and 18 show two matrices of bar plots with

the convergence history for the engineering and science pan-els, respectively. Rows in each matrix represent the differentproperty variables that have been probed. The vertical axison each graph plots the coefficient of variation 𝛽, defined asthe ratio of observed standard deviation 𝜎 normalized by themean value 𝜇 of the sample:

𝛽 = 𝜎

𝜇. (2)

Round number is shown on the x-axis of each graph. Ar-eas of reducible ambiguity are represented by plots wherethe coefficient of variation is reduced significantly betweenrounds. For instance, the variability on science value associ-ated with the maximum horizontal characteristic radius (tra-verse distance—HorizRad Max in Figure 18) shows a de-crease in ambiguity of ∼90% in approximately three rounds.Other measures, such as sample depth on the sample plot,reveal areas of irreducible ambiguity that could not be reducedby framework iterations.Convergence information as shown by the plots is valuable

information for decision makers, regardless of whether theyidentify areas of reducible or irreducible ambiguity. By pro-viding knowledge on reducible ambiguities, system architectsimprove their confidence on their models, and substantiatethe rationale for their conclusions. In cases where areas ofopen debates are identified, the framework allowed systemarchitects to be aware of areas of uncertain value, wherefurther discussion or analysis is required. As discussed inthe Descriptive Systems Architecting Management Frame-work Approach, these are areas where decision makers couldconsider delaying investments or developments, waiting forambiguities to reveal in the future.

4.2.10 Step 10: Documentation and Development ofRecommendationsOnce iterations are brought to a close, sufficient informationis generated for a comprehensive documentation of resultsand development of recommendations to decision makers.Specific recommendations to the MSR case study are notreported here, as their analysis is outside the scope of thispaper.

5. CONCLUSIONS AND FUTURE WORK

Ambiguity is a threat to successful systems architecting. Sys-tem architects need to identify, classify, reduce, and mitigateambiguity in early phases of the design process. Sourcesof ambiguity include the identification and management ofstakeholders, the elicitation of stakeholder needs and needsmapping to system functions (the functional intent of the ar-chitecture). Ambiguities are important to consider early in thedesign process, since this is the phase when future lifecyclecosts and potential for value generation are decided, and thephase of maximum leverage on system outcomes by the ar-chitect. A DB-SAF has been proposed in this paper to informdecision makers in situations of deep ambiguity. DB-SAF is astructured, iterative anonymous tool for informing ambiguitymitigation strategies inspired by the Delphi method in policy-making. This paper presented the approach, and demonstratedits application to aMSR case study.Major challenges encoun-tered in the application of DB-SAF have been (1) ensuringconsistency and objectivity in expert panel sampling and (2)the amount of time and resources required to conduct a study.Objectivity and consistency in selecting expert panels is

key to DB-SAF. As the outputs of the method heavily relyon the inputs provided by experts, it is thus critical to en-sure rigor and to understand the limitations associated withnonprobability sampling methods adopted. Control strategiesneed to be implemented during interviews, in order to ensureconsistency and assess reliability of interviewed experts. Aprotocol for the interview needs to be formulated and fol-lowed during each interview, in order to ensure consistencyacross different sessions. As DB-SAF always implies a partialsampling of the expert population, it must be recognized thatno generalizations, in a scientific sense, can be made fromthe interviews. Nevertheless, as the ambiguity issue at handis highly qualitative and implies consideration of several in-tangible factors, such rigor (typical of the social sciences) isthe best guarantee one can provide on the validity of a study.The amount of effort depends on the number of study

participants, number of Delphi rounds, and the tightness ofthe convergence criteria being chosen. The case study exem-plified how between 1 and 3 rounds are sufficient to conductan early-stage study; the discussion of convergence analysispresented in this paper shows how the setup of the study is atradeoff made by the system architect, based upon resourcesand time available to conduct the study. It is worthwhile tonotice, however, that most ambiguity is reduced in the firstround of iterations, making the proposed method suitable forearly-stage studies on a short schedule.Several avenues of future work are opened by this research.

For instance, it would be interesting to assess the effects ofcultural differences in leadership, gender and age differences,and other behavioral aspects in systems architecture studiesunder ambiguity. As DB-SAF heavily relies on expert elicita-tion, further investigation in its social science aspects wouldguide the development of future case studies involving largedesign teams distributed over multiple countries. Finally, itwould also be interesting to consider the development of anonline version of the DB-SAF framework suited for real-time design studies involving stakeholders. In this context,it would be interesting to apply online software and active



learning approaches to systems architecting under ambiguousobjectives.

ACKNOWLEDGMENTS

This research was conducted as part of the doctoral disserta-tion of the lead author. The authors would like to acknowledgefunding from the MIT-NASA HEOMD grant “Comprehen-sive Analysis and Synthesis of Exploration Architectures,”and all the experts from industry and academia who agreedto participate to the DB-SAF study of the MSR case study.

REFERENCES

A.E. Abbas, Constructing Multiattribute Utility Functions for Deci-sion Analysis, Tutorials in Operations Research, (2010).

M. Adler and E. Ziglio, Gazing into the oracle: The Delphi methodand its application to social policy and public health, JessicaKingsley Publishers, London, 1996.

R. Bellman and L.A. Zadeh, Decision-making in a fuzzy environ-ment, Manage Sci, 17 (1970), 141–164.

R.E. Bellman, Dynamic programming, Princeton University Press,Princeton, NJ, 1957.

P. Biernacki and D. Waldorf, Snowball sampling—problems andtechniques of chain referral sampling, Sociol Methods Res, 10(1981), 141–163.

G. Booch, J. Rumbaugh, and I. Jacobson, The unified modeling lan-guage for object-oriented development, Rational Software Cor-poration, Santa Clara, CA, 1996.

A.M. Brandenburger and B.J. Nalebuff, The right game: Use gametheory to shape strategy, Harvard Bus Rev, 73 (1995), 57–71.

T.R. Browning, E. Fricke, and H. Negele, Key concepts in modelingproduct development processes, Syst Eng, 9 (2006), 104–128.

C. Camerer and M. Weber, Recent developments in modelling pref-erences: Uncertainty and ambiguity, J Risk Uncertainty, 5 (1992),325–370.

B.G. Cameron, S. Catanzaro, and E.F. Crawley, Value based archi-tecture selection, Proceedings of the 57th International Astronau-tical Congress, Valencia, Spain, 2006.

H. Courtney, J. Kirkland, and P. Viguerie, Strategy under uncertainty,Harvard Bus Rev, 75 (1997), 66–79.

E.F. Crawley, ESD.34 System architecture—Lecture notes, Mas-sachusetts Institute of Technology, Cambridge, MA, 2008.

R. de Neufville, O. de Weck, D. Frey, D. Hastings, R. Larson, D.Simchi-Levi, K. Oye, A. Weigel, and R. Welsch, Uncertaintymanagement for engineering systems planning and design, Pro-ceedings of the Engineering Systems Symposium, MIT, Cam-bridge, MA, 2004.

O.L. de Weck, 16.842 Fundamentals of systems engineering—Lecture notes, MIT OpenCourseWare, Cambridge, MA, 2009.

O. De Weck, R. De Neufville, and M. Chaize, Staged deploymentof communications satellite constellations in low Earth orbit, JAerospace Comput, Inform Commun, 1 (2004), 119–136.

D. Dori and E.F. Crawley, Object-process methodology: A holisticssystems paradigm, Springer Verlag, Heidelberg, Germany, 2002.

J.S. Dyer, P.C. Fishburn, R.E. Steuer, J. Wallenius, and S. Zionts,Multiple criteria decision making, multiattribute utility theory:the next ten years, Manage Sci, 38 (1992), 645–654.

W. Edwards, The theory of decision making, Psychol Bull, 51(1954), 380–417.

W. Edwards, How to use multiattribute utility analysis for socialdecision making, IEEE Trans Syst, Man, and Cybernet, 7 (1977),326.

A. Engel and T.R. Browning, Designing systems for adaptability bymeans of architecture options, Syst Eng, 11 (2008), 125–146.

P.C. Fishburn, Utility theory for decision making, Research AnalysisCorp., Mclean VA, 1970.

U. Flick, An introduction to qualitative research, 4th edition, SAGEPublications Ltd, London, UK, 2009.

E.H. Forman and S.I. Gass, The analytical hierarchy process—Anexposition, Oper Res, 49 (2001), 469–487.

E. Fricke and A.P. Schulz, Design for changeability (DfC): Princi-ples to enable changes in systems throughout their entire lifecy-cle, Syst Eng, 8 (2005), 342–359.

B.G. Glaser and A. Strauss, The discovery of grounded theory:Strategies for qualitative research, Aldine Publishing Co., Pis-cataway, NJ, 1967.

J.L. Gooding, R.E. Arvidson, M.I. Zolotov, Physical and chemicalweathering, Mars, A93–27852 (1992) 626–651.

P. Harremoes, Late lessons from early warnings: Precautionary prin-ciple 1896–2000, European Environment Agency, Copenhagen,Denmark, 2001.

D.E. Hastings, A.L. Weigel, and M.A. Walton, Incorporating un-certainty into conceptual design of space system architectures,Proceedings of the ESD Internal Symposium, Working PaperSeries, May 29-30, 2002.

H. Hermanns, “Interviewing as an activity,” in U. Flick and E.V.Kardorff (Editors), A companion to qualitative research, SAGE,London, 2004, pp. 203–208.

C. Hopf, Die pseudo-exploration: Uberlegungen zur Technik quali-tativer Interviews in der Sozialforschung, Zeitschrift fur Soziolo-gie, 7 (1978), 97–115.

R.A. Howard, Decision analysis: Practice and promise, Manage Sci,34 (1988), 679–695.

D. Kahneman and A. Tversky, Prospect theory: An analysis of deci-sion under risk, Econometrica, 42 (1979), 263–291.

R. Keeney and H. Raiffa, Decisions with multiple objectives, JohnWiley and Sons, New York, 1976.

H.Y.B. Koo, A meta-language for systems architecting, Mas-sachusetts Institute of Technology, Cambridge, MA, 2005.

R. Likert, A technique for the measurement of attitudes, Arch Psy-chol, 140 (1932), 1–55.

J. Lin, Exploring Flexible Strategies in Engineering Systems usingScreeningModels—Applications to offshore petroleum projects,Massachusetts Institute of Technology, Cambridge, MA, 2008.

M.N. Marshall, Sampling for qualitative research, Fam Pract, 13(1996), 522–526.

J. Mason, Qualitative researching, Sage Publications, ThousandsOaks, USA, 1996.

M. McCord and R. de Neufville, “Lottery equivalents”: Reductionof the certainty effect problem in utility assessment, Manag Sci,32 (1986), 56–60.

R.G. McGrath and I.C. MacMillan, Discovery-driven planning, Har-vard Bus Rev, 73 (1995), 44–54.

J. Meyer, Two-moment decision models and expected utility maxi-mization, Am Econ Rev, 77 (1987), 421–430.

D.S. Moore, G.P. McCabe, and B. Craig, Introduction to the practiceof statistics, W.H. Freeman, 2007.



G. Morgan, Risk analysis and management, Scientific American,New York, 1993.

G.M. Morgan and M. Henrion, Uncertainty—A guide to dealingwith uncertainty in quantitative risk and policy analysis, Cam-bridge University Press, Cambridge, UK, 1990.

J.Murray, Definition of uncertainty in the Oxford English dictionary,Clarendon Press, Oxford, UK, 1961.

NAS, Environmental impact of stratospheric flight: Biological andclimate effects of aircraft emissions in the stratosphere, NationalResearch Council, Washington, DC, 1975.

NASA, Mars 2018 MAX-C Caching Rover – Planetary ScienceDecadal Survey – Mission Concept Study, 2010.

NASA, MSR Lander Mission – Planetary Science Decadal Survey –Mission Concept Study, 2010.

NASA, MSR Orbiter mission – Planetary Science Decadal Survey –Mission Concept Study, 2010.

NASA, Mars Exploration Program Analysis Group (MEPAG), JPL,Pasadena, CA, 2010.

J. Nash, Equilibrium points in N-person games, Proc Natl Acad SciUSA, 36(1) (1950), 48–49.

NRC, Planetary Science Decadal Survey 2013–2022, NRC, 2011.K. Oye, 17.310J science, technology and public policy—Lecture

notes, MIT, Cambridge, MA, 2010.M.S. Phadke, Quality engineering using robust design, Prentice Hall

PTR Upper Saddle River, NJ, USA, 1995.R.J. Rasmussen, Reactor safety study: An assessment of accident

risks in U.S. Commercial Nuclear Power Plants, Nuclear Regu-latory Commission, Washington, DC, 1975.

A.M. Ross and D.E. Hastings, The tradespace exploration paradigm,Proceedings of the INCOSE 2005 International Symposium,Rochester, NY, July, 2005.

A.M. Ross, D.A. Rhodes, and D. Hastings, Defining changeabil-ity: Reconciling flexibility, adaptability, scalability, modifiability,and robustness for maintaining system lifecycle value, Syst Eng,11 (2008), 246–262.

G. Rowe and G. Wright, The Delphi technique as a forecasting tool:Issues and analysis, Int J Forecast, 15 (1999), 353–375.

G. Rowe, G. Wright, and F. Bolger, Delphi: A reevaluation of re-search and theory, Technol Forecast Social Change 39(3), (1991),235–251.

B. Roy and M.R. McCord, Multicriteria methodology for decisionaiding, Springer, Dordrecht, The Netherlands, 1996.

H.M. Sapolsky, The politics of risk, Daedalus, 119 (1990), 83–96.S. Schenkerman, Avoiding rank reversal in AHP decision-support

models, Eur J Oper Res, 74 (2003), 407–419.M.R. Silver and O.L. deWeck, Time-expanded decision networks: A

framework for designing evolvable complex systems, Syst Eng,10 (2007), 167–186.

W.L. Simmons, A framework for decision support in systems archi-tecting, Massachusetts Institute of Technology, Cambridge, MA,2008.

G. Taguchi, Introduction to quality engineering: Designing qualityinto products and processes, Quality Resources, White Plains,NY, 1986.

T. Terpstra, M.K. Lindell, and J.M. Gutteling, Does communicating(Flood) risk affect (Flood) risk perceptions? Results of a quasi-experimental study, Risk Anal, 29 (2009), 1141–1155.

C.G. Ulrich, Deutungsmusteranalyse und diskursives Interview,Zeitschrift fur Soziologie, 28 (1999), 429–447.

K. Viscusi, Economics of regulation and antitrust, MIT Press, Cam-bridge, MA, 2005.

D. von Winterfeldt and W. Edwards, Decision analysis and be-havioral research, Cambridge University Press, Cambridge, UK,1986.

J. Wallenius, J.S. Dyer, P.C. Fishburn, R.E. Steuer, S. Zionts, andK. Deb, Multiple criteria decision making, multiattribute utilitytheory: Recent accomplishments and what lies ahead, ManageSci, 54 (2008), 1336–1349.

M. Zeleny, Multiple criteria decision making, McGraw-Hill, NewYork, 1982.

Dr. Alessandro Golkar is Assistant Professor at the Skolkovo Institute of Science and Technology (Skoltech) inMoscow,Russian Federation, a private university opened in collaboration with MIT. He received a Ph.D. in Aeronautics andAstronautics from MIT. He is the Director of the Strategic Innovation ResearchGroup (SIRG) at Skoltech. His researchinterests lie in the areas of systems architecture, project management, systems engineering, and spacecraft design anal-ysis and optimization. Alessandro had research and consulting experience at Caltech/NASAJet Propulsion Laboratory,and at the European Space Agency. BeforeMIT, Alessandro received a Laurea degree in Aerospace Engineering in 2006and a Laurea Specialistica degree in Astronautics Engineering in 2008 from University of Rome “La Sapienza”, Italy.

Dr. Edward F. Crawley received an Sc.D. in Aerospace Structures fromMIT in 1981. His early research interests centeredon structural dynamics, aeroelasticity, and the development of actively controlled and intelligent structures. Recently,Dr. Crawley’s research has focused on the domain of the architecture and design of complex systems. From 1996 to2003 he served as the Department Head of Aeronautics and Astronautics at MIT, leading the strategic realignment of thedepartment. Dr. Crawley is a Fellow of the AIAA and the Royal Aeronautical Society (UK), and is a member of threenational academies of engineering. He is the author of numerous journal publications in the AIAA Journal, the ASMEJournal, the Journal of Composite Materials, and Acta Astronautica. He received the NASA Public Service Medal.Recently, Prof Crawley was one of the ten members of the presidential committee led by Norman Augustine to studythe future of human spaceflight in the US.


A Framework for Space Systems Architecture under...

Documents

Transcript of A Framework for Space Systems Architecture under...