11

Strategic Software Strategic Software Engineering:Engineering:

On the Need for a New Software On the Need for a New Software Development ParadigmDevelopment Paradigm

Stuart R. FaulkStuart R. Faulk

Computer and Information ScienceComputer and Information Science

University of OregonUniversity of Oregon

22

DefinitionDefinition

par·a·digm : A collection of the major par·a·digm : A collection of the major assumptions, concepts, and propositions in a assumptions, concepts, and propositions in a substantive area of work or knowledge, a substantive area of work or knowledge, a coherent body of ideas or an approach attracting coherent body of ideas or an approach attracting widespread acceptance and use and providing widespread acceptance and use and providing the conceptual and contextual status for a the conceptual and contextual status for a branch of knowledge. Paradigms serve to orient branch of knowledge. Paradigms serve to orient research and theorizing in an area. Paradigms research and theorizing in an area. Paradigms describe "approaches" and compactly codify describe "approaches" and compactly codify central concepts and their relations central concepts and their relations

33

The “Waterfall” Life CycleThe “Waterfall” Life Cycle

44

With Iteration…With Iteration…

55

With Forethought…With Forethought…

66

Etc….Etc….

77

Common AssumptionsCommon Assumptions

Development focuses on producing specific Development focuses on producing specific code versioncode version– Begin with some form of requirementsBegin with some form of requirements– Develops a custom software solution to meet those Develops a custom software solution to meet those

requirementsrequirements– Ends with deployment, end of maintenance, or new Ends with deployment, end of maintenance, or new

versionversion

Implicit assumption that developing systems in Implicit assumption that developing systems in large organizations is much like writing code in large organizations is much like writing code in small onessmall ones

88

OverviewOverview

““Houston, we have a problem.” – on the nature Houston, we have a problem.” – on the nature of the ongoing software “crisis”of the ongoing software “crisis”No silver bullet – roadblocks to crisis resolutionNo silver bullet – roadblocks to crisis resolutionA road less traveled – going around rather than A road less traveled – going around rather than through the roadblocksthrough the roadblocksDéjà vu all over again - why we cannot Déjà vu all over again - why we cannot understand the future if we will not understand understand the future if we will not understand the pastthe pastThe Art of War - strategy over tacticsThe Art of War - strategy over tactics

99

What is Software Engineering?What is Software Engineering?The purpose of software engineering is to gain The purpose of software engineering is to gain

and maintain intellectual control over the and maintain intellectual control over the products and processes of software products and processes of software

development.development.

““Control” => ability to plan the work then work to the Control” => ability to plan the work then work to the planplan““Intellectual control” - means that we can make Intellectual control” - means that we can make rational choices based on an understanding of the rational choices based on an understanding of the (downstream) effects of those choices.(downstream) effects of those choices.Software engineering Software engineering managementmanagement is related but is related but focuses on control of development focuses on control of development resourcesresources (money, time, personnel).(money, time, personnel).

1010

The SSE HypothesisThe SSE Hypothesis

Effective management and control of Effective management and control of developments in most real organizations developments in most real organizations requires a requires a strategic viewstrategic view of software of software development (i.e., one that spans development development (i.e., one that spans development across multiple products, development cycles, across multiple products, development cycles, and organizational boundaries).and organizational boundaries).– Lemma 1: Necessary control is Lemma 1: Necessary control is unobtainableunobtainable at the at the

project level. project level. – Lemma 2: Some key development problems are Lemma 2: Some key development problems are

artifactsartifacts of our life cycle model of our life cycle model

1111

““Houston, we have a problem.”Houston, we have a problem.”

Have been in “crisis” since the advent of “big” softwareHave been in “crisis” since the advent of “big” software– What we want for software developmentWhat we want for software development

Low risk, predictabilityLow risk, predictabilityLower costs and proportionate costsLower costs and proportionate costsFaster turnaroundFaster turnaround

– What we have:What we have:High risk, high failure rate (40% - 60%)High risk, high failure rate (40% - 60%)Unpredictable schedule, cost, effortUnpredictable schedule, cost, effort

– Characterized by Characterized by lack of control: lack of control: cannot plan the work, cannot plan the work, work the planwork the plan

Three road blocks to getting beyond the crisis:Three road blocks to getting beyond the crisis:– Complex software is Complex software is essentially difficultessentially difficult – Software development is Software development is pre-industrialpre-industrial– Traditional SE processes Traditional SE processes only look at part of the problem only look at part of the problem

1212

I. Software is Essentially DifficultI. Software is Essentially DifficultIn In No Silver BulletNo Silver Bullet Fred Brooks provides a perspective on Fred Brooks provides a perspective on why we have not been able to resolve the “software crisis”why we have not been able to resolve the “software crisis”

– Identifies two kinds of development difficultiesIdentifies two kinds of development difficultiesEssential difficulties - part of the essence of the problemEssential difficulties - part of the essence of the problem

Accidental difficulties - introduced by imperfect practiceAccidental difficulties - introduced by imperfect practice

– Thesis: developing complex software systems is Thesis: developing complex software systems is essentially essentially difficultdifficult – – but not as difficult as we make it!but not as difficult as we make it!

““The hard part of building software [is] the specification, design The hard part of building software [is] the specification, design and testing of [its] conceptual constructs, not the labor of and testing of [its] conceptual constructs, not the labor of

representing it and testing the fidelity of the representing it and testing the fidelity of the representation”representation”

– Lemmas:Lemmas:

1.1. Most new SE technology addresses primarily “accidental Most new SE technology addresses primarily “accidental difficulties”difficulties”

2.2. Improvements here are at the point of diminishing Improvements here are at the point of diminishing returnsreturns

1313

II. Software is Pre-IndustrialII. Software is Pre-IndustrialPre-IndustrialPre-Industrial

Craftsman builds productCraftsman builds product– Builds one product at a timeBuilds one product at a time

– Each product is unique, parts are Each product is unique, parts are not interchangeablenot interchangeable

– Quality depends on craftsman’s Quality depends on craftsman’s skill – training & experienceskill – training & experience

– Unbounded opportunities for errorUnbounded opportunities for error

Focus on individual productsFocus on individual products– Customization is easyCustomization is easy

Scaling is difficultScaling is difficult– Parts are not interchangeableParts are not interchangeable

– No economy of scaleNo economy of scale

– Control problems rise exponentially Control problems rise exponentially with product size!with product size!

Post-IndustrialPost-IndustrialMachines produce productsMachines produce products– Quality depends on machines & Quality depends on machines &

manufacturing processmanufacturing process

– Production requires little training or Production requires little training or experienceexperience

Focus on developing the Focus on developing the means of productionmeans of production– Craftsman builds means to build Craftsman builds means to build

product (tools, factory)product (tools, factory)

– Customization is difficultCustomization is difficult

Easily scalesEasily scales– Parts are interchangeableParts are interchangeable

– Products are alikeProducts are alike

– Economies of scale applyEconomies of scale apply

1414

III. We Only Look at Part of the ProblemIII. We Only Look at Part of the Problem

Know that choices hereKnow that choices here– Are affected by prior choices, experience, productsAre affected by prior choices, experience, products

– Will affect future capabilities: business, technicalWill affect future capabilities: business, technical

Omits critical Omits critical context informationcontext information– Relation to short or long term business goals?Relation to short or long term business goals?

– Effect of decisions on future products?Effect of decisions on future products?

Factors affecting ability to control the process are outside the scopeFactors affecting ability to control the process are outside the scope– Factors Factors affectingaffecting the process are implicit the process are implicit

– Factors Factors affected byaffected by the process are nowhere evident the process are nowhere evident

SoftwareDesign

System Integrationand Testing

Coding

Deployment andMaintenance

RequirementsAnalysis

Product

1515

Consequence: Merry-Go-Round of Consequence: Merry-Go-Round of Sequential DevelopmentSequential Development

Industrial reality:Industrial reality:– Knowledge is not Knowledge is not

institutionalized (tacit)institutionalized (tacit)– Life cycle repeats Life cycle repeats

lessons of the pastlessons of the past

SoftwareDesign

System Integrationand Testing

Coding

Deployment andMaintenance

RequirementsAnalysis

Product

Those who cannot learn from history are doomed to repeat it - Santayana

1616

Sequential Development Over TimeSequential Development Over Time

Deploy/MaintainTest

CodeDesign

Requirements

Product

Time

Deploy/MaintainTest

CodeDesign

Requirements

Product

Deploy/MaintainTest

CodeDesign

Requirements

Product

… a result of “tactical software engineering”

1717

If only we could stop and think…If only we could stop and think…

“Here is Edward Bear, coming downstairs now, bump,bump, bump, on the back of his head, behind Christopher Robin. It is, as far as he knows, the only way of coming downstairs, but sometimes he feels there really is another way, if only he could stop bumping for a moment and think of it”

-- A. A. Milne

1818

A Road Less TraveledA Road Less Traveled

Going around (rather than Going around (rather than through) the software productivity through) the software productivity

barriers. barriers.

1919

Qualities of a SolutionQualities of a SolutionEssential nature of the problemEssential nature of the problem– Building conceptual structures is essentially difficultBuilding conceptual structures is essentially difficult– There are inherent control problems in building large, There are inherent control problems in building large,

complex systems from hand-crafted parts complex systems from hand-crafted parts – Key problems are outside the traditional life cycle scopeKey problems are outside the traditional life cycle scope=> => Cannot solve the problem within traditional life-cycle viewCannot solve the problem within traditional life-cycle view

Implied nature of the solution (== paradigm change)Implied nature of the solution (== paradigm change)– Change Change whatwhat we build ( we build (productproduct) – develop software assets ) – develop software assets

to systematically reuse conceptual structuresto systematically reuse conceptual structures– Change Change howhow we build ( we build (processprocess) )

Use an industrial model of developmentUse an industrial model of developmentExpand the development scope to encompass the whole Expand the development scope to encompass the whole problemproblem

– Change Change whowho does what ( does what (organizationorganization) – organize to build ) – organize to build families of products and processesfamilies of products and processes

2020

I. Change to Build Reusable Assets (what)I. Change to Build Reusable Assets (what)

Observation: sequential development is inefficientObservation: sequential development is inefficient– Much of software “development” is really re-development.Much of software “development” is really re-development.

Software inevitably exists in many versionsSoftware inevitably exists in many versionsSeldom develop truly new applicationsSeldom develop truly new applicationsSeldom start from scratchSeldom start from scratch

– Implication: typically much in common among systems we Implication: typically much in common among systems we build build … but very little is reused!… but very little is reused!

Real progress depends on reusing (rather than re-Real progress depends on reusing (rather than re-creating) conceptual structurescreating) conceptual structures– Exploit commonality to develop reusable assets for families Exploit commonality to develop reusable assets for families

of systemsof systems– Develop infrastructure to support rapid production of Develop infrastructure to support rapid production of

instances of the family from reusable assetsinstances of the family from reusable assets– Avoid the essential difficulties of creating conceptual Avoid the essential difficulties of creating conceptual

structures by reusing themstructures by reusing them

2121

Systematic ReuseSystematic Reuse

Distinguish “systematic” and “opportunistic” Distinguish “systematic” and “opportunistic” reusereuseHigh-leverage reuse depends on High-leverage reuse depends on reusing the reusing the contextcontext– Effort in tailoring near-fit components loses the benefitEffort in tailoring near-fit components loses the benefit– Verbatim reuse requires reusing the contextVerbatim reuse requires reusing the context

I.e., entire structure must be reused, not just componentsI.e., entire structure must be reused, not just components

ImplicationsImplications– Products must be built to be reusedProducts must be built to be reused– Must build reusable components in the context of a Must build reusable components in the context of a

reusable infrastructurereusable infrastructure– Sacrifice some customizability for efficiencySacrifice some customizability for efficiency

2222

II. Exploit Industrial Development ModelII. Exploit Industrial Development Model

Build Products

Factory(Means of Production)

Product

Develop Means of Production Design Manufacturing process

Design Factory Build Factory

Build Product(by hand)

Product

Pre-Industrial Post-Industrial

Historical note: Eli Whitney’s vision

and invention transformed the

product development

process.

2323

Example: Product-Line DevelopmentExample: Product-Line Development

A product-line A product-line development strategy development strategy sets out to sets out to maximize maximize reusereuse over a family of over a family of similar software systems.similar software systems.– Reuse is planned and Reuse is planned and

systematically executedsystematically executed– Common assets are Common assets are

developed as a distinct developed as a distinct productproduct

– The process focuses on The process focuses on developing means of developing means of production before any production before any particular productparticular product

Build Applications(Family Members)

Application EngineeringEnvironment

(Means of Production)

Application

Develop Means of Production Design development process

Design production environment Build reusable assets

A Product-Line Process

2424

Product-Line Development Over TimeProduct-Line Development Over Time

Adapt Assets

PL Requirements

Reusable Assets& Generators

Time

… a result of “strategic” software engineering

PL Design

Create Product

Product

Adapt Assets

Create Product

Product

Adapt Assets

Create Product

Product

2525

III. Extend the Development ScopeIII. Extend the Development Scope

Shift from “tactical” to “strategic” view of developmentShift from “tactical” to “strategic” view of development““Tactical” development maximizes local objectivesTactical” development maximizes local objectives– Focuses on a particular productFocuses on a particular product– Key SE issue: how to make tradeoffs to achieve Key SE issue: how to make tradeoffs to achieve product-product-

relevant goalsrelevant goals (improved quality, reduced cost,) (improved quality, reduced cost,)– Concerns outside the project are ignoredConcerns outside the project are ignored

““Strategic” development focuses on maximizing global Strategic” development focuses on maximizing global concernsconcerns– Focuses on overall organizational objectivesFocuses on overall organizational objectives– Key issue: how to make trade-offs to Key issue: how to make trade-offs to maximize overall returnmaximize overall return

May trade-off near-term goals to maximize overall returnMay trade-off near-term goals to maximize overall returnE.g., pay upfront to develop reusable architecture E.g., pay upfront to develop reusable architecture

– Project is viewed as only one part of the businessProject is viewed as only one part of the business

2626

Strategic Development ContextStrategic Development Context

Organization Mission Level (Business Goals)

Project Level (Product Line Goals)

Project Workflow o Project mission objectives o Productivity over project lifetime - Multiple development iterations - Maintenance, reuse

- Code developed to realize higher-level objectives (flow down)- Decision implications flow up

Organization Workflow o Organizational mission objectives o Productivity software investment - Multiple projects over time - Multiple platforms over time

Code Development Level

2727

Form Follows FunctionForm Follows FunctionThe structure of the organization tends to the The structure of the organization tends to the structure of the product it producesstructure of the product it producesProduct-line development differs in:Product-line development differs in:– Work products producedWork products produced

Means of production as primary productMeans of production as primary productProcesses as productsProcesses as products

– ProcessProcessDistinct phases: domain engineering and application Distinct phases: domain engineering and application engineering activitiesengineering activitiesDifferent methods, success criteriaDifferent methods, success criteria

– Integration of functional areas of an organizationIntegration of functional areas of an organizationClose links between business & technical - must understand Close links between business & technical - must understand business implications of technical decisions and visa versabusiness implications of technical decisions and visa versa

Both organizational structure and relationships must Both organizational structure and relationships must change to address product line developmentchange to address product line development

2828

Components of Strategic Components of Strategic DevelopmentDevelopment

ProcessProcess– Encompasses multiple product developments Encompasses multiple product developments – Reorganized to evolve software product familiesReorganized to evolve software product families

ProductsProducts– Production environment as productProduction environment as product– Processes as product (concurrent engineering)Processes as product (concurrent engineering)

OrganizationOrganization– Link strategic business goals to technical decisionsLink strategic business goals to technical decisions

Management, marketing, customer come into the loopManagement, marketing, customer come into the loop

– Organizational support for strategic asset baseOrganizational support for strategic asset baseSupport common assets independent of any deliverableSupport common assets independent of any deliverableProvide budget, schedule, authority, ownershipProvide budget, schedule, authority, ownership

– Capital investment in means of productionCapital investment in means of production

2929

The Critical RelationshipsThe Critical Relationships

PRODUCTS(what)

PROCESS(how)

ORGANIZATION(who)

• A three-legged stool • Each necessarily reflects the others• Each necessarily drives (or blocks) the others

3030

Déjà vu all over again…Déjà vu all over again…

3131

Existing Technology SufficesExisting Technology Suffices

Can be done with existing technology but Can be done with existing technology but requires requires – Deep understanding of SE principlesDeep understanding of SE principles– Thinking differently about SE problemsThinking differently about SE problems

Logical extension of a common set of SE Logical extension of a common set of SE principles (largely due to Parnas)principles (largely due to Parnas)– design for ease of changedesign for ease of change– information hidinginformation hiding– abstractionabstraction– developing programs as families of systems developing programs as families of systems – separation of concernsseparation of concerns

What differs is how they are appliedWhat differs is how they are applied

3232

Meta-Software EngineeringMeta-Software Engineering

Consider a development process “strategic” if it Consider a development process “strategic” if it considers and invests in work products beyond the considers and invests in work products beyond the scope of the current developmentscope of the current development– Focus on intellectual and managerial control over multiple Focus on intellectual and managerial control over multiple

developments over timedevelopments over time– Product-line software development can be viewed as part of this Product-line software development can be viewed as part of this

familyfamily

Applied “meta-Software Engineering: processes, Applied “meta-Software Engineering: processes, methods, and tools supportingmethods, and tools supporting– Developing products to produce development products Developing products to produce development products

(documentation, code, V&V infrastructure, etc.(documentation, code, V&V infrastructure, etc.– Processes for producing processes (processes as products)Processes for producing processes (processes as products)– Tools for creating domain specific tools …Tools for creating domain specific tools …

3333

An SSD Case Study:An SSD Case Study:Celsius Tech Product Line Celsius Tech Product Line

3434

CelsiusTechCelsiusTechLeading Swedish supplier of command and Leading Swedish supplier of command and control systems (now Saab Tech)control systems (now Saab Tech)Develop SS2000 naval systemsDevelop SS2000 naval systems– Integrated ship-board systemIntegrated ship-board system– Command, communications, weapons controlCommand, communications, weapons control– Variety of customers kinds of ships Variety of customers kinds of ships

Swedish, Danish, Finnish, Australian, Pakistani, Omani Swedish, Danish, Finnish, Australian, Pakistani, Omani NaviesNaviesSubmarines, frigates, patrol vesselsSubmarines, frigates, patrol vessels

Systems vary in size, function, and hardwareSystems vary in size, function, and hardware– Different kinds of vessels, computers, weaponsDifferent kinds of vessels, computers, weapons– Different languages, control systemsDifferent languages, control systems– Common underlying purposeCommon underlying purpose

3535

Development ProblemDevelopment Problem

Conventional, sequential development model having Conventional, sequential development model having difficultiesdifficulties– Failing on increasingly complex systemsFailing on increasingly complex systems– Increasing cost overruns, and schedule slippageIncreasing cost overruns, and schedule slippage

Defining event:simultaneously awarded two major Defining event:simultaneously awarded two major contractscontracts– Swedish and Danish naviesSwedish and Danish navies– Even larger systemEven larger system

Clear that they could not satisfy both contracts with Clear that they could not satisfy both contracts with current methodscurrent methods– Had neither technology nor personnelHad neither technology nor personnel– Budget overruns on fixed-price contracts could sink company Budget overruns on fixed-price contracts could sink company

from excess successfrom excess success

3636

Product-Line ApproachProduct-Line Approach

Could build both systems only if they could Could build both systems only if they could use much in commonuse much in commonGenesis of new business strategyGenesis of new business strategy– Develop systems as members of a familyDevelop systems as members of a family– Technical strategy: create flexible and robust Technical strategy: create flexible and robust

set of building blocks from which new systems set of building blocks from which new systems could be createdcould be created

– Building blocks become core company asset Building blocks become core company asset for rapid, efficient future system developmentfor rapid, efficient future system development

3737

Technical ApproachTechnical Approach

Analyzed what remained the same and Analyzed what remained the same and what differed between different systems what differed between different systems (e.g., for different navies)(e.g., for different navies)

Defined a common architecture for Defined a common architecture for SS2000 product lineSS2000 product line– Organized by increasing likelihood of changeOrganized by increasing likelihood of change– Application specific layer is parameterized, Application specific layer is parameterized,

designed for adaptabilitydesigned for adaptability

3838

Organizational ApproachOrganizational Approach

Reorganized to develop and maintain common Reorganized to develop and maintain common architecturearchitecture

Small team developed initial, common architectureSmall team developed initial, common architecture

Central task is no longer to develop systems but to Central task is no longer to develop systems but to maintain the product line itselfmaintain the product line itself

““Externally, Celsius Tech builds ship systems. Internally, they Externally, Celsius Tech builds ship systems. Internally, they evolve and grow a common asset base that provides the capability evolve and grow a common asset base that provides the capability to turn out ships systems. This mentality [] might sound subtle, but it to turn out ships systems. This mentality [] might sound subtle, but it manifests itself in the configuration control policies, the organization manifests itself in the configuration control policies, the organization

of the enterprise, and the way that new products are marketed.”of the enterprise, and the way that new products are marketed.”- - Software Architecture in PracticeSoftware Architecture in Practice, by Len Bass, Paul Clements, and Rick Kazman, by Len Bass, Paul Clements, and Rick Kazman

3939

ResultsResultsShrinking schedulesShrinking schedules– Overall production time dropped steadily (roughly %60)Overall production time dropped steadily (roughly %60)

– Recent releases were predictably on scheduleRecent releases were predictably on schedule

Code reuseCode reuse– Average of 70-80% of the system code reused verbatimAverage of 70-80% of the system code reused verbatim

Reduced staffingReduced staffing– Built the two systems with roughly same staff as for single projectBuilt the two systems with roughly same staff as for single project

Expanding product lineExpanding product line– Used common architecture to move quickly into related business Used common architecture to move quickly into related business

areasareas

– E.g., Developed new air defense system for Swedish Air Force by E.g., Developed new air defense system for Swedish Air Force by noting that a ground station is a ship whose location doesn’t change noting that a ground station is a ship whose location doesn’t change very often and whose pitch and roll are constantly zero.very often and whose pitch and roll are constantly zero.

4040

Shrinking SchedulesShrinking Schedules

From “A Case Study in Successful Product Line Development, SEI Report SEI-96-TR-016, Clements & Brownsword

Figure: Change in end-to-end system development schedules over time

4141

Increasing ReuseIncreasing Reuse

From “A Case Study in Successful Product Line Development, SEI Report SEI-96-TR-016, Clements & Brownsword

4242

Reduced Staffing NeedsReduced Staffing Needs

From “A Case Study in Successful Product Line Development, SEI Report SEI-96-TR-016, Clements & Brownsword

4343

Why these results?Why these results?ProcessProcess– Change focus to developing core assets (means of Change focus to developing core assets (means of

production)production)– Broaden to include strategic business goals.Broaden to include strategic business goals.

ProductsProducts– New products New products

Adaptable componentsAdaptable componentsProduction infrastructure (compilers, tools)Production infrastructure (compilers, tools)Processes as products!Processes as products!

– New criteria of goodnessNew criteria of goodness

OrganizationOrganization– Must organize according to products and processMust organize according to products and process– Organization must provide the essential inputsOrganization must provide the essential inputs– Organization must support asset ownershipOrganization must support asset ownership– Organization must work together to support strategic goalsOrganization must work together to support strategic goals

4444

SummarySummary

Productivity improvement opportunities largely Productivity improvement opportunities largely outside the scope of current paradigmoutside the scope of current paradigm

Thinking strategically results in fundamentally Thinking strategically results in fundamentally different decisions about development goals, different decisions about development goals, methods, and resourcesmethods, and resources– Focus on infrastructure and assents rather than code Focus on infrastructure and assents rather than code

deliverablesdeliverables– Industrial rather than craftsman process modelIndustrial rather than craftsman process model– Organized around strategic goals (capital investment Organized around strategic goals (capital investment

rather than cost model for software)rather than cost model for software)

4545

Implied ChangesImplied Changes

Relevant Research AreasRelevant Research Areas– Methods, tools, and processes for systematically Methods, tools, and processes for systematically

developing processes as productsdeveloping processes as products– Domain-specific architectures, languages, toolsDomain-specific architectures, languages, tools– Model-based verification, validation for familiesModel-based verification, validation for families– Software product linesSoftware product lines

TeachingTeaching– Re-emphasis on fundamental SE principles Re-emphasis on fundamental SE principles

(information hiding, abstraction, modularization, etc. – (information hiding, abstraction, modularization, etc. – not heuristics!)not heuristics!)

– More realistic case studies encompassing long-term More realistic case studies encompassing long-term development contextdevelopment context

– Understanding of waterfall as degenerate caseUnderstanding of waterfall as degenerate case

4646

EndEnd

Questions?Questions?