Overview of Software EngineeringCS 330

Spring 2007

Key Ingredients in successful organizations PeopleTechnologyProcess

A better viewProcess and Technology supporting peopleProcessesTechnologyPeople

What is software?Computer programs and associated documentation

Software products may be developed for a particular customer or may be developed for a general marketSoftware products may beGeneric/COTS - developed to be sold to a range of different customersCustom- developed for a customer according to their specification

EngineeringEngineering is The application of scientific principles and methods to the construction of useful structures & machinesExamplesMechanical engineeringComputer engineeringCivil engineeringChemical engineeringElectrical engineeringNuclear engineeringAeronautical engineering

Software EngineeringThe term is 35 years old: NATO ConferencesGarmisch, Germany, October 7-11, 1968Rome, Italy, October 27-31, 1969The reality is it is finally beginning to arriveComputer science one the scientific basisYears of studies/experience/statistics provide basis tooMany aspects have been made systematicMethods/methodologies/techniquesLanguagesToolsProcesses

Why Engineer Software ?The problem is complexityMany sources, but size is a key:Mozilla contains 3 Million lines of codeUNIX contains 4 million lines of codeWindows 2000 contains 108 lines of codeSecond is role and combinatorics of stateThird is uncertainty of inputs and their timingFourth is the continuing changing environment and demands. Software engineering is about managing all the sources of complexity to produce effective software.

Software Engineering in a NutshellDevelopment of software systems whose size/complexity warrants team(s) of engineersmulti-person construction of multi-version software [Parnas1987]Scopestudy of software process, development/management principles, techniques, tools and notationsGoalproduction of quality software, delivered on time, within budget, satisfying customers requirements and users needs

What does a software engineer do?

Software engineers should adopt a systematic and organised approach to all aspects of software development.use appropriate tools and techniques depending on the problem to be solved, the development constraints and the resources availableUnderstand and communicate processes for improved software development within their organizationBe effective team members and/or leaders.Can be very technical or more managerial depending on organizational need.

What is the difference between software engineering and computer science?Computer ScienceSoftware Engineeringis concerned withComputer science theories are currently insufficient to act as a complete underpinning for software engineering, BUT it is a foundation for practical aspects of software engineering

What is the difference between software engineering and system engineering?Software engineering is part of System engineeringSystem engineering is concerned with all aspects of computer-based systems development including hardware, software and process engineeringSystem engineers are involved in system specification, architectural design, integration and deployment

Difficulties?SE is a unique brand of engineeringSoftware is malleableSoftware construction is human-intensiveSoftware is intangible and generally invisibleSoftware problems are unprecedentedly complexSoftware directly depends upon the hardwareIt is at the top of the system engineering food chainSoftware solutions require unusual rigorSoftware state means behaviors can depend on history. Software has discontinuous operational nature

Software Engineering Software ProgrammingSoftware programmingSingle developerToy applicationsShort lifespanSingle or few stakeholdersArchitect = Developer = Manager = Tester = Customer = UserOne-of-a-kind systemsBuilt from scratchMinimal maintenance

Software Engineering Software ProgrammingSoftware engineeringTeams of developers with multiple rolesComplex systemsIndefinite lifespanNumerous stakeholdersArchitect Developer Manager Tester Customer UserSystem familiesReuse to amortize costsMaintenance accounts for 60%-80% of overall development costs

Economic and Management Aspects of SESoftware Engineering is about improved ROI (can be Capital and/or Social ROI)Software production = development + maintenanceMaintenance costs 60%-80% of all (successful) development costs20% corrective (12%-16% total costs)30% adaptive (18%-24% total costs)50% perfective (30-40% total costs)Quicker development is not always preferablehigher up-front costs may defray downstream costspoorly designed/implemented software is a critical cost factor in system cost and delays

Relative Costs of Fixing Software Faults

Mythical Man-Monthby Fred BrooksPublished in 1975, republished in 1995Experience managing development of OS/360 in 1964-65Central argumentLarge projects suffer management problems different in kind than small ones, due to division in laborCritical need is the preservation of the conceptual integrity of the product itselfCentral conclusionsConceptual integrity achieved through chief architectImplementation achieved through well-managed effortsoftware developers are not interchangeable work units.Brooks LawAdding personnel to a late project makes it later

Software Engineering:From Principles to Tools

Software QualitiesQualities are goals in the practice of software engineering, and directly relate to many of the guiding principles.

External vs. Internal qualitiesProduct vs. Process qualities

Software QualitiesCritical Quality AttributesCorrectnessMaintainabilityDependabilityUsabilityReliabilityOther Attributes Completeness Compatibility Portability InternationalizationUnderstandability Scalability Robustness Testability Reusability Customizability Efficiency

External vs. Internal QualitiesExternal qualities are visible to the userreliability, usability, efficiency (maybe), robustness, scalability

Internal qualities are the concern of developersthey help developers achieve external qualitiesverifiability, maintainability, extensibility, evolvability, adaptability, portability, testability, reusability

Product vs. Process QualitiesProduct qualities concern the developed artifactsmaintainability, performance, understandability, Process qualities deal with the development activityproducts are developed through processmaintainability, productivity, predictability

Some Software QualitiesCorrectnessideal qualityestablished w.r.t. the requirements/specificationabsoluteReliabilitystatistical propertyprobability that software will operate as expected over a given period of time/inputsrelative

Some Software Qualities (cont.)Robustnessreasonable behavior in unforeseen circumstancessubjectivea specified requirement is an issue of correctness; an unspecified requirement is an issue of robustnessUsabilityability of end-users to easily use software extremely subjective

Some Software Qualities (cont.)Understandabilityability of developers to easily understand produced artifactsinternal product qualitysubjectiveVerifiabilityease of establishing desired properties performed by formal analysis or testinginternal quality

Some Software Qualities (cont.)Performanceequated with efficiencyassessable by measurement, analysis, and simulationEvolvabilityability to add or modify functionalityaddresses adaptive and perfective maintenanceproblem: evolution of implementation is too easyevolution should start at requirements or design

Some Software Qualities (cont.)Reusabilityability to construct new software from existing piecesmust be planned foroccurs at all levels: from people to process, from requirements to codeInteroperabilityability of software (sub)systems to cooperate with otherseasily integratable into larger systemscommon techniques include APIs, distributed programming interfaces (CORBA, DCOM), plug-in protocols, etc.

Some Software Qualities (cont.)Scalabilityability of a software system to grow in size while maintaining its properties and qualitiesassumes maintainability and evolvabilitygoal of component-based development

Process PrinciplesPrescribes all major activitiesUses resources, within a set of constraints, to produce intermediate and final productsMay be composed of sub-processesEach activity has entry and exit criteriaActivities are organized in a sequenceHas a set of guiding principles to explain goalsConstraints may apply to activity, resource or product

Software Development StagesRequirements Analysis & Specification Conceptual/System/Architectural Design Detailed/Program Design Implementation/Coding Unit & Integration Testing System Testing/Validation System Delivery/Deployment MaintenanceNote there are many variations on the names. You are responsible for the main categories above (an on the next pages)..

Software Lifecycle ModelsWaterfall ModelV ModelPhased Development ModelIncremental ModelPrototyping ModelSpiral Model

Software Development LifecycleWaterfall Model Requirements Design Implementation Integration Validation Deployment

V Model[Pfleeger 98]

Phased Development Model[Pfleeger 98]

Software Development Lifecycle Incremental ModelVersion 1:Complete General DesignVersion 2:Design/Implement first set of planned new features.Note overlap with V1 scheduleVersion 3:Design/Implement second set of planned new features

Prototyping Model[Pressman 97]Listen to CustomerCustomerTest-drivesMock-upBuild/ReviseMock-Up

Prototyping Model[Pfleeger 98]

Spiral developmentProcess is represented as a spiral rather than as a sequence of activities with backtracking.Each loop in the spiral represents a phase in the process. No fixed phases such as specification or design - loops in the spiral are chosen depending on what is required.Risks are explicitly assessed and resolved throughout the process.

Spiral model of the software process

Spiral model sectorsObjective settingSpecific objectives for the phase are identified.Risk assessment and reductionRisks are assessed and activities put in place to reduce the key risks.Development and validationA development model for the system is chosen which can be any of the generic models.PlanningThe project is reviewed and the next phase of the spiral is planned.

Evolutionary developmentExploratory development Objective is to work with customers and to evolve a final system from an initial outline specification. Should start with well-understood requirements and add new features as proposed by the customer.Throw-away prototypingObjective is to understand the system requirements. Should start with poorly understood requirements to clarify what is really needed.

Evolutionary development

Evolutionary developmentProblemsLack of process visibility;Systems are often poorly structured;Special skills (e.g. in languages for rapid prototyping) may be required.ApplicabilityFor small or medium-size interactive systems;For parts of large systems (e.g. the user interface);For short-lifetime systems.

Component-based software engineeringBased on systematic reuse where systems are integrated from existing components or COTS (Commercial-off-the-shelf) systems.Process stagesComponent analysis;Requirements modification;System design with reuse;Development and integration.This approach is becoming increasingly used as component standards have emerged.

Reuse-oriented development

Component-Based DevelopmentDevelop generally applicable components of a reasonable size and reuse them across systemsMake sure they are adaptable to varying contextsExtend the idea beyond code to other development artifactsQuestion: what comes first?Integration, then deploymentDeployment, then integration

Different Flavors of ComponentsThird-party software piecesPlug-ins / add-insAppletsFrameworksOpen SystemsDistributed object infrastructuresCompound documentsLegacy systems

Process iterationSystem requirements ALWAYS evolve in the course of a project so process iteration where earlier stages are reworked is always part of the process for large systems.Iteration can be applied to any of the generic process models.Two (related) approachesIncremental delivery;Spiral development.

Incremental deliveryRather than deliver the system as a single delivery, the development and delivery is broken down into increments with each increment delivering part of the required functionality.User requirements are prioritised and the highest priority requirements are included in early increments.Once the development of an increment is started, the requirements are frozen though requirements for later increments can continue to evolve.

Incremental development

Incremental development advantagesCustomer value can be delivered with each increment so system functionality is available earlier.Early increments act as a prototype to help elicit requirements for later increments.Lower risk of overall project failure.The highest priority system services tend to receive the most testing.

Extreme programmingAn approach to development based on the development and delivery of very small increments of functionality.Relies on constant code improvement, user involvement in the development team and pairwise programming.Covered in Chapter 17

Software Development LifecycleWaterfall Model Requirements Design Implementation Integration Validation Deployment

Software specificationThe process of establishing what services are required and the constraints on the systems operation and development.Requirements engineering processFeasibility study;Requirements elicitation and analysis;Requirements specification;Requirements validation.

RequirementsProblem Definition Requirements/Specificationdetermine exactly what the customer and user need (maybe want)Requirements develop a contract with the customerSpecification say what the software product is to doDifficultiesclient is computer/software illiterate (no idea what is doable)client asks for wrong product (want vs need)client is computer/software literate (specifies solution not need)specifications are ambiguous, inconsistent, incompleteStudies have shown that the percentage of defects originating during requirements engineering is estimated at more than 50 percent. The total percentage of project budget due to requirements defects is 25 to 40 percent.

The requirements engineering process

Software design and implementationThe process of converting the system specification into an executable system.Software designDesign a software structure that realises the specification;ImplementationTranslate this structure into an executable program;The activities of design and implementation are closely related and may be inter-leaved.

Design process activitiesArchitectural designAbstract specificationInterface designComponent designData structure designAlgorithm design

The software design process

Structured methodsSystematic approaches to developing a software design.The design is usually documented as a set of graphical models.Possible modelsObject model;Sequence model;State transition model;Structural model;Data-flow model.

Architecture vs. Design[Perry & Wolf 1992]Architecture is concerned with the selection of architectural elements, their interactions, and the constraints on those elements and their interactions necessary to provide a framework in which to satisfy the requirements and serve as a basis for the design.

Design is concerned with the modularization and detailed interfaces of the design elements, their algorithms and procedures, and the data types needed to support the architecture and to satisfy the requirements.

Architecture/DesignRequirements/Specification Architecture/Designarchitecture: decompose software into modules/objects/components with interfacesdesign: develop module/object/component specifications (algorithms, data types) and communication detailsmaintain a record of design decisions and traceabilityspecifies how the software product is to do its tasksDifficultiesmiscommunication between module designersdesign may be inconsistent, incomplete, ambiguousHow to achieve a requirement may be unknown

Planning/SchedulingBefore undertaking cost of development, need to estimate the costs/sizes of various stepsEstimate Code sizeEstimate tools neededEstimate personnelOften Done after Architecture and before rest of design, but revised again after full design. Develop schedule for aspects of project lifecycleIf doing predictive/quantitative SE, build on past experience, considering how to improve process.

Implementation & IntegrationDesign Implementationimplement modules; verify that they meet their specificationscombine modules according to the designspecifies how the software design is realizedDifficultiesmodule interaction errorsorder of integration may influence quality and productivity

Programming and debuggingTranslating a design into a program and removing errors from that program.Programming is a personal activity - there is no generic programming process.Programmers carry out some program testing to discover faults in the program and remove these faults in the debugging process.

The debugging process

Software validationVerification and validation (V & V) is intended to show that a system conforms to its specification and meets the requirements of the system customer.Involves checking and review processes and system testing.System testing involves executing the system with test cases that are derived from the specification of the real data to be processed by the system.

Verification and ValidationAnalysisStaticScienceFormal verificationInformal reviews and walkthroughsTestingDynamicEngineeringWhite box vs. black boxStructural vs. behavioralIssues of test adequacy

The testing process

Testing stagesComponent or unit testingIndividual components are tested independently; Components may be functions or objects or coherent groupings of these entities.System testingTesting of the system as a whole. Testing of emergent properties is particularly important.Acceptance testingTesting with customer data to check that the system meets the customers needs.

Testing phases

Quality AssuranceDone as part of each stepReduce costs by catching errors early. Help determine ambiguities/inconsistenciesHelp ensure quality product.

DeploymentCompleted End-User DocumentationSeparate from Developer documentationInstallation Process(es)Customer test proceduresSupport Processes (help desk, etc)Trouble TrackingRepair/rework to address bugsRegression testing (as bugs are fixed)

Maintenance & EvolutionOperation Changemaintain software during/after user operationdetermine whether the product still functions correctlyDifficultiesRigid or fragile designslack of documentationpersonnel turnover

Software evolutionSoftware is inherently flexible and can change. As requirements change through changing business circumstances, the software that supports the business must also evolve and change.Although there has been a demarcation between development and evolution (maintenance) this is increasingly irrelevant as fewer and fewer systems are completely new.

System evolution

Why I include CASE ToolsComputer Aides Software Engineering tools support good SE processes (e.g. UML)Some tools absolute requirement for scaling e.g. build and configuration management.Integrated CASE (ICASE) tools embody good processes and improve productivity (E.g. Rational tool set)Some tools (e.g. debuggers, Purify) do almost impossible for humans. But.. Tools changeNo SE tools from my first 3 jobs exist (except Fortran/C languages)I use regularly use 3 SE tools from my next set of jobs.Other tools I learned have been replaced with similar but expanded concepts.. Understanding today;s tools gives a basis for learning future ones.

ICASE Design ToolsRational Rose and Rational Unified Development.From UML drawing to code and back.Generates stubs and eventually testing code. Supports multiple languages

public class Car { public Driver theDriver; /** * @roseuid 3EAFF17E035B */ public Car() { }}

public class Driver { /** * @roseuid 3EAFF53F02FD */ public Driver() { }}

Associations are implemented as reference attributes.No explicit role name defined so, Rose adds automatically a role name to the code: theDriver

Templates for the default constructors are provided.(Similar for methods/members when given in the class diagram.)

Marc Conrad - University of Luton

Configuration ManagementCM is a discipline whose goal is to control changes to large software through the functions ofComponent identificationChange trackingVersion selection and baseliningManaging simultaneous updates (team work)Build processes with automated regression testingSoftware manufacture

CM in Action1.0

Build ToolsNecessary for large projects. Keep track of what depends upon on what, and what needs recompiled or regenerated when things change. Important even for small 1-person projects as soon as you have multiple files. Can do much more than just compile, can generate document (if using code-based docs), generate manufactured code (e.g. SOAP interfaces), even send emails or suggest alternatives.E.g. in our IUE project, edit some files compile was one in seconds, edit another and a rebuild taking days would be needed. If more than 30 files impacted, our make process recommend a new branch to avoid conflicts!

Debugging ToolsHow do you see what the code is really doing (not what it seems it should do)? How to you see what happened to code during compiler optimization?How do you find/track down the cause of Segfault/GFP in code youve never seen before?How can you test various possibilities without generating special code or recompiling.How do you track down a memory leak?

Tools, workbenches, environments

The Rational Unified ProcessA modern process model derived from the work on the UML and associated process.Normally described from 3 perspectivesA dynamic perspective that shows phases over time;A static perspective that shows process activities;A practive perspective that suggests good practice.

RUP phase model

RUP phasesInceptionEstablish the business case for the system.ElaborationDevelop an understanding of the problem domain and the system architecture.ConstructionSystem design, programming and testing.TransitionDeploy the system in its operating environment.

RUP good practiceDevelop software iterativelyManage requirementsUse component-based architecturesVisually model softwareVerify software qualityControl changes to software

Static workflows

Workflow

Description

Business modelling

The business processes are modelled using business use cases.

Requirements

Actors who interact with the system are identified and use cases are developed to model the system requirements.

Analysis and design

A design model is created and documented using architectural models, component models, object models and sequence models.

Implementation

The components in the system are implemented and structured into implementation sub-systems. Automatic code generation from design models helps accelerate this process.

Test

Testing is an iterative process that is carried out in conjunction with implementation. System testing follows the completion of the implementation.

Deployment

A product release is created, distributed to users and installed in their workplace.

Configuration and change management

This supporting workflow managed changes to the system (see Chapter 29).

Project management

This supporting workflow manages the system development (see Chapter 5).

Environment

This workflow is concerned with making appropriate software tools available to the software development team.

Computer-aided software engineeringComputer-aided software engineering (CASE) is software to support software development and evolution processes.Activity automationGraphical editors for system model development;Data dictionary to manage design entities;Graphical UI builder for user interface construction;Debuggers to support program fault finding;Automated translators to generate new versions of a program.

Case technologyCase technology has led to significant improvements in the software process. However, these are not the order of magnitude improvements that were once predictedSoftware engineering requires creative thought - this is not readily automated;Software engineering is a team activity and, for large projects, much time is spent in team interactions. CASE technology does not really support these.

CASE classificationClassification helps us understand the different types of CASE tools and their support for process activities.Functional perspectiveTools are classified according to their specific function.Process perspectiveTools are classified according to process activities that are supported.Integration perspectiveTools are classified according to their organisation into integrated units.

Functional tool classification

Tool type

Examples

Planning tools

PERT tools, estimation tools, spreadsheets

Editing tools

Text editors, diagram editors, word processors

Change management tools

Requirements traceability tools, change control systems

Configuration management tools

Version management systems, system building tools

Prototyping tools

Very high-level languages, user interface generators

Method-support tools

Design editors, data dictionaries, code generators

Language-processing tools

Compilers, interpreters

Program analysis tools

Cross reference generators, static analysers, dynamic analysers

Testing tools

Test data generators, file comparators

Debugging tools

Interactive debugging systems

Documentation tools

Page layout programs, image editors

Re-engineering tools

Cross-reference systems, program re-structuring systems

Activity-based tool classification

CASE integrationToolsSupport individual process tasks such as design consistency checking, text editing, etc.WorkbenchesSupport a process phase such as specification or design, Normally include a number of integrated tools.EnvironmentsSupport all or a substantial part of an entire software process. Normally include several integrated workbenches.

Boults view of SESE must balance risks in software development process:Risks of error in requirements specification, design, implementation, and integrationRisks of exceeding available resourcesRisks of being late on delivery or missing the marketDont let push for formality dominate your process. Dont let push for expedience destroy your process.

Software Process QualitiesProcess is reliable if it consistently leads to high-quality productsProcess is robust if it can accommodate unanticipated changes in tools and environmentsProcess performance is productivityProcess is evolvable if it can accommodate new management and organizational techniquesProcess is reusable if it can be applied across projects and organizations

Assessing Software QualitiesQualities must be measurable/quantifiableMeasurement requires that qualities be precisely definedImprovement requires accurate and consistent measurementsFor most SD groups, qualities are informally defined and are difficult to assess

Software Engineering AxiomsAdding developers to a project will likely result in further delays and accumulated costsThe longer a fault exists in softwarethe more costly it is to detect and correctthe less likely it is to be properly correctedUp to 70% of all faults detected in large-scale software projects are introduced in requirements and designdetecting the causes of those faults early may reduce their resulting costs by a factor of 200 or moreBasic tension of software engineeringbetter, cheaper, faster pick any two!functionality, scalability, performance pick any two!Want/Need Managements buy in to formal SE process.If you dont document your process, you dont have one!

Boehms Spiral ModelPLANDEVELOP AND TESTDETERMINE GOALS,ALTERNATIVES,CONSTRAINTSEVALUATE ALTERNATIVESAND RISKSRequirements,life-cycle planBudget1Alternatives1Constraints1Risk analysis1Risk analysis2Risk analysis3Risk analysis4Constraints2Constraints3Constraints4Budget2Budget3Budget4Alternatives2Alternatives3Alternatives4Prototype1Proto-type2Proto-type3Proto-type4Concept ofoperationSoftwarerequirementsValidatedrequirementsDevelopmentplanIntegrationand test planSoftwaredesignValidated,verified designDetaileddesignCodeUnit testSystemtestAcceptancetestImplementationplanstart

Key pointsSoftware processes are the activities involved in producing and evolving a software system. Software process models are abstract representations of these processes.General activities are specification, design and implementation, validation and evolution.Generic process models describe the organisation of software processes. Examples include the waterfall model, evolutionary development and component-based software engineering.Iterative process models describe the software process as a cycle of activities.

Key pointsRequirements engineering is the process of developing a software specification.Design and implementation processes transform the specification to an executable program.Validation involves checking that the system meets to its specification and user needs.Evolution is concerned with modifying the system after it is in use.The Rational Unified Process is a generic process model that separates activities from phases.CASE technology supports software process activities.

Definition of Critical Attributes: Maintainability - possible to evolve software to meet changing needs Dependability - includes range of characteristics: reliability, security and safety; should not cause physical or economic damage in the event of system failure Efficiency - should not make wasteful use of system resources such as memory and process cycles Usability - should have appropriate user interface and adequate documentationOther Attributes:SafetySecurityReliabilityResilienceRobustnessUnderstandabilityTestabilityAdaptabilityModularityComplexityPortabilityReusabilityLearnabilityDefinition of Process: a series of steps involving activities, constraints, and resources that produce an intended output of some kind

Process Attributes: Understandability - To what extent is the process explicitly defined and how easy is it to understand? Visibility - Do the process activities culminate in clear results so that the progress of the process is externally visible? Supportability - To what extent can the process activities be supported by CASE tools? Acceptability - acceptable and usable by engineers? Reliability - Is the process designed in such a way that process errors are avoided or trapped before they result in product errors? Robustness - Can the process continue in spite of unexpected problems? Maintainability - Can the process evolve to reflect changing organizational requirements or identified process improvements? Rapidity - How fast can the process of delivering a system from given specification be completed?

Requirements analysis & specification Conceptual/System design Detailed/Program design Coding/Implementation Unit Testing & Integration System Testing System Delivery Maintenance[German Ministry of Defense 1992]Variation that shows how testing activities relate to analysis and design

Makes more explicit the iteration and rework that are hidden in waterfallIterative vs. Incremental DevelopmentIterative - full system at beginning, changes functionality with each releaseIncremental - subsystems are added & functionality builds with each release

ADVANTAGES: Early training on early release Create markets for non-existing functionality Bundle patches in with frequent releases Development team can focus on different areas with each releaseAll or part of a system constructed quickly to understand or clarify issuesOverall goal: reducing risk and uncertainty in developmentExplore alternative designs[Boehm 88]Viewed in light of risks involvedCombine development with risk MGT