Requirements and Architectural Design

Lecture 2

Requirements and Architectural DesignLASER Summer School on Software Engineering 2013, Elba Island, ItalyPere Mato/CERN

1

Monday, September 9, 13

What is the Scientific Software we need to develop?


The goal is to understand in the most general; that’s usually also the simplest.

- A. Eddington

We use experiments to inquire about what “reality” does.

Theory &Parameters

Reality

We need the software to fill

this gap

3


“Clear statement of how the world works formulated mathematically in term of equations”

Review of Particle PhysicsParticle Data Group,

Barnett et al.

4

Theory


“Clear statement of how the world works formulated mathematically in term of equations”

Additional term goes here

Review of Particle PhysicsParticle Data Group,

Barnett et al.

4

Theory


5

Experiment


0x01e84c10: 0x01e8 0x8848 0x01e8 0x83d8 0x6c73 0x6f72 0x7400 0x00000x01e84c20: 0x0000 0x0019 0x0000 0x0000 0x01e8 0x4d08 0x01e8 0x5b7c0x01e84c30: 0x01e8 0x87e8 0x01e8 0x8458 0x7061 0x636b 0x6167 0x65000x01e84c40: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84c50: 0x01e8 0x8788 0x01e8 0x8498 0x7072 0x6f63 0x0000 0x00000x01e84c60: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84c70: 0x01e8 0x8824 0x01e8 0x84d8 0x7265 0x6765 0x7870 0x00000x01e84c80: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84c90: 0x01e8 0x8838 0x01e8 0x8518 0x7265 0x6773 0x7562 0x00000x01e84ca0: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84cb0: 0x01e8 0x8818 0x01e8 0x8558 0x7265 0x6e61 0x6d65 0x00000x01e84cc0: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84cd0: 0x01e8 0x8798 0x01e8 0x8598 0x7265 0x7475 0x726e 0x00000x01e84ce0: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84cf0: 0x01e8 0x87ec 0x01e8 0x85d8 0x7363 0x616e 0x0000 0x00000x01e84d00: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d10: 0x01e8 0x87e8 0x01e8 0x8618 0x7365 0x7400 0x0000 0x00000x01e84d20: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d30: 0x01e8 0x87a8 0x01e8 0x8658 0x7370 0x6c69 0x7400 0x00000x01e84d40: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d50: 0x01e8 0x8854 0x01e8 0x8698 0x7374 0x7269 0x6e67 0x00000x01e84d60: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d70: 0x01e8 0x875c 0x01e8 0x86d8 0x7375 0x6273 0x7400 0x00000x01e84d80: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d90: 0x01e8 0x87c0 0x01e8 0x8718 0x7377 0x6974 0x6368 0x0000

5

Experiment


0x01e84c10: 0x01e8 0x8848 0x01e8 0x83d8 0x6c73 0x6f72 0x7400 0x00000x01e84c20: 0x0000 0x0019 0x0000 0x0000 0x01e8 0x4d08 0x01e8 0x5b7c0x01e84c30: 0x01e8 0x87e8 0x01e8 0x8458 0x7061 0x636b 0x6167 0x65000x01e84c40: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84c50: 0x01e8 0x8788 0x01e8 0x8498 0x7072 0x6f63 0x0000 0x00000x01e84c60: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84c70: 0x01e8 0x8824 0x01e8 0x84d8 0x7265 0x6765 0x7870 0x00000x01e84c80: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84c90: 0x01e8 0x8838 0x01e8 0x8518 0x7265 0x6773 0x7562 0x00000x01e84ca0: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84cb0: 0x01e8 0x8818 0x01e8 0x8558 0x7265 0x6e61 0x6d65 0x00000x01e84cc0: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84cd0: 0x01e8 0x8798 0x01e8 0x8598 0x7265 0x7475 0x726e 0x00000x01e84ce0: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84cf0: 0x01e8 0x87ec 0x01e8 0x85d8 0x7363 0x616e 0x0000 0x00000x01e84d00: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d10: 0x01e8 0x87e8 0x01e8 0x8618 0x7365 0x7400 0x0000 0x00000x01e84d20: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d30: 0x01e8 0x87a8 0x01e8 0x8658 0x7370 0x6c69 0x7400 0x00000x01e84d40: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d50: 0x01e8 0x8854 0x01e8 0x8698 0x7374 0x7269 0x6e67 0x00000x01e84d60: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d70: 0x01e8 0x875c 0x01e8 0x86d8 0x7375 0x6273 0x7400 0x00000x01e84d80: 0x0000 0x0019 0x0000 0x0000 0x0000 0x0000 0x01e8 0x5b7c0x01e84d90: 0x01e8 0x87c0 0x01e8 0x8718 0x7377 0x6974 0x6368 0x0000

1/5000th of an event in the CMS detector

◦ Get about 400 events per second

5

Experiment


“Address”: what detector element took the reading

“Value”: What the electronics wrote down

Look up type, calibration info

Look up/calculate spatial position

Check valid, convert to useful units/form

Draw

6

0x01e8 0x8848

What does the Data Mean?


7


Raw Data

Theory &Parameters

Reality

A small number of general equations, with specificinput parameters (perhaps poorly known)

The imperfect measurement of a (set of) interactions in the detectorVery strong selection

Observables Specific lifetimes, probabilities, masses,branching ratios, interactions, etc

EventsA unique happening:Run 21007, event 3916 which contains a H -> xx decayPhysical quantities: positions, momentum, energy, etc.

Reconstruction

Analysis

Phenomenology

8

Bridging the Gap


Calculate expected branching ratios

Randomly pick decay paths, lifetimes, etc for a number of events

Calculate what imperfect detector would have seen for those events

Treat that as real data and reconstruct it

Compare to original to understand efficiency

Raw Data

Theory &Parameters

Reality

Observables

Events

Monte Carlo simulation’s role


Data Flow and Processing Stages

10


11

Applications

Event DetDesc. Calib.

Experiment Framework

Simulation AnalysisDataMngmt.

Core Libraries

non-HEP specificsoftware packages

Applications are built on top of frameworks and implementing the required algorithms

Every experiment has a framework for basic services and various specialized frameworks: event model, detector description, visualization, persistency, interactivity, simulation, calibrarion, etc.

Specialized domains that are common among the experiments

Core libraries and services that are widely used and provide basic functionality

General purpose non-HEP libraries

LHC Software Stack


✤ Foundation Libraries✤ Basic types✤ Utility libraries✤ System isolation libraries

✤ Mathematical Libraries✤ Special functions✤ Minimization, Random Numbers

✤ Data Organization✤ Event Data✤ Event Metadata (Event collections)✤ Detector Conditions Data

✤ Data Management Tools✤ Object Persistency✤ Data Distribution and Replication

Simulation ToolkitsEvent generatorsDetector simulation

Statistical Analysis ToolsHistograms, N-tuplesFitting

Interactivity and User InterfacesGUIScriptingInteractive analysis

Data Visualization and GraphicsEvent and Geometry displays

Distributed ApplicationsParallel processingGrid computing

Software Components

12


What are the main Requirements and in which Environment?

13


✤ Large number of physicists and engineers participating actively in the data analysis and for extended period of time✤ Collaborations are formed by 1000s scientists and engineers from 100s

institutes✤ Widely distributed computing environment

✤ Instead of building a single (huge) computing center decided to federate about 200 of various sizes (Worldwide LHC Computing Grid)

✤ Huge quantity of data that has to be distributed and shared by all members of each experiment ✤ Each scientist should have access to the whole data from anywhere in the

world

LHC Computing Characteristics

14


✤ Must cope with the unprecedented conditions and challenges that characterizes these experiments (trigger rate, data volumes, etc.)

✤ In spite of its complexity it should be easy-to-use ✤ Each one of the ~ 4000 LHC physicists (including people from remote/

isolated countries, physicists who have built the detectors, software-old-fashioned senior physicists) should be able to run the software, modify part of it (reconstruction, ...), analyze the data, extract physics results

✤ Design should take into account the >15 years lifetime of the LHC✤ Resilient designs, technology choices will evolve over time

✤ The standard language for physics applications software in all four LHC experiments is C++✤ language choice may change in the future or multi-language could be

introduced

LHC Software Requirements

15


✤ Operate seamlessly in a distributed environment and also be functional in ‘disconnected’ local environments

✤ Favor software re-use✤ Use of existing software should be consistent with the architecture

✤ The software quality of the framework should be at least as good than the internal software of any of the sub-detectors

✤ Supporting multiple platforms✤ Software should be written in a portable manner and complying to the

language standards

LHC software Requirements (2)

16


} Small problems can be solved with simple techniques} For large problems you need

to use different techniquesthat are in general morecomplex and with a up front cost

Software Scale

17


✤ Can be built by one person

✤ Requires

✤ Minimal modeling

✤ Simple process

✤ Simple tools

✤ Little risk

Architecting a dog house

18


✤ Built most efficiently and timely by a team

✤ Requires

✤ Modeling

✤ Well-defined process

✤ Power tools

Architecting a house

19


✤ Built by many companies. Requires:✤ Modeling✤ Simple plans, evolving to blueprints✤ Scale models✤ Engineering plans✤ Well-defined process✤ Architectural team✤ Political planning✤ Infrastructure planning✤ Time-tabling and scheduling✤ Selling space✤ Heavy equipment

✤ Major risks

Architecting a high rise

20


✤ To make communication possible to you need to share a vocabulary✤ Standards for languages, design patterns, architecture, etc.

✤ To work together you need to control the integrity of source code ✤ Tools for code versioning (e.g. CVS, SubVersion, GIT)

✤ To build, test and release a large system can be difficult✤ Tools for creating releases (e.g. CMT, SCRAM, CMake), issue tracking systems

(e.g. Bugzila, Savannah, JIRA)✤ But individual effort is still important

✤ Good tools and methods can help to do a better job✤ License software and tools can be a hurdle

✤ Open source software make life easier for scientific worldwide collaborations

Tools for large projects

21


✤ “More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity”, William Wulf (AT&T Professor)

✤ Overall efficiency is what really matters✤ The cost of the improving the code (people are expensive) should be included

✤ Perceived performance is what really matters✤ Is the system getting the job done or not?

✤ Reminder: Performance assumes correctness✤ A fast program delivering [sometimes] wrong results is not helpful

Performance

22

Performance will be revisited later


✤ Put extra effort into building high quality components✤ Be more efficient by re-using these components✤ Many obstacles to overcome

✤ too broad functionality / lack of flexibility in components✤ organizational - reuse requires a broad overview to ensure unified approach

✤ we tend to split into domains each independently managed✤ cultural

✤ don’t trust others to deliver what we need✤ fear of dependency on others✤ fail to share information with others✤ developers fear loss of creativity

✤ Reuse is a management activity

Importance of Reuse

23

Reinventing the ‘wheel’ is very common in research environments


24

Application Domains✤ Event Processing Applications

✤ Trigger, Simulation, Reconstruction, Event Selection programs

✤ Data Analysis✤ Event/Detector display, data presentation

programs✤ Detector calibration

✤ Calibration and Alignment programs✤ Job configuration, submission, monitoring

and control✤ Grid/Cloud awareness

Mainly batch orientedInteractive for development &

debuggingReal-time

Mainly interactive

Batch and interactive

Mainly interactive


Software Design

25


✤ System Architecture✤ Component design✤ Class design

Software Design

26


✤ Capture major interfaces between subsystems and packages early✤ Be able to visualize and reason about the design in a common

notation✤ Common vocabulary, running scenarios

✤ Be able to break the work into smaller pieces that can be developed concurrently by different teams

✤ Acquire an understanding of non-functional constrains✤ Programming languages, concurrency, database, GUI, component re-use

Architectural Design

27


28

Architecture Defined✤ Definition of [software] architecture [1]

✤ Set or significant decisions about the organization of the software system✤ Selection of the structural elements and their interfaces which compose the

system✤ Their behavior -- collaboration among the structural elements✤ Composition of these structural and behavioral elements into progressively

larger subsystems✤ The architectural style that guides this organization

✤ The architecture is the blue-print (architecture description document)

[1] I. Jacobson, et al. “The Unified Software development Process”, Addison Wesley 1999


✤ Software architecture also involves✤ usage✤ functionality✤ performance✤ resilience✤ reuse✤ comprehensibility✤ economic and technology constraints and tradeoffs✤ aesthetic concerns

Architecture defined (continued)

29

Mary Shaw, Carnegie Mellon UniversityGrady Booch, Philippe Kruchten, Rich Reitman, Kurt Bittner, Rational


http://www.cmu.edu/

http://www.cmu.edu/

✤ A well designed architecture has certain qualities:✤ layered subsystems✤ low inter-subsystem coupling✤ robust, resilient and scalable✤ high degree of reusable

components✤ clear interfaces✤ driven by most important and

risky use cases✤ easy to understand

Architectural Design Qualities

30


Models✤ Models are the language of designer, in many disciplines✤ Models are representations of the system to-be-built or as-

built✤ Models are vehicle for communications with various

stakeholders✤ Visual models, blueprints✤ Scale models✤ Models allow reasoning about some characteristic of the

real system

31


✤ Architecture is many things to many different interested parties✤ end-user✤ customer✤ project manager✤ system engineer✤ developer✤ architect✤ maintainer✤ other developers

✤ Multidimensional reality✤ Multiple stakeholders

✤ Implies multiple views, multiple blueprints

Many stakeholders, many views

32


✤ Select scenarios: criticality and risk✤ Identify main classes and their responsibility✤ Distribute behavior on classes✤ Structure in subsystems, layers,

define interfaces✤ Define distribution and concurrency✤ Implement architectural prototype✤ Derive tests from use cases✤ Evaluate architecture✤ Iterate

Architectural design workflow

33

Use case view

Logical view

Implementation View

Deployment view

Process view


Scenario-based evaluation✤ Scenario is a brief description of an interaction of a stakeholder with a

system

System

34


Scenario-based evaluation✤ Scenario is a brief description of an interaction of a stakeholder with a

system

System

What if…

What if…

What if…What

if…

What if…

34


✤ User scenarios✤ What if I want to run a new track fit algorithm?✤ What if I need to use the newest calibration?

✤ Deployment engineer✤ What if we need to port the software to the Solaris platform?✤ What if we embed the software in real-time systems

✤ Manager✤ What if we need to support some standard data formats✤ What if we integrate a commercial GUI system

Scenarios evaluation examples

35


Sources of architecture

Theft Method

Intuition

Classical system Unprecedented system

Theft

Method

Intuition

36


Architectural style✤ An architecture style defines a family of systems in terms of a pattern

of structural organization.✤ An architectural style defines

✤ a vocabulary of components and connector types✤ a set of constraints on how they can be combined✤ one or more semantic models that specify how a system’s overall properties

can be determined from the properties of its parts

37

Mary Shaw, Carnegie Mellon University


http://www.cmu.edu/

http://www.cmu.edu/

38

Architectural styles✤ General categorization of systems [1]

✤ user-centric

✤ data-centric

✤ computation-centric

[1] G. Booch, “Object Solutions”, Addison-Wesley 1996

focus on the direct visualizationand manipulation of the objects that define a certain domain

focus upon preserving the integrity of the persistent objects in a system

focus is on the transformationof objects that are interestingto the system


39

Different style in different domains✤ The applications in the

different domains may have different emphasis in:✤ Interactivity✤ Database✤ Computation

✤ Elements of all three are present in all applications

User InterfaceInteractivity

Scripting

Object storeDatabase

Data Integrity

AlgorithmsComputation

computation-centric

data-centricuser-

centric


✤ Unified Modeling Language (UML) is a standardized general-purpose modeling language

✤ Includes a set of graphical notation techniques to create visual models of software-intensive systems

✤ Is an open standard✤ Supports the entire software development lifecycle✤ Supports diverse applications areas✤ Is based on experience and needs of the user community✤ Supported by many tools

UML

40

http://www.uml.org


http://www.uml.org

http://www.uml.org

UML Diagrams

41

} Structure diagrams◦ Class◦ Component◦ Deployment◦ Object◦ Package

} Behavior diagrams◦ Activity◦ State machine◦ Use case

} Interaction diagrams◦ Communication◦ Interaction◦ Sequence


Use Case Diagram✤ Captures system functionality as

seen by users✤ Built in early stages

of development✤ Specify the context of a

system✤ Capture the requirements of a system✤ Validate a system’s architecture✤ Drive implementation and generate

test cases✤ Developed by analysts and domain

experts

42


Class Diagram✤ Captures the vocabulary of a

system✤ Built and refined throughout

development✤ Name and model concepts in the

system✤ Specify collaborations✤ Specify logical database schemas

✤ Developed by analysts, designers, and implementers

43


Object Diagram✤ Shows instances and links✤ Built during analysis and

design✤ Illustrate data/object

structures✤ Specify snapshots

✤ Developed by analysts, designers, and implementers

44


Sequence Diagram✤ Captures dynamic behavior (time-

oriented)✤ Purpose

✤ Model flow of control

✤ Illustrate typical scenarios

45


Collaboration Diagram✤ Captures dynamic behavior (message-oriented)

✤ Model flow of control✤ Illustrate coordination of object structure and control

46


✤ Captures dynamic behavior (event-oriented)✤ Purpose

✤ Model object lifecycle✤ Model reactive objects (user interfaces, devices, etc.)

Statechart Diagram

47


✤ Experience✤ In software development✤ In the domain

✤ Pro-active, goal oriented✤ Leadership, authority✤ Architecture team

✤ Balance between technologists, domain experts, users

The Architect Role

48


The Architect Role✤ Not just a top level designer

✤ Need to ensure feasibility✤ Not the project manager

✤ But “joined at the hip”✤ Not a technology expert

✤ Purpose of the system, “fit”✤ Not a lone scientist

✤ Communicator

49


✤ Defining the architecture of the software✤ Maintaining the architectural integrity of the software✤ Assessing technical risks related to the software design✤ Proposing the order and contents of the successive iterations ✤ Consulting services✤ Assisting marketing for future product definition✤ Facilitating communications between project teams

Software architecture team charter

50


Architecture is making decisions

51

The life of a software architect is a long (and sometimes painful) succession of suboptimal decisions made partly in the dark.

Philippe Kruchten, Rational


Take Away Messages✤ Software is essential to process experimental data to do Science✤ In the case of HEP this data processing is rather large and complex

✤ The software stack is composed of many libraries and packages, some of them are general purpose but many of them are specific to the experiment developed by the scientists themselves

✤ A solid architecture is required to fulfill the demanding requirements and to enable the contributions from each scientist ✤ First time in HEP that software architecture was taken very serious✤ Each LHC experiment had/has a software architecture team ✤ UML has been extensively used to express design concepts, articulate

discussions, and document key classes and their interactions✤ The role of the architect (or architecture team) is very wide

✤ Balance between technology, domain expertise and user experience

52


} Grady Booch, Object Solutions, Addison-Wesley, 1995.} Eric Gamma, John Vlissides, Richard Helm, Ralph Johnson, Design

Patterns, Addison-Wesley 1995.} Rational Unified Process 5.0, Rational, Cupertino, CA, 1998} Len Bass, Paul Clements & Rick Kazman, Software Architecture in Practice,

Addison-Wesley, 1998

References

53


Requirements and Architectural Design

Documents

Transcript of Requirements and Architectural Design