Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software...

92
Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari Systä 1

Transcript of Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software...

Page 1: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Software Engineering Methodology Lecture 9, 23.3.2015

Kari Systä

24.2.2014 TIE-21100-6/Kari Systä 1

Page 2: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Today

• Something more about global SW development

• Some notes from last week’s guest lecture

• SW Quality and Quality Systems

• Time permitting also

– Inspections and reviews

– Testing

If no time, the we discuss those later

• Note, all slides will not be discussed

24.2.2014 TIE-21100-6/Kari Systä 2

Page 3: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Lessons Learned from Leading Workshops about Geographically

Distributed Agile Teams

Johanna Rothman, Shane Hastie

IEEE Software March/April 2013

Page 4: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

PL#1: “We Thought Scrum Would Work” • designed for a five- to seven-person, cross-functional,

collocated team

• additional coordination is needed: use agile approaches to experiment and adapt to find what fits best for you.

PL#2: “We’ll Force a Cross-Functional Team to Form” • “Go do Scrum with these people from these silo’d teams. We’ll give you

testers in India, developers in Germany, developers in Sweden, developers in Israel, product owners in Brazil, business analysts in North Carolina, and Scrum masters in California. We empower you to make yourselves a team.”

• ” Even when collocated, it takes time and effort for a group of individuals to become a team” …

• ” The more time zones apart the team members are, the longer it takes to jell. Teams need interaction to be able to storm—to figure out their collaboration process, their ways of communicating, and their shared norms.”

Page 5: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

PL#3: Meeting Expense Is Cheaper than the Cost of Not Meeting • ”What astonished us was how many senior management teams flung people

together, called them a team, … When our workshop participants asked for travel budgets, they were told there was no money for travel. ”

• “The return on investment of the relatively small amount of money spent bringing the teams together could be measured in better collaboration, shortened decision-making cycles, fewer misunderstandings and rework, and higher-quality products with shorter time to market. “

ML#1: Don’t Underpower Infrastructure in Remote Locations • ”We heard astonishing stories about how the remote teams didn’t have the

same access to tools, sources, knowledge, and talent as the teams closer to head quarters”

• ”With distributed teams, the mere ability to collaborate is determined by the tools available to the members with the lowest capability”

Page 6: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

ML#2: Don’t Hire Testers 12 Hours Away to Save Money • ”Managers too often selected testers to be 12 hours away from developers

because they didn’t understand that building software is a creative design activity. Unfortunately, some managers think that software is a manufacturing activity. ”

• ” Distributing work to reduce cost is one of the worst reasons to form distributed teams. Examine why the team is being distributed and ensure that goals are actually achievable.”

ML#3: Managers Can’t Keep Their Hands off People and on Projects • ”As soon as you have a geographically distributed team, it’s crucial to

manage the project portfolio and have work flow through the team. Moving people on and off disturbs the team and makes it difficult for its members to succeed at all.”

Page 7: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

CL#1: Different Isn’t Wrong, but It Takes Effort to Overcome This Bias • “Distributed team members don’t have the same understanding of each

other’s culture. Our natural tendency is to assume that everyone has the same set of unwritten norms as we do. Of course, this is wrong, but that doesn’t matter—unless team members take the time to stop and think about how they communicate and what messages they convey through their behavior. “

• “A lot of research looks at these cultural dimensions6 and recommends spending time in a team talking about each other’s cultures. Respectful curiosity goes a long way to creating understanding. Bringing people together and giving them the space to have these conversations is a great way to start a team building some shared norms. “

Page 8: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

CL#2: Sharing Food Is an Important Bonding Activity

CL#3: Create Team Norms around Risk and Being Done • When we taught the workshops, the participants often asked, “How do we

help our project teams get to ‘done’ at the end of the iteration?”

• Collocated teams have these problems as well, but they can more easily conduct retrospectives and examine their behaviors and artifacts. It’s not

• so easy for a geographically distributed team to do this.

Page 9: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Some notes from the lecture by Juho-Pekka Pöyry (Leonidas)

• UI is the starting phase and initial spec • They use many programming languages • Sales may cause problems • Everybody (client, management) can claim to be an

expert – Doctors vs. IT professional

• Some pitfalls – Too high abstractions hide important facts – Estimates on % left – Process over content (Nokia test -

http://jeffsutherland.com/nokiatest.pdf)

Page 10: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

How do the lectures continue?

02.03.2015 TIE-21100/21106 10

02.03 Modern practices Continuous Integration, Continuous Deployment, DevOps,..

09.03 No lectures - exam week

16.03 Guest Lecture 1: Juho-Pekka Pöyry, Leonidas

23.03 Improving Quality: review practices, testing and quality assurance Quality in general; Quality management systems

30.03 Guest Lecture 2: Ari Jaaksi

06.04 Easter - no lectures

13.04 Dependable and safety-critical systems

20.04 Other methods; RUP, XP, Lean; Kanban.

27.04 Role of software architecture; software maintenance; software evolution

04.05 Software business, software start-ups

11.05 Recap, prepare for the exam.

Page 11: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Learning goals of today

• Develop a holistic view to SW quality

• Learn basics of ”quality management systems” and related standards

• Know what concepts – ISO9000 – series

– CMMI

– SPICE

mean

10.03.2014 TIE-21100/21106; K.Systä 11

Page 12: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

A FEW THOUGHTS ABOUT QUALITY The next four slides are based on presentation of Esko Hannula (CEO of Quentinel)

10.03.2014 TIE-21100/21106; K.Systä 12

Page 13: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Many different definitions

10.03.2014 TIE-21100/21106; K.Systä 13

People Subjective view on what is good and

valuable.

Development Engineer

Meeting of the predefined criteria

Quality manager Operation complies with standardized

quality systems

Production Engineer How big % of the patch fulfils the predefined

criteria-

Page 14: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

SW Quality defined in terms of problems, deviations, bugs … cost

10.03.2014 TIE-21100/21106; K.Systä 14

Implementation Deploy-ment

Production

Cost of bad management

Cost of waste

Cost of quality

assurance

Cost of error correction Cost of

changes and training

Cost of delays

Cost of errors

Cost of error correction

Cost of missing things

Cost of breaks in

operations

Cost of extra maintenance

Cost of work-arounds

Cost of exta features

Page 15: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Stakeholders and quality of software

10.03.2014 TIE-21100/21106; K.Systä 15

User Can perform tasks

- easily - reliably

CEO Cash-flow Revenue

Productivity

CIO Maintenance and operation costs.

User support. Business complaints.

Project manager Schedule is met Budget is met

Page 16: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Quality is about value

10.03.2014 TIE-21100/21106; K.Systä 16

Accepted by end-users

Fits business needs and

ways of working

Implemented and operated

well

Fulfills the (known)

requirements Quality

Page 17: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Software Quality attributes according to ISO 9126 (old standard)

10.03.2014 TIE-21100/21106; K.Systä 17

ISO 9126 –

External and

internal

quality

Functionality

Reliability

Usability

Efficiency

Maintainability

Portability

Analyzability Changeability

Stability Testability

Adaptability Installability Co-existence Replaceability

Time behavior Resource utilization Suitability

Accuracy Interoperability Security

Maturity Fault tolerance Recoverability

Understandability Learnability Operability Attractiveness

Page 18: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Software Quality attributes according to ISO 25010:2011

10.03.2014 TIE-21100/21106; K.Systä

18

ISO 9126 –

External and

internal

quality

Functional suitability

Reliability

Operability

Efficiency

Maintainability

Transferability

Modularity Reusability

Analyzability Changeability

Stability Testability

Portability Adaptability Installability

Time behavior Resource utilization

Suitability Accuracy Interoperability Security

Maturity Fault tolerance Recoverability

Appropriateness Recognisability Ease of use Learnability Attractiveness Technical accessibility

Compatibility

Replaceability Co-existence Interoperability

Security

Confidentiality Integrity Non-repudiation Accountability Authenticity

Page 19: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

QUALITY MANAGEMENT SYSTEMS

10.03.2014 TIE-21100/21106; K.Systä 19

Page 20: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Process view to SW development

10.03.2014 TIE-21100/21106; K.Systä 20

Process In out

control measurement

Process is defined so that it guarantees quality (selected aspects of it)

For example: • All bug reports handled • Requirement changes

communicated

Auditing checks the defined process is really followed.

Page 21: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Process-based quality improvement (figure 24.3 in Sommerville)

10.03.2014 TIE-21100/21106; K.Systä 21

Define process

Develop product

Access product quality

Standardize process

Quality

OK?

Improve process

Page 22: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Question:

Why the focus in on process and not in product (SW)?

10.03.2014 TIE-21100/21106; K.Systä 22

Page 23: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Quality management systems (Laatujärjestelmät)

• Documented description of the development process

• Aim is to provide predictable and sustainable quality

• Customer may be interested in this

• Can be audited by a neutral body

10.03.2014 TIE-21100/21106; K.Systä 23

Page 24: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

ISO9000 Standards • Standard used to develop quality management systems • Targets to all industries – including software • Revisions

– 1987: had the same structure as the UK Standard BS 5750 and influence from US Military standards

– 1994: emphasized quality assurance via preventive actions, instead of just checking final product

– 2000: placed the concept of process management front and center ("Process management" was the monitoring and optimization of a company's tasks and activities, instead of just inspection of the final product)

– 2010: introduced clarifications to the existing requirements of ISO 9001:2000

– Forthcoming 2015 version (http://www.iso.org/iso/home/news_index/news_archive/news.htm?refid=Ref1633)

10.03.2014 TIE-21100/21106; K.Systä 24

Page 25: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Principles behind ISO9000 series (http://www.iso.org/iso/qmp_2012.pdf)

• Principle 1 – Customer focus

• Principle 2 – Leadership

• Principle 3 – Involvement of people

• Principle 4 – Process approach

• Principle 5 – System approach to management

• Principle 6 – Continual improvement

• Principle 7 – Factual approach to decision making

• Principle 8 – Mutually beneficial supplier

10.03.2014 TIE-21100/21106; K.Systä 25

Page 26: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

• ISO 9000, Quality management systems - Fundamentals and vocabulary

• ISO 9001, Quality management systems - Requirements

24.2.2014 TIE-21100-6/Kari Systä 26

Page 27: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

ISO9001 and quality management (figure 24.5 in Sommerville)

10.03.2014 TIE-21100/21106; K.Systä 27

ISO 9001 quality models

Organization quality manual

Project 2 quality plan

Project 1 quality plan

Project 3 quality plan

Organization quality process

Project quality management

instantiated as

Is used to develop

Supports

instantiated as

Page 28: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Certification checks that the company

• has a documented quality (management) system that fulfills the requirements of ISO 9001 -standard. – The document is the ”quality manual”

• can show that it operates according to its quality system.

• Continuously develops and improves its ways of working

Quality system usually

• Set clear goals to the organization and its operation

• Documents ways of working to achieve the goals

• Prepares to show that the organization operates according to the documented ways of working

• Prepares to show that ways of working are continuously improved.

24.2.2014 TIE-21100-6/Kari Systä 28

Page 29: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Auditing and certification An organization must perform internal audits to check how its quality management system is working. An organization may decide to invite an independent certification body to verify that it is in conformity to the standard, but there is no requirement for this. Alternatively, it might invite its clients to audit the quality system for themselves. Certification is not requirement, but certification may: • be a contractual or regulatory requirement • be necessary to meet customer preferences • fall within the context of a risk management programme, and • help motivate staff by setting a clear goal for the development of its

management system. Certification – the provision by an independent body of written assurance (a certificate) that the product, service or system in question meets specific requirements. Accreditation – the formal recognition by an independent body, generally known as an accreditation body, that a certification body is capable of carrying out certification.

10.03.2014 TIE-21100/21106; K.Systä 29

Page 30: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Example from http://www.iso.org/iso/qmp_2012.pdf (process approach)

Key benefits • Lower costs and shorter cycle times • through effective use of resources • Improved, consistent and predictable results • Focused and prioritized improvement opportunities. Means: • Systematically defining the activities necessary to obtain a desired result • Establishing clear responsibility and accountability for managing key

activities • Analysing and measuring of the capability of key activities • Identifying the interfaces of key activities within and between the

functions of the organization • Focusing on the factors – such as resources, methods, and materials –

that will improve key activities of the organization • Evaluating risks, consequences and impacts of activities on customers,

suppliers and other interested parties.

10.03.2014 TIE-21100/21106; K.Systä 30

Page 31: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

ISO 9001 core processes

Product delivery

• Business acquisition

• Design and development

• Test

• Production and Delivery

• Service and Support

Supporting processes

• Business management

• Supplier management

• Inventory management

• Configuration management

10.03.2014 TIE-21100/21106; K.Systä 31

Page 32: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Why new 9001:2015?

• Maintain Relevance • Integration among all standards • Consistent foundation for the long-term (next 10-25 years) • Increase adoption of the standard

• Address increased variety of business users

– Service industries – Office environments

• Address increasing complexity of business environment

– Non-office/virtual office

24.2.2014 TIE-21100-6/Kari Systä 32

Page 33: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Key changes • Emphasis on Risk-based management

– Preventive action

• Increase emphasis on achieving value for organization and its customers

• Documented Information – Decreased emphasis on documentation – Expands concept of documentation – Replaces Documents and Records

• Organizational Context – Responsiveness to business environment

• Outsourcing is now External Provision • Enhanced leadership requirements • No requirement for Management Representative • No requirement for Quality Manual

24.2.2014 TIE-21100-6/Kari Systä 33

Page 34: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Some Dilbert humor

TIE-21100/21106; K.Systä 34 10.03.2014

Page 35: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

And more

10.03.2014 TIE-21100/21106; K.Systä 35

Page 36: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

CMMI

10.03.2014 TIE-21100/21106; K.Systä 36

Page 37: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

http://cmmiinstitute.com/

• A framework developed by Software Entineering Instute (SEI) of Carnegie-Mellon University (CMU)

• CMMI (or Capability Maturity Model Integration) is a proven approach to performance management with decades of results showing it works.

• Organizations using CMMI have predictable cost, schedule, and quality—business results that serve as discriminators among their competitors.

• CMMI is built with practices and goals seen in thousands of real organizations worldwide. Use these practices and goals to evaluate your own performance and decide what to improve for your own business reasons.

10.03.2014 TIE-21100/21106; K.Systä 37

Page 38: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

CMMI and ISO9000-series

• CMMI is a set of best practices ISO9000 set of standards

• CMMI is about software ISO9000 is for all industries

10.03.2014 TIE-21100/21106; K.Systä 38

Page 39: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

CMMI • Sources

– Haikala&Mikkonen:

– http://www.sei.cmu.edu/reports/10tr033.pdf (chapter 1)

• Old version CMM (1987-1997) listed 5 maturity levels 1. Initial

2. Repeatable process

3. Defined process

4. Quantiatively Manged Process

5. Optimizing Process

• New version is called CMMI defines two models for process impovement – Staged

– Continuous

10.03.2014 TIE-21100/21106; K.Systä 39

Page 40: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Software Process improvement (http://www.sei.cmu.edu/reports/10tr033.pdf)

10.03.2014 TIE-21100/21106; K.Systä 40

Page 41: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

10.03.2014 TIE-21100/21106; K.Systä 41

Page 42: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Basic concepts of CMMI

• Capability levels apply to an organization’s process improvement achievement in individual process areas. These levels are a means for incrementally improving the processes corresponding to a given process area. – Continuous representation

• Maturity levels apply to an organization’s process improvement achievement across multiple process areas. These levels are a means of improving the processes corresponding to a given set of process areas. – For staged representation

10.03.2014 TIE-21100/21106; K.Systä 42

Page 43: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

10.03.2014 TIE-21100/21106; K.Systä 43

Capability levels apply to an organization’s process improvement achievement in individual process areas. These levels are a means for incrementally improving the processes corresponding to a given process area.

Continuous representation focuses on process area capability as measured by capability levels and the staged representation focuses on overall maturity as measured by maturity levels.

Maturity levels apply to an organization’s process improvement achievement across multiple process areas. These levels are a means of improving the processes corresponding to a given set of process areas

Page 44: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Capability and Maturity Models

10.03.2014 TIE-21100/21106; K.Systä 44

Level Continuous Representation Capability Levels

Staged representation Maturity Levels

0 Incompete

1 Performed Initial

2 Managed Managed

3 Defined Defined

4 Quantitatively Managed

5 Optimizing

Page 45: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

What are the capability levels Capability Level 0: Incomplete

An incomplete process is a process that either is not performed or is partially performed. One or more of the specific goals of the process area are not satisfied and no generic goals exist for this level since there is no reason to institutionalize a partially performed process.

Capability Level 1: Performed A capability level 1 process is characterized as a performed process. A performed process is a process that accomplishes the needed work to produce work products; the specific goals of the process area are satisfied. Although capability level 1 results in important improvements, those improvements can be lost over time if they are not institutionalized. The application of institutionalization (the CMMI generic practices at capability levels 2 and 3) helps to ensure that improvements are maintained.

10.03.2014 TIE-21100/21106; K.Systä 45

Page 46: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Capability Level 2: Managed

A capability level 2 process is characterized as a managed process. A managed process is a performed process that is planned and executed in accordance with policy; employs skilled people having adequate resources to produce controlled outputs; involves relevant stakeholders; is monitored, controlled, and reviewed; and is evaluated for adherence to its process description.

The process discipline reflected by capability level 2 helps to ensure that existing practices are retained during times of stress.

Capability Level 3: Defined

A capability level 3 process is characterized as a defined process. A defined process is a managed process that is tailored from the organization’s set of standard processes according to the organization’s tailoring guidelines; has a maintained process description; and contributes process related experiences to the organizational process assets.

A critical distinction between levels 2 and 3 is the scope of standards, process descriptions, and procedures. At level 2, the standards, process descriptions, and procedures can be quite different in each specific instance of the process (e.g., on a particular project). At level 3, the standards etc are tailored from the organization’s set of standard processes to suit a particular project or organizational unit and therefore are more consistent,

10.03.2014 TIE-21100/21106; K.Systä 46

Page 47: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Maturity levels Maturity Level 1: Initial

At maturity level 1, processes are usually ad hoc and chaotic. The organization usually does not provide a stable environment to support processes. Success in these organizations depends on the competence and heroics of the people in the organization and not on the use of proven processes. In spite of this chaos, maturity level 1 organizations often produce products and services that work, but they frequently exceed the budget and schedule documented in their plans.

Maturity level 1 organizations are characterized by a tendency to overcommit, abandon their processes in a time of crisis, and be unable to repeat their successes.

10.03.2014 TIE-21100/21106; K.Systä 47

Page 48: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Maturity Level 2: Managed At maturity level 2, the projects have ensured that processes are planned and executed in accordance with policy; the projects employ skilled people who have adequate resources to produce controlled outputs; involve relevant stakeholders; are monitored, controlled, and reviewed; and are evaluated for adherence to their process descriptions. The process discipline reflected by maturity level 2 helps to ensure that existing practices are retained during times of stress. When these practices are in place, projects are performed and managed according to their documented plans. Also at maturity level 2, the status of the work products are visible to management at defined points (e.g., at major milestones, at the completion of major tasks). Commitments are established among relevant stakeholders and are revised as needed. Work products are appropriately controlled. The work products and services satisfy their specified process descriptions, standards, and procedures.

10.03.2014 TIE-21100/21106; K.Systä 48

Page 49: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Maturity Level 3: Defined At maturity level 3, processes are well characterized and understood, and are described in standards, procedures, tools, and methods. The organization’s set of standard processes, which is the basis for maturity level 3, is established and improved over time. These standard processes are used to establish consistency across the organization. Projects establish their defined processes by tailoring the organization’s set of standard processes according to tailoring guidelines. (See the definition of “organization’s set of standard processes” in the glossary.)

A critical distinction between maturity levels 2 and 3 is the scope of … (as capability level 3 in continuous representation)

10.03.2014 TIE-21100/21106; K.Systä 49

Page 50: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Maturity Level 4: Quantitatively Managed At maturity level 4, the organization and projects establish quantitative objectives for quality and process performance and use them as criteria in managing projects. Quantitative objectives are based on the needs of the customer, end users, organization, and process implementers. Quality and process performance is understood in statistical terms and is managed throughout the life of projects.

For selected subprocesses, specific measures of process performance are collected and statistically analyzed. When selecting subprocesses for analyses, it is critical to understand the relationships between different subprocesses and their impact on achieving the objectives for quality and process performance.

10.03.2014 TIE-21100/21106; K.Systä 50

Page 51: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Maturity Level 5: Optimizing At maturity level 5, an organization continually improves its processes based on a quantitative understanding of its business objectives and performance needs. The organization uses a quantitative approach to understand the variation inherent in the process and the causes of process outcomes. Maturity level 5 focuses on continually improving process performance through incremental and innovative pr ocess and technological improvements. A critical distinction between maturity levels 4 and 5 is the focus on managing and improving organizational performance. At maturity level 4, the organization and projects focus on understanding and controlling performance at the subprocess level and using the results to manage projects. At maturity level 5, the organization is concerned with overall organizational performance using data collected from multiple projects.

10.03.2014 TIE-21100/21106; K.Systä 51

Page 52: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

SPICE (ISO 15504)

• E.g. http://en.wikipedia.org/wiki/ISO/IEC_15504 • ISO/IEC 15504 is the reference model for the maturity

models (consisting of capability levels which in turn consist of the process attributes and further consist of generic practices) against which the assessors can place the evidence that they collect during their assessment, so that the assessors can give an overall determination of the organization's capabilities for delivering products (software, systems, and IT services)

• Very similar to Continuous Representation of CMMI (when considered in detailed level discussed in this course)

10.03.2014 TIE-21100/21106; K.Systä 52

Page 53: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

What does this matter

• These SPI approached are not very popular at the moment,

– But used in some companies

• Young software engineers do not need to be experts, but MSc in Software Engineering must have some idea what they are

10.03.2014 TIE-21100/21106; K.Systä 53

Page 54: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Material

Sommerville Chapter 26

Haikala&Mikkonen Chapter 11

• Quality Management Principles: http://www.iso.org/iso/qmp_2012.pdf

• ISO 9000 standard family: http://www.iso.org/iso/iso_9000

• CMMI: http://cmmiinstitute.com/ http://www.sei.cmu.edu/reports/10tr033.pdf

TIE-21100/21106; K.Systä 54 10.03.2014

Page 55: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

REVIEWS AND INSPECTIONS

24.2.2014 TIE-21100-6/Kari Systä 55

Page 56: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Terminology Inspection = tarkastus • Internal event in the project - not strictly tied to project schedules (next

phase may start) • Sole purpose is in detecting defects for early correction • Relatively small amount of work under inspection • The whole material (to be inspected) is scanned through • More "formal", diary/log/minutes is kept • Documents, code, prototypes,... (Technical) Review = katselmointi, katselmus, tekninen katselmus • Formal event to check that a milestone have been reached; makes a

milestone in a project visible (go / no-go) • The entire phase product • Number of participants can be large and different stakeholders less

formal, diary/log/minutes maybe not kept, just "scanning" material. Walkthrough (, walk-thru) = läpikäynti • Informal - “what the designer thinks his/her code does”. Assessment • Usually for processes (will be discussed in next lecture)

24.2.2014 TIE-21100-6/Kari Systä 56

Page 57: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Definitions

• IEEE 610.12-1990: inspection = a static analysis technique that relies on visual examination of development products to detect errors, violations of development standards, and other problems. Types include code inspection; design inspection. review = a process or meeting during which a work product, or set of work products, is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include code review, design review, formal qualification review, requirements review, test readiness review. walk-through = a static analysis technique in which a designer or programmer leads members of the development team and other interested parties through a segment of documentation or code, and the participants ask questions and make comments about possible errors, violation of development standards, and other problems.

24.2.2014 TIE-21100-6/Kari Systä 57

Page 58: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Difference between inspection and reviews

• There are two intentions

– Inspect to find errors

– Review to ensure that milestone has been reached, i.e., all conditions of the milestone are met

• Procedure may very similar

• The meeting in inspections is often called review

24.2.2014 TIE-21100-6/Kari Systä 58

Page 59: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Inspections

• Very efficient way to find errors

• Fagan: "Design and code inspections to reduce errors in program development", IBM Systems Journal, 1976.

• Requires effort – is an investment

24.2.2014 TIE-21100-6/Kari Systä 59

Page 60: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Side note: pair programming This not inspection

• Cockburn, Williams: The Costs and Benefits of Pair Programming • Two programmers work collaboratively on the same algorithm,

design or programming task, sitting side by side at one computer.

• One writes code and the other “inspects”

24.2.2014 TIE-21100-6/Kari Systä 60

Page 61: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

About economy of inspections

• To find defects as early as possible • To make design, central code, documents etc as

stable (and as early) as possible with the help of technical colleagues’ input and experience.

• 5% to 15% of project cost (working time) • Finds up to 80% of the defects in product

(normally less, never 100 %) • Cost effective in improving quality • Testing is too late; all faults in specs, design

documents etc. are already implemented. That is why inspections are needed and are cost effective

24.2.2014 TIE-21100-6/Kari Systä 61

Page 62: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Inspection in practice

24.2.2014 TIE-21100-6/Kari Systä 62

Planning Overview Preparation

Meeting Rework Rework

Page 63: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Sommerville drawing of the process

24.2.2014 TIE-21100-6/Kari Systä 63

Page 64: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Roles (http://www.softwareengineering-9.com/Web/QualityMan/roles.html)

Grady and Van Slack (Grady and Van Slack, 1994) suggest six roles: • Author or owner. The programmer or designer responsible for

producing the program or document. Responsible for fixing defects discovered during the inspection process.

• Inspector. Finds errors, omissions and inconsistencies in programs and documents. May also identify broader issues with the code being inspected such as lack of portability.

• Reader. Presents the code or document at an inspection meeting. • Chairman or moderator. Manages the process and facilitates the

inspection. Reports process results to the chief moderator. • Scribe. Records the results of the inspection meeting. • Chief moderator. Responsible for inspection process improvements,

checklist updating, standards development, etc. Not necessarily involved in all inspections.

24.2.2014 TIE-21100-6/Kari Systä 64

Page 65: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Roles (as instructed in our project course)

Moderator, coordinator, chairman, leader • Leads the meeting; keep the discussion in defects, may

even cancel the meeting at start if it seems to be worthless (e.g. poor material or poor preparation) !

Inspectors, checkers, participants • Voice comments on the section read, and look for errors Secretary • Fill in defect list Author • Get a clear understanding about what needs to be

corrected.

24.2.2014 TIE-21100-6/Kari Systä 65

Page 66: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Good practices (from project course) no need to ask "turn to speak" if you don't have any comments for that section/paragraph/line, no need to say that speak shortly (no coffee table conversations) if you are in doubt (if there is an unclear matter), ask ! "is that an error or not ?", unclear matters should be recorded to diary/log/minutes "a professional attitude" is the best; do it right, do it once, don't waste time. Do do not discuss solution It helps a lot if the diary/log/minutes is visible to all participants all the time • everybody can see and check what was written down • to see if secretary did understand finding/question right • is the speed too fast (or slow) considered to writing diary • no need for repetition/review of diary at the end of inspection. If target is "ready" and people are prepared well, inspection goes fast and easy.

24.2.2014 TIE-21100-6/Kari Systä 66

Page 67: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Typical problems • unprepared attendees ("I'm in a hurry" is no excuse !!!) • unstable (draft) specs, interfaces in code • inspection changes into a design meeting • unrelevant comments • too much material • follow-up not performed • "crying" author or moderator • document prepared in isolation, hopeless quality, somebody should

prechecked • process differences • political decisions (e.g. QA vs. delivery deadlines) • inspections simply not performed • disturbances (phones, people coming late or leaving early,...).

24.2.2014 TIE-21100-6/Kari Systä 67

Page 68: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

http://www.cs.tut.fi/kurssit/OHJ-3500/2012-13/tilastoja.html

Simple data collected from course document inspections • duration of inspection; e.g. 1,75 h • total inspection time (prep+insp); e.g. 22,5 h • number of findings; e.g. 23 • number of pages in document; e.g. 37. From these the following statistics can be calculated easily: • error density; errors / page (e.g. 0,62) • speed of inspection; pages / hour (e.g. 21,14) • speed, number of findings per hour; errors / hour (e.g.

13,14) • time for finding one error; total inspection time / number

of errors (e.g. 0,98 h) • time spent for one page; inspection time / page (e.g. 2,84

min).

24.2.2014 TIE-21100-6/Kari Systä 68

Page 69: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Reviews and agile methods

• The review process in agile software development is usually informal. – In Scrum, for example, there is a review meeting after each

iteration of the software has been completed (a sprint review), where quality issues and problems may be discussed.

• In extreme programming, pair programming ensures that code is constantly being examined and reviewed by another team member.

• XP relies on individuals taking the initiative to improve and refactor code. Agile approaches are not usually standards-driven, so issues of standards compliance are not usually considered.

69 Chapter 24 Quality management

Page 70: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

One Agile method, Feature-Driven Development, recommends use of inspections

Develop features using the following practices: • Domain Object Modeling • Developing by Feature • Component/Class Ownership • Feature Teams • Inspections • Configuration Management • Regular Builds • Visibility of progress and results

24.2.2014 TIE-21100-6/Kari Systä 70

Page 71: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Lean community has developed capture-recapture code inspection (http://leansoftwareengineering.com/2007/06/05/the-capture-recapture-code-inspection/)

24.2.2014 TIE-21100-6/Kari Systä 71

Page 72: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Theory 𝑁 =

𝑛1 ∗ 𝑛2

𝑚

𝑁 = (𝑛1+1)(𝑛2+1)

(𝑚+1) - 1

N = Estimate of total defect population size n1 = Total number of defects discovered by first reviewer n2 = Total number of defects discovered by second reviewer m = Number of common defects discovered by both reviewers

24.2.2014 TIE-21100-6/Kari Systä 72

n1 n2 m N1 N2

1 10 1 10 10

2 9 2 9 9

3 8 2 12 11

4 7 2 14 12

5 6 2 15 13

6 5 2 15 13

7 4 1 28 19

8 3 1 24 17

9 2 1 18 14

10 1 1 10 10

Page 73: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Capture-recapture code inspection

The basic version requires four roles: • the reviewee • reviewer A • reviewer B • review moderator • Approximately 200 lines of code will result in the

highest inspection yield • Code sent to sent to the two reviewers • The reviewers bring their marked-up documents to the

review meeting. Each inspector will enumerate his findings, and the group will validate each defect.

24.2.2014 TIE-21100-6/Kari Systä 73

Page 74: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Walk-through

• More light-weight than inspections

• Informal - “what the designer thinks his/her code does”.

• The second motivation is spreading information

• Very useful for code-segment that all other developers use or are dependent on

24.2.2014 TIE-21100-6/Kari Systä 74

Page 75: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

TESTING

24.2.2014 TIE-21100-6/Kari Systä 75

Page 76: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Testing Disjkstra:

• ”Testing can only show presence of errors, not their absence”

Inspections and testing both have their roles:

• In testing error can mask other errors

• Incomplete versions can inspected

• Inspections can consider broader set of quality attributes

• Tests are easy to repeat

• Testing can discover issues that relate timing, interactions between different parts of software, …

24.2.2014 TIE-21100-6/Kari Systä 76

Page 77: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

• TIE-21200 Ohjelmistojen testaus / TIE-21200 Software Testing

• Learning outcomes of the course (SG*)

• Opiskelija tuntee testaamisen peruskäsitteet ja -tekniikat yksikkö-, integrointi-, järjestelmä- ja hyväksyntätestaustasolla sekä osaa soveltaa niitä ohjelmistotyössä kaikissa elinkaaren vaiheissa. Opiskelija tunnistaa sellaiset testaukseen liittyvät tehtävät, jotka voidaan joka osittain tai kokonaan automatisoida työkalujen avulla. Lisäksi opiskelija osaa käyttää vähintään yhtä automatisointityökalua.

• Students knows fundamental concepts of testing, and techniques for unit, integration, system and acceptance testing and apply these in all phases of software development. Student recognizes testing tasks that can partly or completely automated. In addition, students will lean to use at least one test-automation tool.

24.2.2014 TIE-21100-6/Kari Systä 77

Page 78: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Testing – traditional categorization

20.1.2014

78

Vendor

Customer

TIE-21100&21106/K.Systä

Unit

Testing

Unit

Testing

Unit

Testing

Integration

Testing

System

Testing

Acceptance

Testing

Page 79: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

V-moded

9.9.2013 79

Specification

Architecture

design

Detailed

design

System

testing

Integration

testing

Unit

testing

JOTU/K.Systä

Page 80: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Unit testing

• Software unit is tested before it is integrated to other parts of the

system

• Typically the size of tested unit is 100-1000 lines of code

• To run a unit in isolation developers need use test-beds and test

drivers

• For example if the unit is a class or object, the test should

– Call each method with a representative parameter values

– Set and check values of attributes

– Put object to different states

20.1.2014 TIE-21100&21106/K.Systä 80

Page 81: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Integration testing (Sommerville uses term Component testing)

• Test how units work together

• Usually concentrates in major interfaces in the software

24.2.2014 TIE-21100-6/Kari Systä 81

A B

C

Test cases

Page 82: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Black box and white box testing

24.2.2014 TIE-21100-6/Kari Systä

82

A B

A

Test cases

A B

A

Test cases

Page 83: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

About test coverage

int foo (int x, int y) {

int z = 0;

if ((x>0) && (y>0)){

z = x;

} else if (x*y > 0) {

z = x + 1;

}

return z;

}

24.2.2014 TIE-21100-6/Kari Systä 83

Code coverage Condition/decision coverage Parameter value coverage …

Page 84: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

This weeks weekly exercise

• Tue 11.02.2014 at 10-12 and 12-14

• Wed 12.02.2014 at 12-14

• Thu 13.02.2014 at 12-14 and 14-16

• In TC217!

24.2.2014 TIE-21100-6/Kari Systä 84

Page 85: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

System testing

• System is addresses the complete system

• Compared to functional specification and use manual

• Acceptance testing is a form of system testing that is done together with the customer

– Passing a. test is prerequisite for approval of the delivery

24.2.2014 TIE-21100-6/Kari Systä 85

Page 86: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

System and acceptance testing

• May include

– Field testing

– Performance testing

– Load testing

– Reliability testing

– Installation testing

– User / Usability testing

24.2.2014 TIE-21100-6/Kari Systä 86

Page 87: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Test planning http://www.softwareengineering-9.com/Web/Testing/Planning.html

• The testing process A description of the major phases of the system testing process. This may be broken down into the testing of individual sub-systems, the testing of external system interfaces, etc.

• Requirements traceability Users are most interested in the system meeting its requirements and testing should be planned so that all requirements are individually tested.

• Tested items The products of the software process that are to be tested should be specified.

• Testing schedule An overall testing schedule and resource allocation. This schedule should be linked to the more general project development schedule.

• Test recording procedures It is not enough simply to run tests; the results of the tests must be systematically recorded. It must be possible to audit the testing process to check that it has been carried out correctly.

• Hardware and software requirements This section should set out the software tools required and estimated hardware utilisation.

• Constraints Constraints affecting the testing process such as staff shortages should be anticipated in this section.

• System tests This section, which may be completely separate from the test plan, defines the test cases that should be applied to the system. These tests are derived from the system requirements specification.

24.2.2014 TIE-21100-6/Kari Systä 87

Page 88: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Testing and waterfall

9.9.2013 88 JOTU/K.Systä

System testing

Unit and integration testing

Often independent test team is used because they do not have same preassumptions as developers. Testing – especially writing good test cases is a skill of its own.

Page 89: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Testing and Agile • Testing is a crucial part of agile processes

• The iterative nature means that already changed code need to be tested again => A lot of regression testing needed

=> Test automation is an important tool in Agile

• Test driven development is often used with Agile

24.2.2014 TIE-21100-6/Kari Systä 89

Identifty new functionality

Write tests Run test Implement

Functionality and refactor

Page 90: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

24.2.2014 TIE-21100-6/Kari Systä 90

Page 91: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Test automation & continuous integration

• Continuous integration

– Whenever a task is completed, it is integrated to the system

– Each time the whole set of tests will be executed

Automated tests are important

• Continuous integration is common in Agile and especially Lean approaches

24.2.2014 TIE-21100-6/Kari Systä 91

Page 92: Software Engineering Methodology Lecture 9, 23.3sweng/lectures/2015_09_Quality.pdf · Software Engineering Methodology Lecture 9, 23.3.2015 Kari Systä 24.2.2014 TIE-21100-6/Kari

Summary

• Two ways to improve quality – Inspections & reviews

– Testing

• One big theme was not discussed: – Selection of good test cases that will ne discussed in

the testing course

• Next lecture (10.3): – Quality systems and quality standards

– General about quality

– Maybe more about testing

24.2.2014 TIE-21100-6/Kari Systä 92