T-76.613 Software Testing and Quality Assurance
-
Upload
softwarecentral -
Category
Documents
-
view
113 -
download
1
description
Transcript of T-76.613 Software Testing and Quality Assurance
HELSINKI UNIVERSITY OF TECHNOLOGY
T-76.5613 Software Testing and Quality Assurance
3.10.2007
Testing in Agile Software Development
Juha ItkonenSoberIT
2Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Overview
Agile Testing and Testing in Agile Software DevelopmentChallenges of Testing in Agile DevelopmentHow Agile Quality Practices Work?Agile Unit Testing and TDDAgile Acceptance Testing with FIT
3Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Process models from waterfall to agile
Time
Requirements
Design
Implementation
Testing
Waterfall Incremental,e.g. RUP
Agile - XP
Functionality
Redrawn from Beck. Embracing change with Extreme Programming, 1999.
4Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
V-model
Intuitive and easy to explainQuite adaptable to various situationsMakes a good model for training peopleBeats the code-and-fix approach on any larger project
Requirements
Unittesting
Coding
Moduledesign
Architecturedesign
Functionalspecification
Acceptancetesting
Systemtesting
Integrationtesting
Build Test
5Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Problems in applying V-model
Document drivenRelies on the existence, accuracy, completeness, and timeliness of development documentation
Communicates change poorlyIgnores the fact that, in practice, software is developed in a series of handoffs
Does not fit into modern iterative development processes
6Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Agile Software Development Manifesto
In agile development we value:Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
+ 12 more detailed principles
http://www.agilemanifesto.org/
How do these values affect testing?
7Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Plan-Driven vs. Agile
Wanted result ?
Planned result
8Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Agile Software Development – What About Testing?
Agility in software developmentWorking software is most important deliverableResponding to change – not resisting changeTight development and release cycle
Iterative and incremental processFast feedback, frequent adjustments
Efficient collaboration and communicationWithin the team and with the customer
Agility from the testing viewpointPlaces many challengesTesting must also be agile in agile developmentYou can learn a lot from the agile methods
9Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Two views of agile testing
eXtreme TestingAutomated unit testing
Developers write tests Test first developmentDaily builds with unit tests always 100% pass
Functional testingCustomer-ownedComprehensiveRepeatableAutomaticTimelyPublic
Exploratory TestingUtilizes professional testers’skills and experienceOptimized to find bugsMinimizing time spent on documentationContinually adjusting plans, re-focusing on the most promising risk areasFollowing hunchesFreedom, flexibility and fun for testers
Focus on automated verification – enabling agile software
development
Focus on manual validation – making testing activities
agile
10Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Some Agile Principles
Satisfy the customer through early and continuous delivery of valuable software.Working software is the primary measure of progress.Deliver working software frequently, from a couple of weeks to a couple of months.Welcome changing requirements, even late in development.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Business people and developers must work together daily throughout the project. Simplicity--the art of maximizing the amount of work not done--is essential.
http://www.agilemanifesto.org/
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
11Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Challenges that agile principles place on testing
Agile Principle ChallengeFrequent deliveries of valuable software
- Short time for testing in each cycle- Testing must be time-boxed, too
Responding to change even late in the development
-Testing cannot be designed beforehand based on specifications- Tests must not prevent change
Relying on face-to-face communication
- Getting developers and business people actively involved in testing
Working software is the primary measure of progress
- Quality information is required early and frequently throughout development
Simplicity is essential - Testing practices get easily dropped for simplicity’s sake
12Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Contradictions with traditional testing principles
Testing principle Contradicting practices in agile methodsIndependency of testing and destructive attitude
- Developers write tests for their own code- Developers concentrate on constructive QA practices, i.e., building quality into the product and showing that features work
Testing requires specific skills
- Developers do the testing as part of the development- The customer has a very important and collaborative role and a lot of responsibility for the resulting quality
Oracle problem - Relying on automated tests to reveal defects
Evaluating achieved quality
- Confidence in quality through tracking conformance to a set of good practices
13Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
How Do the Agile Quality Practices Work?
14Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Organizing testing (waterfall vs. agile)
Programmers
Testers
Waterfall model Agile models (XP)
Customer
Programmer
Programmer
Tester
Idea:Testing in collaboration
15Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Quality Assurance in Agile Development
Plan-driven quality assurance does not work in agile context
There is no V-model and waterfallThe role of documentation and specifications is secondaryThe rhythm of development is fast and tightRoles and responsibilities are assigned differently
Cornerstones of agile quality assuranceConstant and tight rhythmCollaboration and communicationRigorous low level (quality) practicesTest-Driven Software Engineering
16Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Rhythm
Continuous unit-level integration and test cycleShort incremental release cyclesCompleting features in short cycles and small increments
Complete, integrated, testedBuilding quality and tracking the quality level in short cycles
t
Release
Iteration Heartbeat
17Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
1 Rhythm – viewpoint of testing
Feedback is faster and more frequentbetween development and testingfrom customers and users
Defects and issues are found earlierFeel the rhythm - tackle tasks before they bunch up
Testing features in small increments is easierLess new defects to hinder testingLess open defects to distract development
Real quality level and real progress better visibleRisks of testing and deployment are smaller
Easier to estimate and plan testing phase (or tasks)
Challenges- Short time for testing in each cycle- Testing must be time-boxed, too- Testing cannot be designed beforehand - Quality information is required early and frequently - Testing practices get easily dropped
18Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
2 Communication
Communication in agile development does not work in a traditional way
Traditionally others communicate to testers with requirements and specification documents and testers respond with defect reports and test reports
“So we can let free of the illusion that documents will save us. We can view them as they are: interesting texts, partly fictional, often useful.” - Brian Marick
19Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Collaboration
Collaboration is effective way of sharing information, knowledge, and understandingImproves common understanding and makes communication easy and efficient
Less guesswork and proceeding based on wrong assumptionsInformation moves faster and with less overhead
Especially effective in the home ground of agile development
When developing new, unfamiliar thingsWhen requirements are vague and changingWhen customer is easily available for collaboration
20Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Communication and collaboration – viewpoint of testing
There are testers also in agile teamsTester’s role is to help the development team to create good enough quality softwareTesters bring expertise and skills as well as the testers destructive viewpoint in the teamBy working together with developers and the customer the reliance on comprehensive documentation is not needed
Challenges- Tests must not prevent change- Getting developers and business people actively involved in testing- Developers write tests for their own code as part of the development- Developers concentrate on constructive QA practices- The customer has a lot of responsibility for the resulting quality- Relying automated tests to reveal defects
21Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
3 Low Level Quality Practices
Quality practices are tightly integrated into the daily heartbeat rhythm of the development
Testing each individual feature as part of its developmentBuilding quality using constructive practicesQuality is part of development – not delayed any furtherE.g. automated unit tests, pair-programming, continuous integrations, refactoring, …
Goal is to achieve high quality level before the code leaves the developer’s desk
22Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Some practices of an agile developer
Integrate early integrate oftenKeep your project releasable at all timesAutomate acceptance testingUse automated unit testsUse it before you build it –TDDWrite code in short edit/build/test cyclesEmphasize collective ownership of codeKeep a solutions log…
Keep it simpleDon’t fall for the quick hack Write code to be clear not cleverWarnings are really errorsProvide useful error messagesBe a mentorShare code only when it’s readyReview codeKeep others informed of the status of your work…
V. Subramaniam & A. Hunt, “Practices of an Agile Developer”,The Pragmatic Bookshelf, 2006.
23Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Low level quality practices – viewpoint of testing
Technical quality of the software is highLess trivial defects – easier to test hard issuesBoring, routine, checking procedures are automatedTesters and developers have a common goal of creating high quality software
Testers help developers <–> developers help testers
Testers can help developers Pair tester, who helps a developer to test each functionalityTester writes tests for the current functionality under development
Tests help developer in implementation and provide fast feedback already during implementation
Challenges- Short time for testing in each cycle- Testing must be time-boxed, too- Tests must not prevent change- Testing practices get easily dropped
24Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
4 Test-Driven Software Engineering
Use it before you build it!
Agile test-driven approach can be applied also on the level of customer requirements and acceptance testing
Tests can help programmers, testers, and business people (customer) communicateTests can by used to guide development workTests act as the true measure of real progress
25Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test-Driven Approach
Testers, programmers, and customer design together good tests based on the customer requirements
How, exactly, the features and core business logic should workMajor problems in the implementation (risks)Documenting the important details and potential problem areas beforehand
Tests guide the programmers in implementation work
What exactly must be done; how special cases should behave; when the feature is doneWhat are the risks of this feature; which issues to pay attention to
Tracking the amount of passing tests communicates the progress of development
As agile values and principles state 0
5
1 0
1 5
2 0
2 5
3 0
1 2 3 4 5 6 7 8 9 1 0
26Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
New Roles for Tests in TDSE
In Test-Driven Software Engineering tests have a clear role in the early requirements and design phasesTests help developers, testers, and customer to communicate
Common vocabulary and understanding
Tests document and concretize the details of requirementsConcrete examples of the functions and logicSpecial cases and exceptionsEasy to extend and update during the development
Tests guide the implementation workFast feedback to developers when automated
Tests provide an honest and agile metric of true progress in terms of working software
27Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Agile Unit Testing and
Test-Driven Development
28Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
No manual tests!
Chapter titled ”Manual Tests” in Testing ExtremeProgramming (Crispin & House 2003)
”No manual tests.”
29Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Unit testing
”A unit is the smallest possible testable software component”
Various interpretations existsProcedure / functionClass / object / methodSmall-sized component
Goal is to ensure that software units are functioning as the developer intended
E.g. a function returns correct value when exercised with valid parametersE.g. function behaves correctly when given an invalid input
Automated unit test is a piece of code that exercises another piece of code
30Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Document driven vs. agile unit testing
”Testing should be carried out by a group that is independent of the development group”
Course book presents a unit testing viewpoint which appreciates this principle
Unit testing should be a separate task conducted after the development is doneUnit test design is based on written specifications
Modern ideas of unit testing represent slightly different viewpoint
Unit tests are written by developerAutomation is essential part of unit testingUnit tests might even be written before the actual program code
This lecture is about agile unit testingRead the course book Ch. 6 and understand the differenceThe same tools are used by both disciplines
31Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Unit testing - a vital testing level
Writing software is difficult and every programmer makes mistakes
There will be defects in all software you writeUnit testing lays the foundation for good quality softwareUnit testing lays foundation for effective testing on higher levels
Defects that slip to upper levels (integration, system or acceptance testing)
More difficult to spotTime consuming to debugUses organizational resources
— Writing, prioritizing and managing test and defect reportsBad unit level quality slows down integration and system testing
32Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
How do you know that your code works correctly?
Do you just trust your luck?Do you use “printf” or ”System.out.println()” outputs to see what happens in your code?Do you use debugger to find out how your code really works?Do you ask your friend/colleague to check through your code?Do you create small (temporary) widget to access the piece of code directly runtime?Do you use some ugly, cumbersome, time consuming practice that leaves trash into your code and is thrown up to the user’s face when you least expect it…
“Getting here is not possible, something is really messed up in this code!!!”“Fatal error 9981: Assertion failed : av_d_dx > 0”
33Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Agile unit testingTest-Driven Development Writing and running tests is integral part of daily development rhythm
Tests are developers’ QA and programming tool
All unit tests are automatedAutomated tests are run as often as possible
Gather all your tests to a test suite and use it for regression testingIntegration, nightly, milestoneDuring development; after every change; new code or bug fix
34Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test-Driven Development
Writing test before code to be tested”a little test, a little code, a little test, a little code, ...”Tests are added gradually during implementation – not in large lump afterwards
Process of writing tests drives low-level design and programming
Tests specify what code should doTests validate that code does what it should
Actually, a design and coding practiceOne of the core practices of Extreme Programming
Developers have been applying TDD for several decades
35Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
TDD episode (1/2)
Proceeds step by stepWrite a test. Design and implement just enough to make the test pass. Repeat.
Testing and coding alternate in very small steps
Duration of one cycle should be a few minutesSmall steps – difficult to make mistake
Write test
Write code
Run tests
Fail
Pass
36Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
TDD episode (2/2)
Episode is over when you can’t write a failing test anymore
Write test for each requirement of the codeWrite test for each point that can possibly break
One cycle at a timeDon’t write a bunch of tests at once
Refactor if you ever see the chance to make the design simplerRun all tests after finishing episode
Make sure you did not brake anything else
37Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test-Driven Development – claimed benefits (1/2)
Close feedback loopTDD cycle is very short – know if code is working right after you programmed it
Task-orientationEncourage programmer to decompose problem into manageable programming tasksHelps to maintain focus
Low-level designProgrammer is forced to think which classes and methods to createhow they are used, how to name them, what arguments are needed, what is the return value
38Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test-Driven Development – claimed benefits (2/2)
Results better codeIf the test is too hard to write, the code being tested is too complicated
Results testable codeProgrammer can’t end up with code that cannot be tested
Effect on qualityTesting becomes part of the development process and gets doneSide effect of TDD is that code gets thoroughly unit tested
39Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Is Test-Driven Development testing?
What is the goal and attitude of a developer?
Do the tests reveal defects?
What are the tests based on?
How is the quality of the unit tests assured?
What information the tests actually produce?
40Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Unit testing frameworks
Convenient frameworks for writing unit tests as part of the programming task
In the same environment with the same language
Tools of xUnit family are the most famous onesJUnitCppUnitNUnit…
The unit testing frameworks provide a convenient way of writing and executing unit tests
Standard and convenient way to write tests, log results, group tests into suites
41Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Basic building blocks of a unit testing framework
A way of writing test cases for each class, method, function, …Test case is a function or a class – a piece of code
A way of managing suites of test casesA data structure that is used to group test cases into logical collections
Assert methods/functions that enable easy logging and reporting test results
The framework provides standard functions that are used to checkthe test resultsAutomated result logging and reporting provided by the framework
A way of executing test cases and reporting the resultsGUI and command-line based execution tools (Integrated to IDEs)Convenient report generators (Xml, html)
Initializing and cleaning upThe framework ensures that init and clean-up functions are properly executedInitialization (or set up) function to initialize the variables, test data, etc. before executing each test caseClean up (or tear down) function to release the used resources and clean up after executing each test case