Agile TestingAn Introduction To The Issues

By

Richard Neeve

<Date removed to protect client identity>

Question 1

“Do testing specialists have any role to play in an Agile world, or should their direct

involvement be ruled out?”

Bias Alert!!!

• Turkeys tend not to vote for Christmas.• So bias is somewhat inevitable.

But….

• I’ve always said that the ultimate success for a test team is to be redundant.

• Some of my most enjoyable work experience was in a test team that was specifically tasked with making itself redundant.

So hopefully I have been adequately objective in my analysis.

Arguments against involvement

• Classifying Agile team members as developers and testers can undermine the sense of collective ownership.

• Many Agile projects have managed without.• Could threaten agility.

No doubt there are other arguments….

Arguments for involvement

• Can bring a tester’s perspective. Valuable if managed properly.• Some customers are asking for it. An emerging trend?• Leading Agile proponents are advocating it.• Some test requirements still require a conventional approach (maybe).• Cynics argue that Agile is just the latest re-packaging of old ideas so

the need for testers is unchanged.

And perhaps the most powerful argument is:

• To rule out test specialists is to put ‘process over people’ which goes against Agile values.

Again there are probably other arguments….

Summary

• Agile deliveries need testing. How? By whom?

• Matter of debate (even amongst testers).

• Market will decide. Jobserve and anecdotes suggests career testers are increasingly getting involved.

• Turkeys don’t vote for Christmas so within the testing community the debate centres on ‘how?’ rather than ‘by whom?’

• Bias is inevitable, but I was genuinely more persuaded by the arguments for involving testers.

But ultimately….

• Customers buy the product not the method, so unless the customer insists on it, we should only do it if we believe it will be valuable.

Question 2

“If we do see some potential for the involvement of testing specialists, how specifically might

they contribute?”

• Providing ‘testability’ consultancy from the outset.• Doing tests that are highly specialised and/or less amenable to

a test-first paradigm (e.g. performance).• Providing continuous critical thinking.• Protecting the customer interest and/or facilitating the customer

relationship.• Consulting on test automation and reporting.• Supporting the project team as required e.g.

– Taking responsibility for the purity of the test environment.

– Helping to elaborate user stories.

_________________________________

• No doubt there are many other possibilities.• Developers could do all of this but is that the best use of skills?• Demands very good technical and soft skills.

Question 3

“If it is accepted that testing specialists can potentially make a useful contribution, what

are some of the potentially relevant techniques that the test domain can offer?”

• Burgeoning area of discussion within the testing community.• Some techniques summarised below (some probably new to <client

name>).• Each one warrants its own presentation.

– Exploratory testing• Simultaneous learning, test design and test execution (not just

giving it a bash).• Unscripted. Requires very specific mental skills.

– Session-based test management• Lightweight management framework for exploratory testing to

provide minimal measurement and accountability.

– Pair testing• Testing equivalent of pair programming.• (2 testers) or (1 tester +1 developer).

– Pairwise testing• Identifies combinations of variables in the input vector that are

statistically most likely to yield a defect.

– Good enough testing• A decision-making framework to maintain focus on doing the

bare minimum.

– FIT (Framework for Integrated Testing)• Defining testable behaviours in tabular format to facilitate rapid

processing.

– Genetic algorithms• Use evolutionary computation to automatically derive tests.

– Chance discovery algorithms• Seek to identify relationships within natural language text.

– Conventional (i.e. scripted) testing• Needed for areas that are controversial or need

customer/management/regulatory approval.• Useful when a high degree of repeatability is required (e.g.

benchmarking).

Apart from conventional testing, these techniques support Agilevalues by:

• Enabling the low-cost identification/generation of test cases. • Helping to absolve us of burdensome paperwork. • Helping us gain the maximum value from the minimum effort.

______________________________

• Scope to apply these techniques in combination.• Learning curve for each (training required).• Need to keep up to date with the latest thinking (through SIGIST etc).

Question 4

“What are some of the values/philosophies by which we might want any Agile testing to be

guided?”

• Could use an Agile testing manifesto. Jonathan Khol suggests:– Bug advocacy over bug counts.– Testable software over exhaustive requirements docs.– Measuring product success over measuring process success.– Team collaboration over departmental independence.

• Agile testers need to be service oriented.

• Purpose of the tests should be to smooth the path of development, rather than to scrutinize the deliverables in a critical way.

• Need collective ownership of testing. Everyone should write tests and develop test support code.

• Some advocate aiming to make defect tracking redundant (doesn’t seem to be a widely held view).

_______________________

• Some interesting ideas there (I don’t like the last one).

• Using tests to smooth development is probably the biggest mindset change for conventional testers.

Question 5

“What experiences (if any) could we draw on from our existing practices to help us ease

any future transition towards Agile testing?”

Anecdotally, conventional testers struggle with:

• Adapting to an apparent loss of independence from the developers.

• The social aspects arising from the increased level of collaboration.

• Losing the comfort zone provided by scripted tests.

• Adopting a more servile role in relation to the developer.

Don’t get the idea that “we’re well on the way already” but some aspects of

our current practices should mitigate these issues. For example:

• There are regular, focused meetings between dev and test.

• There are lots of 121 investigative discussions between development and test engineers prior to defect submission.

• We have lots of experience with unscripted testing.

Question 6

“How exactly would we transition to a point where we can make Agile testing a reality?”

• Depends entirely on which of these ideas get taken forward.• Careful planning and management required.

_______________________

• The accompanying notes provide a reference to an explanation of Kurt Lewin’s ‘freeze phases’ and their associated techniques:

– Unfreeze

– Transition

– Refreeze

• Lewin’s metaphor neatly encapsulates the journey we will have to undertake in order to successfully embrace Agile testing.

Question 7

“If we were to attempt a move to Agile testing, what are likely to be the key challenges along

the way?”

Again it would depend on which of these ideas we were trying to

implement but:

• We need to get to a point where developers test out of desire and not duty.

• We need to support testers (and developers) in making the required changes in their skills, outlook and approach.

• We need to review what qualities/attitudes/skills we look for when recruiting staff.

Other Observations

Some of these are obvious in hindsight but worth bearing in mind:

• Risk-based testing is self-correcting over time.• Not all testing should be risk-based in order to mitigate the risk

of a poor risk analysis.• Automated testing does not provide automatic manual testing.• The Agile community doesn’t seem to have much advice about

how organizations should manage Agile and non-Agile development streams that are co-dependant to some degree.

• Given that the new code we produce today stands a good chance of becoming legacy code tomorrow, we should aim to produce solutions that are amenable to ‘strangling’.

Conclusions• Test specialists have a role to play but overtly labeling them as

‘testers’ in an Agile team might not be wise.• Lots to consider in achieving a successful (and enduring) move

towards Agile testing.• As with test automation and test asset management, the issues

are numerous and often non-technical, whist the scope for failure is large.

• We will need to invest a lot of thought if we are to have a high probability of success.

• Genuine success will require some very radical changes.• There will lots of mistakes and frustration along the way.• It will probably take many years to get it right.• There is a lot of potential for testing to become more interesting.• We need to start the transition now.

Next Steps

• Addressing the issues presented here at a management level.• Training people in Agile testing techniques.• Piloting Agile testing techniques where possible.• More research into the current state of the art.• Maintaining exposure to the latest thinking via internal and

external forums.