Testing SAP:Modern Testing Methodologies

Ethan Jewett

But first!

What’s the problem?

“Enterprise” project trade-offs

Value

Risk

Duration

Budget

Software development trade-offs

Features

Quality

Time

Budget

Client requests

Build FasterAnd produce more Value

Cheaperwith less Risk

Remember…

Value

Risk

Duration

Budget

Are there hidden assumptions?

Yes.

MethodologyPeople and Culture

Technology

Any other assumptions?Project Preparation

Business Blueprint

Project Realization

Go-live and Support

Technical Design

Technical Build Test Fix Transport Go Live

Write test scripts

Write and run unit test

Integration & Regression

TestTraining Live

Support

The topic

Over the next hour and a half, we’ll talk about how testing methodologies, techniques, and technologies can help us

Change these underlying assumptions.Enable modern project methodologies.

Control the basic project trade-off.

The focusProject Preparation

Business Blueprint

Project Realization

Go-live and Support

Technical Design

Technical Build Test Fix Transport Go Live

Write test scripts

Write and run unit test

Integration & Regression

TestTraining Live

Support

The goal

More valueFaster

With less risk

(and more fun)

For example

V-Model methodology

http://en.wikipedia.org/wiki/File:V-model.JPG

More parallel, less repetition

http://en.wikipedia.org/wiki/File:V-model.JPG

Faster

http://en.wikipedia.org/wiki/File:V-model.JPG

More iterative

http://en.wikipedia.org/wiki/File:V-model.JPG

How do we get there?

Modern methodology

• Modern quality assurance and testing– Leading practice in custom development, open

source, and agile projects• Test driven development (TDD), behavior driven

development (BDD), readable tests, executable requirements, continuous integration

– Overview of the SAP test automation technologies– Road we're going to take through testing SAP

Leading Practice

• Context• Questions and problems• Approaches• Tools

Leading Practice

• Context• Questions and problems• Approaches• Tools

Context

The context of modern software development is above all open, agile and decentralized.

Open source projects are usually the furthest along this path, and as such these projects use (or are forced to invent) tools that fit their processes.

Openness

Requirements, specifications, issues reports, test cases, source code, and project tools and metrics are readily open and available to all participants in the project.

(Not necessarily the same as “open source”.)

Open Collaboration within Corporations Using Software Forges - http://www.riehle.org/publications/2009/open-collaboration-within-corporations-using-software-forges

Agility

The values of the Agile Manifesto:– Individuals and interactions over processes and

tools– Working software over comprehensive

documentation– Customer collaboration over contract negotiation– Responding to change over following a plan

http://agilemanifesto.org/

Decentralization

Open-source projects are decentralized by nature, but most organizations today have some element of decentralization

Challenges:– Quality control– Communication– Maintaining commitment

Leading Practice

• Context• Questions and problems• Approaches• Tools

Manual or automatic testing?

How do we ensure that the build complies with

the tests?

How does testing integrate with issue tracking?

Questions and problems

How do we enforce disciplined

testing?

How do we track 100s or 1000s of bugs

(including duplicates)?

How do we determine the cause

of test failure?

How do we make tests relevant?

Do we use unit, functional, integration, or acceptance testing?

Or all of them?

Leading Practice

• Context• Questions and problems• Approaches• Tools

Approaches

Test CoverageTest AutomationTest Driven DevelopmentBehavior Driven Development“Spec as Test” (the test/spec convergence)Exploratory TestingContinuous Integration

Test coverage

The percentage of code or development that is tested.

In code, this might be measured by determining if every branch of code can result in a failing test condition (“Branch coverage”)

Wikipedia - http://en.wikipedia.org/wiki/Code_coverage

Test automation

• Accepted as gospel modern dev communities– Regardless of how good they are at testing, accept

that they should automate as much as possible– Can lead to ignoring non-automatable testing

• In the SAP world we haven’t even accepted that full test coverage is desirable, much less automation

Test automation pushback -http://railspikes.com/2008/7/11/testing-is-overratedhttp://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html

Automated testing story on SDN (never completed) - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4103

A note on the following slides

I’m using bowling as an example because I’m copying from the internet. I don’t know much about bowling, so the examples are probably wrong. My assumption is that a game is usually 10 frames of two balls per frame. If the last frame is a spare or a strike, you get at most two additional frames.

Or something like that.

Test Driven Development (TDD)

Design

• Technical specification

Design

• Write tests based on specs

Design

• Run tests

Test

TDD Example

require ‘test/unit’require ‘test/unit/assertions’require ‘bowling’

class TC_Bowling < Test::Unit::TestCase

def setup bowling = Bowling.new end

def gutter_game 20.times { bowling.hit(0) } assert bowling.score == 0 endend

Behavior Driven Development (BDD)

• Focus on the external behavior of the code• Abstract away implementation details• Additionally, BDD libraries tend to codify the

best practices of unit testing and TDD

BDD example

require ‘spec’require ‘bowling’

describe Bowling do

it "should score 0 for gutter game" do

bowling = Bowling.new 20.times { bowling.hit(0) } bowling.score.should == 0

endend

“Spec as Test” – or writing features

• Recently, the Ruby community has begun developing testing frameworks that are even closer to natural language.– Cucumber

• These frameworks wrap TDD and BDD frameworks and allow for “business-writable”, “business-readable”, executable test scripts.

Feature exampleFeature (visible to the user)

Scenario: Gutter game Given I have bowled 20 balls and I have knocked over 0 pins per ball When I check the score Then I should have 0 points

Implementation (not visible to user)

require ‘spec/expectations’require ‘bowling’

Given /I have knocked over (\d+) pins per ball/ do |pins| @pins_per_ball = pinsend

Given /I have bowled (\d+) balls/ do |balls| @bowling = Bowling.new balls.times { @bowling.hit( @pins_per_ball ) }end

Then /I should have (\d+) points/ do |points| @bowling.score.should == pointsend

Feature example cont.

Note that we can now implement many more tests with no more code/implementation:

Scenario: Perfect game Given I have bowled 12 balls and I have knocked over 10 pins per ball When I check the score Then I should have 300 points

Scenario: Bad game Given I have bowled 20 balls and I have knocked over 2 pins per ball When I check the score Then I should have 40 points

Scenario: Lots of spares

Given I have bowled 24 balls and I have knocked over 5 pins per ball When I check the score Then I should have ??? Points

Or maybe I need something a little different....

Scenario: Too long a game Given I have bowled 30 balls and I have knocked over 0 pins per ball Then I should receive an error

Side-by-sideTDD BDD Spec as Test

require ‘test/unit’require ‘test/unit/assertions’require ‘bowling’

class TC_Bowling < Test::Unit::TestCase

def setup bowling = Bowling.new end

def gutter_game 20.times { bowling.hit(0) } assert bowling.score == 0 endend

require ‘spec/expectations’require ‘bowling’

describe Bowling do

it "should score 0 for gutter game" do

bowling = Bowling.new 20.times { bowling.hit(0) } bowling.score.should == 0

endend

Scenario: Gutter game

Given I have bowled 20 balls and I have knocked over 0 pins per ball

When I view the score

Then I should have 0 points

TDD, BDD, and Test-as-Spec References

• Test Driven Design/Development– http://en.wikipedia.org/wiki/Test-driven_developmen

t– http://www.agiledata.org/essays/tdd.html

• Behavior Driven Development– http://behaviour-driven.org/

• Spec as test– Cucumber - http://cukes.info/– Rspec - http://rspec.info/ – http://www.pragprog.com/titles/achbd/the-rspec-bo

ok

Exploratory Testing

• The practice of trying to break things– Career security for klutzes

• Exploratory testing appears informal, but can be structured and is a very important aspect of software testing.– Probably the most neglected form of testing in

open source projects

http://www.kohl.ca/blog/archives/000185.htmlhttp://www.kohl.ca/articles/ExploratoryTesting_MusicofInvestigation.pdf

Continuous Integration

• The practice of automating not only your tests but your full commit-build-test cycle1. A commit new or changed code2. Triggers a full system build3. And an execution of the entire test suite

Cruisecontrol.rb -http://rubyforge.org/projects/cruisecontrolrbhttp://cruisecontrolrb.thoughtworks.com/

Hudson -https://hudson.dev.java.net/http://www.softwarebloat.com/2008/11/19/continuous-integration-blueprints-how-to-build-an-army-of-killer-robots-with-hudson-and-cucumber

Leading Practice

• Context• Questions and problems• Approaches• Tools

Tools

Source version control (svn, git)Testing libraries (rspec, cucumber, junit,

ABAPunit, jspec)Continuous Integration tools (cruisecontrol.rb,

Hudson)Issue tracking (Sourceforge, trac, Lighthouse,

Google Code)Agile methodologies (Scrum, XP, etc.)

Modern methodology

• Modern quality assurance and testing– Leading practice in custom development, open

source, and agile projects• Test driven development (TDD), behavior driven

development (BDD), readable tests, executable requirements, continuous integration

– Overview of the SAP test automation technologies

– Road we're going to take through testing SAP

The SAP World

• Manual• Unit testing– ABAP Unit– Java

• Functional testing– eCATT

ABAP Unit

• Modeled on Java Unit• Excellent (if a bit unwieldy) for traditional unit

test automation or classes– Works well for TDD

• ABAP Objects (object-oriented ABAP)

• TDD Demo – ZBOWLING & ZBOWLING_TEST

http://help.sap.com/saphelp_nw70ehp1/helpdata/en/a2/8a1b602e858645b8aac1559b638ea4/frameset.htm

Non-SAP

• We can use just about anything via RFCs• For Web Dynpros, BSPs, or Portal applications,

there are a lot of options using web-drivers that can automatically drive browsers

• Address in depth in future sessions

Modern methodology

• Modern quality assurance and testing– Leading practice in custom development, open

source, and agile projects• Test driven development (TDD), behavior driven

development (BDD), readable tests, executable requirements, continuous integration

– Overview of the SAP test automation technologies– Road we're going to take through testing SAP

The Road to the Future

• Developing techniques to support agile, open, decentralized development in an SAP landscape

• Using SAP tools and 3rd party tools• Shorter, more focused and hands-on sessions

Start with this

http://en.wikipedia.org/wiki/File:V-model.JPG

Get more parallel, less repetitive

http://en.wikipedia.org/wiki/File:V-model.JPG

Spec = Test

Get faster

http://en.wikipedia.org/wiki/File:V-model.JPG

Test Automatio

n

Get iterative

http://en.wikipedia.org/wiki/File:V-model.JPG

Continuous

Integration

And end up here(or somewhere similar)

http://en.wikipedia.org/wiki/File:Scrum_process.svg

Thanks

General references for inclusion• Open Collaboration within Corporations Using Software Forges - http://www.riehle.org/publications/2009/open-collaboration-within-corporations-using-software-forges/• Continuous integration tools

– http://rubyforge.org/projects/cruisecontrolrb - http://cruisecontrolrb.thoughtworks.com/ – https://hudson.dev.java.net/– http://www.softwarebloat.com/2008/11/19/continuous-integration-blueprints-how-to-build-an-army-of-killer-robots-with-hudson-and-cucumber

• Exploratory testing– http://www.kohl.ca/blog/archives/000185.html– http://www.kohl.ca/articles/ExploratoryTesting_MusicofInvestigation.pdf

• The ongoing revolution in software testing– http://www.kaner.com/pdfs/TheOngoingRevolution.pdf

• ABAP Unit– Spotlight on ABAP Unit Part 1 - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/1088

• Load testing– http://dannorth.net/2007/02/monkey-business-value

• Joel Spolsky – 12 steps to better code - http://www.joelonsoftware.com/articles/fog0000000043.html• Automated testing story on SDN - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4103• JUnit - http://www.junit.org/ • Open Source

– http://www.paulgraham.com/opensource.html (What business can learn from open source)• Watir

– http://wtr.rubyforge.org/ • Spec as test

– Cucumber - http://cukes.info/– Rspec - http://rspec.info/ – http://www.pragprog.com/titles/achbd/the-rspec-book

• SAP testing– http://www.beteoblog.com/beteo-alm-miniguides/software-quality/– http://raa.ruby-lang.org/project/saprfc/ – Integration Tests in ABAP Development und Tool Support - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/7131 – XSLT - https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/12173

• Test automation as panacea (not)– http://railspikes.com/2008/7/11/testing-is-overrated– http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html

• BDD - http://behaviour-driven.org/• Blue Ruby - https://sap.na.pgiconnect.com/p16473929/ - http://www.slideshare.net/schmerdy/blue-ruby-sdn-webinar-1260181 -

https://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/408a9a3b-03f9-2b10-b29c-f0a3374b19d8