21-25 Feb 2001 SIGCSE 2001
Integrating Testing into the Curriculum – Arsenic in
Small Doses
Edward L. JonesCIS Department
Florida A&M UniversityTallahassee, FL USA
21-25 Feb 2001 SIGCSE 2001
Lessons from ACE2000
• Practice (10 slides)
• SPRAE framework, carefully explained
• Arsenic pills:
– 7 things to do
– Automated grading
– Student mentoring
• Tester certification in the experience factory
• On-going work
21-25 Feb 2001 SIGCSE 2001
Motivation
• Industry need
• Testing experience adds value
• Avoid isolationist view – confine to course
• Salt courses with test related experiences
• Provide opportunity for advanced study
• Impact our curriculum
21-25 Feb 2001 SIGCSE 2001
Possible Approaches
• Teach a course in software testing
• Expose student work to rigorous testing
• Train selected students
• Opportunistic insertion of testing into existing courses
• Formal training environment
21-25 Feb 2001 SIGCSE 2001
The Holistic Approach
Software Test Lab
Core Curriculum
Elective Testing Course
Testing In Action (Automated Grading)
SPRAE Testing Framework
21-25 Feb 2001 SIGCSE 2001
What is Meant by Holistic
• Unifying framework – what’s important
• Experience-based
• Experiences aligned to framework
• Multiple experiences in different contexts
• Multiple means of delivery
21-25 Feb 2001 SIGCSE 2001
The SPRAE Framework
SSpecification – the basis for testing
PPremeditation – a systematic process
RRepeatability – tester independence
AAccountability – documented process, results
EEconomy – cost-effective practices
21-25 Feb 2001 SIGCSE 2001
Test Life Cycle
Analysis
Design
Implementation
Execution
Evaluation
Specification
Test Plan
Test Script, Data, Driver
Defect DataProblem Reports
Test Results, Log
Test Cases
21-25 Feb 2001 SIGCSE 2001
The Software Testing Course
• 80% practice, 20% theory
• 2/3 fine-grained testing (functions)
• 1/3 object and application testing
• Test cases, drivers, and scripts
• Decision tables the "formalism" of choice
• Functional, boundary, white-box testing
• Evaluation via coverage & error seeding
21-25 Feb 2001 SIGCSE 2001
Course -- Lessons Learned
• Advantages– In-depth, continuous concept treatment – Complement to other software skills – Basis for future advanced work
• Deficiencies– Why Johnny can’t test? Programming skills – Not a mainstream course available to all– Students compartmentalize course content
21-25 Feb 2001 SIGCSE 2001
Testing in Action – Automated Program
Grading
Prepare Assignment Implement Grader Grade Programs
Write Program
Grading Log
Grade Report
Student ProgramAssignment
Specification
Assignment Specification
Test plan
Test cases
Test driver
Checker script
Automated Grader
Overhead
21-25 Feb 2001 SIGCSE 2001
Results -- Automated Program Grading
• Student shock & outrage at exactness• Behavior modification – tester mindset • Extra work on teacher
– Specification must be better than average!– Practice what you preach (test process)– Cost amortized via similar assignment styles
• Is grader too strict for CS1/CS2?
• Selling colleagues on the idea!
21-25 Feb 2001 SIGCSE 2001
Opportunistic Insertion into Existing Courses
• Risk of diluting course content• Hard to transfer to colleagues (comfort
zone with subject matter) • Value-Added approach
– Testing strengthens other skills– Testing brings objectivity to student
• Some examples
21-25 Feb 2001 SIGCSE 2001
Simple Things to Try
• Grade another student’s program and justify grade -- in writing.
• (Group) Develop test cases before writing program.
• Treasure Hunt. Find seeded errors, document process followed, give evidence of fix.
• Develop and sell certified components to be used in subsequent assignments.
• Blind testing. Write specification from executable.
21-25 Feb 2001 SIGCSE 2001
Mentoring Students - Experiences
• Hand picked students -- paid• Series of testing projects• Skills taught as needed• Student access to examples from courses, other
students• MANAGEMENT NIGHTHMARE!• Structured environment and process needed!
21-25 Feb 2001 SIGCSE 2001
The Software TestLab
• Environment for discovery & learning
• An evolving laboratory • Tools & Tutorials• Staff (students, faculty)• Test problem/solution test bed
• Students participate in the evolution
• Feedback on lab resources• Create/Refine resources
• Technology insertion into classroom
Vison
21-25 Feb 2001 SIGCSE 2001
TestLab -- The Big Picture
• Marketing • Interns• New Hires
Support
Curriculum
Students
SoftwareTestLab
ResearchPublications
Corporate Sponsors
21-25 Feb 2001 SIGCSE 2001
Student Mentorship Model • Manage skill development
• Set clear achievement goals
• Key Practices x Competency Levels
• Certify levels of progression
• Enable student-student mentoring
• Establish recognition program
Mentorship
21-25 Feb 2001 SIGCSE 2001
Key Practices
• Practitioner - performs defined test.
• Builder - constructs test “machinery”
• Designer - designs test cases.
• Analyst - determines test needs, strategy.
• Inspector - verifies correct process, results.
• Environmentalist - establishes & maintains the test environment.
• Specialist - performs entire test life cycle.
Mentorship
21-25 Feb 2001 SIGCSE 2001
Competency Levels
Practitioner
Test Designer
Test Analyst
Test Inspector
Test Environmentalist
Test SPECIALIST
1
Test Builder 1
1
1
1
1
1
2
2
2
2
2
2
2
3
3
3
3
3
3
3
4
4
4
4
4
4
4
5
5
5
5
5
5
5
Test Practitioner
Mentorship
21-25 Feb 2001 SIGCSE 2001
Test Specialist I
• Practitioner I - Run function test script & document test results.
• Builder I - Develop function test driver.
• Designer I - Develop functional and boundary test cases from decision table.
•Analyst I - Develop decision table from functional specification.
• Inspector I - Verify adherence to function test process and standards.
Mentorship
21-25 Feb 2001 SIGCSE 2001
Certification Testbed
• Repository of testing problems
• Repository of students’ test artifacts
• Best in class promoted to solutions testbed
• Deficient solutions used for future tester certification
Infrastructure
21-25 Feb 2001 SIGCSE 2001
Conclusions
• Testing must compete for air time with existing subject matter
• Opportunities to insert testing exist
• Testing can bring added value to course
• Need rigorous study of value-added hypothesis
• Biggest job may be selling colleagues
21-25 Feb 2001 SIGCSE 2001
On-Going & Future Work
• Evolve TestLab Mentorship Model – Experience Ladder & Certification– Evolving Problem/Solution Artifacts
• Careful study of value-added hypothesis
• Exploit automated grading – student mindset
• Disseminate Results
21-25 Feb 2001 SIGCSE 2001
Training Sequence (1)
• TestLab Environment Basic Training
• Unix basics
• C++/language refresher
• Encapsulation of function under test
• Repositories
Infrastructure
21-25 Feb 2001 SIGCSE 2001
Training Sequence (2)
• Black-Box Function (unit) Testing• Specification• Stimulus-response test cases• Function Test driver (5 styles)• Test driver input/results files• Test script (set-up + perform)• Test log
• Decision tables• Functional (partition) test case design• Boundary test case design
Infrastructure
21-25 Feb 2001 SIGCSE 2001
Training Sequence (3)
• Black-Box Object Testing• Specification of object’s methods
• Analysis of method’s stimulus-response
• Test planning
• Test cases = method + stimulus + response
• Object Test driver
• Object Test driver input/results files
• Operational test scenarios
Infrastructure
21-25 Feb 2001 SIGCSE 2001
Training Sequence (4)
• White-Box Function Testing• Control flow graph basics
• Coverage criteria/measures
• Instrumentation for data collection
• Use of in-house coverage tools
• Coverage analysis
• White-box coverage during black-box test
• Supplemental white-box test cases
Infrastructure
21-25 Feb 2001 SIGCSE 2001
Training Sequence (5)
• Clear-Box Object Testing• Goal is to overcome information hiding
• Test windows into internal object state
• set_state ( … )
• get_state ( … )
• Test case • Stimulus: precondition + method-stimulus
• Response: method-response + postcondition
• Clear-box object test driver
• Clear-box test oracles
Infrastructure
21-25 Feb 2001 SIGCSE 2001
Training Sequence
• TestLab Environment Basic Training
• Black-Box Function (unit) Testing
• Black-Box Object Testing
• White-Box Function Testing
• Clear-Box Object Testing
Infrastructure
21-25 Feb 2001 SIGCSE 2001
TestLab Infrastructure
• SPRAE Framework / Lifecycle
• Software Testing Course
• Training Sequence
• Standards & Techniques
• Student Mentorship Model
• Problem & Solution Testbeds
Status
21-25 Feb 2001 SIGCSE 2001
Standard Products
• Specification (narrative, semiformal)
• Decision Table
• Test Plan
• Test Script
• Test Driver
• Test Driver Input File
• Test Results File
• Test Log
Infrastructure
21-25 Feb 2001 SIGCSE 2001
Techniques
• Functional Equivalence Partitioning
• Boundary Value Analysis
• Function Encapsulation
• Control Flow Analysis
• Error-seeding for tester certification
Infrastructure
21-25 Feb 2001 SIGCSE 2001
What Makes It Holistic?
• Testing an integral part of curriculum
• Goal is multiple test experiences
• At least one experience in each course
• Repetition and reinforcement
• Accumulation of experiences
• Coverage of test lifecycle
21-25 Feb 2001 SIGCSE 2001
Course Outline
1 Course Overview2 Software Quality Assurance3 The Practice of Testing4 Specification-Driven Testing5 Boundary Testing6 Measuring Test Effectiveness7 Testing Object-Oriented Software8 Application Testing9 Course Review & Wrap-Up
21-25 Feb 2001 SIGCSE 2001
Automation Issues
• Does the teacher have the time?
• Is grader too strict for CS1/CS2?
• Additional automation to lower cost.
• The trap: “just a little more automation”
• Selling colleagues on the idea!
21-25 Feb 2001 SIGCSE 2001
Outline
• The Holistic Approach
• The SPRAE Testing Framework
• The Software Testing Course
• Automated Program Grading
• Opportunistic Insertion
• The Software Test Lab
• Conclusions / Future Work
21-25 Feb 2001 SIGCSE 2001
Example - Pay (S)
Specification: Compute pay for an employee, given Hours worked and hourly pay Rate; overtime is 1.5 times hourly Rate, for Hours above 40.
Hours
Rate
PayComputePay
21-25 Feb 2001 SIGCSE 2001
Principle P
Premeditation: Use a systematic process to devise test cases based on the specification.
Our Technique:• Decision analysis -- identify behaviors• One test case per behavior• Determine expected result
21-25 Feb 2001 SIGCSE 2001
Example - PayTest Case Design: Decision Table. Columns identify behaviors to test.
Decision Table Behaviors
Hours > 40 Y N Conditions
Pay = Hours * Rate X Actions
Pay = 40 * Rate + 1.5 *Rate * (Hours – 40)
X
10 10 Rate
40 50 Hours
400 550 Expected Pay
Test Cases 1 2
21-25 Feb 2001 SIGCSE 2001
Principle R
Repeatability: Processes for test case creation and test execution must yield equivalent results, independently of the tester.
21-25 Feb 2001 SIGCSE 2001
Principle AAccountability: Keep records that document test process and artifacts.
Documentation answers:• What tests were planned?• Which tests were conducted?• Who did what testing, and when?• What were the results?• How were the results interpreted?
21-25 Feb 2001 SIGCSE 2001
Example - PayRepeatability/Accountability:
Test Script Test Log
User Action: Enter
ExpectedResult Test ID /Outcomes
Step Rate Hours Pay 01 021 10 40 400 P P2 10 50 550 F P
Error/Discrepancy LogTest Step Description
01 2 Result Pay=500. OT not calculated.
Top Related