Efficient Techniques for Software Testing Jeff Lei CSE@UTA April 13, 2007.
-
Upload
juliet-chapman -
Category
Documents
-
view
217 -
download
0
Transcript of Efficient Techniques for Software Testing Jeff Lei CSE@UTA April 13, 2007.
Efficient Techniques for Software Testing
Jeff Lei
CSE@UTA
April 13, 2007
Outline
• Software Testing• Concurrency Testing
– Testing Concurrent Programs– Testing Synchronization Components
• T-Way Interaction Testing• Other Work
– Model Checking, Protocol Validation, Pervasive Computing
Software Engineering
• Software has become pervasive in modern society– Directly contributes to quality of life– Malfunctions cost billions of dollars every year,
and have severe consequences in a safe-critical environment
• All about building better software in better ways, especially for large-scale development– Requirements, design, coding, testing,
maintenance, configuration, documentation, deployment, and etc.
Software Testing
• A dynamic approach to detecting software faults– Alternatively, static analysis can be performed,
which is however often intractable
• Involves sampling the input space, running the test object, and observing the runtime behavior
• Perhaps the most widely used approach in practice– Labor intensive, and often consumes more than
50% of development cost
Research Problems
• How to detect as many faults as possible spend with as little effort as possible?– How to generate a good set of test inputs?– How to perform actual test executions?– How to evaluate test outcomes?
Outline
• Software Testing• Concurrency Testing
– Testing Concurrent Programs– Testing Synchronization Components
• T-Way Interaction Testing• Other Work
– Model Checking, Protocol Validation
Concurrent Programs
• Consists of multiple threads of execution that proceed in parallel– multithreaded programs, multi-process (or
distributed) programs
• Advantages: – Better resource utilization, increased computation
efficiency, and providing simplified solutions for many problem domains
• Notoriously difficult to test due to their non-deterministic behavior
What Is Unique
• Non-determinism– Multiple executions with the same data input might
display different behaviors
• Synchronization– How to ensure that concurrent events are
exercised in an expected order?
• Communication– How to exchange information between different
threads?
Concurrency Testing Strategies
• Non-deterministic testing - execute the same program with the same input for multiple times:– easy, but inefficient, and some errors cannot be detected
• Deterministic testing - select a set of test sequences and then force them to be exercised:– can detect more subtle errors, but requires additional effort
for test sequence selection and runtime control
• State exploration – explore the state space in a systematic manner– suffers the well-known state explosion problem– state representation is difficult for programs written in a full-
fledged programming language
Reachability Testing
• Combines non-deterministic and deterministic testing– Generating test sequences dynamically, without
constructing any static model– Dealing with partial orders directly, and thus no
redundant interleavings!
• Can also be considered as a state exploration technique– Exercising every possible synchronization
sequence of a program with a given input
The Framework
1. Execute a program P with a given input non-deterministically to collect a trace Q
2. Identify the race conditions in Q and compute its race variants
3. For each variant, conduct a prefix-based test run to collect a new trace Q’
4. Repeat 2, 3, and 4 for each newly collected trace Q’.
Example
Main Results
• Developed three reachability testing algorithms – Exhaustive RT, Combinatorial RT, and Random
RT
• Built a reachability testing tool called RichTest– Portable implementation, applicable to most
common synchronization constructs
• Conducted an empirical evaluation– Performed significantly better than VeriSoft (from
Bell Labs)
• Publications: STVR07a, TSE06, ISSRE05
Dinning Philosophers
Outline
• Software Testing• Concurrency Testing
– Testing Concurrent Programs– Testing Synchronization Components
• T-Way Interaction Testing• Other Work
– Model Checking, Protocol Validation
Synchronization Components
• Synchronization logic is often encapsulated into one or more components
• Synchronization components need to be tested separately– Obtain necessary confidence prior to integration– helps to reduce complexity by incremental testing
• A different problem than testing complete programs– The ways in which a component could be used is
typically unknown
Java Monitor
• A high-level synchronization construct that supports data encapsulation and information hiding– A Java class that defines one or more
synchronized methods
• Mutual exclusion automatically enforced by the Java runtime
• Conditional synchronization implemented by the programmer, using wait and notify operations
Java Monitor Semantics
• Consists of three components: an entry queue, a critical section, a waiting queue
• A thread must enter the CS before it executes a synchronized method; it has to wait in the entry queue if the CS is not empty
• The wait and notify operations can only be executed inside the CS– wait: puts the current thread in the waiting queue– notify: awakens one thread in the waiting queue
Example
class BoundedBuffer { private int fullSlots=0; private int capacity = 0; private int[] buffer = null; private int in = 0, out = 0; public BoundedBuffer(int bufferCapacity) {1. capacity = bufferCapacity; 2. buffer = new int[capacity]; } public synchronized void deposit (int value) {3. while (fullSlots == capacity) {4. try { wait(); } catch (InterruptedException ex) {} } 5. buffer[in] = value;6. in = (in + 1) % capacity;7. if (fullSlots++ == 0) {8. notifyAll(); } } public synchronized int withdraw () {9. int value = 0;10. while (fullSlots == 0) {11. try { wait(); } catch (InterruptedException ex) {} } 12. value = buffer[out];13. out = (out + 1) % capacity;14. if (fullSlots-- == capacity) {15. notifyAll(); }16. return value; } }
Graphic View
Critical Section
Entry Queue
W
Condition Queue
D
Assume that the buffer is initially empty.
Research Questions
• What is unique to testing Java monitors?– Used to synchronize, and thus typically accessed
by, multiple threads simultaneously
• Can existing work on OO testing apply?– OO testing methods typically assume a single-
threaded test driver
• How to effectively test Java monitors?
Technical Challenges
• To simulate possible scenarios in which a Java monitor can be accessed, multiple threads need to be created– Some faults can only be detected by a certain
minimum number of threads. But how many?– Multiple threads can display non-deterministic
behavior. How to describe and control thread behavior?
A State Exploration-Based Approach
• Explores the state space of a monitor as an open system– Threads introduced on-the-fly, and as needed, to
simulate race conditions– State abstraction used to ensure termination and
to control state explosion– Thread behavior is controlled at finer granularity
than method calls
Main Results
• Developed an algorithm that implements the state exploration-based approach
• Built a prototype tool, called MonitorExplorer, and conducted an empirical evaluation– Empirical results suggest that this approach can
be very effective
• Ongoing work: Extend to general monitors, and general synchronization components
• Publication: ISSRE06
Outline
• Software Testing• Concurrency Testing
– Testing Concurrent Programs– Testing Synchronization Components
• T-Way Interaction Testing• Other Work
– Model Checking, Protocol Validation, Pervasive Computing
Combinatorial Testing
• Creates tests by combining different parameter values– The input space of a test object is represented as
a set of parameters and their values
• Advantages: light specification, test input generation can be fully automated, requires no access to source code
• How to deal with the combinatorial explosion problem?
T-Way Interaction Testing
• Motivation: Not every parameter contributes to every fault– Many faults can be exposed by interactions
involving a few parameters
• Given any t parameters, every combination of values of these parameters be covered by at least one test– Allows faults triggered by at most t parameters to
be detected
• Makes an excellent trade-off between test effort and test coverage
T-Way Testing - Example
P1 P2 P3 0 0 0 0 0 1 0 1 0 0 1 11 0 01 0 11 1 01 1 1
P1 P2 P3 0 0 0 0 1 1 1 0 11 1 0
Three parameters, each with values 0 and 1
2-way testing
exhaustive testing
State of the Art
• Greedy construction: Each step tries to cover as many combinations as possible– Involves explicit enumeration of all possible
combinations
• Algebraic Construction: test sets are constructed using pre-defined rules
• Most approaches focus on 2-way (or pairwise) testing
Strategy In-Parameter-Order
• Input: A set of parameters and values; Output: a t-way test set
• Construct a t-way test set for the first t parameters
• Extend the test set to cover each of the remaining parameters one by one– Horizontal growth - extends each existing test by
adding one value for the new parameter– Vertical growth – adds new tests, if needed, to
make the test set complete
Example
(a)
P1 P2 P30 0 0 0 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1
(b)
P1 P2 P3 P40 0 0 0 0 0 1 10 1 0 20 1 1 01 0 0 1 1 0 1 2 1 1 0 01 1 1 1
P1 P2 P3 P40 0 0 0 0 0 1 10 1 0 20 1 1 01 0 0 1 1 0 1 2 1 1 0 01 1 1 11 0 1 0 0 1 0 1 0 0 1 2 1 1 0 2 * 0 0 2 * 1 1 2
(c)
Horizontal growth Vertical growth
• Four parameters: P1, P2, P3, and P4• P1, P2, and P3 have 2 values• P4 has 3 values
3-way test set
Main Results
• Developed two algorithms, IPOG and IPOG-D, for general t-way testing– Has the lowest algorithmic complexity among the
existing algorithms
• Built a t-way testing tool called FireEye, and conducted an empirical evaluation– Performed significantly better than existing tools,
e.g., ITCH, TConfig, Jenny, TestVector
• Publications: TSE02, ECBS07, STVR07b
TCAS Example
t-way
FireEye ITCH Jenny TConfig TVG
Size Time Size Time Size Time Size Time Size Time
2 100 0.8 120 0.73 108 0.001 108 >1 hour 101 2.75
3 400 0.36 2388 1020 413 0.71 472 >12 hour 9158 3.07
4 1361 3.05 1484 5400 1536 3.54 1476 >21 hour 64696 127
5 4219 18.41 NA >1 day 4580 43.54 NA >1 day 313056 1549
6 10919 65.03 NA >1 day 11625 470 NA >1 day 1070048 12600
TCAS: Seven 2-value parameters, two 3-value parameters, one 4-value parameter, two 10-value parameters
Outline
• Software Testing• Concurrency Testing
– Testing Concurrent Programs– Testing Synchronization Components
• T-Way Interaction Testing• Other Work
– Model Checking, Protocol Validation, Pervasive Computing
Other Work
• Model checking: How to deal with the state explosion problem?– True concurrency model vs interleaving-based concurrency
model– Publication: FM05
• Protocol validation: A more specific problem than model checking– Allows multiple threads to proceed simulatenously during
state exploration– Publication: CJ06
• Pervasive computing– Context-awareness: How to derive high level events from a
sequence of low level events?– Power-constraints: How to reduce the power consumption
on small mobile devices?
References
• [STVR07a] Y. Lei, R. Carver, R. Kacker, and D. Kung. A Combinatorial Testing Strategy for Concurrent Programs. Accepted for publication in Software Testing, Verification, and Reliability, 2007.
• [STVR07b] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. Two deterministic strategies for multi-way software testing. Software Testing, Verification, and Reliability, pending review.
• [ECBS07] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. A general strategy for multi-way software testing. Int’l Conf. on Engineering of Computer-Based Systems, 2007.
• [ISSRE06] Yu Lei, R. H. Carver, D. Kung, V. Gupta, M. Hernandez. A State Exploration-Based Approach to Testing Java Monitors. Int’l Symposium Software Reliability Engineering, 256-265, 2006.
• [CJ06] Q. Ye, Y. Lei, and D. Kung. A Blocking-Based Approach to Protocol Validation, The Computer Journal, 49:541-553, 2006.
• [TSE06] Y. Lei and R. Carver, Reachability testing of concurrent programs, IEEE Transactions On Software Engineering, 32(6):382-403, 2006.
• [FM05] Y. Lei and P. Iyer. An Approach to Unfolding Asynchronous Communication Protocols, Proc. 13th Intl. Symp. on Formal Methods, pp. 334-349, 2005.
• [TSE02] K. C. Tai and Y. Lei, "A test generation strategy for pairwise testing", IEEE Transactions on Software Engineering, Vol. 28, No. 1, pp. 109-111., Jan. 2002.