Research in Concurrent Software Testing: A Systematic Review

Systematic Review Planning and Conduction Results Conclusion Bibliography

Research in Concurrent Software Testing:A Systematic Review

Simone Souza1

Maria Brito1 Rodolfo Silva1 Paulo Souza1 Ed Zaluska2

1Universidade de Sao Paulo - Brazil2University of Southampton - UK

[email protected]

PADTAD 2011, Toronto, Canada, July 17, 2011


Outline

1 Systematic Review

2 Planning and Conduction

3 Results

4 Conclusion


Systematic Review

What is a systematic review?

Why do I need of a systematic review?

How to do a systematic review?


Systematic Review

A systematic literature review is a means of identifying,evaluating and interpreting all available research relevant to aparticular research question, or topic area, or phenomenon ofinterest [Kitchenham and Charters, 2007].


Reasons for performing a Systematic Literature Review

Most research starts with a literature review: this review mustbe as complete as possible!

To summarise the available existing evidence for a researcharea

Evidences about: most used programming language, testingtool...

To identify any gaps in current research and suggest areas forfurther investigation


Systematic Review process

This process is based on systematic review used in medicinearea

Kitchenham has adapted this process for Software Engineeringarea [Kitchenham, 2004, Kitchenham and Charters, 2007]


Systematic Review process

1 Planning the reviewIdentification of the need for a reviewSpecifying the research question(s)Definiton of inclusion and exclusion criteriaDeveloping a review protocol

2 Conducting the reviewSelection of the primary studiesData extraction and synthesis

3 Reporting the review


Systematic review planning

1. Definition of the research questions and search strings

The most important part of any systematic review

The research questions drive the entire systematic reviewmethodology

The review results are strictly related to the quality of thesearch strings



1. Research questions defined in our review protocol:

What testing approaches have been proposed to testconcurrent programs?

What fault taxonomies related to concurrent programs havebeen identified?

What tools have been developed to test concurrent programs?



2. Search strings defined in our review protocol:

String 1: ((”parallel program*”OR ”concurrent program*”OR”multithread*”OR ”multi-thread*”)AND (”test*”))

String 2: ((”concurren*”OR ”parallel”) AND (”bug*”OR”defect*”OR ”error*”) AND (”taxonomy”))



3. Selection of the digital source databases:

ACM Digital Library (portal.acm.org)

IEEE eXplore (ieeexplore.ieee.org)

SCOPUS (scopus.com)

CITESEER (citeseerx.ist.psu.edu)



4. Definition of a Control List

Composed of already-known primary studies related to theresearch subject

Used to check if the research results contain all papers presentin this control list

Provides a mechanism to evaluate the effectiveness of thesearch strings



5. Definition of selection criteria1 Inclusion criteria

(IC1) Primary studies that present testing approaches forconcurrent programs(IC2) Primary studies that characterize specific bugs related toconcurrent programs(IC3) Primary studies that propose tools for supportingconcurrent program testing

2 Exclusion criteria(EC1) primary studies that present testing approaches notrelated to concurrent programs(EC2) primary studies that present fault types not related toconcurrent programs(EC3) primary studies that propose testing tools not related toconcurrent program program testing


Systematic review execution

The review was performed between April/2011 and May/2011

Searches returned 1166 primary studies

JabRef1 was used during the selection process

JabRef tool is an open source bibliography reference manager.The native file format used is BibTeX, the standard LaTeXbibliography format.

1http://jabref.sourceforge.net/index.php



Four participants performed the primary studies selection

Two selection interations were performed:

Initial selection: 314 primary studies selectedFinal selection: 175 primary studies selected



Classification of the primary studies:

Testing approach (136)Testing tool (43)Fault taxonomy (7)Others - surveys, benchmarks, experimental studies (5)


Result: Testing approach for multithreaded programs

Technique Static/dynamic References

structural testing static [Yang and Pollock, 2003, Bron et al., 2005, Sung, 1988, Chung et al., 1996,Sarmanho et al., 2008, Yang et al., 1998, Taylor et al., 1992,Takahashi et al., 2008, Yang and Pollock, 1997b, Zhao, 1999,Sherman et al., 2009, Saleh et al., 2001, Lu et al., 2007]

mutation testing,failure injection

static [Artho et al., 2006, Gligoric et al., 2010, Bradbury et al., 2006,Sen and Abadir, 2010]

formal techniques static/dynamic [Wood et al., 2010, Xu and Liang, 2007, Carver and Durham, 1995,Kavi et al., 2002, Chen and RoA§u, 2007, Chen, 2000a, Yang and Sakallah, 2010,Wu et al., 2009, Rakamaric, 2010, Rose et al., 2005]

model checking dynamic [Musuvathi and Qadeer, 2007, Yang et al., 2008, Artho, 2001,Boroday et al., 2005, Chen and MacDonald, 2007, Gradara et al., 2004,Khurshid et al., 2003, Li et al., 2009, Musuvathi and Qadeer, 2008, Sen, 2007,Yang and Sakallah, 2008, Rodrıguez et al., 2004]

model-based tes-ting

static/dynamic [Aichernig et al., 2009a, Aichernig et al., 2009b, Campbell et al., 2005,Chen, 2000b, Seo et al., 2006, Sohn et al., 1999]

static analysis,symbolic analysis

static/dynamic [Kundu et al., 2010, Rungta et al., 2009, Chen et al., 2009, Chen, 2009,Chen et al., 2000, Flanagan et al., 2005]

execution analysis dynamic [Krena et al., 2010, Barnhart et al., 2008, McMinn, 2009, Burckhardt et al., 2010,Sen and Agha, 2006]

testing driven de-velopment

static/dynamic [Dantas et al., 2008a, Ricken, 2007, Ricken and Cartwright, 2009,Jagannath et al., 2010a]

controlled execu-tion

dynamic [John and Chapin, 2000, Cai and Chen, 2000, Yang and Pollock, 1997a,Edelstein et al., 2003, Eytani and Latvala, 2007, Ball et al., 2009a,Dantas, 2008, Dantas et al., 2008b, Emmi et al., 2010, Pugh and Ayewah, 2007,Yu and Narayanasamy, 2009, Wang and Stoller, 2006, Stoller, 2002,Kamil and Yelick, 2010, Rungta and Mercer, 2009, Sridharan and Rinard, 2002,Joshi et al., 2009, Tai, 1989, Carver and Tai, 1989, Carver and Tai, 1991,Devietti et al., 2009, Seo et al., 2001, Burckhardt et al., 2010]


Result: Testing approach for message-passing programs


structural testing static [Yang and Chung, 1992, Yang and Chung, 1990a, Yang and Chung, 1990b,Koppol and Tai, 1996, Koppol et al., 2002, Souza et al., 2008b,Kojima et al., 2009, Krawczyk and Wiszniewski, 1996, Wang et al., 1997,Shih et al., 1996, Katayama et al., 1999, Katayama et al., 1997,Katayama et al., 1995, Katayama et al., 1996, Katayama et al., 1998,Liang et al., 2000, Yuan et al., 2006]

mutation testing static [Jagannath et al., 2010b]

model-based testing static [Chung et al., 1999]

controlled execution dynamic [Oberhuber and Munchen, 1995, Vuduc et al., 2006, Tai and Karacali, 2001,Lei, 2003, Olsson, 1999]

race detection dynamic [Jianxin and Dingxing, 1996]

static analysis static [Christakis and Sagonas, 2011]


Result: Testing approach for both paradigms


reachability testing dynamic [Lei and Carver, 2006, Lei et al., 2007, Carver and Lei, 2004,Carver and Lei, 2010b, Lei and Carver, 2004, Hwang et al., 1995, Li et al., 2004,Wong and Lei, 2008, Carver and Lei, 2010a, Gong et al., 2007, Wong et al., 2005,Chen et al., 2003, Pu and Xu, 2008, Wei et al., 2006]

test case generation static/dynamic [Ding et al., 2008, Tan et al., 2009, Xiaoan et al., 2009]

monitoring and exe-cution analysis

dynamic [Faure et al., 2010, Farchi and Kliot, 2005, Cao et al., 2001, Cao et al., 2006,Karacali and Tai, 1999, Lu et al., 2006]


Result: Concurrent software testing tool I

Language Tool Technique References

AdaCATS reachability analysis [Young et al., 1992]TSG formal verification [Carver and Durham, 1995]TCgen structural testing [Katayama et al., 1998]

MPI

GEM dynamic verification of relevant interlea-vings

[Humphrey et al., 2010]

JitterBug controlled execution [Vuduc et al., 2006]MPIRace-Check race detection [Park et al., 2007]MARMOT race and deadlock detection [Krammer et al., 2004]

PVM/MPI ValiPar structural testing [Souza et al., 2008b]

PVMValiPVM structural testing [Souza et al., 2008a]STEP structural testing [Krawczyk and Wiszniewski, 1996]

C

monitoring tool Ordered Sequence Testing Criterion [Itoh et al., 1996]Inspect model checker [Yang et al., 2008]C2Petri formal verification [Kavi et al., 2002]Direct sensitivity analysis and runtime manager [Chatterjee et al., 2010]

C/C++ViP formal verification [Dingel and Liang, 2004]ConMem race and atomicity detection [Zhang et al., 2010]

C++MultiRace race detection [Pozniansky and Schuster, 2003]ThreadSanitizer race detection [Serebryany and Iskhodzhanov, 2009]

.NET frameworkGAMBIT formal verification [Coons et al., 2010]CHESS systematic testing [Ball et al., 2009b,

Musuvathi et al., 2007,Musuvathi et al., 2008,Musuvathi and Qadeer, 2007,Ball et al., 2009a,Musuvathi and Qadeer, 2008]


Result: Concurrent software testing tool II

Language Tool Technique References

Java

jCUTE race detection [Sen and Agha, 2006]

RichTest reachability testing [Lei and Carver, 2006,Carver and Lei, 2004]

MuTMuT mutation testing [Gligoric et al., 2010]ConTest controlled execution [Edelstein et al., 2003,

Farchi et al., 2003,Copty and Ur, 2005,Krena et al., 2010,Krena et al., 2007]

ThreadControl testing driven development [Dantas et al., 2008a]Kivati atomicity violation [Chew and Lie, 2010]HAVE atomicity violation [Chen et al., 2009]CTrigger atomicity violation [Park et al., 2009]PENELOPE atomicity violation [Sorrentino et al., 2010]DEJAVU replay testing [deok Choi and Zeller, 2002]ConCrash replay testing and unit testing [Luo et al., 2010]Enforcer failure injection [Artho et al., 2006]Java PathFinder,TIE

model checking [Maheswara et al., 2010,Havelund and Rosu, 2001]

CalFuzzer active testing [Joshi et al., 2009, Sen, 2007]Bandera formal verification [Corbett et al., 2000]FindLocks formal verification [Rose et al., 2005]

POSIX Threads API

Kendo deterministic testing [Olszewski et al., 2009]ValiPthread structural testing [Sarmanho et al., 2008]Gadara deadlock detection [Wang et al., 2008]Della Pasta all-du-path testing [Yang et al., 1998]


Result: Concurrent software bugs

Bugs Paradigm References

processor activation errors, processor coordina-tion

multithreaded [Sung, 1988]

errors and computation errors

ordering bug, atomicity violation and deadlock multithreaded [Burckhardt et al., 2010]

interleaving error and assumed never to occur, multithreaded [Farchi et al., 2003]deadlock, code segment unprotected

atomicity-violation or order-violation, multithreaded [Lu et al., 2008]others, bug manifestation, concerns totransacional memory and deadlock

interleaving error, data race error, multithreaded [Lu et al., 2007]wrong def-use relation

faults in monitors: enter, wait, signal-exit multithreaded [Cao et al., 2001]and internal process termination faults;inconsistent states; logic and runtime faults

observability and locking error both [Bogdan et al., 1994]

testing patterns for software multithreaded [Lourenco and Cunha, 2007]transactional memory (consideringtransactions and variables)


Contributions by year


Contributions by research subject


Results


Geography distribution of the authors


Authors Relationship Diagram


Distribution of the primary studies by publication channel

Publication channel Publisher Type Number %

Asia Pacific Software Engineering Conference IEEE conference 9 5,56

International Workshop on Parallel and Distributed Systems: Testing and De-bugging (PADTAD)

ACM conference 8 4,94

International Conference on Architectural support for programming languages andoperating systems

ACM conference 7 4,32

Concurrency and Computation: Practice and Experience Eslevier journal 5 3,09

IEEE Transactions on Software Engineering IEEE journal 5 3,09

International Conference on Software Engineering ACM conference 5 3,09

ACM SIGSOFT Symposium on the Foundations of Software Engineering ACM conference 4 2,47

International Conference on Software Testing, Verification and Validation IEEE conference 4 2,47

International Symposium on Software Engineering for Parallel and Distributed Sys-tems (PDSE)

IEEE conference 4 2,47

Electronic Notes in Theoretical Computer Science Eslevier journal 3 1,85

International Computer Software and Applications Conference IEEE conference 3 1,85

International Conference on Automated Software Engineering IEEE conference 3 1,85

International Conference on Computer Aided Verification CAV LNCS conference 3 1,85

International Conference Fundamental Approaches to Software Engineering (FASE) LNCS conference 3 1,85

International Haifa Verification Conference LNCS conference 3 1,85

International Parallel and Distributed Processing Symposium IEEE conference 3 1,85

ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming ACM conference 2 1,23

ACM SIGSOFT International Symposium on Software Testing and Analysis ACM conference 2 1,23

Asia-Pacific Conference on Quality Software IEEE conference 2 1,23

IEEE International Conference on Systems, Man and Cybernetics IEEE conference 2 1,23

Information and Software Technology Eslevier journal 2 1,23

International Conference on Computational Science LNCS conference 2 1,23

International Conference on Dependable Systems and Networks IEEE conference 2 1,23

International Conference on Formal Engineering Methods LNCS conference 2 1,23


Difficulties and Limitations

Definition of the search string:

the words of the strings must represent the expected primarystudies set (hard task!)the search string must be adapted for each source database

The abstracts, titles and key words do not always representthe subject of the paper

Example: debugging X testing

The selection of the primary studies is done manually


Conclusions

Suggestions for further investigation:

Cost/efficacy evaluation of the proposed testing techniques;Definition of benchmarks to support testing techniqueevaluation;Integration of the testing approaches, considering differenttesting levels;Automatic generation of test data;


Conclusions

The systematic review identifies different groups of authorsworking in important and challenging subjects:

Coverage testing, model checking, controlled execution, bugclassification and testing tools

76% of the primary studies are contributions for the definitionof new approaches for concurrent program testing

The results confirm that the PADTAD Workshop has selectedsignificant papers in current research areas


Thank you for your attention!Simone Souza

Universidade de Sao [email protected]

PADTAD 2011, Toronto, Canada, July 17, 2011


Bibliography I

Aichernig, B., Griesmayer, A., Schlatte, R., and Stam, A. (2009a).

Modeling and testing multi-threaded asynchronous systems with creol.Electronic Notes in Theoretical Computer Science, 243:3–14.

Aichernig, B. K., Griesmayer, A., Johnsen, E., R., S., and Stam, A. (2009b).

Conformance testing of distributed concurrent systems with executable designs.Formal Methods for Components and Objects, 5751 LNCS:61–81.

Artho, C. (2001).

Finding faults in multi-threaded programs.Technical report, Swiss Federal Institute of Technology.

Artho, C., Biere, A., and Honiden, S. (2006).

Enforcer - efficient failure injection.International Symposium on Formal Methods, 4085 LNCS:412–427.

Ball, T., Burckhardt, S., Coons, K. E., Musuvathi, M., and Qadeer, S. (2009a).

Preemption sealing for efficient concurrency testing.Technical report, Microsoft Research, TechReport Number: MSR-TR-2009-143.

Ball, T., Burckhardt, S., De Halleux, J., Musuvathi, M., and Qadeer, S. (2009b).

Deconstructing concurrency heisenbugs.International Conference on Software Engineering - ICSE 2009, pages 403–404.

Barnhart, R., Trefftz, C., Jorgensen, P., and Tao, Y. (2008).

An investigation on testing of parallelized code with OpenMP.International Workshop on OpenMP (2007), Lecture Notes in Computer Science, 4935 LNCS:125–128.


Bibliography II

Bogdan, H. K., Wiszniewski, B., and Mork, P. (1994).

Classification of software defects in parallel programs.Technical report, Faculty of Electronics, Technical University of Gdansk, Poland.

Boroday, S., Petrenko, A., Singh, J., and Hallal, H. (2005).

Dynamic analysis of java applications for multithreaded antipatterns.ACM SIGSOFT Software Engineering Notes, 30:1–7.

Bradbury, J., Cordy, J., and Dingel, J. (2006).

Mutation operators for concurrent Java (J2SE 5.0).Workshop on Mutation Analysis, page 11.

Bron, A., Farchi, E., Magid, Y., Mir, Y., and Ur, S. (2005).

Applications of synchronization coverage.ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPOPP, pages 206–212.

Burckhardt, S., Kothari, P., Musuvathi, M., and Nagarakatte, S. (2010).

A randomized scheduler with probabilistic guarantees of finding bugs.International Conference on Architectural Support for Programming Languages and Operating Systems -ASPLOS, pages 167–178.

Cai, X. and Chen, J. (2000).

Control of nondeterminism in testing distributed multithreaded programs.Asia-Pacific Conference on Quality Software, pages 29 –38.

Campbell, C., Veanes, M., Huo, J., and Petrenko, R. (2005).

Multiplexing of partially ordered events.IFIP International Conference on Testing of Communicating Systems, TestCom, 3502:97–110.


Bibliography III

Cao, J., Chan, A., Chan, S., and Cheung, N. (2006).

A robust monitor construct with runtime fault detection.Concurrency Computation Practice and Experience, 18(5):471–500.

Cao, J., Cheung, N., and Chan, A. (2001).

Run-time fault detection in monitor based concurrent programming.International Conference on Dependable Systems and Networks (DSN 2001), pages 357 –366.

Carver, R. and Durham, R. (1995).

Integrating formal methods and testing for concurrent programs.Annual Conference on Computer Assurance (COMPASS ’95): Systems Integrity, Software Safety andProcess Security, pages 25 –33.

Carver, R. and Lei, J. (2010a).

A stateful approach to testing monitors in multithreaded programs.International Symposium on High-Assurance Systems Engineering (HASE), pages 54 –63.

Carver, R. and Lei, Y. (2004).

A general model for reachability testing of concurrent programs.International Conference on Formal Engineering Methods, ICFEM 2004, 3308:76–98.

Carver, R. and Lei, Y. (2010b).

Distributed reachability testing of concurrent programs.Concurrency Computation Practice and Experience, 22(18):2445–2466.


Bibliography IV

Carver, R. and Tai, K. C. (1989).

Deterministic execution testing of concurrent ada programs.Conference on Tri-Ada ’89: Ada technology in context: application, development, and deployment, pages528–544.

Carver, R. and Tai, K.-C. (1991).

Replay and testing for concurrent programs.IEEE Software, 8(2):66 –74.

Chatterjee, K., De Alfaro, L., Raman, V., and Sanchez, C. (2010).

Analyzing the impact of change in multi-threaded programs.International Conference Fundamental Approaches to Software Engineering (FASE), 6013 LNCS:293–307.

Chen, F. and RoA§u, G. (2007).

Parametric and sliced causality.International Conference on Computer Aided Verification CAV, 4590 LNCS:240–253.

Chen, H. Y., Sun, Y. X., and Tse, T. (2003).

A scheme for dynamic detection of concurrent execution of object-oriented software.IEEE International Conference on Systems, Man and Cybernetics, 5:4828 – 4833 vol.5.

Chen, J. (2000a).

On verifying distributed multithreaded java programs.Hawaii International Conference on System Sciences (HICSS), page 8 pp. vol.1.

Chen, J. (2000b).

A study on static analysis in network of synchronizing fsms.Asia-Pacific Software Engineering Conference (APSEC), pages 489 –493.


Bibliography V

Chen, J. (2009).

Guided Testing of Concurrent Programs Using Value Schedules.PhD thesis, University of Waterloo.

Chen, J. and MacDonald, S. (2007).

Testing concurrent programs using value schedules.International Conference on Automated Software Engineering, pages 313–322.

Chen, Q., Wang, L., Yang, Z., and Stoller, S. (2009).

Have: Detecting atomicity violations via integrated dynamic and static analysis.International Conference Fundamental Approaches to Software Engineering (FASE), 5503:425–439.

Chen, Z., Xu, B., Yang, H., Liu, K., and Zhang, J. (2000).

An approach to analyzing dependency of concurrent programs.Asia-Pacific Conference on Quality Software, pages 39 –43.

Chew, L. and Lie, D. (2010).

Kivati: Fast detection and prevention of atomicity violations.EuroSys’10 - Proceedings of the EuroSys 2010 Conference, pages 307–319.

Christakis, M. and Sagonas, K. (2011).

Detection of asynchronous message passing errors using static analysis.International Symposium Practical Aspects of Declarative Languages (PADL), Austin, TX, USA, 6539LNCS:5–18.

Chung, C.-M., Shih, T., Wang, Y.-H., Lin, W.-C., and Kou, Y.-F. (1996).

Task decomposition testing and metrics for concurrent programs.International Symposium on Software Reliability Engineering, pages 122 –130.


Bibliography VI

Chung, I. S., Kim, H. S., Seop, B. H., Kwon, Y. R., and Lee, B. S. (1999).

Testing of concurrent programs based on message sequence charts.International Symposium on Software Engineering for Parallel and Distributed Systems, pages 72–82.

Coons, K., Burckhardt, S., and Musuvathi, M. (2010).

Gambit: Effective unit testing for concurrency libraries.ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 45(5):15–24.

Copty, S. and Ur, S. (2005).

Multi-threaded testing with AOP is easy, and it finds bugs!International Euro-Par Conference, 3648:740–749.

Corbett, J. C., Dwyer, M. B., Hatcliff, J., and Robby (2000).

Bandera: A source-level interface for model checking java programs.International Conference on Software Engineering, pages 762–765.

Dantas, A. (2008).

Improving developers’ confidence in test results of multi-threaded systems: avoiding early and lateassertions.Conference on Object-Oriented Programming Systems, Languages, and Applications, OOPSLA, pages899–900.

Dantas, A., Brasileiro, F., and Cirne, W. (2008a).

Improving automated testing of multi-threaded software.International Conference on Software Testing, Verification and Validation, ICST 2008, pages 521–524.


Bibliography VII

Dantas, A., Gaudencio, M., Brasileiro, F., and Cirne, W. (2008b).

Obtaining trustworthy test results in multi-threaded systems.Simposio Brasileiro de Engenharia de Software, pages 1–10.

deok Choi, J. and Zeller, A. (2002).

Isolating failure-inducing thread schedules.International Symposium on Software Testing and Analysis, pages 210–220.

Devietti, J., Lucia, B., Ceze, L., and Oskin, M. (2009).

Dmp: deterministic shared memory multiprocessing.International Conference on Architectural support for programming languages and operating systems, pages85–96.

Ding, Z., Zhang, K., and Hu, J. (2008).

A rigorous approach towards test case generation.Information Sciences, 178(21):4057–4079.

Dingel, J. and Liang, H. (2004).

Automating comprehensive safety analysis of concurrent programs using verisoft and txl.ACM SIGSOFT International Symposium on the Foundations of Software Engineering.

Edelstein, O., Farchi, E., Goldin, E., Nir, Y., Ratsaby, G., and Ur, S. (2003).

Framework for testing multi-threaded java programs.Concurrency Computation Practice and Experience, 15(3-5 SPEC.):485–499.

Emmi, M., Qadeer, S., and Rakamarie, Z. (2010).

Delay-bounded scheduling.Symposium on Principles of Programming Languages, pages 411–422.


Bibliography VIII

Eytani, Y. and Latvala, T. (2007).

Explaining intermittent concurrent bugs by minimizing scheduling noise.International Haifa Verification Conference, HVC, 4383 LNCS:183–197.

Farchi, E. and Kliot, G. (2005).

Effective testing and debugging techniques for a group communication system.International Conference on Dependable Systems and Networks, pages 80–85.

Farchi, E., Nir, Y., and Ur, S. (2003).

Concurrent bug patterns and how to test them.International Parallel and Distributed Processing Symposium, page 7 pp.

Faure, E., Benabdenbi, M., and Pecheux, F. (2010).

Distributed online software monitoring of manycore architectures.International On-Line Testing Symposium (IOLTS), pages 56 –61.

Flanagan, C., Freund, S., and Qadeer, S. (2005).

Exploiting purity for atomicity.IEEE Transactions on Software Engineering, 31(4):275 – 291.

Gligoric, M., Jagannath, V., and Marinov, D. (2010).

Mutmut: Efficient exploration for mutation testing of multithreaded code.International Conference on Software Testing, Verification and Validation (ICST), pages 55 –64.

Gong, X., Wang, Y., Zhou, Y., and Li, B. (2007).

On testing multi-threaded java programs.International Conference on Software Engineering, Artificial Intelligence, Networking, andParallel/Distributed Computing (SNPD/ACIS ), 1:702 –706.


Bibliography IX

Gradara, S., Santone, A., Villani, M., and Vaglini, G. (2004).

Model checking multithreaded programs by means of reduced models using a tool based on the selectivemu-calculus logic to check systems described through the ccs specification language.Electronic Notes in Theoretical Computer Science, 110:55–74.

Havelund, K. and Rosu, G. (2001).

Java PathExplorer - a runtime verification tool.International Symposium on Artificial Intelligence, Robotics and Automation in Space: A New SpaceOdyssey, page 2001.

Humphrey, A., Derrick, C., Gopalakrishnan, G., and Tibbitts, B. R. (2010).

GEM: graphical explorer of mpi programs.International Symposium on Software Visualization, pages 217–218.

Hwang, G.-H., chung Tai, K., and lu Huang, T. (1995).

Reachability testing: An approach to testing concurrent software.International Journal of Software Engineering and Knowledge Engineering, 5:493–510.

Itoh, E., Furukawa, Z., and Ushijima, K. (1996).

A prototype of a concurrent behavior monitoring tool for testing of concurrent programs.Asia-Pacific Software Engineering Conference, pages 345 –354.

Jagannath, V., Gligoric, M., Jin, D., Rosu, G., and Marinov, D. (2010a).

Imunit: improved multithreaded unit testing.International Workshop on Multicore Software Engineering, pages 48–49.


Bibliography X

Jagannath, V., Gligoric, M., Lauterburg, S., Marinov, D., and Agha, G. (2010b).

Mutation operators for actor systems.International Conference on Software Testing, Verification, and Validation Workshops, pages 157–162.

Jianxin, X. and Dingxing, W. (1996).

Analyzing nondeterminacy of message passing programs.International Symposium on Parallel Architectures, Algorithms, and Networks, pages 547 –549.

John, D. B. and Chapin, J. (2000).

Systematic testing of multithreaded programs.Technical report, MIT/LCS Technical Memo, LCSTM 607.

Joshi, P., Naik, M., seo Park, C., and Sen, K. (2009).

Calfuzzer: An extensible active testing framework for concurrent programs.International Conference on Computer Aided Verification CAV, pages 675–681.

Kamil, A. and Yelick, K. (2010).

Enforcing textual alignment of collectives using dynamic checks.International Workshop Language and Compilers for Parallel Computing, LCPC, 5898 LNCS:368–382.

Karacali, B. and Tai, K.-C. (1999).

Automated test sequence generation using sequencing constraints for concurrent programs.International Symposium on Software Engineering for Parallel and Distributed Systems, pages 97 –108.

Katayama, T., Furukawa, Z., and Ushijima, K. (1995).

Event interactions graph for test-case generations of concurrent programs.Asia-Pacific Software Engineering Conference, pages 29 –37.


Bibliography XI


A method for structural testing of ada concurrent programs using the event interactions graph.Asia-Pacific Software Engineering Conference, pages 355 –364.


Test-case generation method for concurrent programs including task-types.Asia-Pacific Software Engineering Conference and International Computer Science Conference, APSEC andICSC, pages 485–494.


Design and implementation of test-case generation for concurrent programs.Asia-Pacific Software Engineering Conference, pages 262 –269.

Katayama, T., Itoh, E., Furukawa, Z., and Ushijima, K. (1999).

Test-case generation for concurrent programs with the testing criteria using interaction sequences.Asia-Pacific Software Engineering Conference (APSEC ’99), pages 590 – 597.

Kavi, K., Moshtaghi, A., and Chen, D.-J. (2002).

Modeling multithreaded applications using petri nets.International Journal of Parallel Programming, 30(5):353–371.

Khurshid, S., Pasareanu, C. S., and Visser, W. (2003).

Generalized symbolic execution for model checking and testing.International Conference on Tools and Algorithms for the Construction and Analysis of Systems,2619:553–568.


Bibliography XII

Kitchenham, B. (2004).

Procedures for performing systematic reviews.Technical Report TR/SE-0401, Keele University, UK.

Kitchenham, B. and Charters, S. (2007).

Guidelines for performing systematic literature reviews in software engineering.Technical Report EBSE 2007-001, Keele University and Durham University.

Kojima, H., Kakuda, Y., Takahashi, J., and Ohta, T. (2009).

A model for concurrent states and its coverage criteria.International Symposium on Autonomous Decentralized Systems(ISADS ’09), pages 1 –6.

Koppol, P., Carver, R., and Tai, K.-C. (2002).

Incremental integration testing of concurrent programs.IEEE Transactions on Software Engineering, 28(6):607–623.

Koppol, P. V. and Tai, K.-C. (1996).

An incremental approach to structural testing of concurrent software.ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 14–23.

Krammer, B., Mulller, M., and Resch, M. (2004).

MPI application development using the analysis tool MARMOT.International Conference on Computational Science, ICCS, 3038:464–471.

Krawczyk, H. and Wiszniewski, B. (1996).

A method for determining testing scenarios for parallel and distributed software.Technical report, Faculty of Electronics, Telecommunications and Informatics, Technical University ofGdansk, Poland, Tech Report N. 193/1996.


Bibliography XIII

Krena, B., Letko, Z., Tzoref, R., Ur, S., and Vojnar, T. (2007).

Healing data races on-the-fly.International Workshop on Parallel and Distributed Systems: Testing and Debugging, PADTAD’07, pages54–64.

Krena, B., Letko, Z., Vojnar, T., and Ur, S. (2010).

A platform for search-based testing of concurrent software.International Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging (PADTAD2010), pages 48–58.

Kundu, S., Ganai, M., and Wang, C. (2010).

Contessa: Concurrency testing augmented with symbolic analysis.International Conference on Computer Aided Verification CAV, 6174 LNCS:127–131.

Lei, J. (2003).

Non-deterministic testing of concurrent programs.International Symposium on Software Reliability Engineering (ISSRE), page 2 pp.

Lei, Y. and Carver, R. (2004).

Reachability testing of semaphore-based programs.International Computer Software and Applications Conference (COMPSAC ), pages 312 –317 vol.1.

Lei, Y. and Carver, R. (2006).

Reachability testing of concurrent programs.IEEE Transactions on Software Engineering, 32(6):382–403.


Bibliography XIV

Lei, Y., Carver, R., Kacker, R., and Kung, D. (2007).

A combinatorial testing strategy for concurrent programs.Software Testing Verification and Reliability, 17(4):207–225.

Li, J., Hei, D., and Yan, L. (2009).

Correctness analysis based on testing and checking for openmp programs.ChinaGrid Annual Conference (ChinaGrid ’09), pages 210 –215.

Li, S. Q., Chen, H. Y., and Sun, Y. X. (2004).

A framework of reachability testing for java multithread programs.IEEE International Conference on Systems, Man and Cybernetics, 3:2730 – 2734 vol.3.

Liang, Y., Li, S., Zhang, H., and Han, C. (2000).

Timing-sequence testing of parallel programs.Journal of Computer Science and Technology, 15(1):94–95.

Lourenco, J. a. and Cunha, G. (2007).

Testing patterns for software transactional memory engines.International Workshop on Parallel and Distributed Systems: Testing and Debugging, pages 36–42.

Lu, S., Jiang, W., and Zhou, Y. (2007).

A study of interleaving coverage criteria.ACM SIGSOFT Symposium on the Foundations of Software Engineering, 6th joint meeting of the Europeansoftware engineering conference, pages 533–536.


Bibliography XV

Lu, S., Park, S., Seo, E., and Zhou, Y. (2008).

Learning from mistakes: a comprehensive study on real world concurrency bug characteristics.International conference on Architectural support for programming languages and operating systems,42:329–339.

Lu, S., Tucek, J., Qin, F., Zhou, Y., and Bugs, V. (2006).

Avio: Detecting atomicity violations via access-interleaving invariants.International Conference on Architectural support for programming languages and operating systems(ASPLOS), pages 37–48.

Luo, Q., Zhang, S., Zhao, J., and Hu, M. (2010).

A lightweight and portable approach to making concurrent failures reproducible.International Conference Fundamental Approaches to Software Engineering (FASE), 6013 LNCS:323–337.

Maheswara, G., Bradbury, J., and Collins, C. (2010).

Tie: An interactive visualization of thread interleavings.Conference on Computer and Communications Security, ACM, pages 215–216.

McMinn, P. (2009).

Search-based failure discovery using testability transformations to generate pseudo-oracles.Annual Genetic and Evolutionary Computation Conference, GECCO-2009, pages 1689–1696.

Musuvathi, M. and Qadeer, S. (2007).

Iterative context bounding for systematic testing of multithreaded programs.ACM SIGPLAN Conference on Programming language design and implementation, 42:446–455.


Bibliography XVI

Musuvathi, M. and Qadeer, S. (2008).

Fair stateless model checking.ACM SIGPLAN Programming Language Design and Implementation (PLDI).

Musuvathi, M., Qadeer, S., and Ball, T. (2007).

Chess: A systematic testing tool for concurrent software.Technical report, Microsoft Research, Tech. Report MSR-TR-2007-149.

Musuvathi, M., Qadeer, S., Nainar, P. A., Ball, T., Basler, G., and Neamtiu, I. (2008).

Finding and reproducing heisenbugs in concurrent programs.USENIX Conference on Operating Systems Design and Implementation, pages 267–280.

Oberhuber, M. and Munchen, T. (1995).

Elimination of nondeterminacy for testing and debugging parallel programs.International Workshop on Automated and Algorithmic Debugging (AADEBUG’95.

Olsson, R. (1999).

Reproducible execution of sr programs.Concurrency and Computation: Practice and Experience, 11(9):479 – 507.

Olszewski, M., Ansel, J., and Amarasinghe, S. (2009).

Kendo: efficient deterministic multithreading in software.International Conference on Architectural support for Programming languages and operating systems, ACMSIGPLAN Notes, 44:97–108.

Park, M.-Y., Shim, S., Jun, Y.-K., and Park, H.-R. (2007).

MPIRace-check: Detection of message races in MPI programs.International Conference Advances in Grid and Pervasive Computing, GPC, 4459 LNCS:322–333.


Bibliography XVII

Park, S., Lu, S., and Zhou, Y. (2009).

Ctrigger: exposing atomicity violation bugs from their hiding places.International Conference on Architectural Support for Programming Languages and Operating Systems(ASPLOS, pages 25–36.

Pozniansky, E. and Schuster, A. (2003).

Efficient on-the-fly data race detection in multithreaded C++ programs.International Parallel and Distributed Processing Symposium, page 8 pp.

Pu, F. and Xu, H.-Y. (2008).

A feasible strategy for reachability testing of internet-based concurrent programs.IEEE International Conference on Networking, Sensing and Control (ICNSC), pages 1559 –1564.

Pugh, W. and Ayewah, N. (2007).

Unit testing concurrent software.International Conference on Automated Software Engineering (ASE), pages 513–516.

Rakamaric, Z. (2010).

Storm: static unit checking of concurrent programs.International Conference on Software Engineering, ACM/IEEE, pages 519–520.

Ricken, M. and Cartwright, R. (2009).

ConcJUnit: Unit testing for concurrent programs.International Conference on Principles and Practice of Programming in Java, PPPJ 2009, pages 129–132.

Ricken, M. G. (2007).

A framework for testing concurrent programs.Technical report, RICE UNIVERSITY.


Bibliography XVIII

Rodrıguez, E., Dwyer, M. B., Hatcliff, J., and Robby (2004).

A flexible framework for the estimation of coverage metrics in explicit state software model checking.International Workshop on Construction and Analysis of Safe, Secure and Interoperable Smart Devices.

Rose, J., Swamy, N., and Hicks, M. (2005).

Dynamic inference of polymorphic lock types.Science of Computer Programming, 58(3):366–383.

Rungta, N. and Mercer, E. (2009).

A meta heuristic for effectively detecting concurrency errors.International Haifa Verification Conference, HVC, 5394 LNCS:23–37.

Rungta, N., Mercer, E., and Visser, W. (2009).

Efficient testing of concurrent programs with abstraction-guided symbolic execution.International SPIN Workshop, 5578 LNCS:174–191.

Saleh, K., Boujarwah, A. A., and Al-Dallal, J. (2001).

Anomaly detection in concurrent java programs using dynamic data flow analysis.Information and Software Technology, 43(15):973–981.

Sarmanho, F. S., Souza, P. S. L., Souza, S. R. S., and Simao, A. S. (2008).

Structural testing for semaphore-based multithread programs.International Conference on Computational Science, 5101 LNCS(PART 1):337–346.

Sen, A. and Abadir, M. (2010).

Coverage metrics for verification of concurrent systemc designs using mutation testing.IEEE International High Level Design Validation and Test Workshop (HLDVT), pages 75 –81.


Bibliography XIX

Sen, K. (2007).

Effective random testing of concurrent programs.International Conference on Automated Software Engineering, ASE’07, pages 323–332.

Sen, K. and Agha, G. (2006).

A race-detection and flipping algorithm for automated testing of multi-threaded programs.International Haifa Verification Conference, pages 166–182.

Seo, H.-S., Chung, I., and Kwon, Y. (2006).

Generating test sequences from statecharts for concurrent program testing.IEICE Transactions on Information and Systems, E89-D(4):1459–1469.

Seo, H.-S., Chung, I. S., Kim, B. M., and Kwou, Y. R. (2001).

The design and implementation of automata-based testing environment for java multi-thread programs.Asia-Pacific Software Engineering Conference (APSEC), pages 221 – 228.

Serebryany, K. and Iskhodzhanov, T. (2009).

Threadsanitizer: data race detection in practice.Workshop on Binary Instrumentation and Applications, pages 62–71.

Sherman, E., Dwyer, M. B., and Elbaum, S. (2009).

Saturation-based testing of concurrent programs.ACM SIGSOFT Symposium on the Foundations of Software Engineering, joint meeting of the EuropeanSoftware engineering, pages 53–62.

Shih, T., Chung, C.-M., Wang, Y.-H., Kuo, Y.-F., and Lin, W.-C. (1996).

Software testing and metrics for concurrent computation.Asia-Pacific Software Engineering Conference, pages 336 –344.


Bibliography XX

Sohn, H. W., Kung, D., and Hsia, P. (1999).

State-based reproducible testing for CORBA applications.International Symposium on Software Engineering for Parallel and Distributed Systems, pages 24–35.

Sorrentino, F., Farzan, A., and Madhusudan, P. (2010).

Penelope: Weaving threads to expose atomicity violations.ACM SIGSOFT Symposium on the Foundations of Software Engineering, pages 37–46.

Souza, P. L., Sawabe, E. T., Simao, A., Vergilio, S. R., and Souza, S. R. S. (2008a).

ValiPVM - a graphical tool for structural testing of PVM programs.European PVM/MPI Users’ Group Meeting, Recent Advances in Parallel Virtual Machine and MessagePassing Interface, 5205 LNCS:257–264.

Souza, S. R. S., Vergilio, S. R., Souza, P. S. L., Simao, S., and Hausen, A. C. (2008b).

Structural testing criteria for message-passing parallel programs.Concurrency Computation Practice and Experience, 20(16):1893–1916.

Sridharan, M. and Rinard, C. (2002).

Dynamic datarace detection for object-oriented programs.Technical report, MASSACHUSETTS INSTITUTE OF TECHNOLOGY.

Stoller, S. (2002).

Testing concurrent java programs using randomized scheduling.Electronic Notes in Theoretical Computer Science, 70(4):149–164.

Sung, A. (1988).

Testing shared-memory parallel programs.Symposium on the Frontiers of Massively Parallel Computation, pages 559 –566.


Bibliography XXI

Tai, K. (1989).

Testing of concurrent software.International Computer Software and Applications Conference, pages 62–64.

Tai, K.-C. and Karacali, B. (2001).

On godefroid’s state-less search technique for testing concurrent programs.International Symposium on Autonomous Decentralized Systems, pages 77 –84.

Takahashi, J., Kojima, H., and Furukawa, Z. (2008).

Coverage based testing for concurrent software.International Conference on Distributed Computing Systems Workshops (ICDCS ’08), pages 533 –538.

Tan, R., Nagpal, P., and Miller, S. (2009).

An automated black box testing tool for a parallel programming library.International Conference on Software Testing, Verification, and Validation, ICST 2009, pages 307–316.

Taylor, R. N., Levine, D. L., and Kelly, C. D. (1992).

Structural testing of concurrent programs.IEEE Transactions on Software Engineering, 18(3):206–215.

Vuduc, R., Schulz, M., Quinlan, D., Supinski, B. D., and Sæbjørnsen, A. (2006).

Improving distributed memory applications testing by message perturbation.International Workshop on Parallel and Distributed Systems: Testing and Debugging (PADTAD).

Wang, L. and Stoller, S. (2006).

Runtime analysis of atomicity for multithreaded programs.IEEE Transactions on Software Engineering, 32(2):93 – 110.


Bibliography XXII

Wang, Y., Kelly, T., Kudlur, M., Lafortune, S., and Mahlke, S. (2008).

Gadara: Dynamic deadlock avoidance for multithreaded programs.USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 281–294.

Wang, Y.-H., Chung, C.-M., Shih, T., Keh, H.-C., and Lin, W.-C. (1997).

Software testing and metrics for concurrent computation through task decomposition.IEEE International Conference on Intelligent Processing Systems (ICIPS ’97), 2:1857 –1861 vol.2.

Wei, W., Wu, Y., Lejun, Z., and Lin, G. (2006).

Testing path generation algorithm with network performance constraints for nondeterministic parallelprograms.International Conference on Web-Age Information Management Workshops, WAIM 2006.

Wong, W., Lei, Y., and Ma, X. (2005).

Effective generation of test sequences for structural testing of concurrent programs.IEEE International Conference on Engineering of Complex Computer Systems, ICECCS, pages 539–548.

Wong, W. E. and Lei, Y. (2008).

Reachability graph-based test sequence generation for concurrent programs.International Journal of Software Engineering and Knowledge Engineering, 18(6):803–822.

Wood, B. P., Sampson, A., Ceze, L., and Grossman, D. (2010).

Composable specifications for structured shared-memory communication.International Conference on Object oriented Programming systems languages and applications, pages140–159.


Bibliography XXIII

Wu, M., Zhou, B., and Shi, W. (2009).

A self-adaptive test framework for concurrent programs.International Conference on Management of Emergent Digital EcoSystems, MEDES ’09, pages 456–457.

Xiaoan, B., Na, Z., and Zuohua, D. (2009).

Test case generation of concurrent programs based on event graph.International Joint Conference on INC, IMS, and IDC (NCM 2009), pages 143–149.

Xu, K. and Liang, D. (2007).

Formally defining a graphical language for monitoring and checking object interactions.International Conference Model Driven Engineering Languages and Systems, MoDELS, 4735 LNCS:620–634.

Yang, C. and Pollock, L. L. (1997a).

Identifying redundant test cases for testing parallel language constructs.Technical Conference Constructs, ARL-ATIRP.

Yang, C. S. D. and Pollock, L. L. (1997b).

An algorithm for all-du-path testing coverage of shared memory parallel programs.Asian Test Symposium, pages 263–268.

Yang, C.-S. D. and Pollock, L. L. (2003).

All-uses testing of shared memory parallel programs.Software Testing, Verificatio, AND RELIABILITY JOURNAL, 13:3–24.

Yang, C.-S. D., Souter, A. L., and Pollock, L. L. (1998).

All-du-path coverage for parallel programs.ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 153–162.


Bibliography XXIV

Yang, R.-D. and Chung, C.-G. (1990a).

The analysis of infeasible concurrent paths of concurrent ada programs.International Computer Software and Applications Conference (COMPSAC), pages 424 –429.

Yang, R.-D. and Chung, C.-G. (1990b).

A path analysis approach to concurrent program testing.International Phoenix Conference on Computers and Communications, pages 425 –432.

Yang, R.-D. and Chung, C.-G. (1992).

Path analysis testing of concurrent programs.Information and Software Technology, 34(1):43–56.

Yang, Y., Chen, X., and Gopalakrishnan, G. (2008).

Inspect: A runtime model checker for multithreaded c programs.Technical report, School of Computing University of Utah, n. UUCS-08-004.

Yang, Z. and Sakallah, K. (2008).

Smt-based symbolic model checking for multi-threaded programs.Workshop Exploiting Concurrency Efficiently and Correctly.

Yang, Z. and Sakallah, K. (2010).

Trace-driven verification of multithreaded programs.International Conference on Formal Engineering Methods, ICFEM, 6447 LNCS:404–419.

Young, M., Taylor, R. N., Levine, D. L., Forester, K., and Brodbeck, D. (1992).

A concurrency analysis tool suite: Rationale, design, and preliminary experience.Technical report, ACM Transactions on Software Engineering and Methodology.


Bibliography XXV

Yu, J. and Narayanasamy, S. (2009).

A case for an interleaving constrained shared-memory multi-processor.International Symposium on Computer Architecture, pages 325–336.

Yuan, Y., Li, Z., and Sun, W. (2006).

A graph-search based approach to BPEL4WS test generation.International Conference on Software Engineering Advances, ICSEA’06.

Zhang, W., Sun, C., and Lu, S. (2010).

Conmem: Detecting severe concurrency bugs through an effect-oriented approach.International Conference on Architectural Support for Programming Languages and Operating Systems -ASPLOS, pages 179–191.

Zhao, J. (1999).

Multithreaded dependence graphs for concurrent java programs.International Symposium on Software Engineering for Parallel and Distributed Systems, pages 13 –23.

Research in Concurrent Software Testing: A Systematic Review

Documents

Transcript of Research in Concurrent Software Testing: A Systematic Review