Software Reliability Methods
description
Transcript of Software Reliability Methods
Software Reliability Methods
Sorin Lerner
Software reliability methods: issues
• What are the issues?
Software reliability methods: issues
• What is software reliability? How to measure it?– Bug counts ? Will we ever have bug-free software?– How many 9’s ?– Service Level Agreements ?
• What is a bug?– Adherence to specifications– But what is a specification…– User unhappy: is that a bug?– Different levels of severity
Software reliability methods: issues
• Cost of the methods for achieving reliability– Independently develop 5 versions of the software, run
them all in parallel ) less likely that they fail at the same time in the same way. But… cost… is… high
– For tools, cost of development of the tools
• Burden on the programmer– fully automated vs. semi-automated methods– allow progressive adoption
Software reliability methods: issues
• Level of guarantee provided by the method– Hard guarantees, statistical guarantees, no formal
guarantee– What if tool is broken: trusted computing base
• When is the method used?– compile-time, link-time, load-time, run-time
• What does the tool see?– source code, assembly, the whole program or part of
the program
One way of dividing the spectrum
Compiler
if (…) { x := …;} else { y := …;}…;
010010110100101011011
One way of dividing the spectrum
CompilerCompiler
if (…) { x := …;} else { y := …;}…;
010010110100101011011
Static techniques
Testing techniques
Run-time techniques
Compiler
if (…) { x := …;} else { y := …;}…;
010010110100101011011
One way of dividing the spectrum
CompilerCompiler
if (…) { x := …;} else { y := …;}…;
010010101001011010010101001010110111011011
Static techniques
Testing Testing techniquestechniques
Run-time Run-time techniquestechniques
010010110100101011011
Testing techniques
Run-time techniques
Static techniques
Static Techniques
• Spec: says what code should and should not do
• Complete spec: specifies all behaviors (hard to formalize)
• Incomplete spec: only defines some behaviors– e.g. “no null derefs”, “requests received are eventually processed”
• Many formalisms exist for specs (Pre/Post conditions, FSMs, Temporal Logic, Abstract State Machines etc.)
if (…) { x := …;} else { y := …;}…;
Spec
«¬
$ \ rt l
Code satisfies spec?
Static Techniques
• Language Design– Clean language
design– Type Systems– Domain-specific
languages– …
if (…) { x := …;} else { y := …;}…;
Spec
«¬
$ \ rt l
Code satisfies spec?
• Program Analysis– Dataflow analysis– WP/SP– Model checking– Automated
Theorem Proving– …
Interaction between the
two
CleanLTSysDSL
DFAWP/SPMCATP
ESC/Java [Leino et al PLDI 2002]
object Foo {
//@ PRE (FORMULA) method bar(...) {
...
} //@ POST (FORMULA)}
Compute Weakest Precondition
WP(POST, bar) = weakest condition Q such that Q at entry to bar establishes POST at exist
)
• Programmer annotates code with pre- and post-conditions, tool verifies that these hold
Automated Theorem Prover
CleanLTSysDSL
DFAWP/SPMCATP
ParserCodeGen
Compiler
DSLOpt
DSLOpt
DSLOpt
DSLOpt
DSLOpt
DSLOpt
Checker Checker Checker
Rhodium [Lerner et al POPL 2005]
CleanLTSysDSL
DFAWP/SPMCATP
ParserParserCodeCodeGenGen
CompilerCompiler
DSLOpt
DSLOpt
DSLOpt
Checker
DSLOpt
Checker
DSLOpt
Checker
DSLOpt
Rhodium [Lerner et al POPL 2005]
CleanLTSysDSL
DFAWP/SPMCATP
Rhodium [Lerner et al POPL 2005]
VCGen
LocalVC
AutomaticTheoremProver
Rdm Opt
Checker
LemmaFor any Rhodium opt:
If Local VC is trueThen opt is OK
Proof
«¬
$
\ rt l
Opt-independent
Opt-dependent
CleanLTSysDSL
DFAWP/SPMCATP
ESP [Das et al PLDI 2002]
Interface usage rules in documentation
–Order of operations, data access–Resource management –Incomplete, wordy, not checked
Violated rules ) crashes–Failed runtime checks–Unreliable software
CleanLTSysDSL
DFAWP/SPMCATP
ESP [Das et al PLDI 2002]
C Progra
m
Safe Not Safe
Rules
ESP
CleanLTSysDSL
DFAWP/SPMCATP
ESP [Das et al PLDI 2002]
• ESP is a program analysis that keeps track of object state at each program point– e.g.: is file handle open or closed?
• Challenge: scale to large programs– One of scalability issues: merge nodes– Always analyze both sides of merge node )
exponential (or non-terminating) program analyses
• ESP has a heuristic for handling merges that– avoids exponential blow-up and runs fast in practice– maintains enough precision to verify programs
CleanLTSysDSL
DFAWP/SPMCATP
BLAST [Henzinger et al POPL 2000]
Interface usage rules in documentation
–Order of operations, data access–Resource management –Incomplete, wordy, not checked
Violated rules ) crashes–Failed runtime checks–Unreliable software
BLAST [Henzinger et al POPL 2000]
C Progra
m
SafeError Trace
Rules
BLAST
BLAST [Henzinger et al POPL 2000]
C Progra
m
SafeError Trace
Rules
BLAST
BLAST [Henzinger et al POPL 2000]
Perform “Predicate
Abstraction” C Program
Rules
start with a set of predicates
Safe
Refine set of predicates
No errors found Analyze
trace
Trace feasible
Error Trace
Trace infeasible
BLAST
error trace found
augmented set of predicates
BLAST [Henzinger et al POPL 2000]
Perform “Predicate
Abstraction” C Program
Rules
start with a set of predicates
Safe
Refine set of predicates
No errors found Analyze
trace
Trace feasible
Error Trace
Trace infeasible
BLAST
error trace found
augmented set of predicates
CleanLTSysDSL
DFAWP/SPMCATP
Type Systems
• What is a type system?
• A discipline for writing code that can be mechanically checked, and can prevent certain kinds of run-time errors
• For example, java type system prevents calling methods that don’t exists, or calling methods with parameters of the wrong type
CleanLTSysDSL
DFAWP/SPMCATP
Type Systems
• Type systems can track and provide guarantees about many other aspects of computation:– Safe explicit memory management (Crary, Walker
and Morrisett, POPL 99)– Execution time bounds (Crary and Weirich, POPL 00)– Information flow (Myers, POPL 00)– Security automata (Walker, POPL 00)
CleanLTSysDSL
DFAWP/SPMCATP
Type Systems
• MultiJava [Clifton et al 2000] adds to Java:– multi-methods: methods that dispatch symetrically on
the type of all params, not just the first– open classes: classes
• Adding these features makes modular type checking harder, and required innovations on the type system side
• Interplay between language design and type systems
CleanLTSysDSL
DFAWP/SPMCATP