Andreas Kuehlmann

Software Integrity Group, Synopsys

July 19, 2015

Making a Practical Impact in SW Verification

© 2015 Synopsys, Inc. 2

Introduction

© 2015 Synopsys, Inc. 3

A Typical SW Project

Complex Software

More Bugs

More Complex SW

More Bugs

Fix Bugs

Fix Bugs

New Features

New Features

The Grand Challenge:• There seems to be only a way down• Most projects don’t know where they

are on that path

© 2015 Synopsys, Inc. 4 Automotive

PrivacyFSIHealthCare

Apollo 11: 145 kLOC (lines of code)

Microsoft Windows: 2.3 MLOC (1992) 40 MLOC (2009)

Automobile: 50 kLOC (1981) 100 MLOC (2014)

Size of SW applications continues to grow Number of SW developers increases steadily

SW compliance regulations start emergingSoftware tools market grows

Security testing growing fastest Dynamic ready for disruption

The Challenges and Opportunities

© 2015 Synopsys, Inc. 5

Six Non-technical Challenges in SW Testing

© 2015 Synopsys, Inc. 6

Challenge 1:

• Can’t train them to become experts in temporal logic– This was tried in the HW world with little success

Developers don't think functionally, they thinkoperationally

© 2015 Synopsys, Inc. 7

Challenge 1 (cont.):

• Can’t expect them to understand global loop invariants and be able to maintain them

• Can’t explain bugs in such terms

• Developers understand gdb– Explain bugs in gdb’s terms!

http://homepages.ius.edu/rwisman/C455/html/notes/Chapter2/Loops/lpinv.htm

Developers don't think functionally, they thinkoperationally

© 2015 Synopsys, Inc. 8

Challenge 2:

Developers (like all humans) have a very limited memory and context switch is expensive!

In 1885, Herman Ebbinghaus did an experiment where subjects memorized a list of meaningless three letter words and tracked how quickly his subjects forgot the words.

Source: Gerald Weinberg: Quality Software Management: Systems Thinking

© 2015 Synopsys, Inc. 9

Challenge 2 (cont):

• Test the code while it is fresh in your mind– TDD

– Real-time code analysis

– …

Developers (like all humans) have a very limited memory and context switch is expensive!

Applied Software Measurement, Capers Jones, 1996

© 2015 Synopsys, Inc. 10

Challenge 3:

• Developers have a low tolerance for false bug alarms–1st false bug report

“Well, it didn’t get this right”

–2nd false bug report

“Annoying”

–3rd false bug report

“This tool is useless”

–4th false bug report

“Boss, why are you wasting my time, get rid of that tool!”

Don’t get in the developer’s way!

© 2015 Synopsys, Inc. 11

Challenge 4:

“My code is right! – I don’t need any tools!”

Yet

“What idiot broke the build again!”

Developers have big egos!

© 2015 Synopsys, Inc. 12

Challenge 5:

• Need to enable the tail end and pull them to the left– “Guard-rails” for developers

Not all developers are A or B grade

© 2015 Synopsys, Inc. 13

Challenge 6:

Competing factors for project delivery of a SW product:

1. Number and complexity of features

2. Cost to develop them

3. Time needed for developing them

4. Quality and security of the delivered product

1, 2, and 3 have clearly measurable metrics

W/o metrics, 4 is the weak player and often sacrificed

Management has often little insight into the quality of the SW code developed by their teams

© 2015 Synopsys, Inc. 14

From my Personal Point of View….

If you want to make a broader practical impact…

• Don’t start from a solution and then chase a problem (“the worm and the elephant”)

– This leading to success is the exception

– Instead, pick an urgent and broad problem

– Even at the expense of “academic beauty”

• Try to get ahead of the train

– Instead of following which I have seen too often in the SW verification research community

– Security, IoT, Self-driving cars, ….

• Stay connected with reality

– There are millions of SW developers out there in the “stone age”

– There are a huge amount of low-hanging fruits

• There is a lot of luck involved – use your instinct!

– Don’t be afraid for change!