Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make...

39
1 1 Random input testing with R Patrick Burns Patrick Burns http://www.burns http://www.burns- stat.com stat.com 2011 August Given at useR!2011 at the University of Warwick on 2011 August 17 in the Programming session, Uwe Ligges presiding.

Transcript of Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make...

Page 1: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

11

Random input testing with R

Patrick BurnsPatrick Burnshttp://www.burnshttp://www.burns--stat.comstat.com

2011 August

Given at useR!2011 at the University of Warwick on 2011 August 17 in the Programming session, Uwe Ligges presiding.

Page 2: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

2

The goal is for us to have fewer bugs in our code.

Photo from istockphoto.com

Page 3: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

3

We’ll start with a bit about the application that drove me in the direction that I’ve come.

The application is portfolio optimization in finance. The textbook version of this is quadratic programming – that’s an easy problem.

The reality is that it is a species of the knapsack problem: what is the best bundle of stuff that you can carry with you? That’s a hard problem.

How hard?

Let’s think about doing it via brute force.

Photo by David Weekly via everystockphoto.com

Page 4: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

4

We’ll take a small problem.

Photo via istockphoto.com

Page 5: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

5

We’ll make some blue sky, absolutely optimistic assumptions.

Photo via istockphoto.com

Page 6: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

6

Unit timeof

Now we need a unit of time to state how long this process will take.

I have one in mind, what is it?

Page 7: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

7

The first guess from the audience was right: the age of the universe.

There is a combinatorial explosion that makes this a fantastically hard problem.

Wilkinson Microwave Anisotropy Probe image from NASA

Page 8: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

8

Not everyone is keen on the wait.

Photo from istockphoto.com

Page 9: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

9

So in my code I have a number of shortcuts to speed up the process.

Nothing could possible go wrong here.

Photo by jurvetson via everystockphoto.com (I think)

Page 10: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

10

Utilities

Maximise information ratioMinimum varianceMaximum returnMean-variance utilityMean-volatility utilityMinimize distance to a portfolioWith or without transaction costs

Linear costsPolynomial costsMultiple terms with arbitrary exponentsSeparate costs for long-buy, long-sell, short-buy, short-sell

With or without multiple expected return vectorsWith or without multiple variance matricesWith or without multiple target portfoliosChoice of quantile in multiple utility casesCan use weights in multiple utility cases

When doing optimization, we need a utility. So pick one.

Page 11: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

11

Constraints

Long-only (or not)Value of gross, net, long, shortTurnoverMaximum weight (per asset)Number of assets tradedNumber of assets in portfolioThreshold (portfolio and/or trade)Linear

On portfolio or tradeOn net or grossOn long-side, on short-side, or bothCount constraints

Forced tradesAssets to tradeNumber of positions closedRound lotting (automatic)Risk fractions

Expected returnVarianceDistanceSum of n largest weightsCost

There will also be a set of constraints which is a subset of all possible constraints.

So there is a combinatorial explosion of inputs that is similar to the combinatorial explosion that makes it a hard problem in the first place.

And we want to test to see if the code works.

Page 12: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

12

Fixed problem

Right result?

The standard approach to software testing is to produce a fixed problem and ask if we get the correct answer.

Perhaps you can spot that already I have a problem.

Page 13: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

13

Fixed problem

Rightish result?

If it takes billions of years to get one right answer, perhaps my test suite will be a bit thin for a while.

I can’t ask if I have the right answer. I can only ask if the answer is sort of right.

Page 14: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

14

Rightish answers

Sanity check shortcutsSanity check resultClose to best observed?

To see if an answer appears right, there is a large number of checks in test mode that squawk if something doesn’t look right. In particular almost all shortcuts are also computed the long way in test mode and the two results compared.

There are checks to see if the result makes sense.

Since my optimization algorithm is random, it will potentially get different answers when using different seeds. Hence we can see if the current answer is close to the best answer that has been observed. This gives us a sense of how well the algorithm works on the particular problem.

Page 15: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

15

testthat

RUnit

svUnit

If you are creating a standard type test suite for R code, then there are a few packages that can help you.

Page 16: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

16

Fixed problem

Right result?

I have a standard test suite. But no matter how expansive my suite becomes, I will only ever test a tiny fraction of the possible combinations of inputs.

And writing a standard test suite is quite labor-intensive.

Plus there can be bias in the test. As someone said to me after the talk, we may have an unconscious desire NOT to find bugs.

Hence looking for alternatives to standard testing seems like a worthwhile endeavor.

Page 17: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

17

Random problem

Any explosions?

One alternative type of test is to create random inputs and then smell for smoke.

Page 18: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

18

Create inputs

Call code

Look at output

There are three steps in doing this.

If the code you are testing is not R code, then R is still a great place to do the first step of creating the random inputs.

If you are testing R code, then there are two key functions in performing the second step of calling the code.

Page 19: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

19

do.call

If you don’t know this function, you should.

It allows you to call a function with an actual list of arguments.

So in the present case, we create a list containing the random inputs that were created in step 1, and then call the function being tested.

Page 20: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

20

try

tryCatch

or

These are slightly different ways of doing the same thing.

They catch the errors from the test cases so that the errors don’t leak out into the actual test suite.

Page 21: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

21

When doing random testing, there are several things that can happen. Here is the tree of those things.

Page 22: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

22

Get answer

We can get an answer as opposed to an error.

Page 23: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

23

Get answerokay

Boring.

Page 24: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

24

Get answerwrong

Not boring.

Page 25: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

25

Get answerwrong

Bad, bad, bad computer!

We don’t like to be here. But it does make the effort of testing seem worthwhile.

If we are here, then we want to fix the problem. There are two more things we should do as well:

1. Create a case to put in the standard test suite for this problem. Random testing is an adjunct to standard testing, not a replacement.

2. Look through the code for other occurrences of this same sort ofproblem.

Page 26: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

26

Get answerinefficient

We can get the right answer via an inefficient (in time or memory) route. In my current application I’m not going to know this, but in other applications it is feasible to spot this.

Page 27: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

27

Get error

We can get an error rather than an answer.

This is a wonderful reason to do random testing. Standard test suites seldom to never exercise the code that throws errors. Random testing can exercise it a lot.

You could create a test where you don’t even expect to get an answer –you’re just testing the error throwing. This, of course, is entirely divorced from my original motivation for this sort of testing.

Page 28: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

28

Get errorCaught

When we get an error, it can be caught by the code we’ve written.

Page 29: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

29

Get errorCaught

Not clear

If the error message is not clear, then we should change the error message.

Page 30: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

30

Get errorCaught

Clear

If the error message is clear, then we should question that assumption.

When we are coding, we have the curse of knowledge: we know everything that is going on. So any error message that we write is going to make sense to us.

We should try to put ourselves in the frame of mind of a user who is not especially clued in.

Page 31: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

31

Get errorNot caught

If the error message is not caught by our code,

Page 32: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

32

Get errorNot caught

Not clear

Page 33: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

33

Get errorNot caught

Clear

we should consider catching it because it is even harder in that case for the message to be clear to our not-so-clued-in user.

Page 34: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

34

I find writing test suites to be as exciting as digging ditches.

Of all the tasks involved in creating software, this is the lowest of the low for me.

I had expected the same to be true for random testing.

Photo by KayPat via stock.xchng

Page 35: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

35

But I found random testing to have a quite different ambience to it. It is much more like playing.

Standard tests are basically static – write them once, periodically add bits to them, use them many times.

Random testing is more like an interactive game: do a bit, see how it goes, do some more, see how it goes, …

Keeping track of the different versions of the random tests is a good idea. You can go back and use a primitive test on a new version of the software. You can also use a primitive version as the basis for going off in a different testing direction.

I hope to see you on the playground.

Photo by forst747 via stock.xchng

Page 36: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

36

One of the good things about giving a talk is that the audience tells you the name for what you are doing. “Fuzz testing” is a common word for random input testing.

One of the key questions is what distribution to use. The distribution will depend on your goal. It seems like there are three basic goals:

1. Test the correctness of answers2. Test error throwing code3. Test for security issues

The last of these is most closely associated with “fuzz testing”.

Page 37: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

37

If your goal is one of the latter two, then the distributions should be very wide. That is, lots of stuff that need not even make any sense at all.

If you are trying to test the error throwing, then you probably want a more refined test as well that has arguments closer to what is expected.

If your goal is to test answers, then it seems to me that there are two possibilities:

• A distribution aiming at what is likely to be done in practice• A distribution aiming at most likely to generate bugs

The reason that random testing is more fun is because it is really programming – writing the code that creates the distributions.

Page 38: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

38

In my tests I have them start by doing two things:• Print the set-seed number so that results are reproducible• Print the version of the test being used

My tests so far have used the iteration variable as the set-seed number. Martin Maechler had a better idea: generate a random set-seed number. That way there is no need to keep track of what iterations have already been done with each test.

Page 39: Random input testing with R · Bad, bad, bad computer! We don’t like to be here. But it does make the effort of testing seem worthwhile. If we are here, then we want to fix the

39

The business end of my random test function looks like:

ans <- try(do.call(“trade.optimizer”,

c(thisdata, extra.args)))

if(inherits(ans, “try-error”)) {

cat(“\n error with seed”, the.seed, “\n\n”)

} else {

pprobe.check(ans)

}


Yesterday, we asked “are all chemicals bad?”

Yesterday, we asked “are all chemicals bad?”

COVID-19. Why things are not as bad as they seem and where we … · 1 COVID-19: Why things are not as bad as they seem - and where we might go from here. Insights from the scientific

COVID-19. Why things are not as bad as they seem and where we … · 1 COVID-19: Why things are not as bad as they seem - and where we might go from here. Insights from the scientific

We are here!

We are here!

We are here.

We are here.

WE ARE HERE · WE ARE HERE WORSHIP We are here - frail, flawed and far from perfection - yet we are here. We are here to worship you - the source of all that we know to be true and

WE ARE HERE · WE ARE HERE WORSHIP We are here - frail, flawed and far from perfection - yet we are here. We are here to worship you - the source of all that we know to be true and

Fantastic food and careful HERE! - Teach Early YearsFood is something that everyone understands, and we all know what we believe to be good and bad when it comes to what we fill our

Fantastic food and careful HERE! - Teach Early YearsFood is something that everyone understands, and we all know what we believe to be good and bad when it comes to what we fill our

HERE (?) WE ARE

HERE (?) WE ARE

We Lived Here

We Lived Here

People forget. We are here to act. We are here to deliver ...

People forget. We are here to act. We are here to deliver ...

Here we are!

Here we are!

“You Have to be Bad or Dumb to Get in Here”: Reconsidering ...€¦ · “You Have to be Bad or Dumb to Get in Here”: Reconsidering the In-School and Out-of-School Literacy

“You Have to be Bad or Dumb to Get in Here”: Reconsidering ...€¦ · “You Have to be Bad or Dumb to Get in Here”: Reconsidering the In-School and Out-of-School Literacy

‘Here be dragons, here be savages, here be bad plumbing ... · ‘Here be Dragons, Here be Savages, Here be bad Plumbing’: Australian Media Representations of Sport and Terrorism

‘Here be dragons, here be savages, here be bad plumbing ... · ‘Here be Dragons, Here be Savages, Here be bad Plumbing’: Australian Media Representations of Sport and Terrorism

We are not that bad

We are not that bad

Here We Go:

Here We Go:

“Teaching is not so bad!!!” “Maybe God doesn’t want me here?”

“Teaching is not so bad!!!” “Maybe God doesn’t want me here?”

Why Are We Here? How Did We Get Here? Where Do We Go Next?

Why Are We Here? How Did We Get Here? Where Do We Go Next?

Here we are !

Here we are !

We Are Here!

We Are Here!

reform PEIA now! How did we get here? How did we get here? Where we are! Where we are! Where do we go from here? Where do we go from here?

reform PEIA now! How did we get here? How did we get here? Where we are! Where we are! Where do we go from here? Where do we go from here?

(and We Should Feel Bad) - USENIX

(and We Should Feel Bad) - USENIX