Who will test your tests?
-
Upload
yahya-poonawala -
Category
Engineering
-
view
125 -
download
2
Transcript of Who will test your tests?
![Page 1: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/1.jpg)
WHO WILL TEST YOUR TESTS ?
Yahya Poonawala @meetykp Priti Biyani @pritibiyani
![Page 2: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/2.jpg)
2 https://flic.kr/p/9uZhDS 2
![Page 3: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/3.jpg)
TESTS Code works
Reduce Bugs
Reduce cost of change
Allow Refactoring
Faster development
Reduce fear
3
![Page 4: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/4.jpg)
NOW WHEN YOU HEAR ….
4
![Page 5: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/5.jpg)
Oh, it failed?
Don’t worry. Just re-trigger it!
5
![Page 6: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/6.jpg)
Oh, it failed?
Strange; it passed on my machine!
6
![Page 7: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/7.jpg)
Oh, it failed?
Run it on agent 007! It’s an agent issue!
7
![Page 8: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/8.jpg)
Oh, it failed?
I know. It fails at 12AM!
8
![Page 9: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/9.jpg)
9
![Page 10: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/10.jpg)
Now, again,
WHY DO WE WRITE TESTS?
10
![Page 11: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/11.jpg)
Tests prove that the code actually works
Tests prove that the code sometimes works
11
![Page 12: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/12.jpg)
Tests reduce bugs
Tests itself might have bugs
12
![Page 13: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/13.jpg)
Tests reduce the cost of change
Change? Are you kidding?
13
![Page 14: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/14.jpg)
Tests helps in faster development.
Testing is slowing me down L
14
![Page 15: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/15.jpg)
NON-DETERMINISTIC
TESTS
A.K.A “Flaky Tests”
15
![Page 16: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/16.jpg)
16 http://www.trovatten.com/long-game/
![Page 17: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/17.jpg)
17
![Page 18: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/18.jpg)
18
http://tom-doherty.com/wp-content/uploads/2014/02/craigslist-waste-time.jpg
![Page 19: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/19.jpg)
19
![Page 20: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/20.jpg)
ILLUSION OF SUCCESS!
20
![Page 21: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/21.jpg)
ILLUSION OF FAILURE!
21
![Page 22: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/22.jpg)
22
![Page 23: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/23.jpg)
23
![Page 24: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/24.jpg)
DO NOT DELETE YOUR FLAKY TESTS!!
24
![Page 25: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/25.jpg)
CAUSES OF FLAKINESS
25
![Page 26: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/26.jpg)
Thread 1:: !" " if( name is not Yahya ) { !" " " " " "… !" " " " " "fputs( name, file ) !" " " " " "… !" " " " "} !!Thread 2:: !" " Initialize name to Yahya!!
![Page 27: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/27.jpg)
CONCURRENCY
27
![Page 28: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/28.jpg)
Initialize lock; !!Thread 1:: !" " Acquire lock !" " " " "if( name is not Yahya) { !" " " " " ". . . !" " " " " "fputs( name, file ) !" " " " " ". . . !" " " " "} !" " " " "Release lock !!Thread 2:: !" " Acquire lock !" " " " " Initialize name to"Yahya!" " " " " Release lock !!!
![Page 29: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/29.jpg)
TIME
29
http://imgs.xkcd.com/comics/bug.png
![Page 30: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/30.jpg)
EXTERNAL SYSTEM CALLS
30
![Page 31: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/31.jpg)
UNORDERED COLLECTIONS
31
![Page 32: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/32.jpg)
32 https://www.vardot.com/sites/default/files/2.jpg
![Page 33: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/33.jpg)
! !# pseudo-code ! !!! !makeAsyncCall; ! !sleep(2000); ! !readResponse; !
![Page 34: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/34.jpg)
ASYNCHRONOUS WAITING
34
![Page 35: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/35.jpg)
!!"make asynchronous call; !!"While (response received?) !" "if timeout then !" " " "throw Test Timeout Exception !"!" else !" " "sleep(10) ! . . . !!"read response !!!
![Page 36: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/36.jpg)
RESOURCE LEAKS
36
![Page 37: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/37.jpg)
# //pseudo-code !FILE_NAME = “content.txt”!!// Test 1:!!"FileReader.read(FILE_NAME).to be empty !!// Test2: Write to the file!!"data = “Priti” !"FileWriter.write(FILE_NAME, data) !"FileReader.read(FILE_NAME) == data !!// Test 3: Updates the file!!"append_data = “Biyani” !"complete_data = “Priti Biyani” ! FileWrite.append(FILE_NAME, append_data) ! FileReader.read(FILE_NAME) == complete_data !!
![Page 38: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/38.jpg)
TEST ORDER DEPENDENCY
38
![Page 39: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/39.jpg)
INPUT / OUTPUT
39
![Page 40: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/40.jpg)
40
![Page 41: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/41.jpg)
WHAT’S YOUR STRATEGY?
41
![Page 42: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/42.jpg)
STOP CALLING YOUR BUILD FLAKY!!
42
![Page 43: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/43.jpg)
Identify Flaky tests
Quarantine
Plan Fix Tests
Monitor
43
![Page 44: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/44.jpg)
IT’S NOT ALWAYS YOUR TESTS
44
![Page 45: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/45.jpg)
DO NOT GET OVERWHELMED BY FAILING TESTS
45
![Page 46: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/46.jpg)
REMEMBER!
46
![Page 47: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/47.jpg)
LOVE YOUR TEST CODE !
47
![Page 48: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/48.jpg)
There is no such thing as FLAKY TEST!
If its flaky, its not a test ! 48
![Page 49: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/49.jpg)
AND THE FUTURE…
49
![Page 50: Who will test your tests?](https://reader034.fdocuments.in/reader034/viewer/2022042602/55d38b8bbb61eb784c8b4613/html5/thumbnails/50.jpg)
REFERENCES • An Empirical Analysis of Flaky tests • Eradicating Non-Determinism in Tests – Martin
Fowler
• No more flaky tests on the GO team - Pavan Sudarshan
• Top 12 reasons to write unit tests
• http://tech.just-eat.com/2015/03/30/how-to-fix-flaky-tests/
50