Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012
-
Upload
eurostar-software-testing-conference -
Category
Technology
-
view
107 -
download
2
description
Transcript of Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012
![Page 1: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/1.jpg)
www.eurostarconferences.com
Matt Archer, Independent Software Tester, UK
How to regression test a billion rows of
financial data every sprint
@esconfs #esconfs
![Page 2: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/2.jpg)
Big data (a pension scheme example)
2
x x = 100,000 people
12 months
100 years
1.2 million estimates
1.2 million estimates
10s of financial models
x A Very Large
Database! = Billions of
database rows
![Page 3: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/3.jpg)
Creating a website
3
Page 1
Page 2
Page 3
Pages to develop + test
Sprint 1
![Page 4: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/4.jpg)
Creating a website
4
Page 1
Page 2
Page 3
Page 4 Page 6 Page 5
Pages to regression test
Pages to develop + test
Sprint 1 Sprint 2
![Page 5: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/5.jpg)
Creating a website
5
Page 1
Page 2
Page 3
Page 4 Page 6 Page 5
Pages to regression test
Pages to develop + test
Page 8 Page 7 Page 9
Sprint 1 Sprint 2 Sprint 3
![Page 6: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/6.jpg)
Page 11 Page 10 Page 12
Creating a website
6
Page 1
Page 2
Page 3
Page 4 Page 6 Page 5
Pages to regression test
Pages to develop + test
Page 8 Page 7 Page 9
Sprint 1 Sprint 2 Sprint 3 Sprint 4
![Page 7: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/7.jpg)
Page 11 Page 10 Page 12
Creating a website
7
Page 1
Page 2
Page 3
Page 4 Page 6 Page 5
Pages to enhance + test + regression test
Page 8 Page 7 Page 9
Pages to regression test
Pages to regression test
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5
![Page 8: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/8.jpg)
A common approach
8
Regression Test Strategy 1 (“The Regression Pack”)
1. Re-run tests from previous sprints.
2. Select those tests using a risk-based heuristic (rule of thumb).
3. Automate as much as possible, increasing the coverage over time.
![Page 9: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/9.jpg)
An example web page with financial data
9
![Page 10: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/10.jpg)
Example of a data-checking test
Given… …an organisation with approved data
When… … a user selects an Information Type of “Present Value” a Timeframe of “Weekly” and a Grouping of “Individual”
Then… … the data should be aggregated accordingly*
10
132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987
132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987
132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987
132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987
132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987
132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987
132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987
132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987 132473943987
* Much better BDD examples exist. Opted for brevity over clarity.
![Page 11: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/11.jpg)
2 x 3 x 3 = 18 Data permutations
11
18
![Page 12: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/12.jpg)
How many permutations for an entire site!?
12
Page 1
Page 2
Page 3
Page 4 Page 6 Page 5
Pages to enhance + test + regression test
Page 8 Page 7 Page 9
Page 11 Page 10 Page 12
Pages to regression test
Pages to regression test
![Page 13: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/13.jpg)
The thoroughness vs. maintenance conundrum
13
…create those tests if you
Think they’ll find bugs!
Follow Your
Instincts…
BUT more tests = more maintenance
![Page 14: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/14.jpg)
An alternative approach
So, what is the alternative?
14
![Page 15: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/15.jpg)
An alternative approach
15
Regression Test Strategy 2 (“Capture and Compare”)
1. Take snapshots of the data as it is
displayed in a known high-quality build
2. Store the snapshots somewhere safe
3. Use the snapshot to help identify unforeseen changes (bugs) in future release candidates and live releases
![Page 16: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/16.jpg)
Step 1: Identify a gold build
16
Page 4 Page 6 Page 5
Page 8 Page 7 Page 9
Page 11
Page 10
Page 12
Page 1
Page 2
Page 3
The “Gold” Build (1.3.2.14)
I’m so great, I’ve been the subject of…
BDD Unit Tests
Integration Tests Exploratory Testing
UX Inspections SME Reviews
Customer Demos
And maybe even… production use
(with NO complaints!)
![Page 17: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/17.jpg)
Step 2: Specify the data permutations to capture
17
<WebPage path = “/analytics”>
<QueryStringParam name = “InformationType”>
<QueryStringValue value = “PresentValue”/>
<QueryStringValue value = “Cashflow”/>
</QueryStringParam>
…
…
</WebPage>
![Page 18: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/18.jpg)
Step 3: Capture and clean the HTML tables
18
Capture specifications
Capture
String Host String Spec_Location String Save_Location
Void Capture() Void Clean()
Data snapshots
<table> <tr> <td>4.24</td> <td>15.93</td> <td>12.67</td> … </tr> … </table>
Selenium
“Gold” build
![Page 19: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/19.jpg)
Step 4: Synchronise the data
19
Database
Database Synchronise
Release Candidate “Gold” Build
![Page 20: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/20.jpg)
Step 5: Compare the release against the snapshot
20
Compare
String Host String Spec_Location String Snap_Location
Void Compare() Void Highlight()
Data snapshots
Capture specifications
Highlighted release candidate
Release candidate
![Page 21: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/21.jpg)
21
Step 6: Manual validation
What is a bug? What is a deliberate change?
![Page 22: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/22.jpg)
What to capture, what to test?
What to capture? What to test?
The final build from the previous sprint
The final build from the current sprint
The build currently deployed in live
The build currently deployed in staging
The build deployed in live prior to release
The build deployed in live after the release
22
• The technique can support a variety of team milestones
![Page 23: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/23.jpg)
Summary
• Beware generic regression testing (or any testing) strategies
• Look for more efficient ways of detecting regression bugs
• Balance your regression techniques
23
![Page 24: Matt Archer - How To Regression Test A Billion Rows Of Financial Data Every Sprint - EuroSTAR 2012](https://reader035.fdocuments.in/reader035/viewer/2022081401/558932f3d8b42ac7138b465b/html5/thumbnails/24.jpg)
24
Questions?
Matt Archer
August 2012
Twitter: @MattArcherUK
Blog: mattarcherblog.wordpress.com
Email: [email protected]