Adaptive Testing Methodology [ ATM ]
-
Upload
daniel-miessler -
Category
Software
-
view
658 -
download
3
Transcript of Adaptive Testing Methodology [ ATM ]
![Page 1: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/1.jpg)
ATM: Adaptive Testing Methodology
Daniel MiesslerDirector of Advisory ServicesIOActive
![Page 2: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/2.jpg)
Web hacking in pictures
Image from stopherdingcats.com
![Page 3: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/3.jpg)
Concepts
![Page 4: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/4.jpg)
“ I used to think we had security problems, and then we figured out how to integrate the security solution.
Actually, the security basics are long figured out, it’s the integration that's killing us. We don't have a security problem with integration requirements. We have an integration problem with security requirements.
~ Gunnar Peterson
http://1raindrop.typepad.com/1_raindrop/2013/11/there-are-no-security-problems.html
![Page 5: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/5.jpg)
My take on Gunnar’s thought
1. Security is an integration problem
2. It’s not that we don’t know what to do
3. It’s that we don’t know how to integrate what we know (or learn) into what we do
![Page 6: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/6.jpg)
Security is an integration problem
![Page 7: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/7.jpg)
Two ways to learn: Osmotic vs. Algorithmic
VS
![Page 8: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/8.jpg)
Osmotic learning
1. Consume a talk/book/video about testing SAP2. Don’t fall asleep3. Mostly pay attention to the content4. Say, “hmm…” to yourself 1-3 times5. Maybe jot something down on a piece of paper
you’ll never see again6. Don’t remember any/most of it when you do
the task next
![Page 9: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/9.jpg)
Algorithmic learning
1. You already care about testing SAP a lot2. For this reason, you already have an algorithm for
doing so3. You also like to learn more about it (seminars/etc.)4. When you learn something new, you immediately
update your methodology with anything legit5. The very next time you test SAP, you have
directly benefitted from the talk/video/book you consumed
![Page 10: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/10.jpg)
![Page 11: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/11.jpg)
Algorithmic vs. Osmotic learning
![Page 12: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/12.jpg)
Web methodologies are monolithic
199 Pages
94 Pages
=
![Page 13: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/13.jpg)
Web methodologies lack context
![Page 14: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/14.jpg)
Web methodologies lack empathy
“The customer wants you to find everything you can in 13 minutes.”
![Page 15: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/15.jpg)
Methodologies are hard to update
![Page 16: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/16.jpg)
Review– Security is an integration problem
![Page 17: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/17.jpg)
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies
![Page 18: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/18.jpg)
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic
![Page 19: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/19.jpg)
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive
![Page 20: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/20.jpg)
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have
![Page 21: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/21.jpg)
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have– Methodologies are hard to update
![Page 22: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/22.jpg)
Review (propositions, challenges)– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have– Methodologies are hard to update
![Page 23: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/23.jpg)
Adaptive Testing Methodology (ATM)
![Page 24: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/24.jpg)
Methodology
![Page 25: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/25.jpg)
Methodology (project)
https://github.com/danielmiessler/ATM
![Page 26: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/26.jpg)
Methodology (content)
1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470
![Page 27: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/27.jpg)
Methodology (content)
1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470
2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf
![Page 28: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/28.jpg)
Methodology (content)
1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470
2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf
3. OWASP Web Testing Guidehttps://www.owasp.org/images/5/52/OWASP_Testing_Guide_v4.pdf
![Page 29: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/29.jpg)
Methodology (content)
1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470
2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf
3. OWASP Web Testing Guidehttps://www.owasp.org/images/5/52/OWASP_Testing_Guide_v4.pdf
4. Jason Haddix’s Bughunter Methodologyhttps://appsecusa2015.sched.org/event/3kXN/the-bug-hunters-methodology
** initial compilation / curation done by me
![Page 30: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/30.jpg)
ATM Concepts
![Page 31: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/31.jpg)
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)
Image by knotlikeyou2 of Deviant Art
![Page 32: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/32.jpg)
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools
![Page 33: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/33.jpg)
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT
![Page 34: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/34.jpg)
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT– Flexibility based on conditions
![Page 35: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/35.jpg)
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT– Flexibility based on conditions– Transparency
![Page 36: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/36.jpg)
Methodology (structure)
![Page 37: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/37.jpg)
Methodology (structure)
![Page 38: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/38.jpg)
Methodology (structure) [ technology ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.
![Page 39: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/39.jpg)
Methodology (structure) [ technology ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.Apache | 30M | 1H | 1D | 2D | UL | Check text here.
![Page 40: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/40.jpg)
Methodology (structure) [ technology ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.Apache | 30M | 1H | 1D | 2D | UL | Check text here.Wordpress PHP | 30M | 1H | 1D | 2D | UL | Check text here.
![Page 41: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/41.jpg)
Methodology (structure) [ time ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.
![Page 42: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/42.jpg)
Methodology (structure) [ time ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.PHP | 30M | 1H | 1D | 2D | UL | Check text here.
![Page 43: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/43.jpg)
Methodology (structure) [ time ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.PHP | 30M | 1H | 1D | 2D | UL | Check text here.Express | 30M | 1H | 1D | 2D | UL | Check text here.
![Page 44: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/44.jpg)
Execution
![Page 45: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/45.jpg)
Execution
1. Client makes a request to ATM service
![Page 46: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/46.jpg)
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
![Page 47: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/47.jpg)
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
3. ATM service tests the domain for its stack
![Page 48: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/48.jpg)
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
3. ATM service tests the domain for its stack4. ATM service receives stack information
![Page 49: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/49.jpg)
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
3. ATM service tests the domain for its stack4. ATM service receives stack information5. ATM service parses the current
methodology for rules that match the stack and time combination given
![Page 50: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/50.jpg)
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
3. ATM service tests the domain for its stack4. ATM service receives stack information5. ATM service parses the current
methodology for rules that match the stack and time combination given
6. ATM service returns the custom set of methodology checks to the client
![Page 51: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/51.jpg)
Execution (visual)
SITE
CLIENT ATM
![Page 52: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/52.jpg)
Execution (visual)
SITE
CLIENT ATM(send domain/time)
[1]
![Page 53: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/53.jpg)
Execution (visual)
SITE
CLIENT ATM(send domain/time)
[1][2]
(checks site stack)
![Page 54: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/54.jpg)
Execution (visual)
SITE
CLIENT ATM(send domain/time)
[1][2]
(checks site stack)
[3] ATM parses checks
![Page 55: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/55.jpg)
Execution (visual)
SITE
CLIENT ATM(send domain/time)
[1][2]
(checks site stack)
[3] ATM parses checks
(returns checks to client)
[4]
![Page 56: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/56.jpg)
Demo
![Page 57: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/57.jpg)
- Contextual security testing- Crowdsourced updates via Github- Adjusts to technology stack- Adjusts to your time constraints- Produces customized testing for your app
![Page 58: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/58.jpg)
Next steps
Improve the methodologies (currently alpha, way more to add) Add additional factors (continuous monitoring via proxy logs) Add other types of context (besides stack and time) Add other types of testing (network/forensics/etc) Determine best time increments (community) Improve performance of the stack detection (multithreading) Create this is public service infrastructure that can be used with
various methodologies and clients Determine if I should do anything with the domains (stack-
check.com / adaptivetestingmethodology.com) (meh) Explore local implementations (non-service-based) for product
integrations
![Page 59: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/59.jpg)
Announcement: Portswigger and ATM
Coming to Burpsuite Soon!
** Ask me about RobotsDisallowed and SecLists integration as well
![Page 60: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/60.jpg)
Thanks
Daffyd Stuttard and the WAHH OWASP for ASVS and Web Testing Methodology Jason Haddix for the Bug Hunter’s Methodology Nestor Mata Cuthbert for help with Wordpress IOActive for being a phenomenal place to work
![Page 61: Adaptive Testing Methodology [ ATM ]](https://reader036.fdocuments.in/reader036/viewer/2022062412/587199d61a28ab044e8b5743/html5/thumbnails/61.jpg)
- [ PROJECT ] github.com/danielmiessler/ATM- [ SERVICE ] danielmiessler.com/services/atm
- [ TWITTER ] twitter.com/danielmiessler- [ MAIL ] [email protected] [ MAIL ] [email protected]