2015 National Startup Startup by Startup Ireland with Amarach Research
Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14
-
Upload
leanstartupconf -
Category
Business
-
view
460 -
download
0
Transcript of Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14
![Page 1: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/1.jpg)
Designing EffectiveExperiments
Grace Ng | @uxceo
Test your business idea in just 5 minutes
![Page 2: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/2.jpg)
Photo Q&A
![Page 3: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/3.jpg)
What is this?
![Page 4: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/4.jpg)
Minimum Viable Product An experiment that uses the least amount of effort
to get the most amount of customer feedback
![Page 5: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/5.jpg)
Minimum Viable Product An experiment that uses the least amount of effort to test the
Riskiest Assumption against real customer feedback
![Page 6: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/6.jpg)
Do people even have visual questions to post?
Riskiest Assumption: Have photo Qs to post.
Ask Question
Upload Photo
![Page 7: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/7.jpg)
What kind of bird is this? (Is it even a bird??)
![Page 8: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/8.jpg)
Will there be people who are able to answer the photo questions that are posted?
Riskiest Assumption: Someone can answer within given time.
![Page 9: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/9.jpg)
What kind of bird is this? (Is it even a bird??)
![Page 10: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/10.jpg)
Will this scale? Can this be replicated?
![Page 11: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/11.jpg)
![Page 12: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/12.jpg)
![Page 13: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/13.jpg)
![Page 14: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/14.jpg)
LEARN MEASURE
BUILD
![Page 15: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/15.jpg)
I Don’t Care...
![Page 16: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/16.jpg)
![Page 17: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/17.jpg)
$2M. 2 Years.
![Page 18: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/18.jpg)
Mistakes
![Page 19: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/19.jpg)
Effective Interviews
1. Tested the wrong assumptions
Mistakes
![Page 20: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/20.jpg)
Effective Interviews
2. Asked the wrong questions
Mistakes
![Page 21: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/21.jpg)
Effective Interviews
3. Didn’t define success
Mistakes
![Page 22: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/22.jpg)
Effective Interviews
4. Don’t need to build to test
Mistakes
![Page 23: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/23.jpg)
Case Study
![Page 24: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/24.jpg)
Founders UX Help
![Page 25: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/25.jpg)
Experiment: 1
Customer: Bootstrapped startup founders
![Page 26: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/26.jpg)
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
![Page 27: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/27.jpg)
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution:
![Page 28: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/28.jpg)
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Bootstrapped startup founders
Riskiest Assumption: No reliable source
![Page 29: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/29.jpg)
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Bootstrapped startup founders
Riskiest Assumption: No reliable source
Method & Success Criterion Interview. 6/10 frustrated
![Page 30: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/30.jpg)
1. Cost• Project Cost• Opportunity Cost
2. Existing metrics• Business KPI• Specific user behavior
How to MEASURE success?How to measure success?
![Page 31: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/31.jpg)
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Bootstrapped startup founders
Riskiest Assumption: No reliable source
Method & Success Criterion Interview. 6/10 frustrated
![Page 32: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/32.jpg)
Exploration Interviews
![Page 33: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/33.jpg)
Effective Interviews
1. I’m doing some research around...
2. Have you ever [experienced problem]?
3. Tell me about that time, what happened?
4. What did you do to resolve it?
5. How satisfied were you with the outcome?
6. If unsatisfied, how else have you tried to solve it?
Effective Interviews
![Page 34: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/34.jpg)
Collect Data
• Level of pain/frustration
• Frequency of encounter with pain
• Existing behavior to solve problem
• Time/money spent searching for a solution
Collect Data
![Page 35: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/35.jpg)
Experiment: 1
Result & Decision 9/12. Persevere
Learning Friends. Classes. Pay for UX help
Riskiest Assumption: No reliable source
Method & Success Criterion Interview. 6/10 frustrated
GET OUT OF THE BUILDING!
![Page 36: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/36.jpg)
Experiment: 2
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
![Page 37: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/37.jpg)
Experiment: 2
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Virtual UX feedback
Riskiest Assumption: Will they pay?
Method & Success Criterion Pitch. 8/10 will pay
![Page 38: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/38.jpg)
$10
![Page 40: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/40.jpg)
Experiment: 2
Result & Decision 10/10 paid. Test 2nd side of market
Learning Don’t mind virtual. Price is cheap.
Riskiest Assumption: Will they pay?
Method & Success Criterion Pitch. 8/10 will pay
GET OUT OF THE BUILDING!
![Page 41: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/41.jpg)
Experiment: 3
Customer: UX designers
Problem: Getting clients for consulting
![Page 42: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/42.jpg)
Experiment: 3
Customer:
Problem:
Solution: Virtual UX feedback
Riskiest Assumption: Will consult virtually
Method & Success Criterion Pitch. 8/10 will sign up
UX designers
Getting clients for consulting
![Page 43: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/43.jpg)
Seriously?
![Page 44: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/44.jpg)
Experiment: 3
Result & Decision 0/10. Pivot!
Learning Focus on < 5 years. 5-10 years have established client base.
Riskiest Assumption: Will consult virtually
Method & Success Criterion Pitch. 8/10 will sign up
GET OUT OF THE BUILDING!
![Page 45: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/45.jpg)
Experiment: 4
Customer: Transition to UX designers
Problem: Getting clients for consulting
![Page 46: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/46.jpg)
Experiment: 4
Customer:
Problem:
Solution: Virtual UX feedback
Riskiest Assumption: Will consult virtually
Method & Success Criterion Pitch. 60% will sign up
Transition to UX designers
Getting clients for consulting
![Page 48: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/48.jpg)
Google Ads
![Page 49: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/49.jpg)
No Backend
![Page 51: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/51.jpg)
3 Data Points@uxceo @nycuxpa #ngnycuxpa@uxceo
3 Data Points
![Page 52: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/52.jpg)
Experiment: 4
Result & Decision 37/184 emails. Persevere *
Learning * Did not meet success metric, but visitors referred friends
Riskiest Assumption: Will consult virtually
Method & Success Criterion Pitch. 60% will sign up
GET OUT OF THE BUILDING!
![Page 55: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/55.jpg)
Experiment: 5
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Virtual UX feedback
![Page 56: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/56.jpg)
Experiment: 5
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Virtual UX feedback
Riskiest Assumption: Online service is sufficient
Method & Success Criterion Concierge. 60% satisfied
![Page 57: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/57.jpg)
Concierge@uxceo [email protected]
Concierge
Customer submits request
UX designer gets email and submits feedback
![Page 58: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/58.jpg)
Manually@uxceo [email protected]
Manually
Customer submits request
UX designer gets email and submits feedback
![Page 59: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/59.jpg)
Experiment: 5
Result & Decision 20% satisfied. Iterate
Learning Solved problem, but wanted more interaction with designer
Riskiest Assumption: Online service is sufficient
Method & Success Criterion Concierge. 60% satisfied
GET OUT OF THE BUILDING!
![Page 60: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/60.jpg)
High PainLow Pain
Low Frequency
High Frequency
Focus
![Page 62: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/62.jpg)
![Page 63: Tactics for Truly Effective Experiment Design by Grace Ng - The Lean Startup Conference 12/11/14](https://reader034.fdocuments.in/reader034/viewer/2022052623/559ee6f01a28abcb618b4574/html5/thumbnails/63.jpg)
Grace Ng | @uxceo
Test your business idea in just 5 minutes
Thank You!