Measuring Real-World Accuracies and Biases in Modeling ... · Guessability in Practice. 24 Single...
114
1 Blase Ur , Sean M. Segreti, Lujo Bauer, Nicolas Christin, Lorrie Faith Cranor, Saranga Komanduri, Darya Kurilova, Michelle L. Mazurek, William Melicher, Richard Shay Measuring Real-World Accuracies and Biases in Modeling Password Guessability
Transcript of Measuring Real-World Accuracies and Biases in Modeling ... · Guessability in Practice. 24 Single...
1
Blase Ur, Sean M. Segreti, Lujo Bauer, Nicolas Christin, Lorrie Faith Cranor, Saranga Komanduri, Darya Kurilova, Michelle L. Mazurek, William Melicher, Richard Shay
Measuring Real-World Accuracies and Biases in
Modeling Password Guessability
2
How strong is a
particular password?
3
iloveyou
4
iloveyou
5
n(c$JZX!zKc^bIAX^N
6
j@mesb0nd007!
7
Why Measure Password Strength?
8
Why Measure Password Strength?
• Eliminate bad passwords
– Organizational password audits
9
Why Measure Password Strength?
• Eliminate bad passwords
– Organizational password audits
• Help users make better passwords
10
Why Measure Password Strength?
• Eliminate bad passwords
– Organizational password audits
• Help users make better passwords
– Determine if interventions are effective
11
Why Measure Password Strength?
• Eliminate bad passwords
– Organizational password audits
• Help users make better passwords
– Determine if interventions are effective
– Provide users feedback
12
Password-Strength Metrics
13
Password-Strength Metrics
• Statistical approaches
– Traditionally: Shannon entropy
– Recently: α-guesswork
14
Password-Strength Metrics
• Statistical approaches
– Traditionally: Shannon entropy
– Recently: α-guesswork
• Disadvantages for researchers
– No per-password estimates
– Huge sample required
15
Parameterized Guessability
• How many guesses a particular cracking
algorithm with particular training data
would take to guess a password
16
j@mesb0nd007!
Guess # 366,163,847,194
17
Guess # past cutoff
n(c$JZX!zKc^bIAX^N
18
Guessability Plots
19
Guessability Plots
20
Guessability Plots
21
Advantages of Guessability
• Straightforward
• Models an attacker
• Per-password strength estimates
22
Guessability in Practice
23
Guessability in Practice
24
Single Cracking Approach
25
Default Configuration
26
Questions About Guessability
27
Questions About Guessability
1) How does guessability used in research
compare to an attack by professionals?
28
Questions About Guessability
1) How does guessability used in research
compare to an attack by professionals?
2) Would substituting another cracking
approach impact research results?
29
Approach
30
4 password sets
5 password-cracking approaches
Approach
password
iloveyou
teamo123
…
passwordpassword
1234567812345678
!1@2#3$4%5^6&7*8
…
Pa$$w0rd
iLov3you!
1QaZ2W@x
…
pa$$word1234
12345678asDF
!q1q!q1q!q1q
…
31
Four Password Sets
32
Four Password Sets
• Basic (3,062): 8+ characters
password
33
Four Password Sets
• Basic (3,062): 8+ characters
password
• Complex (3,000): 8+ characters, 4 classes
Pa$$w0rd
34
Four Password Sets
• Basic (3,062): 8+ characters
password
• Complex (3,000): 8+ characters, 4 classes
Pa$$w0rd
• LongBasic (2,054): 16+ characters
passwordpassword
35
Four Password Sets
• Basic (3,062): 8+ characters
password
• Complex (3,000): 8+ characters, 4 classes
Pa$$w0rd
• LongBasic (2,054): 16+ characters
passwordpassword
• LongComplex (990): 12+ characters, 3+ classes
pa$$word1234
36
Five Cracking Approaches
• John the Ripper
• Hashcat
• Markov models
• Probabilistic Context-Free Grammar
• Professionals
37
• Guesses variants of input wordlist
John the Ripper
38
• Guesses variants of input wordlist
• Wordlist mode requires:
– Wordlist (passwords and dictionary entries)
– Mangling rules
John the Ripper
39
• Guesses variants of input wordlist
• Wordlist mode requires:
– Wordlist (passwords and dictionary entries)
– Mangling rules
• Speed: Fast
John the Ripper
40
• Guesses variants of input wordlist
• Wordlist mode requires:
– Wordlist (passwords and dictionary entries)
– Mangling rules
• Speed: Fast
– 1013 guesses
John the Ripper
41
• Guesses variants of input wordlist
• Wordlist mode requires:
– Wordlist (passwords and dictionary entries)
– Mangling rules
• Speed: Fast
– 1013 guesses
• “JTR”
John the Ripper
42
John the Ripper
wo
rdlis
t
rule
s
gue
sse
s
43
John the Ripper
usenix
security
wo
rdlis
t
rule
s
gue
sse
s
44
John the Ripper
usenix
security
[ ]
[add 1 at end]
[change e to 3]
wo
rdlis
t
rule
s
gue
sse
s
45
usenix
security
[ ]
[add 1 at end]
[change e to 3]
usenix
security
usenix1
security1
us3nix
s3curity
John the Ripper
wo
rdlis
t
rule
s
gue
sse
s
46
usenix
security
[ ]
[add 1 at end]
[change e to 3]
usenix
security
usenix1
security1
us3nix
s3curity
John the Ripper
wo
rdlis
t
rule
s
gue
sse
s
47
usenix
security
[ ]
[add 1 at end]
[change e to 3]
usenix
security
usenix1
security1
us3nix
s3curity
John the Ripper
wo
rdlis
t
rule
s
gue
sse
s
48
Hashcat
• Guesses variants of input wordlist
49
Hashcat
• Guesses variants of input wordlist
• Wordlist mode requires:
– Wordlist (passwords and dictionary entries)
– Mangling rules
50
Hashcat
• Guesses variants of input wordlist
• Wordlist mode requires:
– Wordlist (passwords and dictionary entries)
– Mangling rules
• Speed: Fast
51
Hashcat
• Guesses variants of input wordlist
• Wordlist mode requires:
– Wordlist (passwords and dictionary entries)
– Mangling rules
• Speed: Fast
– 1013 guesses
52
Hashcat
wo
rdlis
t
rule
s
gue
sse
s
53
Hashcat
usenix
security
[ ]
[add 1 at end]
[change e to 3]
wo
rdlis
t
rule
s
gue
sse
s
54
usenix
security
[ ]
[add 1 at end]
[change e to 3]
usenix
usenix1
us3nix
security
security1
s3curity
Hashcat
wo
rdlis
t
rule
s
gue
sse
s
55
usenix
security
[ ]
[add 1 at end]
[change e to 3]
usenix
usenix1
us3nix
security
security1
s3curity
Hashcat
wo
rdlis
t
rule
s
gue
sse
s
56
Markov Models
• Predicts future characters from previous
57
Markov Models
• Predicts future characters from previous
• Approach requires weighted data:
– Passwords
– Dictionaries
58
Markov Models
• Predicts future characters from previous
• Approach requires weighted data:
– Passwords
– Dictionaries
• Ma et al. IEEE S&P 2014
59
Markov Models
• Predicts future characters from previous
• Approach requires weighted data:
– Passwords
– Dictionaries
• Ma et al. IEEE S&P 2014
• Speed: Slow
60
Markov Models
• Predicts future characters from previous
• Approach requires weighted data:
– Passwords
– Dictionaries
• Ma et al. IEEE S&P 2014
• Speed: Slow
– 1010 guesses
61
Probabilistic Context-Free Grammar
• Generate password grammar
– Structures
– Terminals
62
Probabilistic Context-Free Grammar
• Generate password grammar
– Structures
– Terminals
• Kelley et al. IEEE S&P 2012
– Based on Weir et al. IEEE S&P 2009
63
Probabilistic Context-Free Grammar
• Generate password grammar
– Structures
– Terminals
• Kelley et al. IEEE S&P 2012
– Based on Weir et al. IEEE S&P 2009
• Speed: Slow Medium
64
Probabilistic Context-Free Grammar
• Generate password grammar
– Structures
– Terminals
• Kelley et al. IEEE S&P 2012
– Based on Weir et al. IEEE S&P 2009
• Speed: Slow Medium
– 1014 guesses
65
Probabilistic Context-Free Grammar
• Generate password grammar
– Structures
– Terminals
• Kelley et al. IEEE S&P 2012
– Based on Weir et al. IEEE S&P 2009
• Speed: Slow Medium
– 1014 guesses
• “PCFG”
66
Professionals (“Pros”)
67
Professionals (“Pros”)
• Contracted KoreLogic
– Password audits for Fortune 500 companies
– Run DEF CON “Crack Me If You Can”
68
Professionals (“Pros”)
• Contracted KoreLogic
– Password audits for Fortune 500 companies
– Run DEF CON “Crack Me If You Can”
• Proprietary wordlists and configurations
69
Professionals (“Pros”)
• Contracted KoreLogic
– Password audits for Fortune 500 companies
– Run DEF CON “Crack Me If You Can”
• Proprietary wordlists and configurations
– 1014 guesses
70
Professionals (“Pros”)
• Contracted KoreLogic
– Password audits for Fortune 500 companies
– Run DEF CON “Crack Me If You Can”
• Proprietary wordlists and configurations
– 1014 guesses
– Manually tuned, updated
71
4 password sets 5 approaches
Approach
password
iloveyou
teamo123
…
passwordpassword
1234567812345678
!1@2#3$4%5^6&7*8
…
Pa$$w0rd
iLov3you!
1QaZ2W@x
…
pa$$word1234
12345678asDF
!q1q!q1q!q1q
…
72
Outline of Results
• Importance of Configuration
• Comparison of Approaches
• Impact on Research Analyses
73
Configuration Is Crucial
LongComplex
74
Configuration Is Crucial
LongComplex
75
Configuration Is Crucial
LongComplex
76
Configuration Is Crucial
LongComplex
77
Configuration Is Crucial
LongComplex
78
Outline of Results
• Importance of Configuration
• Comparison of Approaches
• Impact on Research Analyses
79
Comparison for Basic Passwords
80
Comparison for Basic Passwords
81
Comparison for Basic Passwords
82
Comparison for Basic Passwords
83
Comparison for Basic Passwords
84
Comparison for Basic Passwords
85
Comparison for Complex Passwords
86
Comparison for Complex Passwords
87
Comparison for Complex Passwords
88
Comparison for Complex Passwords
89
Comparison for Complex Passwords
90
Comparison for Complex Passwords
91
Comparison for Complex Passwords
92
Min_auto Conservative Proxy for Pros
93
Outline of Results
• Importance of Configuration
• Comparison of Approaches
• Impact on Research Analyses
94
Impact on Research
• Coarse-grained analyses
• Fine-grained analyses
• Analysis of one password
95
Impact on Research
• Coarse-grained analyses same results
• Fine-grained analyses
• Analysis of one password
96
Impact on Research
• Coarse-grained analyses same results
• Fine-grained analyses different
• Analysis of one password
97
Impact on Research
• Coarse-grained analyses same results
• Fine-grained analyses different
• Analysis of one password different
98
Per-Password Highly Impacted
P@ssw0rd!
99
Per-Password Highly Impacted
• JTR guess # 801
P@ssw0rd!
100
Per-Password Highly Impacted
• JTR guess # 801
• Not guessed in 1014 PCFG guesses
P@ssw0rd!
101
Per-Password Highly Impacted
• JTR guess # 801
• Not guessed in 1014 PCFG guesses
P@ssw0rd!
102
Per-Password Highly Impacted
12345678password
103
Per-Password Highly Impacted
• PCFG guess # 130,555
12345678password
104
Per-Password Highly Impacted
• PCFG guess # 130,555
• Not guessed in 1010 JTR guesses
12345678password
105
Conclusions
106
Conclusions
• Running a single approach is insufficient
– Especially out of the box
107
Conclusions
• Running a single approach is insufficient
– Especially out of the box
• Min_auto conservative proxy for pros
108
Conclusions
• Running a single approach is insufficient
– Especially out of the box
• Min_auto conservative proxy for pros
• Coarse-grained analyses same results
• Fine-grained analyses different
• Analysis of one password different
109
Password Guessability Service (PGS)
• Guessability of plaintext passwords
https://pgs.ece.cmu.edu
110
Password Guessability Service (PGS)
• Guessability of plaintext passwords
https://pgs.ece.cmu.eduasdfF123#
P@ssw0rd!
Qwertyuiop!1
…
111
Password Guessability Service (PGS)
• Guessability of plaintext passwords
https://pgs.ece.cmu.eduasdfF123#
P@ssw0rd!
Qwertyuiop!1
…
"Guess #", "Password"
"127188816", "Qwertyuiop!1"
"1853004462", "asdfF123#"
"2251762491", "P@ssw0rd!"
...
112
Password Guessability Service (PGS)
• Guessability of plaintext passwords
https://pgs.ece.cmu.edu
"Guess #", "Password"
"127188816", "Qwertyuiop!1"
"1853004462", "asdfF123#"
"2251762491", "P@ssw0rd!"
...
asdfF123#
P@ssw0rd!
Qwertyuiop!1
…
113
Blase Ur, Sean M. Segreti, Lujo Bauer, Nicolas Christin, Lorrie Faith Cranor, Saranga Komanduri, Darya Kurilova, Michelle L. Mazurek, William Melicher, Richard Shay
https://pgs.ece.cmu.edu
Measuring Real-World Accuracies and Biases in
Modeling Password Guessability
114
• Per Thorsheim and Jeremi Gosney (PasswordsCon)
• Hashcat / JTR developers
• Matt Marx (@tehnlulz)
• Jerry Ma, Weining Yang, Ninghui Li (Purdue)
• KoreLogic (@CrackMeIfYouCan)
• Dustin Heywood (@Evil_Mog)
• Jonathan Bees
• Michael Stroucken and Chuck Cranor (CMU)
114
Acknowledgments
