Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research...
Transcript of Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research...
![Page 1: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/1.jpg)
Format-Transforming Encryption(more than meets the DPI)
Tom Shrimpton
Florida Institute for Cybersecurity ResearchUniversity of Florida
![Page 2: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/2.jpg)
In-place encryption of CC database
“Looks benign, let it pass”.
Encrypt
“HTTP: … free+speech+democracy …”
TCP/IP ciphertext payload
Circumvention of nation-state internet censorship
1234 5678 9876 5432 4417 1234 5678 9112Encrypt
Deep-packet inspection (DPI)
Monday
Today
![Page 3: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/3.jpg)
3
Encrypt
key
plaintext
Traditional encryption is ill-suited for these tasks
ciphertext
Natively, plaintexts are bit strings(not 16-digit decimal strings) Traditional security goal:
make ciphertexts indistinguishable from random bit strings
(not well-formatted HTTP messages or CC #s)
![Page 4: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/4.jpg)
4
FTE
key
ciphertext
Format-Transforming Encryption
plaintext format
ciphertext format
FTE is like traditional encryption, with the extra operational requirement that ciphertexts abide by the ciphertext format
(“target”)
(“helper info”)
(inspired by Bellare et al. “Format-Preserving Encryption”)
in the specified format
plaintext
A format is a set.
![Page 5: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/5.jpg)
Flexibility is “baked in” to the syntax
FTE
key
plaintext format
ciphertext format(“target”)
(“helper info”)
plaintextciphertext
To change the “look” of ciphertexts, just change the ciphertext format.The system doesn’t (necessarily) need to change.
5
![Page 6: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/6.jpg)
Let’s consider the censorship-circumvention setting
FTE
TCP/IP FTE ciphertext payload
DPI
6
![Page 7: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/7.jpg)
In this setting, shouldn’t assume anything about plaintext formats…
FTE
TCP/IP ciphertext payload
FTE
key
ciphertext{0,1}*
ciphertext format(“target”)
(“helper info”)
plaintext
7
![Page 8: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/8.jpg)
8
FTE
TCP/IP ciphertext payload
… so let’s focus on this simpler API
FTE
key
ciphertext
ciphertext format(“target”)
plaintext
![Page 9: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/9.jpg)
9
FTE
TCP/IP ciphertext payload
“FTP” ciphertext format
Our goal: to cause real DPI systems to reliably misclassify plaintext traffic
for example, HTTP misclassified as FTP
“This is an FTP message.”
![Page 10: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/10.jpg)
FTE
TCP/IP ciphertext payload
Our goal: to cause real DPI systems to reliably misclassify our (plaintext) traffic as whatever protocol we want
(while still having good throughput, low latency…)
arbitrary ciphertext format
![Page 11: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/11.jpg)
We wondered:How do real DPI devices determine to what protocol a message belongs?
“This is an _____ message.”
System Classification Tool Price
appid free
l7-filter free
YAF free
bro free
nProbe ~300 Euros
DPI-X ~$10K
Enterprise grade DPI, well-known company
11
![Page 12: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/12.jpg)
We wondered:How do real DPI devices determine to what protocol a message belongs?
“This is an _____ message.”
Regular langauges/expressionsfigure heavily in state-of-the-art DPI classification tools
System Classification Tool Price
appid Regular expressions free
l7-filter Regular expressions free
YAF Regular expressions (sometimes hierarchical)
free
broSimple regular expression triage,
then additional parsing and heuristicsfree
nProbe Parsing and heuristics (many of them “regular”) ~300 Euros
DPI-X ??? ~$10K
12
![Page 13: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/13.jpg)
13
FTE
key
plaintext ciphertext in L(R)
How should we realize regex-based FTE?
We want:Cryptographic protection for the plaintext
Ciphertexts in L(R)
Regular-expression-based FTE
regex R
Regex defines the ciphertext formatL(R)
![Page 14: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/14.jpg)
14
key
plaintext ciphertext in L(R)
regex R
encryption
How should we realize regex-based FTE?
We want:Cryptographic protection for the plaintext
Ciphertexts in L(R)
Realizing regex-based FTE
![Page 15: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/15.jpg)
L(R)
Ranking a Regular Language
0 1 2 |L(R)|-1i
Let L(R) be lexicographically orderedx0< x1 < … < xi < … < x|L(R)-1|
xi
x2
rank(xi)=i
unrank(2)=x2
With precomputed tables,rank, unrank are O(n)
[Goldberg, Sipser ’85][Bellare et al. ’09]
rank: L(R) {0,1,…,|L(R)|-1}unrank: {0,1,…,|L(R)|-1} L(R)
such that rank( unrank(i) ) = iand unrank( rank(xi) ) = xi
Given a DFA for L(R), there are efficient algorithms
15
![Page 16: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/16.jpg)
key
plaintext ciphertext in L(R)
regex R
Realizing regex-based FTE
encryption
unrank
regex-to-DFA
Intermediate ciphertext, interpreted as an integer n…
…outputs nth string in lexicographic orderingof L(R)
16
![Page 17: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/17.jpg)
regex-basedFTE
key
plaintext a string in L(R)
regex R
Now all we need are good regular expressions
We considered three options :
1. If the DPI is open source (appid, l7-filter, YAF), try to extract them, directly!
2. Build them manually, using RFCs and (when possible) DPI source code.
3. Learn them from traffic that was allowed by the DPI.
17
![Page 18: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/18.jpg)
Use case: Browsing the web through an FTE tunnel
RtargetFTE client FTE proxy
Rtarget
Internet
FTE “wins” if the DPI classifies the stream it sees as the target protocol
FTE ciphertextsregular expressions for HTTP, SSH, SMB, … messages
Using each “target” format, we visited each of the Top 50 websites five times.
18
![Page 19: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/19.jpg)
RtargetFTE client
input protocol stream
FTE proxy
input protocol stream
Rtarget
Punchline: regex-based FTE can make realDPI say whatever we want it to ~100% of
the time.
“Help!”
19
![Page 20: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/20.jpg)
Browser experiencethrough FTE tunnel
⇡ Browser experience through SSH tunnel
FTE library is open-source, runs on multiple platforms/OS, and is fully integrated with major circumvention efforts
Eric Schmidt gave us a sizable unsolicited research gift
![Page 21: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/21.jpg)
A field test…
FTE client
Internet
FTE proxy
Ran various tests every 5 minutes for one month, no sign of detection in logs. (We shut it down after that.)
Used FTE to download Tor bundle:
Tor without FTE: “active blacklisting” attack on proxyTor through FTE: no problems
Without FTE tunnel, we tried Facebook, YouTube, Tor website, banned search queries…
With FTE tunnel, we tried Facebook, YouTube, Tor website, banned search queries…
21
![Page 22: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/22.jpg)
1234 5678 9876 54324417 1234 5678 9112
What about in-place encryption of CC database?
regex-basedFTE
key
regex for language of 16-decimal digit CC #s
22
![Page 23: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/23.jpg)
1234 5678 9876 5432
CC# regex
|plaintext language| = |ciphertext language|
keyencryption
unrank
regex-to-DFA
4417 1234 5678 9112
1) valid 16-digit number in, valid 16-digit number out
2) conventional encryption takes bit strings as input
encoding of valid 16-digit strings into bitstrings
expands the effective plaintext space
3) conventional encryption has ciphertext stretchcan have exponential number of AE ciphertexts that
cannot be unranked!
Not quite handled by “simpler” FTE construction
![Page 24: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/24.jpg)
24
FTE
key
ciphertext
Recall the full FTE API…
plaintext format
ciphertext format
(“helper info”)
plaintext
![Page 25: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/25.jpg)
“rank-encrypt-unrank” FTE construction
encrypt
unrankregex-to-DFA
regex-to-DFA
rank
key
ptxtin L(R1)
ctxtin L(R2)ptxt regex
R1ctxt regex
R2
(generalization of Bellare et al. SAC’09)
ranking provides optimal compression of L(R)
25
![Page 26: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/26.jpg)
“rank-encrypt-unrank” FTE construction
Great potential… but developers face many hard questions:
-- Can I even use R1 and R2 together? (Requires |L(R1)| ≤ |L(R2)|)
-- Will both R1 and R2 admit time/space efficient implementations of (un)ranking?
-- Should “encrypt” be deterministic (i.e. a cipher) or can I use traditional encryption?
-- …
encrypt
unrankregex-to-DFA
regex-to-DFA
rank
key
ptxtin L(R1)
ptxt regex R1
ctxt regex R2
ctxtin L(R2)
26
![Page 27: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/27.jpg)
The space/memory issue
encrypt
unrankregex-to-DFA
regex-to-DFA
rank
key
ptxt
regexR1
regexR2
ctxt
regex NFA DFA
For some regular expressions, this works out just fine…
unranking requires space linear in the size of the DFA,
27
![Page 28: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/28.jpg)
encrypt
unrankregex-to-DFA
regex-to-DFA
rank
key
ptxt
regexR1
regexR2
ctxt
…for others, you can have an exponential space blow-up
regex NFA DFA
unranking requires space linear in the size of the DFA,
The space/memory issue
28
![Page 29: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/29.jpg)
encrypt
regex-to-NFA
regex-to-NFA
rank
key
ptxt
regexR1
regexR2
ctxt
Wanted: efficient (un)ranking methods that work directly from the NFA representation
unrank
The space/memory issue
regex NFA DFA
Problem: (un)ranking from NFAs (or directly from a regex) is PSPACE-complete
29
![Page 30: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/30.jpg)
relaxed rank-encrypt-unrank FTE construction
encryptrelaxedunrank
regex-to-NFA
regex-to-NFA
relaxedrank
key
ptxt
regexR1
regexR2
ctxt
Wanted: efficient (un)ranking methods that work directly from the NFA representation
Problem: (un)ranking from NFAs (or directly from a regex) is PSPACE-complete
We side-step this by developing a new “relaxed ranking” algorithm
regex NFA DFA
30
![Page 31: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/31.jpg)
Ranking of a language from a DFA
0 1 2 |I|-1i
xi rank(xi)=i and unrank(i)=xi
(strings)
original representation
p
intermediate representation
(accepting DFA paths)
L(R) I
=|L(R)|-1
1-1 correspondence between strings and accepting paths
efficient alg. for 1-1 mapping between paths and integers+
31
![Page 32: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/32.jpg)
0 1 2 |I|-1i
xi
(strings)
original representation
p
intermediate representation
(accepting DFA paths)
L(R) I
=|L(R)|-1
deterministic encrypt
unrankregex-to-DFA
regex-to-DFA
rank
R
R
EncK(i)
encrypt and decrypt are done over this set
c
“Rank”
32
![Page 33: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/33.jpg)
0 1 2 |I|-1i
xi
(strings)
original representation
p
intermediate representation
(accepting DFA paths)
L(R) I
=|L(R)|-1
EncK(i)
c
xc
(strings)
targetrepresentation
q
intermediate representation
(paths)
L(R)I
“rank” ”encrypt” ”unrank”
33
![Page 34: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/34.jpg)
Ranking of a language from an NFA
0 1 2 |I|-1i
xi
(strings)original representation
p
intermediate representation(accepting NFA paths)
L(R)I
p
p
p
34
![Page 35: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/35.jpg)
RelaxedRanking of a language from an NFA
0 1 2 |I|-1i
xi
(strings)original representation
p
intermediate representation(accepting NFA paths)
L(R)I
p
p
p
First, need a 1-1 mapping from strings to distinguished paths…
…Then, can use a modified version of path-ranking algorithm for a 1-1 map from I to [0…|I|-1]
35
![Page 36: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/36.jpg)
0 1 2 |I|-1i
xi
(strings)original representation
p
intermediate representation(accepting NFA paths)
L(R)I
>> |L(R)|-1
First, need a 1-1 mapping from strings to distinguished paths…
Image of L(R) under first mapping
…Then, can use a modified version of path-ranking algorithm for a 1-1 map from I to [0…|I|-1]
RelaxedRanking of a language from an NFA
36
![Page 37: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/37.jpg)
0 1 2 |I|-1i
xi
(strings)original representation
p
intermediate representation(accepting NFA paths)
L(R)I
>> |L(R)|-1
encrypt
unrankregex-to-NFA
regex-to-NFA
rank
R
R
j
r
Image of L(R) under first mapping
We use “cycle-walking” andrejection sampling tricks to deal with this sort of problem
encrypt and decrypt are done over this set…
RelaxedRanking of a language from an NFA
37
![Page 38: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/38.jpg)
0 1 2 |I|-1i
xi
(strings)original representation
p
intermediate representation(accepting NFA paths)
L(R)I
>> |L(R)|-1
encrypt
unrankregex-to-NFA
regex-to-NFA
rank
encrypt and decrypt are done over this set…
j
r
Image of L(R) under first mapping
We use “cycle-walking” andrejection sampling tricks to deal with this sort of problem
R
R
RelaxedRanking of a language from an NFA
38
![Page 39: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/39.jpg)
LibFTE (https://libfte.org)
encrypt (relaxed) unrank
regex-to-{N,D}FA
regex-to-{N,D}FA
(relaxed) rank
key
ptxt
regexR1
regexR2
ctxtdeterm./randomized,
cycle-walking,rej. sampling
LibFTE is a library (python, C++ APIs) that supports this framework
Provides a configuration tool to help developers make good, well informed design choices
![Page 40: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/40.jpg)
LibFTE Configuration Assistant
26
input: input format, output format, and optional restrictions (e.g., encryption must be randomized/deterministic)
output: an error OR a list of schemes that satisfy the user-specified constraints, with statistics (no. cycle walks, etc.)
$ ./configuration-assistant \ > --input-format "(a|b)*a(a|b){16}" 0 64 \ > --output-format "[0-9a-f]{16}" 0 16 !==== Identifying valid schemes ==== No valid schemes. ERROR: Input language size greater than output language size. $ !!!!!!
error
$ ./configuration-assistant \ > --input-format "(a|b)*a(a|b){16}" 0 32 \ > --output-format "[0-9a-f]{16}" 0 16 !==== Identifying valid schemes ==== WARNING: Memory threshold exceeded when building DFA for input format VALID SCHEMES: T-ND, T-NN, T-ND-$, T-NN-$ !==== Evaluating valid schemes ==== SCHEME ENCRYPT DECRYPT ... MEMORY T-ND 0.32ms 0.31ms ... 77KB T-NN 0.39ms 0.38ms ... 79KB … $
OR
success
LibFTE configuration assistant
Input: input regex, output regex, operational restrictions (e.g. encryption must be randomized/deterministic)
Output: ERROR or a list of predefined FTE schemes that satisfy the restrictions, with statistics
![Page 41: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/41.jpg)
Tackling the next challenge
uniform random
bits
encrypt
unrankregex-to-DFA
regex-to-DFA
rank
key
ptxt
regexR1
regexR2
ctxt
uniform integer
uniform ciphertext
![Page 42: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/42.jpg)
Tackling the next challenge
uniform random
bits
encrypt
encode
ptxttranslation
key
ptxt
“translationinfo.”
ctxtformat
ctxt
Prob
abili
ty
Ciphertext
‘a’ ‘b’ ‘c’ ‘d’
0.25 0.25 0.25 0.25
Let’s expand the idea of a format from a set to a distribution…
…and generalize unranking to encoding of bits into the ctxt format
![Page 43: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/43.jpg)
Tackling the next challenge
uniform random
bits
encrypt
encode
ptxttranslation
key
ptxt
“translationinfo.”
ctxtformat
ctxt
How should we handle this?
Prob
abili
ty
Ciphertext
‘a’ ‘b’ ‘c’ ‘d’
0.24
0.49
0.25
0.01 0.01
‘e’
How does one invertibly sample from a non-uniform distribution using uniform bits? (Additionally, when the number of ctxts in the format is not a power of two?)
![Page 44: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/44.jpg)
Prob
abili
ty
Ciphertext
‘a’ ‘b’ ‘c’ ‘d’
0.24
0.49
0.25
0.01 0.01
‘e’
Prob
abili
ty
Ciphertext
‘a’‘b’ ‘c’ ‘d’
0.24
0.49
0.25
0.01 0.01
‘e’
Prob
abili
ty
{‘b’} {‘c’,‘a’} {‘d’ ‘e’}
0.49 0.49
0.02
1. Sort the ciphertexts by probability mass
2. Collect into bins that are (a) a power of two in size, (b) all ciphertexts within a bin have probabilities
that are “close” (this is a controllable parameter)
Prob
abili
ty
{‘b’} {‘c’,‘a’} {‘d’ ‘e’}
0.49 0.49
0.02
3. Sample a bin according to itstotal probability mass
Prob
abili
ty
{‘b’} {‘c’,‘a’} {‘d’ ‘e’}
0.49 0.49
0.02
4. Sample within the bin using (uniform) input bits
Bin
Bin Bin
![Page 45: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/45.jpg)
Prob
abili
ty
Ciphertext
‘a’ ‘b’ ‘c’ ‘d’
0.24
0.49
0.25
0.01 0.01
‘e’
Prob
abili
ty
{‘b’} {‘c’,‘a’} {‘d’ ‘e’}
0.49 0.49
0.02
Bin
Pr[A] = Pr[A | { C,A }] Pr[{C,A }]= (0.5)(0.49)
= 0.245
Note: roughly half the time we encode zero bits!
On average, 0.51 bits per sample
![Page 46: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/46.jpg)
Prob
abili
ty
Ciphertext
‘a’ ‘b’ ‘c’ ‘d’
0.24
0.49
0.25
0.01 0.01
‘e’
Prob
abili
ty
{‘b’,‘c’} {‘a’,‘d’ } {‘e’}
0.74
0.260.01
Bin
Pr[A] = Pr[A | { A,D }] Pr[{A,D }]= (0.5)(0.26)
= 0.13
On average, 1 bit per sample
Bin size Fidelity of sampling
Bigger/heavier bins,more bits encoded!
Smaller/lighter bins,smaller sampling error!
vs.
![Page 47: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/47.jpg)
encryptencode
ptxttranslation
key
ptxt
“translationinfo.”
ctxtformat
ctxtApproximate, invertible sampling
Determining the format can be quite challenging…
Distribution depends on granularity/alphabet
How do you actually assert a particular distribution on acompact set-representation (e.g. a regex?)
![Page 48: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/48.jpg)
Markov Chain
Probabilistic CFG
![Page 49: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/49.jpg)
Markov Chain
Probabilistic CFG Machine-Learned Models
![Page 50: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/50.jpg)
encryptencode
ptxttranslation
key
ptxt
“translationinfo.”
ctxtformat
ctxtApproximate, invertible sampling
In submission: using machine-learned generative models as formats.
![Page 51: Tom Shrimpton - Radboud Universiteit · Tom Shrimpton Florida Institute for Cybersecurity Research ... DPI source code. 3. ... regex NFA DFA For some regular ...](https://reader031.fdocuments.in/reader031/viewer/2022021801/5b31291c7f8b9ab5728bfdee/html5/thumbnails/51.jpg)
Format-Transforming Encryption(more than meets the DPI)
Tom Shrimpton
Florida Institute for Cybersecurity ResearchUniversity of Florida