Wirespeed: Extending The Aff4 Container Format For ...
Transcript of Wirespeed: Extending The Aff4 Container Format For ...
![Page 1: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/1.jpg)
DIGITAL FORENSIC RESEARCH CONFERENCE
Wirespeed: Extending The Aff4 Container Format
For Scalable Acquisition And Live Analysis
By
Bradley Schatz
Presented At
The Digital Forensic Research Conference
DFRWS 2015 USA Philadelphia, PA (Aug 9th - 13th)
DFRWS is dedicated to the sharing of knowledge and ideas about digital forensics research. Ever since it organized
the first open workshop devoted to digital forensics in 2001, DFRWS continues to bring academics and practitioners
together in an informal environment. As a non-profit, volunteer organization, DFRWS sponsors technical working
groups, annual conferences and challenges to help drive the direction of research and development.
http:/dfrws.org
![Page 2: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/2.jpg)
Wirespeed:))Extending)the)AFF4)forensic)container)format)for)scalable)acquisi<on)and)live)
analysis))
Dr.)Bradley)Schatz)Director,)Schatz)Forensic)
)DFRWS)Conference)2015)©)Schatz)Forensic)2015)
)
![Page 3: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/3.jpg)
©)2015)Schatz)Forensic)
Overview)
• The)current)approach)to)forensic)acquisi<on)is)a)boOleneck)in)the)forensic)process)
• Propose)addi<ons)to)the)AFF4)container)format)to)support:)– Par<al)acquisi<on)– Acquisi<on)at)maximal)I/O)rates)
• Empirical)results)of)the)proposed)approach)
![Page 4: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/4.jpg)
Background+
+
+
![Page 5: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/5.jpg)
©)2015)Schatz)Forensic)
Pick)one)of)the)below)You)can’t)have)both)
Latency)
Completen
ess)
Physical)Acquisi<on)
Triage)
You)preserve)everything)but)
analysis)will)have)to)wait)
Near)immediate)results)at)the)expense)of)
poten<ally)missing)evidence)
Live)forensics)
![Page 6: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/6.jpg)
©)2015)Schatz)Forensic)
How)can)we)reduce)latency?)While)maximising)completeness)
Latency)
Completen
ess)
Physical)Acquisi<on)
Triage)
Increase)I/O)
throughput?)
Live)analysis)while)we)acquire?)
Dynamic)par<al)acquisi<on?))
Acquire/Analyse)by)priority?)
Live)forensics)
![Page 7: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/7.jpg)
©)2015)Schatz)Forensic)
Why)can’t)I)have)both?)The)deZfacto)standard)evidence)containers)are)a)limi<ng)factor.)
• Linear)complete)bit)stream)– All)or)nothing)preserva<on)choice)– Prevents)nonZlinear/priori<sed)preserva<on)of)evidence))
• Heavyweight)compression)(Inflate))– Limi<ng)factor)on)current)CPU’s)(even)with)mul<Zcore)threading))
• Linear)bytestream)hash)– Prevents)nonZlinear/priori<sed)preserva<on)– Hashing)is)limi<ng)factor)at)high)bitrates)and)with)low)CPU)resources)
• Single)storage)device)– Evidence)output)device)I/O)rate)o_en)limi<ng)factor)
• Logical)imaging)– Missing)raw)filesystem)and)volume)metadata)
)
![Page 8: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/8.jpg)
I/O+Throughput+
+
+
![Page 9: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/9.jpg)
©)2015)Schatz)Forensic)
Is)there)actually)a)problem)with)throughput)here?)
• Research)publica<ons)– FastDD)<=)110)MB/s)[Bertasi)&)Zago)2013])
• Prac<<oner)reports)– Low)100’s)MB/s)[Zimmerman)2013])
• Vendor)marke<ng)– Hardware)devices)promising)250MB/s)[Tableau)2014])
![Page 10: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/10.jpg)
©)2015)Schatz)Forensic)
I/O)throughput)in)acquisi<on)is)a)systems)problem)
Target)Storage) Compression) Hashing) Evidence)
storage)
Target+Storage+ Max+Read+
Current)genera<on)3.5”)7200rpm)SATA) 200)MB/s)
Intel)730)SSD) 550)MB/s)
Macbook)Pro)1TB)(real)data)) 100MB/s)
Macbook)Pro)1TB)(sparse)) 1)GB/s)
RAID)15000rpm)SAS) >)1)GB/s)
![Page 11: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/11.jpg)
©)2015)Schatz)Forensic)
Inflate)compression)is)costly)in)CPU)resources)
Target)Storage) Compression) Hashing) Evidence)
storage)
Algorithm+ Throughput+MB/s*+
Inflate) 39.42)Snappy)(Google)BigTable/MapReduce)) 1,405.42)LZO)(ZFS)) 1,538.31)
*Single)core)of)quad)core)i7Z4770)3.4Ghz)
![Page 12: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/12.jpg)
©)2015)Schatz)Forensic)
Hashing)is)the)next)most)expensive)acquisi<on)opera<on)
Target)Storage) Compression) Hashing) Evidence)
storage)
*Single)core)of)quad)core)i7Z4770)3.4Ghz)
Algorithm+ Throughput+MB/s+
SHA1)) 619.23)MD5)) 745.65)Blake2b)) 601.87)
![Page 13: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/13.jpg)
©)2015)Schatz)Forensic)
I/O)Rate)of)acquisi<on)is)a)systems)problem)
Target)Storage) Compression) Hashing) Evidence)
storage)
Output+ Gb/s+ MB/s+
SATA3) 6) 600)USB3) 5) 500)Commodity(SATA(7200(rpm( 200(Gigabit)Ethernet) 1) 100)USB2) .48) 48)
![Page 14: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/14.jpg)
Our+proposal+
Extensions+to+AFF4+
+
![Page 15: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/15.jpg)
©)2015)Schatz)Forensic)
AFF4)in)a)nutshell)
Image)(Map))
Compressed)block)storage)
• Virtual)block)storage)(Maps))– NonZlinear,)composable)
• Compressed)block)storage)(Streams))• Globally)unique)naming)scheme)• There)is)an)object)represen<ng)each)en<ty))
Zero)filled)storage)(aff4:Zero))
/mapID/map)/mapID/index)
/streamID/00000)/streamID/00000/index)/streamID/00001)/streamID/00001/index)…))
![Page 16: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/16.jpg)
©)2015)Schatz)Forensic)
Faster)compression)Symbolic)sec<ons)
• Extension:)we)define)virtual)bytestreams)“aff4:SymbolicStream00”)to)“aff4:SymbolicStreamFF”))
• Synonyms)aff4:Zero)and)aff4:FF)– Use)case:)Zero)filled)sectors)and)erased)flash)blocks)
0,0,aff4://0466b8fb-9af0-4ef2-b36c-8b0d90fc0ac2> 4096,0,aff4:SymbolicStreamFF 8192,4096, aff4://0466b8fb-9af0-4ef2-b36c-8b0d90fc0ac2>
AFF4+Map+example+
![Page 17: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/17.jpg)
©)2015)Schatz)Forensic)
Par<al)acquisi<on)
• Challenge:)Represen<ng)what)we)didn’t)acquire)• Extensions:)we)define)two)symbols)– aff4:UnreadableData):)Blocks(that(we(tried(to(read(but(couldn’t(
– aff4:UnknownData):)Blocks(that(we(never(even(tried(to(read(
0,0,aff4://0466b8fb-9af0-4ef2-b36c-8b0d90fc0ac2> 4096,0,aff4:UnknownData 8192,4096, aff4://0466b8fb-9af0-4ef2-b36c-8b0d90fc0ac2>
AFF4+Map+example+
![Page 18: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/18.jpg)
©)2015)Schatz)Forensic)
Faster)compression)More)speedZefficient)algorithms)
• Extension:)the)storage)stream)now)has)property)called)“aff4:compressionMethod”(
<aff4://0466b8fb-9af0-4ef2-b36c-8b0d90fc0ac2> a aff4:stream ;
aff4:CompressionMethod <http://code.google.com/p/snappy/> ;
aff4:chunk_size "32768"^^xsd:int ; aff4:size "294912"^^xsd:long ;
AFF4+Stream+example+
![Page 19: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/19.jpg)
©)2015)Schatz)Forensic)
Faster)hashing)NonZlinear)parallelised)block)hashing))
• Our)proposal:)– Deprecate)the)linear)bytestream)hash)– Parallelise)hashing)by)using)segment)hashes)– Hashing)symbolic)chunks)is)a)waste)of)CPU)resources)–)hash)the)map)instead)
– Take)a)singular)hash)of)the)above)hashes)
![Page 20: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/20.jpg)
©)2015)Schatz)Forensic)
Faster)hashing)NonZlinear)parallelised)block)hashing))
0) 1) 3) Image)(Map))
Stream)B)
2) 4)1)
1) 0) 2)4)
Stream)B)block)hashes)
h1) h0) h2)h4) blockHashesHash=sha256(h1,h0,h4,h2…))
mapPointHash=sha256(mapID/map))mapIndexHash=sha256(mapID/index))
blockHash=sha256(blockHashesHash0)..))blockHashesHashn).)mapPointHash).)mapIndexHash))))
![Page 21: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/21.jpg)
©)2015)Schatz)Forensic)
Aggregate)Output)Channels)• Use)the)aggregate)I/O)capacity)of)the)device)
Image)(Map))
Stream)B)Container+B+
Disk+B+
Disk+A+
Stream)A) Container+A+
![Page 22: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/22.jpg)
Experimental+validaFon+
+
![Page 23: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/23.jpg)
©)2015)Schatz)Forensic)
Methodology)• We)built)a)forensic)acquisi<on/analysis)system))
– NonZlinear,)par<al)acquisi<on)&)live)analysis)()called)Wirespeed)))• Prepare)testbed)
– Target)disk:)Intel)730)240G)SSD)(max)read)530)MB/s))– Des<na<on)disks:)Toshiba)2TB)7200RPM)SATA)(max)write)near)200)MB/s))
– Computer)1:)4)core)i7Z4770R)3.20GHz))– Computer)2:)2)core)i5Z3337U)1.80GHz)
• Prepare)test)sample)• Test,)varying)on)
– CPU)– IO)Channels)
![Page 24: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/24.jpg)
Test)standard)composi<on)Stored)block)size)–vZ)LBA)address)
Windows+8.1+
10.2G+
Govdocs1+
(1O75,1O40)+
59.8G+
/dev/random+
38.4G+
Empty+space+
(zeros)+
![Page 25: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/25.jpg)
Compression)is)faster)than)raw)Single)output)drive)
High)entropy)data)write)I/O)
limited)
Medium)entropy)data)throughput)exceeds)
max)output)throughput)Low)entropy)
data)read)throughput)limited)
![Page 26: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/26.jpg)
Mul<ple)output)channels)increases)throughput)Especially)for)uncompressible)data)
High)entropy)data)
![Page 27: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/27.jpg)
Block)based)hashing)beats)linear)stream)hashing)with)low)powered)mul<core)CPU’s))
Dual)core)i5)
Sparse)data)Max)CPU)hash)throughput)
Sparse)data)Read)I/O)limited)
![Page 28: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/28.jpg)
NonZlinear)par<al)imaging)gives)significant)gains)over)linear)
![Page 29: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/29.jpg)
©)2015)Schatz)Forensic)
The)proposed)approach)gives)significant)throughput)gains)over)
current)implementa<ons.)
AcquisiFon+
applicaFon+
I7O4770R+3.2+GHz+
system+
I5O3337U+1.8GHz+
system+
FTK)Imager) 20:10)(198)MB/s))
37:38)(106MB/s))
XZWays)Forensics) 13:58)(286)MB/s))
33:23)(120)MB/s))
Wirespeed)(linear)) 11:29))(384)MB/s))
15:08)(264)MB/s))
![Page 30: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/30.jpg)
©)2015)Schatz)Forensic)
Compara<ve)acquisi<on)speeds)
1+Stripe+ 2+Stripes+ 3+Stripes+
Wirespeed)(linear))
11:29))(384)MB/s))
8:00)(500)MB/s))
7:30)(533)MB/s))
FTK)Imager) 20:10)(198)MB/s))
N/A) N/A)
XZWays)Forensics)
13:58)(286)MB/s))
N/A) N/A)
Wirespeed)(allocated))
8:21)(229)MB/s))
4:42)(408)MB/s))
4:17)(447)MB/s))
![Page 31: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/31.jpg)
Conclusions+
+
+
![Page 32: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/32.jpg)
©)2015)Schatz)Forensic)
Conclusion)• Exis<ng)image)formats)are)a)limita<on)– Linear)byte)stream)hash))– Inflate)algorithm)
• Extensions)to)the)AFF4)format)proposed)– Faster)hashing)and)compression)– Par<al)images)– Exploita<on)of)aggregate)IO)channels)
• Proof)of)concept)demonstrated)significant)throughput)gains)and)improved)latency)
• Our)implementa<on)is)available)at:)– hOp://wirespeed.io)
©)2014)Schatz)Forensic)
![Page 33: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/33.jpg)
©)2015)Schatz)Forensic)
Conclusion)
©)2014)Schatz)Forensic)
![Page 34: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/34.jpg)
©)2015)Schatz)Forensic)
Conclusion)
©)2014)Schatz)Forensic)
![Page 35: Wirespeed: Extending The Aff4 Container Format For ...](https://reader034.fdocuments.in/reader034/viewer/2022051204/62783fd1d184e13a687dbf6e/html5/thumbnails/35.jpg)
©)2015)Schatz)Forensic)
Conclusion)
©)2014)Schatz)Forensic)