Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.
-
Upload
erika-perkins -
Category
Documents
-
view
216 -
download
2
Transcript of Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.
![Page 1: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/1.jpg)
HashingTHEN AND NOW
MIKE SMORUL – ADAPT PROJECT
![Page 2: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/2.jpg)
Commodity Storage Performance
2003 JetStor III IDE-FC62MB/s large block
2013 218MB/s workstation SSDPerc 6/MD1000, 400MB/s+
![Page 3: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/3.jpg)
Chip Speed
2003: Pentium 4 3.2Ghz
2013: Core i7 Extreme3.5Ghz
![Page 4: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/4.jpg)
Hashing Performance
SHA-256 HashingJava: 85MB/sCrypto++: 111-134MB/s
Real World PenaltyJava: 20-40% penalty on
slow seek disk
![Page 5: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/5.jpg)
Implications
Flipped bottlenecks
![Page 6: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/6.jpg)
How to overcome
Faster/weaker digestsSimultaneous transfers
Data locality, tape?Improve single stream
performance
![Page 7: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/7.jpg)
Parallelize Single Stream
Independent IO and digest threads
Always have work for the digest algorithm.
Large files saw over 95% of algorithm potential.
Small files unchanged.
![Page 8: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/8.jpg)
Securing Data in Motion
?
![Page 9: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/9.jpg)
Where to apply fixity
Internal integrity servicesAt Transfer via manifestsEnd to End?
![Page 10: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/10.jpg)
Operational Integrity
Internal AuditingProve your hardwareError, not malice
detectionPeer-Auditing
Prove your friends
![Page 11: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/11.jpg)
Transporting Integrity
Manifest ListsTransfer validation
Digital SignaturesProve identity
Token BasedProve time
![Page 12: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/12.jpg)
Chronopolis Integrity
Current:Producer supplied
authoritative manifestPeers locally monitor
integrityManually trace back to
point of ingest
![Page 13: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/13.jpg)
Chronopolis Integrity
In-progressSingle integrity token back
to ingestIdeal
Tokens issued prior to arrival‘Prove’ the state of data to
point before Chronopolis
![Page 14: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/14.jpg)
Manifests 2.0
Beyond simple transfer listToken manifestsPortable, embeddable
Python, etc
![Page 15: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/15.jpg)
Cloud Integrity
Digests in a cloud validate transfer only
Http headers can pass extended integrity informationEnd-user verification
![Page 16: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/16.jpg)
Integrity as provenance
Integrity checking forward in timeConsumer level verification of
data
Integrity from object creationStart integrity checking before
archiving
![Page 17: Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT.](https://reader031.fdocuments.in/reader031/viewer/2022013101/56649efb5503460f94c0e2a0/html5/thumbnails/17.jpg)
Closing
Why are you hashing?What do you want to
prove?Hashing Cost/performance