(SPOT208) Managing the Pace of Innovation: Behind the Scenes at AWS | AWS re:Invent 2014
Managing terabytes on aws
-
Upload
baruch-sadogursky -
Category
Technology
-
view
641 -
download
0
description
Transcript of Managing terabytes on aws
![Page 1: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/1.jpg)
![Page 2: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/2.jpg)
![Page 3: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/3.jpg)
![Page 4: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/4.jpg)
![Page 5: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/5.jpg)
![Page 6: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/6.jpg)
![Page 7: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/7.jpg)
![Page 8: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/8.jpg)
![Page 9: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/9.jpg)
![Page 10: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/10.jpg)
![Page 11: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/11.jpg)
![Page 12: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/12.jpg)
![Page 13: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/13.jpg)
Request
Storage File Listing Checksum Storage
![Page 14: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/14.jpg)
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
![Page 15: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/15.jpg)
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
![Page 16: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/16.jpg)
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
GET request Cache check, FS request
DB + FS request
![Page 17: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/17.jpg)
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
GET request Cache check, FS request
DB + FS request
PUT request FS request DB + FS request
![Page 18: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/18.jpg)
![Page 19: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/19.jpg)
![Page 20: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/20.jpg)
![Page 21: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/21.jpg)
![Page 22: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/22.jpg)
![Page 23: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/23.jpg)
![Page 24: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/24.jpg)
![Page 25: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/25.jpg)
![Page 26: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/26.jpg)
![Page 27: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/27.jpg)
![Page 28: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/28.jpg)
![Page 29: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/29.jpg)
![Page 30: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/30.jpg)
![Page 31: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/31.jpg)
![Page 32: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/32.jpg)
![Page 33: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/33.jpg)
![Page 34: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/34.jpg)
![Page 35: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/35.jpg)
![Page 36: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/36.jpg)
![Page 37: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/37.jpg)
![Page 38: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/38.jpg)
EBS + LVM
= WIN!
![Page 39: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/39.jpg)
/data
250GB 250GB
![Page 40: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/40.jpg)
/data
250GB 250GB 250GB 250GB
![Page 41: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/41.jpg)
/data
250GB 250GB 250GB 250GB 250GB 250GB
![Page 42: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/42.jpg)
![Page 43: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/43.jpg)
![Page 44: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/44.jpg)
![Page 45: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/45.jpg)
![Page 46: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/46.jpg)
![Page 47: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/47.jpg)
![Page 48: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/48.jpg)
![Page 49: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/49.jpg)
![Page 50: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/50.jpg)
![Page 51: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/51.jpg)
Backup files
![Page 52: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/52.jpg)
Backup files
Backup database
![Page 53: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/53.jpg)
Backup files
Backup database
Keep them in sync!
![Page 54: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/54.jpg)
Backup files
Backup database
Keep them in sync!
![Page 55: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/55.jpg)
Backup only deltas
![Page 56: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/56.jpg)
Backup only deltas
Manage threads to
ec2 efficiently
![Page 57: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/57.jpg)
Backup only deltas
Manage threads to
ec2 efficiently
Keep the data
isolated
![Page 58: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/58.jpg)
Backup only deltas
Manage threads to
ec2 efficiently
Keep the data
isolated
![Page 59: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/59.jpg)
![Page 60: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/60.jpg)
![Page 61: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/61.jpg)
![Page 62: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/62.jpg)
![Page 63: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/63.jpg)
![Page 64: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/64.jpg)
![Page 65: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/65.jpg)
Customer EBS
Customer EBS
Customer EBS
![Page 66: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/66.jpg)
De-duped
EBS
HA Storage
Customer EBS
Customer EBS
Customer EBS
![Page 67: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/67.jpg)
![Page 68: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/68.jpg)
De-duped Glacier
De-duped
EBS Backup
HA Storage
Customer EBS
Customer EBS
Customer EBS
![Page 69: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/69.jpg)
![Page 70: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/70.jpg)
![Page 71: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/71.jpg)
![Page 72: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/72.jpg)
![Page 73: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/73.jpg)
![Page 74: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/74.jpg)
Create empty Artifactory instance
![Page 75: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/75.jpg)
Create empty Artifactory instance
Populate from back
storage on demand
![Page 76: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/76.jpg)
Create empty Artifactory instance
Populate from back
storage on demand
Drink
Cosmopolitan
![Page 77: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/77.jpg)
Conclusions time
1. Be lazy.
![Page 78: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/78.jpg)
![Page 79: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/79.jpg)
Conclusions time
1. Be lazy.
2. Be smart.
![Page 80: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/80.jpg)
Conclusions time
1. Be lazy.
2. Be smart.
3. Be agile.
![Page 81: Managing terabytes on aws](https://reader033.fdocuments.in/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/81.jpg)
Drink
Cosmopolitan