WORKSHOP SUMMARY NHGRI Training and Career Development Programs
NHGRI Cloud Computing talk
-
Upload
deepak-singh -
Category
Technology
-
view
3.244 -
download
0
description
Transcript of NHGRI Cloud Computing talk
![Page 1: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/1.jpg)
Scien&fic Compu&ng with Amazon Web ServicesDeepak Singh
NHGRI Cloud Compu&ng Mee&ng, Bal&more, 2010
![Page 2: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/2.jpg)
AWS + science = win
![Page 3: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/3.jpg)
![Page 4: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/4.jpg)
![Page 5: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/5.jpg)
![Page 6: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/6.jpg)
![Page 7: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/7.jpg)
![Page 8: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/8.jpg)
scale has implications
![Page 9: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/9.jpg)
data management
![Page 10: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/10.jpg)
data processing
![Page 11: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/11.jpg)
data sharing
![Page 12: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/12.jpg)
Image: Chris Dagdigian
![Page 13: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/13.jpg)
![Page 14: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/14.jpg)
amazon web services
![Page 15: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/15.jpg)
the cloud
![Page 16: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/16.jpg)
has_many :definitions
![Page 17: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/17.jpg)
infrastructure as a service
![Page 18: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/18.jpg)
![Page 19: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/19.jpg)
ComputeAmazon Elastic Compute
Cloud (EC2)- Elastic Load Balancing- Auto Scaling
StorageAmazon Simple
Storage Service (S3)- AWS Import/Export
Your Custom Applications and Services
Content DeliveryAmazon CloudFront
MessagingAmazon Simple
Queue Service (SQS)
PaymentsAmazon Flexible Payments Service
(FPS)
On-Demand Workforce
Amazon Mechanical Turk
Parallel ProcessingAmazon Elastic
MapReduce
MonitoringAmazon CloudWatch
ManagementAWS Management Console
ToolsAWS Toolkit for EclipseAWS Toolkit for .NET
Isolated NetworksAmazon Virtual Private
Cloud
DatabaseAmazon RDS and
SimpleDB
![Page 20: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/20.jpg)
• Lower pricing tiers for Cloudfront • AWS Management Console
• New SimpleDB Features • FPS General Availability
• EC2 Reserved Instances • EC2 with Windows • EC2 in EU • AWS Toolkit for Eclipse
• Reserved Instances in EU • Elastic MapReduce • SQS in EU
• AWS Import/Export • Monitoring, Auto Scaling, and Elastic Load Balancing • CloudFront adds access logging
• AWS Security Center • Console support for Cloudfront
• Elastic MapReduce in EU
• AWS Multi Factor Authentication • Virtual Private Cloud private beta • Lower Reserved Instance Pricing • Console Support for CloudWatch
• EBS Shared Snapshots • SimpleDB in EU • Monitoring in EU • Auto Scaling in EU • Elastic Load Balancing in EU • AWS Solutions Provider program
• RDS Launched • High Memory Instances • Reduced EC2 Pricing • EMR Apache Hive support
• SAS 70 Type II Audit • AWS SDK for .NET • CloudFront Private Content • APAC announced
• Boot from EBS • US West Region • VPC Unlimited Beta • ELB Support in Console • CloudFront streaming • EC2 Spot Instances • Windows 2008 Support • Lowered Prices • AWS Economics Center
![Page 21: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/21.jpg)
elasticity
![Page 22: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/22.jpg)
3000 CPU’s for one firm’s risk management application
!"#$%&'()'*+,'-./01.2%/'
344'+567/'(.'
8%%9%.:/'
;<"&/:1='
>?,3?,44@'
A&B:1='
>?,>?,44@'
C".:1='
>?,D?,44@'
E(.:1='
>?,F?,44@'
;"%/:1='
>?,G?,44@'
C10"&:1='
>?,H?,44@'
I%:.%/:1='
>?,,?,44@'
3444JJ'
344'JJ'
![Page 23: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/23.jpg)
scale
![Page 24: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/24.jpg)
> 1PB of data in S3
![Page 25: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/25.jpg)
![Page 26: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/26.jpg)
highly availability
![Page 27: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/27.jpg)
Image: Chris Dagdigian
![Page 28: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/28.jpg)
![Page 29: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/29.jpg)
“Everything fails, all the time”-- Werner Vogels
![Page 30: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/30.jpg)
![Page 31: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/31.jpg)
“Things will crash. Deal with it”-- Jeff Dean
![Page 32: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/32.jpg)
2-4% of serverswill die annually
Source: Jeff Dean, LADIS 2009
![Page 33: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/33.jpg)
1-5% of disk drives will die every year
Source: Jeff Dean, LADIS 2009
![Page 34: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/34.jpg)
human errors
![Page 35: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/35.jpg)
human errors~20% admin issues have unintended consequences
Source: James Hamilton
![Page 36: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/36.jpg)
scalable & available
![Page 37: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/37.jpg)
assume sw/hw failure
![Page 38: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/38.jpg)
design apps to be resilient
![Page 39: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/39.jpg)
automation & alarming
![Page 40: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/40.jpg)
US East Region
Availability Zone A
Availability Zone B
Availability Zone C
Availability Zone D
!"#$%&'()*+
T
TT
![Page 41: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/41.jpg)
![Page 42: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/42.jpg)
elastic load balancing
CloudWatch
auto scaling
elastic block store
elastic IP
SQS
![Page 43: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/43.jpg)
flexibility
![Page 44: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/44.jpg)
![Page 45: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/45.jpg)
on-demand instancesreserved instances
spot instances
![Page 46: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/46.jpg)
![Page 47: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/47.jpg)
some implications
![Page 48: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/48.jpg)
computing platforms
![Page 49: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/49.jpg)
![Page 50: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/50.jpg)
![Page 51: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/51.jpg)
http://cyclecomputing.comhttp://wiki.github.com/documentcloud/cloud-crowd
sudo gem install cloud-crowd
![Page 53: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/53.jpg)
Input S3 bucket
Output S3 bucket
Amazon S3
Hadoop
Amazon EC2 Instances
Input dataset
outputresults
Deploy Application
Web Console, Command line tools
End
Notify
Get ResultsInput Data
Amazon Elastic MapReduce
Hadoop Hadoop
Hadoop
Hadoop
Hadoop
Elastic MapReduce
Elastic MapReduce
![Page 54: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/54.jpg)
application platforms
![Page 57: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/57.jpg)
![Page 58: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/58.jpg)
Image: O’Reilly Radar
![Page 59: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/59.jpg)
software distribution
![Page 61: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/61.jpg)
http://bitbucket.org/galaxy/galaxy-central/wiki/Home
![Page 62: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/62.jpg)
data distribution
![Page 63: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/63.jpg)
![Page 64: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/64.jpg)
http://aws.amazon.com/publicdatasets/
![Page 65: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/65.jpg)
to conclude
![Page 66: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/66.jpg)
![Page 67: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/67.jpg)
built for scale
![Page 68: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/68.jpg)
built for availability
![Page 69: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/69.jpg)
shared dataspacescommon namespaces
![Page 70: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/70.jpg)
task-based resources
![Page 71: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/71.jpg)
new software architectures
![Page 72: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/72.jpg)
new computing platforms
![Page 73: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/73.jpg)
Data Platform
App Platform
![Page 74: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/74.jpg)
available today
![Page 76: NHGRI Cloud Computing talk](https://reader034.fdocuments.in/reader034/viewer/2022052523/55514ec0b4c905c6268b5464/html5/thumbnails/76.jpg)
[email protected] Twi2er:@mndoci Presenta7on ideas from James Hamilton, @mza, and @lessig
Thank you!