Cloud Computing Lec2 - Arvutiteaduse instituut · Amazon Elastic Compute Cloud (EC2) • One of the...
Transcript of Cloud Computing Lec2 - Arvutiteaduse instituut · Amazon Elastic Compute Cloud (EC2) • One of the...
Basics of Cloud Computing – Lecture 2
Cloud Providers, SciCloud and
Research on Cloud at UTResearch on Cloud at UT
Satish Srirama
Outline
• Cloud computing services – recap
• Amazon cloud services
– Elastic Compute Cloud (EC2)
– Storage services - Amazon S3 and EBS– Storage services - Amazon S3 and EBS
• Cloud managers
• Eucalyptus
• SciCloud and Research at UT
10.04.2013 2/45Satish Srirama
Cloud Computing – Services - Recap
• Software as a Service – SaaS– A way to access applications
hosted on the web through your web browser
• Platform as a Service – PaaS– A pay-as-you-go model for IT
resources accessed over the
SaaS
Facebook, Flikr, Myspace.com,
Google maps API, Gmail
Level of
Abstraction
resources accessed over the Internet
• Infrastructure as a Service –IaaS– Use of commodity computers,
distributed across Internet, to perform parallel processing, distributed storage, indexing and mining of data
– Virtualization
PaaS
Google App Engine,
Force.com, Hadoop, Azure, Amazon S3, etc
IaaSAmazon EC2, SciCloud,
Joyent Accelerators, Nirvanix
Storage Delivery Network, etc.
10.04.2013 3/45Satish Srirama
Cloud Infrastructure
• Provisioning of computing resources
– CPU, Memory, Processing
– Basically an “Operating System” on demand
• Usually billed on a per-hour usage model• Usually billed on a per-hour usage model
• Players in this space
– Amazon EC2, Rackspace, GoGrid, Eucalyptus based SciCloud
• Management providers: RightScale, ElasticFox, Amazon Management Console
10.04.2013 4/45Satish Srirama
Cloud Storage
• Provisioning of data storage
– Either file/object based or Database like functionality
• Billed on bandwidth and storage consumed
• Players in the space• Players in the space
– Amazon S3, Amazon EBS
– Amazon’s SimpleDB, Google’s BigTable, Apache
Cassandra
• Management Providers: , Amazon Management
Console, Jungle Disk, Elephant Disk, PutPlace.com
10.04.2013 5/45Satish Srirama
Cloud Platforms
• Provides a complete software stack
– An IDE for the cloud
• Takes care of: Runtimes, Load balancing, Resource provisioningResource provisioning
• Players in the space
– Google AppEngine - Python (initially, now also Java)
– Force.com (SalesForce)
– Hadoop
10.04.2013 6/45Satish Srirama
Cloud Applications
• Applications that are completely ‘online’
• Operate on data that is stored in the ‘cloud’ or ‘ether’
• No client software generally required
• Billing: Ad. Revenue, Premium Services• Billing: Ad. Revenue, Premium Services
• Players in this space– Google Apps - Gmail / Google Docs
– Apple’s MobileMe
– Microsoft’s Live - Hotmail, Live Spaces
– SalesForce.com
10.04.2013 7/45Satish Srirama
Other cloud services
• Provides services, which other applications
can utilize
• Usually free for non-commercial use
• Players in the space• Players in the space
– Google/Yahoo Maps
– Google / Yahoo Web Services
– Amazon Merchant Services
– Amazon Simple Queue Service
10.04.2013 8/45Satish Srirama
Providers we focus at
• Amazon Web Services– Amazon EC2
– Amazon S3
– Amazon EBS
• Private cloud enabling technologies• Private cloud enabling technologies– Eucalyptus
– OpenStack
– SciCloud
• Management providers– ElasticFox
– RightScale
10.04.2013 9/45Satish Srirama
Amazon Elastic Compute Cloud (EC2)
• One of the very early pioneers of cloud computing
• In a nutshell:
– On Demand “Operating System”
• Complete virtual computer with CPU, Memory and disk spacespace
– Based on the XEN virtual image platform
• Variety of operating systems available
– Linux (Fedora, Ubuntu, CentOS, etc)
– Open Solaris
– Microsoft Windows
10.04.2013 10/45Satish Srirama
EC2 continued…
• Very simple pricing structure– CPU hours
– Machine size
– Bandwidth in and out of cloud
• Extremely FAST start up• Extremely FAST start up– 2-3 minutes from start to finish
• Instance snapshotting
• Very large/generous disk space provisioning– 160GB minimum for the standard instances
• Flexible API to control everything
• Wide range of virtual machine types
10.04.2013 11/45Satish Srirama
EC2 Instance types*
Instance CPU Memory Storag
e
Platform API Name Price (per h.)
Small
(default)
1 EC2
computing
unit
1.7 GB 160 GB 32/64 bit m1.small $0.06 (*nix)
$0.091 (win)
Medium 2 EC2 units 3.75 GB 410 GB 32/64 bit m1.medium $0.12 (*nix)
$0.182 (win)$0.182 (win)
Large 4 EC2 units 7.5 GB 850 GB 64 bit m1.large $0.24 (*nix)
$0.364 (win)
Extra Large 8 EC2 units 15 GB 1 690
GB
64 bit m1.xlarge $0.48 (*nix)
$0.728 (win)
Micro Up to 2 EC2
units
613 MB EBS
storage
only
32/64 bit t1.micro $0.02 (*nix)
$0.02 (win)
*Data taken on 08.04.2013 http://aws.amazon.com/ec2/instance-types/
10.04.2013 12/45Satish Srirama
EC2 advanced Instance types*
Instance CPU Memor
y
Storag
e
Platfor
m
API Name Price (per h.)
High-Memory
Instances
6.5 - 26
EC2
units
17.1 –
68.4 GB
420 -
1690
GB
64 bit m2.xlarge
m2.2xlarge
m2.4xlarge
Available at:
http://aws.a
mazon.com/
ec2/pricing/
High-CPU 5, 20 1.7, 7 350, 64 bit c1.medium Same as
• Other types– High I/O Instances– High Storage Instances
High-CPU
Instances
5, 20
EC2
units
1.7, 7
GB
350,
1690
GB
64 bit c1.medium
c1.xlarge
Same as
above
Cluster
Compute
Instances
88 EC2
units
60.5 GB 3370
GB
64 bit cc2.8xlarge Same as
above
*Data taken on 08.04.2013 http://aws.amazon.com/ec2/instance-types/10.04.2013 13/45Satish Srirama
EC2 Terminology
• Having an account– Access Key, Secret Key, Security group
• Availability Zone
• Amazon Machine Image (AMI)– A Virtual Machine File
– ami-XXXX – ami-XXXX
– Stored in a special bucket in Amazon’s S3
• Public and Private instances available– Private instances incur only S3 storage costs
• A Running Machine– Amazon Instance ( i-XXXX )
– Booted/Created from an Amazon Image
• Elastic IP addresses
10.04.2013 14/45Satish Srirama
Troubles with EC2
• On power-off all hard disk data is lost
• IP addresses are assigned at random
• Can’t turn off public IP address
• Do not forget to terminate the instances• Do not forget to terminate the instances
10.04.2013 15/45Satish Srirama
Simple Storage Service (S3)
• Enables you to upload, download, and store data across the Internet
• Buckets store data
– Buckets are the fundamental container in Amazon S3 for data storagedata storage
– 100 buckets for account
– No limit on no of objects that can be stored in a bucket
– Can store up to 5 TB of data in one object
– Object stores Data and Metadata
– Objects stored in a Region never leave the Region
– You cannot modify or append data to an existing object
10.04.2013 16/45Satish Srirama
Simple Storage Service (S3) - continued
• Objects are retrieved via a unique, developer-assigned key
• Keys– Example: http://doc.s3.amazonaws.com/2006-03-01/AmazonS3.wsdl
"doc" is the bucket name and "2006-03-01/AmazonS3.wsdl" is the key.
• Prices– Storage: $0.095 per GB/Month–
– Reduced Redundancy Storage : $0.076 per GB/Month
– Data Transfer 0ut from S3 to AWS different zone: $0.02/GB
– PUT, COPY, POST, or LIST Requests: $0.005 per 1,000 Requests
– GET requests: $0.004 per 10,000 Requests
• Download Data– Downloading is possible from any where
– You can enable others to download and can charge them (Using Amazon DevPay)
http://aws.amazon.com/s3/
10.04.2013 17/45Satish Srirama
Elastic Block Storage (EBS)
• The answer to the persistence problem
• Raw unformatted external block devices
• Allocate 1GB to 1TB volumes
• 20 volumes per account
• Format with your own choice of file system • Format with your own choice of file system
• Attach to running instances in the same availability zone
• Automatically replicated to prevent data loss
• Create snapshots for backup, or to create new volumes from
10.04.2013 18/45Satish Srirama
Basic ec2-api-tools
• ec2-describe-images
• ec2-describe-instances
• ec2-run-instances
• ec2-create-volume• ec2-create-volume
• ec2-attach-volume
• ec2-allocate-address
• ec2-associate-address
• ec2-terminate-instances
10.04.2013 19/45Satish Srirama
Other cloud services from Amazon
• AWS management console
• Amazon Simple Queue Service (SQS)
• Amazon DynamoDB
• Amazon Relational Database Service• Amazon Relational Database Service
• Amazon CloudWatch
• Amazon Elastic MapReduce
• Elastic Load Balancing
• etc.
10.04.2013 20/45Satish Srirama
Getting started
• Signup for an AWS account
– Use your existing Amazon.com account if you
want
• Register a credit card• Register a credit card
– Billed on the 1st of every month for previous
month usage
• http://aws.amazon.com/ec2
10.04.2013 21/45Satish Srirama
10.04.2013 22/45Satish Srirama
Getting started – for free
• http://aws.amazon.com/free/ - free tier
• Valid for 12 months
• 750 hours of EC2 Linux Micro Instance usage
• 5 GB of S3 standard storage, 20,000 Get Requests, and 2,000 Put Requests2,000 Put Requests
• 30 GB of Amazon EBS
• 25 Amazon SimpleDB Machine Hours and 1 GB of Storage
• Be careful
– You cross the limits, you are charged without any notice
10.04.2013 23/45Satish Srirama
ElasticFox
• Mozilla Firefox extension for managing your Amazon EC2 account
• Supported services
– List available AMIs – List available AMIs
– List your running instances
– Launch new instances of an AMI
– Manage security groups and launch permissions associated with your instances
– Manage Amazon EBS volumes
10.04.2013 24/45Satish Srirama
ElasticFox - GUI
10.04.2013 25/45Satish Srirama
Independent software vendors (ISV)
for S3
S3 Explorer filicio.us Jungle Disk
S3 Firefox Organizer MyOwnDB
10.04.2013 26/45Satish Srirama
RightScale
• RightScale cloud management platforms
• Automated configuration– From boot to production on auto-pilot
– Server templates & Right scripts
– Cloning/ replication of set-ups– Cloning/ replication of set-ups
• Automated management– Monitoring, alerts, SLAs -> actions
• Have several examples
• Can login as a developer https://my.rightscale.com
10.04.2013 27/45Satish Srirama
PRIVATE CLOUD ENABLING
TECHNOLOGIES
10.04.2013 28Satish Srirama
Eucalyptus
• Are you OK with using your credit card?
• Open source project
• Elastic Utility Computing Architecture Linking Your Programs To Useful Systems
• Web services based implementation of EC2 • Web services based implementation of EC2 infrastructure
• For establishing private clouds
• Functions as a software overlay– Existing installation should not be violated (too much)
• Focus on installation and maintenance
10.04.2013 29/45Satish Srirama
Eucalyptus Usage
• Experimentation vehicle prior to buying commercial services– Provide development, debugging, and “tech preview”
platform for Public Clouds
• Homogenize local IT environment with Public Clouds– AWS functionality locally makes moving & using Amazon – AWS functionality locally makes moving & using Amazon
AWS easier, cheaper, and more sustainable
• Provide a basic software development platform for the open source community– E.g. the “Linux Experience”
• Not designed as a replacement technology for AWS or any other Public Cloud service
10.04.2013 30/45Satish Srirama
Eucalyptus - architecture
Amazon EC2 Interface
10.04.2013 31/45Satish Srirama
Basic euca2ools
• euca-describe-images
• euca-describe-instances
• euca-run-instances
• euca-create-volume• euca-create-volume
• euca-attach-volume
• euca-terminate-instances
• euca-describe-availability-zones
10.04.2013 32/45Satish Srirama
OpenStack
• Founded by NASA and Rackspace
• The open source cloud computing platform
• Feature-rich and massively scalable
• Powers cloud storage, compute, and • Powers cloud storage, compute, and
networking
• A world-wide open source collaboration
10.04.2013 33/45Satish Srirama
OpenStack has 105+ participating
companies
…and over 1,350 individual contributors. [source: Piston]
10.04.2013 34/45Satish Srirama
3 Major OpenStack Components
• OpenStack Compute/Nova: provision and manage large networks of virtual machines
• OpenStack Object Store/Swift: Create petabytes of reliable storage using standard petabytes of reliable storage using standard servers
• OpenStack Image Service/Glance: Catalog and manage large libraries of server images
• Other components: Dashboard, Load Balancing, Authentication...
10.04.2013 35/45Satish Srirama
Working with OpenStack
• Last week keys are created for you all
• This week you work with the OpenStack
private cloud
• Just like Eucalyptus euca2ools are sufficient to • Just like Eucalyptus euca2ools are sufficient to
invoke the OpenStack services
10.04.2013 36/45Satish Srirama
SCICLOUD & UT’S RESEARCH ON
CLOUD
10.04.2013 37Satish Srirama
Scientific Computing Cloud (SciCloud) [Srirama et al, CCGrid 2010]
• Mobile Cloud Lab owned private cloud infrastructure
• Collection of smaller clouds built using Eucalyptus/OpenStack platformsEucalyptus/OpenStack platforms
• Goal of the project
– To efficiently use the already existing resources of universities
– To address computationally intensive scientific, mathematical, and academic problems
http://mc.cs.ut.ee/mcsite/projects/scicloud
10.04.2013 38/45Satish Srirama
Things that we do in cloud
• Research the utilization of cloud computing platforms for HPC [Srirama et al, SPJ 2011]
• Work with different Cloud computing frameworks– MapReduce (Lecture 3)
– Adapting scientific computing problems to cloud using – Adapting scientific computing problems to cloud using MapReduce [Srirama et al, FGCS 2012]
• Desktop to cloud migration (D2CM) tool [Srirama et al, CCSA 2012]
– Seamless migration of desktop virtual machine images to the Cloud
– Transform local VM images into cloud VM images
10.04.2013 39/45Satish Srirama
Things that we do in cloud - continued
• Mobile Cloud
– Delegate resource-intensive processing to the cloud [Flores et al, IJPCC 2012]
• To enrich the functionality of mobile applications
• Building social network and mashup applications on • Building social network and mashup applications on Android/iOS devices [Srirama et al, SOCA 2012]
– Mobile Cloud Middleware [Flores et al, MoMM 2011]
• Cloud interoperability
• Hiding the complexity of cloud from mobiles
– Code offloading
10.04.2013 40/45Satish Srirama
Things that we do in cloud - continued
• Distributed troubleshooting [Shor & Srirama, CLOSER 2011]
– Finding memory leaks in distributed applications
– Adapting the tools to cloud solutions
• Migrating Enterprise Applications to the Cloud• Migrating Enterprise Applications to the Cloud
– Refactoring enterprise applications for cloud
• ESB based applications
– Framework for monitoring performance of cloud
based web applications [Vasar et al, NORDICLOUD 2012]
• Data mining and Bioinformatics on the cloud
10.04.2013 41/45Satish Srirama
How can you contribute…
• We are always open to new research ideas
• Several theses are open…
– Check at project home page or
http://mc.cs.ut.ee/mcsite/theseshttp://mc.cs.ut.ee/mcsite/theses
– Interested students can approach me
• Join and participate in our seminars
– Mobile and Cloud Computing Seminar -
MTAT.03.280
10.04.2013 42/45Satish Srirama
This week in lab
• You work with SciCloud
– Creating instances
– Connecting to instances through ssh
– Installing software– Installing software
– Preparing images and snapshots
10.04.2013 43/45Satish Srirama
Next Lecture
• Introduction to MapReduce
10.04.2013 44/45Satish Srirama
References
• Amazon Web (Cloud) Services – documentation http://aws.amazon.com/documentation/
• SciCloud – homepage http://mc.cs.ut.ee/mcsite/projects/scicloud
• Eucalyptus User's Guide http://www.eucalyptus.com/docs/3.1/ug-3.1.0.pdf
• S. N. Srirama, O. Batrashev, E. Vainikko: SciCloud: Scientific Computing on the Cloud, 10th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2010), May 17-20, 2010, pp. 579. IEEE CS.
• S. N. Srirama, O. Batrashev, P. Jakovits, E. Vainikko: Scalability of Parallel Scientific Applications on the Cloud, Scientific Programming Journal, Special Issue on Science-driven Cloud Computing, 19(2-3):91-105, 2011. IOS Press.
• S. N. Srirama, P. Jakovits, E. Vainikko: Adapting Scientific Computing Problems to Clouds using MapReduce, Future Generation Computer Systems Journal, 28(1):184-192, 2012. Elsevier press.
• S. N. Srirama, C. Willmore, V. Ivanistsev, P. Jakovits: Desktop to Cloud Migration of Scientific Experiments, 2nd • S. N. Srirama, C. Willmore, V. Ivanistsev, P. Jakovits: Desktop to Cloud Migration of Scientific Experiments, 2nd International Workshop on Cloud Computing and Scientific Applications (CCSA) @ 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2012), May 13-16, 2012.
• H. Flores, S. N. Srirama, C. Paniagua: Towards Mobile Cloud Applications: Offloading Resource-Intensive Tasks To Hybrid Clouds, International Journal of Pervasive Computing and Communications, ISSN: 1742-7371, 8(4):344-367, 2012. Emerald Group Publishing Limited.
• S. N. Srirama, C. Paniagua, H. Flores: Social Group Formation with Mobile Cloud Services, Service Oriented Computing and Applications Journal, ISSN: 1863-2386, 6(4):351-362, 2012. Springer.
• H. Flores, S. N. Srirama, C. Paniagua: A Generic Middleware Framework for Handling Process Intensive Hybrid Cloud Services from Mobiles, The 9th International Conference on Advances in Mobile Computing & Multimedia (MoMM-2011), December 5-7, 2011, pp. 87-95. ACM.
• V. Shor, S. N. Srirama: A Statistical Approach For Identifying Memory Leaks In Cloud Applications, First International Conference on Cloud Computing and Services Science (CLOSER 2011), May 7-9, 2011, pp. 623-628. SciTePress.
• M. Vasar, S. N. Srirama, M. Dumas: Framework for Monitoring and Testing Web Application Scalability on the Cloud, Nordic Symposium on Cloud Computing & Internet Technologies (NORDICLOUD), 2012, pp. 53-60. ACM.
10.04.2013 45/45Satish Srirama