CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce:...
Transcript of CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce:...
![Page 1: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/1.jpg)
CS15-319 / 15-619 Cloud Computing
Recitation 4
September 17th & 20th, 2013
![Page 2: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/2.jpg)
Announcements
• Encounter a general bug:
– Post on Piazza
• Encounter a grading bug:
– Post Privately on Piazza
• Post feedback on OLI
![Page 3: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/3.jpg)
Unit 2: Data Centers • Data Centers Unit
– Module 3: Data Center Trends
– Module 4: Data Center Components
• Read and complete:
– Module 5: Design Considerations
– Unit 2: Checkpoint Quiz
• Timed Quiz (120 minutes), due 19th September, 2013
UNIT 2: Data Centers
Module 3: Data Center Trends
Module 4: Data Center Components
Module 5: Design Considerations
Quiz 2: Data Centers Checkpoint Available 9/16/13 12:01 AM Due 9/19/13 11:59 PM
![Page 4: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/4.jpg)
Project 1 Student Progress
• Introduction to Big Data:
– Sequential Analysis: Average is: 98%
– Elastic MapReduce: 96%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
10
1
10
6
11
1
11
6
12
1
12
6
13
1
13
6
14
1
14
6
15
1
15
6
![Page 5: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/5.jpg)
Piazza Questions
• Elastic MapReduce Billing Question
– Normalized Hours (Elastic MapReduce)
![Page 6: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/6.jpg)
Piazza Questions
• Elastic MapReduce Billing Question
– Elastic MapReduce Pricing
![Page 7: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/7.jpg)
Piazza Questions
• Elastic MapReduce Debugging
![Page 8: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/8.jpg)
Piazza Questions
• Elastic MapReduce Debugging
![Page 9: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/9.jpg)
Piazza Questions
• Elastic MapReduce Debugging
![Page 10: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/10.jpg)
Piazza Traffic
Posts Per Day Total Questions Asked
![Page 11: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/11.jpg)
OLI Feedback
• Students want to know more about MapReduce
![Page 12: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/12.jpg)
Scenario
• Online Photo Verification, PixVerify
Sales Reliability
![Page 13: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/13.jpg)
Current Infrastructure
Application Server
Database Server
User
![Page 14: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/14.jpg)
Cloud Infrastructure
Middleware (Virtualization)
Small Medium
Large Instance Types
Bare Metal Resources
![Page 15: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/15.jpg)
Maximize Requests per Dollar
Small Medium
Large Instance Types
X REQUESTS Y REQUESTS Z REQUESTS
COST A COST B COST C
GOAL: MAXIMIZE (REQUESTS/DOLLAR)
![Page 16: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/16.jpg)
Load Testing Request & Response Flow
Small
Medium
Large
Launchpad
Benchmark Script
![Page 17: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/17.jpg)
Provisioning Instances
• Manual Provisioning: Online Interface
![Page 18: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/18.jpg)
Amazon APIs
• Provisioning & Monitoring:
– EC2 API
– CloudWatch API
• Supported APIs
– Command Line Interface API Tools
– AWS SDK for Java
– AWS SDK for Python
![Page 19: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/19.jpg)
Provisioning and Monitoring
Small
Medium
Large
Launchpad (Simulate User Load)
Your Script/Program - Provision Test Instance
- Run Benchmark Script
- Store Benchmark Output into a file
- CloudWatch to retrieve CPUUtilization
- Terminate Test Instance
- Repeat for Other Test Instances
![Page 20: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/20.jpg)
Demo Outline
• 1. Launch an instance using command line
– Set up command line
– Launch an instance
– CloudWatch command line
• 2. Launch an instance using Java API
– Set up in eclipse
– Amazon EC2 API
– CloudWatch class
![Page 21: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/21.jpg)
Setup ec2 command line tool
• Set JAVA_HOME environment variable – Slightly different in Mac OS if you would like to use the
command line tool locally.
• Set EC2_HOME & system PATH variable • Tell system your access key and secret key • Set private key & certificate • Verify
– ec2-describe-regions
• http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SettingUp_CommandLine.html
![Page 22: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/22.jpg)
Launch an instance
• ec2-run-instances ami_id [-n instance count] [-k keypair] [--instance-type instance_type] [--availability-zone zone]
• http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html
![Page 23: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/23.jpg)
Cloud Watch Command Line
• Download cloud watch
– http://aws.amazon.com/developertools/2534
• Fill your access key and secret key in $AWS_CLOUDWATCH_HOME/credential-file-path.template
• You will have to find out how to use this command line
– Hint: mon-get-stats
![Page 24: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/24.jpg)
Set up AWS Toolkit for Eclipse
• Plug-in for the Eclipse Java IDE
• Help -> Install New Software
• Enter ‘http://aws.amazon.com/eclipse’ in ‘Work with’
• Select ‘AWS Toolkit for Eclipse’
• http://aws.amazon.com/eclipse/
![Page 25: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/25.jpg)
Amazon EC2 API
• Load AWS Credentials
– BasicAWSCredentials
• Create Amazon EC2 Client
– AmazonEC2Client
• Create and Configure Instance Request
– RunInstancesRequest
• Launch Instance
– RunInstanceResult
![Page 26: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/26.jpg)
Amazon CloudWatch API
• Load AWS Credentials – BasicAWSCredentials
• Create Amazon CloudWatch Client – AmazonCloudWatchClient
• Create and Configure Metric Request – GetMetricStatisticsRequest
• Get Metric Result – GetMetricStatisticsResult
• http://aws.amazon.com/sdkforjava/
![Page 27: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/27.jpg)
Discussion
• Your questions…
![Page 28: CS15-319 / 15-619 Cloud Computingmsakr/15319-f13/lectures/Recitation4.p… · –Elastic MapReduce: 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 6 11 16 21 26 31 36 41 46 51](https://reader033.fdocuments.in/reader033/viewer/2022060600/60541579610d747ac5786491/html5/thumbnails/28.jpg)
Upcoming Deadlines
• Unit 2: Timed Quiz (120 minutes)
• Project 2:
UNIT 2: Data Centers
Module 3: Data Center Trends
Module 4: Data Center Components
Module 5: Design Considerations
Quiz 2: Data Centers Checkpoint Available 9/16/13 12:01 AM Due 9/19/13 11:59 PM
Project 2
Introduction and APIs
Single Instance Benchmarks Checkpoint
Available Now Due 9/22/13 11:59 PM