Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013
-
Upload
amazon-web-services -
Category
Technology
-
view
409 -
download
1
description
Transcript of Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013
![Page 1: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/1.jpg)
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Joey Parsons @joeyparsons
November 14th, 2013
Scaling a Mobile Web App to 100 Million Clients and Beyond
Friday, November 15, 13
![Page 2: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/2.jpg)
Friday, November 15, 13
![Page 3: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/3.jpg)
YOUR PERSONAL MAGAZINE
Friday, November 15, 13
![Page 4: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/4.jpg)
The ultimate way to discover, consume & share content on the mobile, social web
Wednesday, June 5, 13Friday, November 15, 13
![Page 5: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/5.jpg)
Friday, November 15, 13
![Page 6: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/6.jpg)
Friday, November 15, 13
![Page 7: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/7.jpg)
Friday, November 15, 13
![Page 8: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/8.jpg)
How are mobile apps different?• WiFi vs Slow connectivity
• Variances in bandwidth and global carriers
• Taking advantage of the local cache • Control your behavior during latency
• Fast devices — significant opportunity for client computation
Friday, November 15, 13
![Page 9: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/9.jpg)
Prototype Phase: From 0 to 1M users
Friday, November 15, 13
![Page 10: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/10.jpg)
- Amazon EC2- Amazon S3- Amazon RDS
Friday, November 15, 13
![Page 11: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/11.jpg)
The Initial Launch Night
Friday, November 15, 13
![Page 12: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/12.jpg)
Things we should have done…• Make sure to prepare for Amazon limits if you need to
scale quickly
• Make sure your external partners understand the volumes you’ll be accessing them
Friday, November 15, 13
![Page 13: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/13.jpg)
Challenges• Understanding the scale of our services• Little to no insight into performance• Beginning to build out tooling for Amazon EC2 but still
in its infancy• No centralized logging or way of detecting errors
Friday, November 15, 13
![Page 14: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/14.jpg)
Getting Started:From 1M to 10M Users
Friday, November 15, 13
![Page 15: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/15.jpg)
- Amazon EC2- Amazon RDS- Amazon S3 - Amazon CloudFront
Friday, November 15, 13
![Page 16: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/16.jpg)
Architecture Changes• Different services have different scale profiles — began
the shift towards microservices• Image content moved to CloudFront• Moved primary data store to MySQL via Amazon RDS• Home grown bash scripts for deploys• Focus on instrumentation
• Logging, Metrics, Monitoring followed suit
Friday, November 15, 13
![Page 17: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/17.jpg)
Host[i-76e33611] - Amazon Instance ID name[tsd01] - Name of the instance owner[ops] - Who owns the instance? service[OOS] - IS for in service, OOS for out of service ami[ami-3f622f56] - What AMI was used type[m1.xlarge] - Type of EC2 instance loc[us-east-1a] - Region and Availability Zone role[flops] - Type of role subclass[opentsdb] - Subclass of role group[0] - Group of node pool[production] - Production, staging, dev public[50.16.58.220] - Public IP address private[10.60.43.18] - Private IP Address
SimpleDB for CMDB
Friday, November 15, 13
![Page 18: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/18.jpg)
# fl-inst-describe -r flip -p production -g 0 -s IS -o ops
Domain[flipboard.prod.instances] has count[1] hosts meeting criteria=======================================Servers of role flip=======================================
Host[i-5b8ae323]: name[flip05] owner[ops] service[IS] public[54.226.44.212] private[10.78.167.211] role[flip] group[0] pool[production] subclass[standard] type[c1.xlarge]
Querying our CMDB
Friday, November 15, 13
![Page 19: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/19.jpg)
The iPhone Launch Night
Friday, November 15, 13
![Page 20: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/20.jpg)
Scaling Fast:10M to 100M Users
Friday, November 15, 13
![Page 21: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/21.jpg)
Storm Kafka GraphiteKibana
Friday, November 15, 13
![Page 22: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/22.jpg)
Architecture Changes• Heavy focus on instrumentation of all services• Pipeline of batch processing using Hadoop• Pipeline of real-time processing using Storm + Kafka• Keen focus on using appropriately sized EC2
instances• Moving off of bash scripts, moving to puppet
Friday, November 15, 13
![Page 23: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/23.jpg)
Mobile application instrumentation
Friday, November 15, 13
![Page 24: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/24.jpg)
All at once?
fl-inst-upgrade -r flip -p production-q
… or …
By group?
fl-inst-upgrade -r flip -p production -g 0 -q
Deploy by groups
Friday, November 15, 13
![Page 25: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/25.jpg)
Using CloudWatch metrics for errors
Friday, November 15, 13
![Page 26: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/26.jpg)
fl-inst-upgrade -r flip -p production -g 1 -q
Continued your deploy
Friday, November 15, 13
![Page 27: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/27.jpg)
Graphite for all metrics
Friday, November 15, 13
![Page 28: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/28.jpg)
Millions of metrics with Graphite
Friday, November 15, 13
![Page 29: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/29.jpg)
d3.js + cubism.js
Friday, November 15, 13
![Page 30: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/30.jpg)
Monitoring via CloudWatchAlarm in PagerDuty
Details available in PagerDuty
Friday, November 15, 13
![Page 31: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/31.jpg)
Lessons Learned• Use Amazon services when possible (Amazon RDS,
Amazon Redshift, Amazon Route 53)• Use SSDs where applicable• Understand your scale and your needs going forward
and invest in Reserved Instances (3 years!)• But, allow flexibility for changing needs and instance
types
Friday, November 15, 13
![Page 32: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/32.jpg)
Amazon Technologies Used• Amazon CloudFront• Amazon Route 53• Amazon EC2• Amazon S3• Amazon Redshift
• Amazon RDS• Amazon SimpleDB• Amazon SQS• ElastiCache• Amazon CloudWatch
Friday, November 15, 13
![Page 33: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/33.jpg)
Beyond:From 100M Users to 1B!
Friday, November 15, 13
![Page 34: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/34.jpg)
What’s next?• Better use of Auto Scaling groups• Predictive analytics — lots of signals• Automated remediation• Heavy focus on using the right instance types for each
service• Take advantage of new AWS products
Friday, November 15, 13
![Page 35: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/35.jpg)
The unknown is exciting …
Friday, November 15, 13
![Page 36: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/36.jpg)
Questions?
Friday, November 15, 13
![Page 37: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/37.jpg)
AWS re:Invent 2013Magazine
http://flip.it/NSNEi
Friday, November 15, 13
![Page 38: Scaling a Mobile Web App to 100 Million Clients and Beyond (MBL302) | AWS re:Invent 2013](https://reader033.fdocuments.in/reader033/viewer/2022051818/54b3a7fb4a79594d0a8b461f/html5/thumbnails/38.jpg)
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
MBL302 Thank You
Friday, November 15, 13