Big Data and OpenStack, a Love Story: Michael Still, Rackspace
-
Upload
openstack -
Category
Technology
-
view
219 -
download
0
Transcript of Big Data and OpenStack, a Love Story: Michael Still, Rackspace
![Page 1: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/1.jpg)
OPENSTACK AND BIG DATA, A LOVE STORY
Michael Still Senior Software Development Manager [email protected] or @mikal on twitter
![Page 2: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/2.jpg)
2
WHO IS THIS GUY? • A Canberran born and bred • An OpenStack developer since 2011, first commit
merged January 2012 - https://review.openstack.org/#/c/2899/
• A Compute Core Reviewer, former Compute PTL, and have served on the OpenStack Technical Committee
• Manager for a team of OpenStack developers spread across Australia
![Page 3: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/3.jpg)
3
CAUTION, THIS BIT IS A TEST
![Page 4: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/4.jpg)
4
![Page 5: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/5.jpg)
5
![Page 6: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/6.jpg)
6
WHO IS RACKSPACE? • Do any of you guys know who Rackspace is and how they fit into the
OpenStack story?
![Page 7: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/7.jpg)
7
BIG DATA • Hopefully we’re all familiar with the term
• That said, the basic idea is to store and process large amounts of data on commodity equipment
• Pioneered by Internet companies • But now used by many ”more traditional” organizations
![Page 8: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/8.jpg)
IMAGE PLACEHOLDER 1280X1080
8
THE OLD WAY
![Page 9: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/9.jpg)
IMAGE PLACEHOLDER 1280X1080
9
THE NEW WAY
![Page 10: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/10.jpg)
10
BIG DATA • The most obvious thing here is that machine counts are increasing… • We’re talking about hundreds or thousands of machines instead of the one
big machine
![Page 11: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/11.jpg)
11
BIG DATA • The most obvious thing here is that machine counts are increasing… • We’re talking about hundreds or thousands of machines instead of the one
big machine
• And our operational budgets are not increasing with machine count (of course)
![Page 12: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/12.jpg)
12
BIG DATA • The most obvious thing here is that machine counts are increasing… • We’re talking about hundreds or thousands of machines instead of the one
big machine
• And our operational budgets are not increasing with machine count (of course)
• So we need to automate more
![Page 13: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/13.jpg)
13
OPENSTACK COMPUTE • From day zero OpenStack supported running virtual machines • We call them instances
![Page 14: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/14.jpg)
14
OPENSTACK COMPUTE • From day zero OpenStack supported running virtual machines • We call them instances
• Virtual machines aren’t a great choice for most big data applications though - For example, its nice if you replicate your data - But what if all the VMs containing replicas are on the same hypervisor? - There are performance costs as well
![Page 15: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/15.jpg)
15
OPENSTACK COMPUTE • From day zero OpenStack supported running virtual machines • We call them instances
• Virtual machines aren’t a great choice for most big data applications though - For example, its nice if you replicate your data - But what if all the VMs containing replicas are on the same hypervisor? - There are performance costs as well
• Big data is about bulk, not artisanal machine orchestration
![Page 16: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/16.jpg)
16
OPENSTACK BAREMETAL • A research project started in 2012
![Page 17: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/17.jpg)
17
OPENSTACK BAREMETAL • A research project started in 2012 • It was… horrible • But has been deployed. Yahoo has tens of thousands of machines running
this code.
![Page 18: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/18.jpg)
18
OPENSTACK BAREMETAL • A research project started in 2012 • It was… horrible • But has been deployed. Yahoo has tens of thousands of machines running
this code.
• Luckily some adults came along and turned that research project into a productionized thing in 2013
![Page 19: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/19.jpg)
19
OPENSTACK BAREMETAL • The new implementation is a separate OpenStack project • Manages machines by talking IPMI / DRAC / iLO / other things • Integrates with OpenStack Compute so that the same APIs work
everywhere
![Page 20: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/20.jpg)
20
WHICH MEANS…
![Page 21: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/21.jpg)
21
API CONTROL OF BULK INFRASTRUCTURE • We can now build images for all our various big data machine types
- Management nodes - Zookeeper nodes - Data storage / worker nodes
• And then manage them with simple command line tools
![Page 22: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/22.jpg)
22
API CONTROL OF BULK INFRASTRUCTURE • I’ve spent the last year helping a customer of ours do something like this
![Page 23: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/23.jpg)
23
API CONTROL OF BULK INFRASTRUCTURE • I’ve spent the last year helping a customer of ours do something like this
• Why a year? • Well, they wanted some other stuff like continuous deployment of
OpenStack as well, and that was a lot harder than the Hadoop bits
![Page 24: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/24.jpg)
24
API CONTROL OF BULK INFRASTRUCTURE • That said, based on a simpler version of their deployment, I think I have
some recommendations now for how to approach a project like this…
![Page 25: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/25.jpg)
25
API CONTROL OF BULK INFRASTRUCTURE • That said, based on a simpler version of their deployment, I think I have
some recommendations now for how to approach a project like this…
• Zookeeper nodes are harder than I thought • Management nodes are even harder • But data and processing nodes are easy
![Page 26: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/26.jpg)
26
API CONTROL OF BULK INFRASTRUCTURE • That said, based on a simpler version of their deployment, I think I have
some recommendations now for how to approach a project like this…
• Zookeeper nodes are harder than I thought • Management nodes are even harder • But data and processing nodes are easy
Luckily, this is the vast majority of your machines
![Page 27: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/27.jpg)
27
API CONTROL OF BULK INFRASTRUCTURE • That said, based on a simpler version of their deployment, I think I have
some recommendations now for how to approach a project like this…
• Zookeeper nodes are harder than I thought • Management nodes are even harder • But data and processing nodes are easy
Luckily, this is the vast majority of your machines
And this is possible, just harder
![Page 28: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/28.jpg)
28
API CONTROL OF BULK INFRASTRUCTURE • Data and processing nodes
- Golden image deployments are the way to go - Keep your data on non-boot disks - To update the OS / image, just rebuild the image and the use nova rebuild - Use keep-ephemeral to avoid re-syncing data during a rollout
![Page 29: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/29.jpg)
29
API CONTROL OF BULK INFRASTRUCTURE • Zookeeper nodes
- This is harder because all the machines in the zookeeper cluster need a shared config listing all their peers
- We solved this by using an overlay network - But floating IPs would probably work in a simpler environment
![Page 30: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/30.jpg)
30
CANBERRA OPENSTACK MEETUP
Tuesday 29 November 7pm to 9pm
https://goo.gl/nxW62K
![Page 31: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/31.jpg)
31
![Page 32: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/32.jpg)
32
Copyright © 2016 Rackspace | Rackspace® Fanatical Support® and other Rackspace marks are either registered service marks or service marks of Rackspce US, Inc. in the United States and other countries. Features, benefits and pricing presented depend on system configuration and are subject to change without notice. Rackspace disclaims any representation, warranty or other legal commitment regarding its services except for those expressly
stated in a Rackspace services agreement. All other trademarks, service marks, images, products and brands remain the sole property of their respective holders and do not imply endorsement or sponsorship.
ONE FANATICAL PLACE | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
US
![Page 33: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/33.jpg)
33
Copyright © 2016 Rackspace | Rackspace® Fanatical Support® and other Rackspace marks are either registered service marks or service marks of Rackspce US, Inc. in the United States and other countries. Features, benefits and pricing presented depend on system configuration and are subject to change without notice. Rackspace disclaims any representation, warranty or other legal commitment regarding its services except for those expressly
stated in a Rackspace services agreement. All other trademarks, service marks, images, products and brands remain the sole property of their respective holders and do not imply endorsement or sponsorship.
ONE FANATICAL PLACE | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
US
Feel free to contact me at: [email protected] or @mikal on twitter
![Page 34: Big Data and OpenStack, a Love Story: Michael Still, Rackspace](https://reader031.fdocuments.in/reader031/viewer/2022030316/587587661a28ab901c8b5043/html5/thumbnails/34.jpg)
34
DATA CENTERS
10 Worldwide
GLOBAL FOOTPRINT
Customers in 150 Countries
PORTFOLIO
Dedicated • Hybrid • Cloud
EXPERTS
6,200 Rackers
REVENUE
Over $2B in Annualized Revenue
FORTUNE 100
We serve the majority of the Fortune 100
WHO WE ARE
3,000+ Cloud Experts