Stairway to the cloud or can we take the highway? Taivo Liik.
-
Upload
ethelbert-shields -
Category
Documents
-
view
212 -
download
0
Transcript of Stairway to the cloud or can we take the highway? Taivo Liik.
![Page 1: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/1.jpg)
Stairway to the cloud or can we take the highway?
Taivo Liik
![Page 3: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/3.jpg)
Current Situation
• 3000 physical servers• 25 000 Virtual Machines• Over 100 deployable application
servers
![Page 4: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/4.jpg)
Current Situation (2)
• Developments happen across the Globe• New versions and fresh features every
week• How to scale them easily?• How to cut down the cost?
![Page 5: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/5.jpg)
Current Approach
• Moving more and more to micro services approach• New micro services are developed constantly• Needs to:
– Be scalable– Be discoverable– Have connectivity to different locations– Be highly available– Be configured– Have ability to store the logs for troubleshooting– Have up to date middleware
![Page 6: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/6.jpg)
Drawing a parallel
![Page 7: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/7.jpg)
Platform as a Service
• Requirements– Easy deployment– Run and scale applications– Possibility to make fast application upgrades
• There are several cloud products to accomplice it.• Do we need:– Open sourced?– Commercial?– Public?– Private?
![Page 8: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/8.jpg)
Open sourced or
commercial?
![Page 9: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/9.jpg)
Public or
private?
![Page 10: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/10.jpg)
What did we choose?
• Open sourced
• Private
![Page 11: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/11.jpg)
Cloud product
• We chose Cloud Foundry as our cloud platform
![Page 12: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/12.jpg)
Drawing a parallel (2)
![Page 13: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/13.jpg)
From VM to cloud
• No permanent storage in Cloud Foundry for applications
• One port opened for application to be accessed outside the cloud
• Flexible scaling – application lifecycle can be very small
• Application can connect to outside world over as many ports as needed
![Page 14: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/14.jpg)
From VM to cloud (2)• Where to store application start-up
configuration?• Where logs can be written?• How application monitoring will work?• How middleware is deployed?• How application deployment changes?• How to know how many application nodes are
running? Do we need to know?
![Page 15: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/15.jpg)
Application logs
• No permanent storage like in VM• Solution needed due–We have more than 100 deployable
application servers– It quite an hassle to read logs from different
data sources
![Page 16: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/16.jpg)
• Push logs to Apache Kafka–Clustered for high availability
Log solution
![Page 17: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/17.jpg)
• Camus jobs are pulling the logs from Kafka to Hadoop Distributed File System (HDFS)
• Reading the logs can be done in single place• On top of Hadoop Hue with Hive and Oozie
Log solution (2)
![Page 18: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/18.jpg)
Keystores & Configuration
• Needs to be stored somewhere • Can be packaged inside the
application package• Can be stored in some site central
location
![Page 19: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/19.jpg)
Keystores & Configuration (2)
• We are using Git
![Page 20: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/20.jpg)
Keystores & Configuration (3)
• Keystores needs to be in a secure location• They are different per site• Needed for making trusted
connection• We are using Vault
![Page 21: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/21.jpg)
Monitoring
• Status was pulled form apps• One port – cannot be wasted• Also having metrics running to the
Graphite on top of Whisper database
![Page 22: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/22.jpg)
![Page 23: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/23.jpg)
Monitoring (2)
• Needs to be highly available• Use Graphite with Influx database• The switch gave us additional tools
like Grafana• Monitoring pulling service can pull
status from Influx DB
![Page 24: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/24.jpg)
The road so far
![Page 25: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/25.jpg)
The road so far (2)
![Page 26: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/26.jpg)
The road so far (3)
![Page 27: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/27.jpg)
The road so far (4)
![Page 28: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/28.jpg)
Service Discovery
• Information about nodes was in the database
• How it works with dynamic scaling?
![Page 29: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/29.jpg)
Middleware
• In virtual machine we had middleware preinstalled• Upgrades took time
![Page 30: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/30.jpg)
Middleware (2)
• manifest.yml• Cloud Foundry middleware in its
blobstore• Custom build packs are supported
![Page 31: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/31.jpg)
Load balancer
• Past–Each VM needed to be in the LB pool–Each VM needed to have firewall hole
opened–DNS records needed to be created
• Now–Define that the application is in the cloud–Cloud infrastructure makes the rest
![Page 32: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/32.jpg)
Security
• Each application is
–In specific org•Inside specific space–Inside application container
![Page 33: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/33.jpg)
Orgs• Orgs can be used for different environments in
single cloud:–Production– Staging– Integration–Testing–Development–Performance– Some other purpose
![Page 34: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/34.jpg)
Did we take the stairway or highway?• We needed to change:–VM based procedures–Middleware and application deployments–Modify existing automation–Keystore storing policy– Error troubleshootings–Network layout–Capacity calculations– Service discovery
![Page 35: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/35.jpg)
Did we take the stairway or highway? (2)
• For single application it isHIGHWAY
• Cutting down the cost infrastructure dependencies• Taking private cloud approach it is
STAIRWAY
![Page 36: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/36.jpg)
Sometimes the stairs have to be taken to get to the highway for faster developments and deployments.
Questions?
![Page 37: Stairway to the cloud or can we take the highway? Taivo Liik.](https://reader036.fdocuments.in/reader036/viewer/2022070415/5697bfec1a28abf838cb8734/html5/thumbnails/37.jpg)
Thank You