CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming...
Transcript of CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming...
![Page 1: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/1.jpg)
CLOUD 39. Berner Architekten-Treffen
+41 79 422 11 87 | [email protected] | CTO
+41 76 498 59 07 | [email protected] | CEO
2017-03-23
![Page 2: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/2.jpg)
ENTERPRISE-WIDE DATA LOGISTICS, DONE RIGHT
WE DELIVER DATA SO YOU CAN FOCUS ON VALUE
+41 76 498 59 07 | [email protected] | CEO
![Page 3: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/3.jpg)
![Page 4: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/4.jpg)
1SITUATION
![Page 5: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/5.jpg)
The world’s most valuable resource is no longer oil, but data. - The Economist
![Page 6: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/6.jpg)
Data = Resource
![Page 7: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/7.jpg)
OR
Every IT Project
Creates more Data
Consumes existing Data
![Page 8: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/8.jpg)
Explosion of Sources and Consumers
Isolated
DataIsolated
Consumers
![Page 9: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/9.jpg)
Isolated
DataIsolated
Consumers
We observe a Gap in between
?
![Page 10: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/10.jpg)
You don’t know what data assets your company has
You already know the Gap, if…
You have to go through an IT project to connect to data
You have no idea how good or bad data quality is
You don’t know who has access to sensitive data
Sensitive data ends up in test environments
Or if you create DB dumps to allow access to your data
You have nightmares about GDPR
![Page 11: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/11.jpg)
We call the Gap…
DATA LOGISTICS
![Page 12: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/12.jpg)
We deliver data so you can create value.
![Page 13: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/13.jpg)
2SPOUD DATA MARKET
![Page 14: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/14.jpg)
Connector
System
Syst
em
System
Compliance and Governance
Collaborative, Streaming Data-Market for your Company.
SPOUD Data Market
![Page 15: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/15.jpg)
Your company data is easy to understand
and at your service with 1-click.
Imagine a world where…
![Page 16: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/16.jpg)
INSURANCERETAIL SPACE SPORTS
Customers
![Page 17: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/17.jpg)
CLOUD! WHAT’S THE QUESTION?
WE DELIVER DATA SO YOU CAN FOCUS ON VALUE
+41 79 422 11 87 | [email protected] | CTO
![Page 18: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/18.jpg)
From Zero to Cloud and Back
GET OUT! FAST!
#TIME-TO-MARKET
#MVP
#PYTHON
#AWS
#1-WEEK
#ETL
#SIMPLE-IOT-ANALYTICS
1 2 3 4
January 2016
65 7
Today
![Page 19: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/19.jpg)
Simple IoT Analytics - Need
- DB - Server for APIs and website
Webserver
![Page 20: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/20.jpg)
Simple IoT Analytics - What we got
WebserverWebserverWebserver
Load
balanc
er
- Auto Scaling - Health Checks - Monitor and Alerting - Managed Database (Backups, Updates, …) - Managed Server (Updates, Cleanup, …
![Page 21: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/21.jpg)
From Zero to Cloud and Back
REINVENTING US! AGILE!
#EVENT-DRIVEN
#MORE-DATA
#ELASTICSEARCH
#AWS-SERVICES
#1-MONTH
#KAFKA
#MORE-ANALYTICS
#REALTIME
#FLINK
1 2 3 4
April 2016
65 7
Today
![Page 22: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/22.jpg)
More Data, more Analytics
- Instead of vertical scaling, redesign architecture - Distributed Log, Message-Driven - Use of Realtime Data Processing with Flink - Use SaaS for everything with standard interfaces (e.g. MySQL, Redis, )
ETL
![Page 23: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/23.jpg)
~ 10 - 30 Instances
![Page 24: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/24.jpg)
From Zero to Cloud and Back
OVERLOAD! SCALE!
#HIGH-VVV
#GT-10-CUSTOMERS
#GT-2000-EVENTS-PER-SEC
#NOMAD
#ELASTIC-BEATS
#HA
#TERRAFORM
January 2017 Today
1 2 3 4 65 7
![Page 25: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/25.jpg)
Handling Load
- 10 customers, 500 events/s on average, 2000 events/s peak - High latency and backpressure! Cause? CPU or I/O? Which
application? - CPU on Flink Node because bad partitioning (and bug) - Vertical scaling as quick fix - Horizontal scaling needs an even load distribution - Partitioning is key - Later, saturated network I/O on message broker
![Page 26: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/26.jpg)
Management & Automation
- Using the existing AWS services for easy management no longer reasonable
- Welcome to the container orchestration level - Welcome to more complexity
~ 50 - 90 Instances
![Page 27: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/27.jpg)
~ 50 - 90 Instances
![Page 28: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/28.jpg)
From Zero to Cloud and Back
THE COSTS!
#PRICING-JUNGLE
#MONEY
#OPTIMIZE
#AUTOMATE
1
April 2017
65 7
Today
2 3 4
![Page 29: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/29.jpg)
2017
4
April 2017 2018
$
![Page 30: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/30.jpg)
2K 10K 12K 12K
0.5K 1K 1.5K 3K
43M 86M 129M 259M
2017
4
April 2017 2018
$
events/sec max
events/sec avg
events/day avg
~ 3000 bytes / event
![Page 31: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/31.jpg)
Cost Factors
- Guarantees are not free (e.g. Message delivery, availability, order) - It’s too easy to use Cloud products :) - No cleanup - Flexibility has its price - Most expensive technologies are Kafka, Flink, ElasticSearch - Idle time, because no reasonable automatic downscaling
![Page 32: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/32.jpg)
Cost Optimisation
- Clean Cloud - Everyone should be aware of the costs and budget - Know your cost indicator (e.g events/s and retention time) - Emerging patterns and products (e.g. serverless) - Automation - Auto Scaling
Cloud
![Page 33: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/33.jpg)
From Zero to Cloud and Back
RESIST COZY LOCK-INS!
#FOCUS
#LOCK-IN
#ENABLER
#STANDARDS
1
August 2017
65 7
Today
2 3 4
![Page 34: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/34.jpg)
Lock-Ins
- Do not depend on specific proprietary services for our base stack - CH and On-Premise is around the corner - Handle specific services as optional volatilities
![Page 35: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/35.jpg)
From Zero to Cloud and Back
ENTERPRISES THINK DIFFERENT!
#EVALUATION
#CH-CLOUD
#ON-PREMISE
#ORGANIZATION
1
September 2017
65 7
Today
2 3 4
![Page 36: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/36.jpg)
Evaluation Cloud Provider CH
- We are spoiled by GCP, AWS, AZ - Be aware of legacy Hosting Business Models with a Cloud paint - Focus on scalability, cost transparency and automation - Our customers are ready to pay more
![Page 37: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/37.jpg)
On-Prem?
- Container to the rescue (LCD) - On-Prem is always highly specific - Be aware of the organisational structures
![Page 38: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/38.jpg)
From Zero to Cloud and Back
EVERYTHING! CROSS-CLOUD!
#CROSS-SILO
#PRIVATE-CLOUD
#ON-PREMISE
#BRIDGE
#MULTI-CLOUD
#EDGE-COMPUTING
1 65 7
Today
2 3 4
![Page 39: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/39.jpg)
Private Cloud, Multi Cloud, Everything
- We have to use different services to survive. Think AI. - Focus on Data in-motion and Event-Driven Architectures - Data Logistics is bridging the silos
![Page 40: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/40.jpg)
Bridging through Data Logistics
Logistics Layer
Infrastructure 1 Infrastructure 2
Logistics Layer
Service AService B
![Page 41: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/41.jpg)
CLOUD! What’s the question?
+41 79 422 11 87 | [email protected] | CTO
+41 76 498 59 07 | [email protected] | CEO
![Page 42: CLOUD - berner-architekten-treffen.chberner-architekten-treffen.ch/archiv/39/BAT39... · Streaming Data-Market for your Company. SPOUD Data Market. ... -It’s too easy to use Cloud](https://reader033.fdocuments.in/reader033/viewer/2022042219/5ec58779c758b01deb0848b5/html5/thumbnails/42.jpg)
ENTERPRISE-WIDE DATA LOGISTICS, DONE RIGHT
+41 79 422 11 87 | [email protected] | CTO
+41 76 498 59 07 | [email protected] | CEO