MANTL Data Platform, Microservices and BigData Services
-
Upload
cisco-devnet -
Category
Technology
-
view
942 -
download
0
Transcript of MANTL Data Platform, Microservices and BigData Services
![Page 1: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/1.jpg)
Dmitri Chtchourov,
MANTL Data Platform, Microservices and BigData Services
Innovation Architect, CIS CTO Group
![Page 2: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/2.jpg)
Agenda
Problem & Opportunity
What do we want to do?
What is in it for us?
How does it work?
What have we done so far?
Anatomy of a Service
Reference Architectures and real use cases
Putting it all together
![Page 3: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/3.jpg)
Problem & Opportunity
Rapid innovation in computing and application development services
No single service is optimal for all solutions
Customers want to run multiple services in a single cluster and run multiple clusters in Intercloud environment
...to maximize utilization ...to share data between services
…Complex/BigData and Microservices together
![Page 4: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/4.jpg)
Technologies matrix* Service Product Cloud/Virtualization CIS/AWS/Metacloud/UCS… Provisioning Open Stack/Terraform Automation Ansible
Clustering & Resource Management Mesos, Marathon, Docker Load Balancing Avi Networks ETL & Data Shaping StreamSets Log Data Gathering Logstash Metrics Gathering CollectD, Avi Networks Messaging Kafka, Solace Data Storing (Batch) HDFS Data Storing (OLTP/Real-time) Cassandra Data Storing (Indexing) Elastic search Data Processing Apache Spark Visualization Zoomdata*Subset example
![Page 5: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/5.jpg)
Cloud Management
Data Collect
Data Storage
Data Processing
Visualisation
Technologies stack
![Page 6: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/6.jpg)
Datacenter and solution today
VM7or
BM7
VM8or
BM8
VM4 or
BM4
VM5or
BM5
VM6or
BM6
VM1 or
BM1
Visualization Service
Data Ingestion Service
Analytics Service
• Configuration and management of 3 separate clusters
• Resources stay idle if service is not active
• Need to move data between clusters for each service
VM2 or
BM2
VM3 or
BM3
VM1 or
BM1
VM2 or
BM2
VM3 or
BM3
![Page 7: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/7.jpg)
What do we want to do?
Data Ingestion Service
Analytics Service
Visualization Service
….to maximize utilization...to share data between services
Shared clusterMultiple clusters
![Page 8: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/8.jpg)
Shared Cluster
CIS/AWS/Metapod/UCS…
VM1 or
BM1
VM2 or
BM2
VM3 or
BM3
VM4 or
BM4
VM5 or
BM5
![Page 9: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/9.jpg)
What is in it for us?
Maximize utilizationDeliver more services with smaller footprint
Shared clusters for all services Easier deployment and management with unified service platform
Shared data between services Faster and more competitive services and solutions
Combine paradigms for flexibility and functionality Run complex services and microservices in the single environment
![Page 10: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/10.jpg)
How does this work?
Mesos Slave
Spark Task Executor Mesos Executor
Mesos Slave
Docker Executor Docker Executor
Mesos Master
Task #1 Task #2 ./python XYZ java -jar XYZ.jar ./xyz
Mesos Master Mesos Master
Spark Service Scheduler Marathon Service SchedulerZookeeper quorum
![Page 11: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/11.jpg)
How does this work?
Mesos provides fine grained resource isolation
Mesos Slave Process
Spark Task Executor Mesos Executor
Task #1 Task #2 ./python XYZ
Compute Node
ExecutorContainer(cgroups)
![Page 12: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/12.jpg)
How does this work?
Mesos provides scalability
Mesos Slave Process
Spark Task Executor
Task #1 Task #2 ./ruby XYZ
Compute Node
Python executor finished,more available resourcesmore Spark
Container(cgroups)
Task #3 Task #4
![Page 13: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/13.jpg)
How does this work?
Mesos has no single point of failure
Mesos MasterMesos MasterMesos Master
VM1or
BM1
VM2 or
BM2
VM3or
BM3
VM4or
BM4
VM5or
BM5
Services keep running if VM fails!
![Page 14: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/14.jpg)
How does this work?
Master node can failover
Mesos MasterMesos MasterMesos Master
VM1or
BM1
VM2or
BM2
VM3or
BM3
VM4or
BM4
VM5or
BM5
Services keep running if Mesos Master fails!
![Page 15: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/15.jpg)
How does this work?
Slave process can failover Tasks keep running if Mesos Slave Process fails!
Mesos Slave Process
Spark Task Executor
Task #1 Task #2 ./ruby XYZ
Compute Node
Task #3 Task #4
![Page 16: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/16.jpg)
How does this work?
Can deploy in many environments Get orchestrated by Openstack, Ansible (scripts), Cloudbreak
True Hybrid Cloud deployment: CIS, AWS, UCS, vSphere, other
CIS/
CIS/AWS/Metpod/vSphere/UCS…
Terraform
REST API REST API
Scripted provisioning
Direct provisioning
Policy, Auto-scaling
VM1or
BM1
VM2or
BM2
VM3or
BM3
VM4or
BM4
VM5or
BM5
![Page 17: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/17.jpg)
How does this work?
Microservices managed and scaled separately Microservices managed by Mesos in a single platform
Microservices architecture for Mesos frameworks and other components
CIS/AWS/Metacloud/vSphere/UCS…Terraform
Spark Executor
N
Spark Executor 1
Spark Scheduler
Kafka Broker N
Kafka Broker 1
Kafka Scheduler
Docker Docker TraefikMicroservices …
REST API REST API
Scripted provisioning
Direct provisioning
Policy, Auto-scaling
VM1or
BM1
VM2or
BM2
VM3or
BM3
VM4or
BM4
VM5or
BM5
![Page 18: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/18.jpg)
What have we done so far?Working with partners on adopting and co-developing services
Partners Co-development Partners
![Page 19: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/19.jpg)
Anatomy of the service/framework
Riak is Basho Technologies distributed highly available database
Optimized Multi-Datacenter operation
We are working together with Basho Labs on developing and testing their Mesos Service version of the product
![Page 20: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/20.jpg)
Riak Use Cases
Online / Commerce● Session Control● Shopping Cart● Product Ratings and Reviews
Internet of Things● Connected Device Data● Sensor Data● Log Data
Content Management● Storing Unstructured Data● Content Personalization● Advertising Data
Gaming● Store Leaderboard Info● Store Bet Transactions● Online Chat
Digital Communications● Online Community Chat● Notification and Alerting● Mobile Messaging
![Page 21: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/21.jpg)
Development phases
Phase 0: Package application in Docker container to deploy on Mesos
Phase 1: Convert application to Microservices Architecture to deploy as Mesos application with multiple components
Phase 2: Create an intelligent scalable Mesos service based on the application
![Page 22: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/22.jpg)
Riak Service: Components
![Page 23: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/23.jpg)
Riak Service: Architecture
![Page 24: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/24.jpg)
Riak Service: Persistence
![Page 25: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/25.jpg)
Riak Service: Operational Simplicity
![Page 26: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/26.jpg)
Riak Service: Highly ScalableE-commerce Application with Varying Traffic
![Page 27: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/27.jpg)
Anatomy of the service/framework
Zoomdata is distributed highly available large scale visualization platform
Optimized very big data set micro-query analytics
We are working together with Zoomdata on developing and testing their Mesos Service version of the product
![Page 28: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/28.jpg)
Zoomdata Service: Components
![Page 29: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/29.jpg)
Zoomdata Service: Mesos + Kubernetes
Mesos Slave
Mesos Master
Mesos SlaveMesos Slave
Zoomdata web app
MongodbSpark Worker
Spark Executor
Spark Executor
Proxy(haproxy, nginx)
KubernetesMongoService/RC
KubernetesSpark-ProxyService/RC
Spark-Proxy
Zoomdata web app
Zoomdata web app
Kubernetes Framework
KubernetesZoomdataSchedulerService/RC
Zoomdata Scheduler
ProxyGen Script
User
● Every component (Zoomdata App, MongoDB, Spark-Proxy, Scheduler) must be started in independent K8s Pod and there must be exactly one MongoDB, Spark-Proxy and Scheduler Pods meanwhile Zoomdata App can be scaled with help Kubernetes Replication Controller.
● There must be defined Kubernetes Service for MongoDB, Spark-Proxy, Scheduler as they will be used in Zoomdata’s App Pod. Every docker container will have env variables for every present Service injected automatically.
![Page 30: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/30.jpg)
Anatomy of the service/framework
StreamSets is an open source continuous big data ingest infrastructure
Accelerates time to analysis with unprecedented transparency and processing to data in motion.
Cluster deployments
JVM, Docker, Spark Streaming on Mesos
Continuous Operations to Minimize downtime
![Page 31: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/31.jpg)
Advantages of Streamsets
Adaptable Data flow - Design and execute intent-driven data flows in a graphical IDE
Instream Sanitization - transform and process the data on the fly
Intelligent Monitoring - Get early warnings, detect anomalies and take action
Link origins to destinations with in-stream data preparation
![Page 32: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/32.jpg)
Streamsets Data Pipeline
MESOS
Streamsets Data Collector (SDC) ArchitectureCluster Streaming modeData Collector runs as an application within Spark Streaming, Spark Streaming runs on Mesos cluster manager to process data from a Kafka cluster.
The Data Collector uses a cluster manager and a cluster application to spawn workers as needed.
Cluster Batch Mode :Data Collector processes all available data from HDFS and then stops the pipeline.
MapReduce generate additional worker nodes as needed.
Standalone modeSingle Data Collector process runs the pipeline. A pipeline runs in standalone mode by default.
![Page 33: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/33.jpg)
MANTL Data Platform OverviewA modern, batteries included platform for rapidly deploying globally distributed services.Mantl’s goal is to provide a fully functional, instrumented, and portable container based PaaS for your business at the push of a button
1) Easy deployment and configuration on different
platforms
2) High availability and self-healing
3) Multi-datacenter support
4) Linear scalability
5) Smart resource management
6) Wide range of supported frameworks
![Page 34: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/34.jpg)
MANTL nodes
Consul for service discoveryMesos cluster managerMarathon for cluster management Docker container runtimeZookeeper for configuration management
Docker containersAny Mesos-based workloads
Traefik for proxying external traffic into services running in the cluster
![Page 35: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/35.jpg)
Security & Operation
Frameworks
Platform Support
Mantl Components
CoreComponents
➢ Data Storage - Riak, Cassandra, HDFS➢ Data processing - Spark➢ Security - Vault
➢ Data ingestion – Kafka➢ Metrics collection - Collectd➢ Logs forwarding - Logstash
➢ Provisioning - Terraform, Ansible➢ Cluster management – Mesos, Marathon➢ Service discovery and configuration management - Consul, Zookeeper,
Traefik➢ Container runtime - Docker
➢ Cisco Cloud Services, Cisco MetaCloud➢ Amazon Web Services➢ Google Compute Engine
➢ Openstack➢ DigitalOcean➢ Bare Metal
➢ Autoscaling and high availability➢ Application load balancer➢ Application dynamic firewall
➢ Manage Linux user accounts➢ Authentication and authorization
for Consul, Mesos, Marathon
![Page 36: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/36.jpg)
Long Running Services
Big Data Processing
Batch Scheduling
Supported Mesos Frameworks
Data Storage
Mesos makes it easy to develop distributed systems by providing high-level building blocks.
![Page 37: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/37.jpg)
ANALYTICS PLATFORM MANAGEMENT
Data Ingestion•Kafka, Streamsets configurators
Data StorageRiak, Cassandra, HDFS
Model DevOps Machine learningMLLib, Spark
Model Deployment•Model loading, versioning
Cluster Management & Scheduling
Cluster managerMesos
Cluster Management long running service
Marathon
Service DiscoveryConsul
Distributed Virtual networkCalico ETCD
ADVANCED ANALYTICS APPS
Analytics Accelerators as Apps•Forecasting, NLP, optimization, enrichment etc.
SPECIALIZED ADVANCED ANALYTICS MODELS
Consulting Services Design, Build, Deploy Maintain, Manage Performance
DASHBOARDS
ZoomData Tableau, Qlik, Spotfire, Excel/BI Cubes …
BUSINESS APPS
Custom ZoomData Visualizations (D3) Custom Applications Customer System Integration
CUST
OM
IZAT
ION
&M
ANAG
ED S
ERVI
CES
CISCO INTERCLOUD
Customization MANTL Data Platform
![Page 38: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/38.jpg)
Sample Architecture for Batch Data Processing
Cassandra
Elastic Search
Spark
Spark Mllib Riak
Kibana Dashboard
VisualisationStorage
Stream Sets
I/P in multiple formats Text, logs and json from various storage source.
Spark application process data and store to elastic search or Cassandra or Riak storage for visualization else it stores in HDFS
Machine learning algorithm for data science application
ZoomdataData
Discovery
D3Web
Application
HDFS
StreamSets Data Collector runs as an application in Spark Streaming to pull data from origin to spark
CSV, Tab delimited etc.
LOG file
JSON
TEXT
![Page 39: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/39.jpg)
Sample Architecture for Data Streaming
Kafka
Cassandra
Elastic Search
Spark
Spark Mllib
Riak
Kibana Dashboard
VisualisationStorage
Stream Sets
Streaming network from different sources
Kafka is used for collecting streaming data and data is consumed through consumer API by Streamset for further processing.
Spark application process data and store to elastic search or Cassandra or Riak storage.
Machine learning algorithm for data science application
ZoomdataData
Discovery
D3Web
Application
![Page 40: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/40.jpg)
Use Case 1 - Shipped Analytics
Collect log metric from cluster to analyze and drive Alert/Recommend engine
• Alert Engine - produces alert messages on a basis of some conditions.
• Trend Engine - produces trend messages related to data aggregation. • Policy Engine – derives from Alert and Trend Engines produces policy
messages which contain recommendations.
![Page 41: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/41.jpg)
Use Case 1 - Shipped Analytics Architecture
Central Cluster
Probe
Probe
Probe
![Page 42: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/42.jpg)
DataCollector
DataCollector
DataCollector
node
node
node
node
node
Use Case 1 - Shipped Analytics Data Flow
![Page 43: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/43.jpg)
• Identify the top technology trends by analyzing public data and open source projects
• Use machine learning to process a wide range of public data available on the world wide web and identify high potential emerging technologies
• Publish results to a web-based dashboards and refresh results regularly
Use case 2 - Emerging Top technology using Public data
![Page 44: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/44.jpg)
Use Case 2 - Analysis Through Public data
![Page 45: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/45.jpg)
Use Case 2 - Dataflow
APIs
RSS feeds
Scraping
Numeric Network Data
Text data (articles, blogs)
Staging tables
Interactive D3 Dashboards
Websites
Data Sources Data Extractors
Data Storage Data Processing Machine Learning Visualization
Below we used the framework to execute the project in CIS Data Platform
![Page 46: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/46.jpg)
Lambda Reference Architecture
Monitoring / Analytics Cluster (local, Texas-3)
Global Monitoring / Analytics Cluster (global, Texas-1)
Monitoring / Analytics Cluster (local, Ams. -1 )
Monitoring / Analytics Cluster (local, Lon.-1)
Local components and deployment is the same as global, just smaller
Real-time and batch processing (Lambda), anomaly detection, visualization
SSL
Kafka
SSL
SSL
MQTT
![Page 47: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/47.jpg)
MANTL Data Platform in Practice: putting it all together
Working on advanced enabling technology – Mesos, K8S, Orchestration
Working on developing individual components – dev & co-dev: Zoomdata, Riak, Streamsets, etc.
Putting together reference architectures and real solutions to test and further develop the technology
Provide innovation and advanced services to customers
Platform to develop and deliver Microservices and Data applications
![Page 48: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/48.jpg)
Q/A
![Page 49: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/49.jpg)
Next stepsContinue partnerships and co-devlopment efforts with industry leaders to deliver innovation
Continue applying new developed technology to real use cases and PoC with customers and partners
Continue working closely with A&E and Product teams on productization roadmap
Work with A&E team closely on prioritization of our R&D activities to stay closely aligned
![Page 50: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/50.jpg)
Anatomy of the service/framework
Elasticsearch is a highly scalable open-source full-text search and analytics engine
Allows to store, search, and analyse big volumes of data quickly and in near real time
Underlying technology in application to Optimize complex search in Big data
We are working together with Elastic developing and testing their UTILIZING Mesos cluster to run Elasticsearch
![Page 51: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/51.jpg)
Elasticsearch on Mesos Cluster
Elastic framework scheduler
Marathon framework scheduler
Chronos framework scheduler
Zookeper
Chronos ExecutorMarathon ExecutorHA Proxy node
Step 1: Mesos Cluster with Marathon & Chronos running
Step 2:Elastic framework installation on MESOS Master with a configured # of mesos slaves to be launched
Step 3: Deploys the ES executore in MESOS slaves
Step 4: ES nodes discovery and Zookeper pugin in ES nodes
Step 5 Using plugin nodes find each other and search is optimized at cluster level
Elasticsearch executor & Zookeper pugin
![Page 52: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/52.jpg)
MANTL Architecture – DatacentersControl nodes manage the cluster and resource nodes. Containers automatically register themselves into DNS so that other services can locate them.
Once WAN joining is configured, each cluster can locate services in other data centres via DNS or Consul API
Single Datacentre Multiple Datacentre
![Page 53: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/53.jpg)
Client Client Client
RPC over DNSmask
RPC over DNSmask
LAN gossipover DNSmask
Server Server(Leader) Server
replication replication
Lead forwarding
Internet
Server Server(Leader) Server
replication replication
Lead forwarding
Datacenter 1
Datacenter 2
Remote DC forwarding
WAN gossip
TCP&UDP
Consul
➢ Service discovery➢ Client health-checking➢ Key-value store for
configurations➢ Multi-datacenter support
![Page 54: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/54.jpg)
Mesos features
Mesos makes it easy to develop distributed systems by providing high-level building blocks.
➢ Scalability➢ Fault-tolerance and self-healing➢ Resource isolation➢ Fine Grained resource elasticity
![Page 55: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/55.jpg)
Mesos architecture
![Page 56: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/56.jpg)
Mesos setup for developing application
![Page 57: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/57.jpg)
ZK
ZK
ZK
Zookeeper quorum
JN
JN
JN
Shared edits
DataNode
DataNode
ActiveNameNode
ZookeeperFailover Controller
ActiveNameNode
ZookeeperFailover Controller
DataNode
Heartbeat Heartbeat
Write Read
Active NN state monitoring
Standby NN state monitoring
Monitor and maintain active lock
Monitor and try to take active
lock➢ Used to store and distribute data accross a cluster
➢ Is a base for batch analytic processing
➢ Is highly available and fault tolerant
➢ Automatically scaled and self-healing with Mesos
HDFS framework
![Page 58: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/58.jpg)
* https://github.com/datastax/spark-cassandra-connector
Mesos Framework for Spark and Cassandra
![Page 59: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/59.jpg)
* Smart broker.id assignment* Preservation of broker placement* Rolling restarts* Easy cluster scale-up
Mesos framework for Kafka
![Page 60: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/60.jpg)
Fault tolerant job scheduler handles dependencies and ISO8601 based
schedules Flexible Job Scheduling Supports arbitrarily long dependency chain supports the definition of jobs triggered by
the completion of other jobs
Mesos framework for Chronos
![Page 61: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/61.jpg)
How MANTL Data Platform for business application• Cisco Data Platform can be used to build custom applications or
service for various analysis and Data analytics initiative.• Companies can streamline Data ingestion, process,
manipulate , analyse and visualize data all in single Infrastructure
![Page 62: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/62.jpg)
Yali Load Testing FrameworkYali
Elasticsearch
Kafka
Cassandra
HDFS
Plugins
Kafka
Cassandra
HDFS
Storage
Elasticsearch
Generate data to load test storage
![Page 63: MANTL Data Platform, Microservices and BigData Services](https://reader036.fdocuments.in/reader036/viewer/2022062316/587998c71a28ab95318b6f23/html5/thumbnails/63.jpg)
Elasticsearch Plugin Testing Results
Job Host config
Elasticsearch config
Execution threads
Batches Records/batch/thread
Average response from ES, s
Records/s Record size, b
Records generated * 10^6
Execution time, min
Win7, 4 cpu, 16 ram
Cluster: CentOs 6.7, Elasticsearch 2.1.1, VPN network, 2 master(4 cpu, 16 ram), 15 worker nodes(8 cpu, 32 ram)
12 60 50000 78 6804 280 36 84
Local: CentOs 6.4, Elasticsearch 2.1.1, VMware virtual network, single node (2 Core cpu, 8 Gb ram)
4 60 10000 1,6 14768 280 2,4 2,5
Records