Characterizing and Contrasting Kuhn-tey-ner Awr-kuh-streyt-ors

download Characterizing and Contrasting Kuhn-tey-ner Awr-kuh-streyt-ors

of 66

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Characterizing and Contrasting Kuhn-tey-ner Awr-kuh-streyt-ors

  • Characterizing andContrasting Kuhn-tey-ner


    All Day DevOps, November 2016

    Lee Calcote

  • Lee Calcote


    clouds, containers, infrastructure,applications and their management

  • [kuhn-tey-ner][awr-kuh-streyt-or]



  • FleetNomadSwarm

    Kubernetes Mesos+Marathon


    @lcalcote(Stay tuned for updates to presentation and book)

    Joyent TritonDocker Datacenter

    AWS ECSAzure Container Service

    Rackspace Carina

  • Onesize does notfitall.

    A strict apples-to-apples comparison is inappropriate and notthe objective, hencecharacterizingandcontrasting.


  • Let's not go here today.

    Container orchestrators may be intermixed.


  • Categorically Speaking


    Genesis & PurposeSupport & MomentumHost & Service Discovery

    Modularity & ExtensibilityUpdates & MaintenanceHealth MonitoringNetworking & Load-BalancingSecrets ManagementHigh Availability & Scale


  • Hypervisor Manager Elements


    Container Orchestrator


    Host (Node)Task



    ServiceVirtual IP

    Secret / ConfigVolume


  • CoreCapabilities

    Cluster Management

    Host Discovery

    Host Health Monitoring


    Orchestrator Updates and Host


    Service Discovery

    Networking and Load-Balancing

    Multi-tenant, multi-region

    AdditionalKey CapabilitiesApplication Health Monitoring

    Application Deployments

    Application Performance Monitoring


  • Nomad

  • Genesis & Purposedesigned for both long-lived services and short-livedbatch processing workloads. cluster manager with declarative job specifications. ensures constraints are satisfied and resourceutilization is optimized by efficient task packing. supports all major operating systems and virtualized,containerized or standalone workloads. written in Go and with a Unix philosophy.


  • Support & MomentumProject began June 2015 has 113 contributors over 16months

    Current release v0.4v0.5 to be released any day nowNomad Enterprise offering aimed for Q1-Q2 next year.

    Supported and governed by HashiCorpHashiconf US '15 had ~300 attendeesHashiconf EU '16 had ~320 attendeesHashiConf US '16 had ~ 500 attendees

    @lcalcoteNomad is a single binary, both for clients

    and servers, and requires no externalservices for coordination or storage.

  • Nomad Architecture

    services for coordination or storage.


  • Host & Service Discovery

    Host Discovery

    Gossip protocol - Serf is usedDocker multi-host networking and Swarmkit use Serf, too

    Servers advertise full set of Nomad servers to clientsheartbeats every 30 seconds

    Creating federated clusters is simple

    Service Discovery

    Nomad integrates with to provide servicediscovery and monitoring.



  • Scheduling

    two distinct phases, feasibility checking and ranking. optimistically concurrent

    enabling all servers to participate in scheduling decisionswhich increases the total throughput and reduces latency

    three scheduler types used when creating jobs:service, batch and system

    nomad plan point-in-time-view of what Nomad will do


  • Modularity & ExtensibilityTask drivers

    Used by Nomad clients to execute a task and provideresource isolation. By having extensible task drivers are important forflexibility to support a broad set of workloads. Does not currently support pluggable task drivers,

    Have to iImplement task driver interface and compileNomad binary.


  • Updates &Maintenance


    Drain allocations on a running node.integrates with tools like Packer, Consul, and Terraform tosupport building artifacts, service discovery, monitoring andcapacity management.


    Log rotation (stderr and stdout)

    no log forwardsupport, yet

    Rolling updates (via the `update` block in the job specification).


  • Health MonitoringNodes

    Node health monitoring is done via heartbeats, soNomad can detect failed nodes and migrate theallocations to other healthy clients.


    currently http, tcp and script

    In the future Nomad will add support for more Consulchecks.

    nomad alloc-status reports actual resource utilization


  • Networking&Load-Balancing


    Dynamic ports are allocated in a range from 20000 to 60000.Shared IP address with Node


    Consul provides DNS-based load-balancing


  • Secrets Management

    Nomad agents provide secure integration with Vaultfor all tasks and containers it spins up

    gives secure access to Vault secrets through aworkflow which minimizes risk of secret exposureduring bootstrapping.


  • High Availability & Scale

    distributed and highly available, using both leaderelection and state replication to provide availability inthe face of failures. shared state optimistic scheduler

    only open source implementation.

    1,000,0000 across 5,000 hosts and scheduled in 5 min.

    Built for managing multiple clusters / cluster federation.


  • easier to usea single binary for both clients andserverssupports different non-containerizedtasksarguably the most advanced schedulerdesignupfront consideration of federation /hybrid cloudbroad OS support

    Outside of scheduler, comparatively lesssophisticated

    Young project

    Less relative momentum

    Less relative adoption

    Less extensible / pluggable


  • Docker Swarm

  • Docker Swarm 1.12aka

    Swarmkit or Swarm mode


  • Genesis & PurposeSwarm is simple and easy to setup. Responsible for the clustering and scheduling aspectsof orchestration. Originally an imperative system, now declarative Swarms architecture is not complex as those ofKubernetes and Mesos Written in Go, Swarm is lightweight, modular andextensible


  • Docker Swarm 1.11 (Standalone)

    Docker Swarm Mode 1.12 (Swarmkit)


  • Support & Momentum

    Contributions:Standalone: ~3,000 commits, 12 core maintainers (140 contributors)

    Swarmkit: ~2,000 commits, 12 core maintainers (40 contributors)

    ~250 Docker meetups worldwide Production-ready:

    Standalone announced ~12 months ago (Nov 2015)

    Swarmkit announced ~3 month ago (July 2016)


  • Host & Service DiscoveryHost Discovery

    used in the formation of clusters by the Manager to discover forNodes (hosts).

    Like Nomad, uses Hashicorp's go for storing cluster state

    Pull model - where worker checks-in with the Manager

    Rate Control - of checks-in with Manager may be controlled atManager - add jitter

    Workers don't need to know which Manager is active; FollowerManagers will redirect Workers to Leader

    Service DiscoveryEmbedded DNS and round robin load-balancing

    Services are a new concept