Layers

Yelp’s MissionConnecting people with great

local businesses.

(Abstraction) Layers(related: separation of concerns)

Abstractions?

"Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise." - Edsger Dijkstra

All non-trivial abstractions, to some degree, are leaky. - Joel Spolsky

Any problem in computer science can be solved with another layer of indirection. But that usually will create another problem - David Wheeler

Danger: I don’t know as much as Rich Hickey

Identify the Author’s Purpose

A. Persuade the audience that his/her design recommendations are better than the current status quo

B. Inform others about the history of architecture decisions in computer science

C. Inspire engineers to think outside the box when designing systems

D. Entertain the audience with historic anecdotes

E. None of the above

DevOps

DevOpsEst. 2009

Let’s Talk About Devops Features

Wireframes

Tasks

Code / Unit Tests

Build + Release

Deployment | Monitoring | Alerting

QA | Acceptance Tests

PM

UX

Lead Dev

Devs

QA

Release Engineering

Operations

“What if we got rid of some of the layers?”

?

Let’s Talk About Devops (no-ops?)

Features

Wireframes

Tasks

Code / Unit Tests

QA | Acceptance Tests

Build + Release

Deployment | Monitoring | Alerting

PM

UX

DevsSeparation

OfConcerns

Why?

● Increased “Performance” (deploys per day)● Better ownership of issues● Harder to hire● Massive increase in developer complexity● Large inter-team communication overhead

Filesystems

Let’s talk about filesystems

files

ext4

dm-crypt

Software Raid

Raw Block Devices

LVM

/mnt/stuff/foo

/mnt/stuff

/dev/mapper/luks-17c8...

/dev/mapper/lv0

/dev/md0

/dev/sd{a,b}

“What if we got rid of some of the layers?”

Let’s talk about zfs

files

zfs

(zraid)(encryption)(checksums)

(pools)(vdevs)

Raw Block Devices

/mnt/stuff/foo

/mnt/stuff

/dev/sd{a,b}

SeparationOf

Concerns

Now What?

● Better error detection/recovery● We can deduplicate things● More efficient snapshots / Repair● All new tooling is not interoperable

New Concept: “Sympathetic Abstraction”

● “Opposite” of “Leaky”?● https://mechanical-sympathy.blogspot.com/

~ $ ssh -vv leb1.xkyle.com -- trueOpenSSH_7.7p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017debug1: Reading configuration data /home/kyle/.ssh/configdebug1: /home/kyle/.ssh/config line 1: Applying options for leb1.xkyle.comdebug1: Reading configuration data /etc/ssh/ssh_configdebug1: /etc/ssh/ssh_config line 19: Applying options for *debug2: resolving "leb1.xkyle.com" port 4242debug2: ssh_connect_direct: needpriv 0debug1: Connecting to leb1.xkyle.com [2607:8b00:0:96::d0eb:ec0a] port 4242....debug1: Entering interactive session.debug1: pledge: networkdebug2: channel_input_open_confirmation: channel 0: callback startdebug2: fd 3 setting TCP_NODELAYdebug2: client_session2_setup: id 0debug1: Sending environment.debug1: Sending env LANG = en_US.UTF-8debug2: channel 0: request env confirm 0debug1: Sending command: true...Transferred: sent 2696, received 2516 bytes, in 0.1 secondsBytes per second: sent 23959.5, received 22359.8debug1: Exit status 0

https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/configCassandra_yaml.html?hl=commitlog_sync

commit_failure_policy

(Default: stop) Policy for commit disk failures:

● die● Shut down gossip and Thrift and kill the JVM, so the node can be replaced.● stop● Shut down gossip and Thrift, leaving the node effectively dead, available for inspection using JMX.● stop_commit● Shut down the commit log, letting writes collect but continuing to service reads (as in pre-2.0.5 Cassandra).● ignore● Ignore fatal errors and let the batches fail.

disk_optimization_strategy

(Default: ssd) The strategy for optimizing disk reads. Possible values: ssd or spinning.

disk_failure_policy

(Default: stop) Sets how Cassandra responds to disk failure. Recommend settings: stop or best_effort. Valid values:

● die● Shut down gossip and Thrift and kill the JVM for any file system errors or single SSTable errors, so the node can be

replaced.● stop_paranoid

Kyle’s Abstraction Law:

“Over time, layers tend toward becoming more sympathetic (or merged) in the pursuit of performance.”

HTTP(S)

OSI Layer Model History

● Published in 1984 under ISO7498 (35 years ago)● Mostly by Charles Bachman from his experience with ARPANET, etc● Written with the purpose of giving protocol designers a Map to follow

Layer 6: Presentation

Layer 5: Session

Layer 4: Transport

Layer 2: Data link

Layer 1: Physical

Layer 3: Network

SSL

N/A (Right? HTTP is “stateless”)

TCP (Transmission Control Protocol)

IP (Internet Protocol)

802.11 (Wi-Fi?)

802.11 (Wi-Fi?)

Layer 7: ApplicationHTTP (Hypertext Transfer Protocol)HTTPS in the OSI Network Model

https://en.wikipedia.org/wiki/Internet_protocol_suite#Comparison_of_TCP/IP_and_OSI_layering

The IETF protocol development effort is not concerned with strict layering. Some of its protocols may not fit cleanly into the OSI model, although RFCs sometimes refer to it and often use the old OSI layer numbers.

The IETF has repeatedly stated that Internet protocol and architecture development is not intended to be OSI-compliant. RFC 3439, addressing Internet architecture, contains a section entitled: "Layering Considered Harmful".

RFC 1122, Internet STD 3

(1989)

Cisco Academy[28]

Kurose,[29] Forouzan[30]

Comer,[31] Kozierok[32]

Stallings[33] Arpanet Reference Model (RFC 871)

OSI model

Four layers Four layers Five layers Four+one layers Five layers Three layers Seven layers

"Internet model" "Internet model" "Five-layer Internet model" or "TCP/IP protocol suite"

"TCP/IP 5-layer reference model"

"TCP/IP model" "Arpanet reference model"

OSI model

Application Application Application Application Application Application/Process Application

Presentation

Session

Transport Transport Transport Transport Host-to-host or transport

Host-to-host Transport

Internet Internetwork Network Internet Internet Network

Link Network interface Data link Data link (Network interface)

Network access Network interface Data link

Physical (Hardware) Physical Physical

Layer 6: Presentation

Layer 5: Session

Layer 4: Transport

Layer 2: Data link

Layer 1: Physical

Layer 3: Network

SSL

N/A (Right? HTTP is “stateless”)

TCP (Transmission Control Protocol)

IP (Internet Protocol)

802.11 (Wi-Fi?)

802.11 (Wi-Fi?)

Layer 7: ApplicationHTTP (Hypertext Transfer Protocol)HTTP in the OSI Network Model

“What if we got rid of some of the layers?”

Transport Layer (OSI 4)

Link Layer (OSI 1,2)

Internet Layer (OSI 3)

UDP (User Datagram Protocol)

IP (Internet Protocol)

802.11 (Wi-Fi)

Application Layer (OSI 5,6,7)

EncryptionSessions

MultiplexingError Correction

QUIC(Quick UDP Internet Connections)

QUIC in the IP Model

SeparationOf

Concerns

Now What?

● Roaming sessions● 0-RRT Handshakes● Better throughput● Better loss recovery● You can’t debug via tcpdump anymore!● Only a few companies has dared to ship this!● Very poor tooling / ecosystems around it!● Must have faith in QUIC-crypto!

(Micro) Services

Rails App

Model View

Controller

Users

Mysql

“What if we ADDED some more layers?”

Users

API Gateway

Users Service SearchTweetsAds Moments

Ads Mysql User

MysqlSearch

Infra MomentsSQLTweets

NoSQL

Followers

Graph DB

Timeline

Now What?

● No more atomic cross-cutting changes● Simplified deploy process? (Simple for who?)● Polyglot stack (easy to hire for)● No more unified tooling● You scale out components (scale to millions of rps)● You scale out development (1000 engineers)

k8s

Layer 3: Application Code

Layer 1: IaaS VM

Layer 0: IaaS Provider Bare Metal

Layer 2: OS

Java Jar

Ubuntu x86_64

c5.xlarge

N/A

Traditional Cloud

Deployment

“What if we ADDED EVEN MORE layers?”

Layer 4: Pod

Layer 3: Container Orchestration

Layer 1: IaaS VM

Layer 0: IaaS Provider Bare Metal

Layer 2: OS

pod-foo

kublet

Ubuntu x86_64

c5.xlarge

N/A

Layers of a k8s Infra

Layer 5: Containerdocker

Layer 6: Processjava -jar

Layer 5: k8s federation

Layer 4: k8s cluster

Layer 3: Deployment / ReplicaSet

Layer 1: Docker Container

Layer 0: Process

Layer 2: Pod

Prod k8s federation

uswest1-prod

Frontend replicaset

myapp-pod

FROM apache2

httpd

K8s Layers

“What if we got rid of some of the layers?”

AWS Metal Instances

Layer 0/1: IaaS Provider Bare MetalHardware

Future Layers

Layer 0/1: IaaS Provider Bare MetalHardware

Future Layers Extreme Example -

Unikernels?

Layer 2-7: Unikernel(No OS, just a directly booting

application)MirageOS, etc

Layer 0/1: IaaS Provider Bare MetalHardware

Future Layers - AWS Lambda / FaaS?

Layer 2-5: Amazon / k8s StuffSame Stuff As Before

Layer 7: Your ProcessYour Code

Layer 6: ContainerFirecracker / Docker

What? We Are Back To Where We Started?

● Where is the mechanical sympathy?● What happened to Layer-Smashing, Kyle?● Three Reasons why we are moving this direction...

Layer 0/1: IaaS Provider Bare MetalHardware

Reason 1:You have a layer for just the stuff you care about

Layer 2-5: Amazon / k8s StuffSame Stuff As Before

Layer 7: Your ProcessYour Code

Layer 6: ContainerFirecracker / Docker

k8s pods

Apache

API Gateway + FaaS Controller(Request Routing)(Compute Dispatch)

Reason 2: Sympathetic routing and dispatchOr “Functions as a Service (Serverless)” Layer Smashing

Autoscaling Group(Compute Dispatch)

Apache

Elastic Load Balancer(Request Routing)

Apache

Reason 3:Sympathetic Compute Scheduling Layers(k8s operators)

Autoscaling Group

MySQL

MySQL

?

K8s MySQL Operator

MySQL

MySQL

I know what I

must do

Conclusions?

1. “Sympathetic Layers” are the best layers!

2. k8s operators and FaaS as re-slicing where the layers are (for the greater good!)

3. You should watch “Simple Made Easy” before changing your mindhttps://www.infoq.com/presentations/Simple-Made-Easy

Hold for (shameless) applause,

Then

Questions?

SEPARATION OF

CONCERNS

IS OVERRATED

Attribution● OSI Layers - http://www.gargasz.info/osi-model-how-internet-works/● Simple Made Easy - The Art of Unix Programming (2003) - Eric S. Raymond

X

Linux Kernel

GPU

X Server

X Client (GUI App)

X11 Protocol over Socket /tmp/.X11-unix/X0

Ioctl via libdrm/dev/dri/card0

3d Game

Ioctl via libdrm/dev/dri/card0

Window Manager

“Mechanism, not policy”

“What if we got rid of some of the layers?”

Now What?

● Lower latency GUI Interactions● NO TEARING