Exploring Twitter’s Finagle technology

stack for microservicesTomasz Kogut / @almendar

XII Tricity Scala User Group Meetup

About me:• Software Engineer at

• Working in the internet advertisement industry

• I try to create distributed, highly performant systems that will run on the JVM

• Programming in Scala since ~2011

Microservices

questionnaireWho uses μ-services approach?

Who has a system that would like to break down into smaller pieces?

What we already know?

• Working on huge monolithic projects is hard

• Complexity and interdependencies become incomprehensible at some point

• We all fell in our guts that breaking it down is a way to go

• Single responsibility principle but for services

The plan1. Create multiple git repositories

2. Create/copy/move code around repos

3. Pour it all with a fair share of REST/HTTP

4. Extract common code/models into libs

5. Run it!

Let’s go to production!

So what happened?

Paying for the lunch

• Easy to develop

• Hard to operate

• More fine-grained services boost both traits

Things to address

• A lot of stuff from “Fallacies of distributed computing"

• Tracing request across different services

• Service discovery and tracing service topology

• Message serialization

• Distributing load across nodes

• Measuring performance and finding bottlenecks is not easy

• Interface contracts can introduce hidden coupling

• Duplication of effort

• Historically we were a .Net shop

• Think spring-like approach

• Moved to API first approach (although we were running some kind of μ-service approach to some extend)

• Splitting one of the business features into small-chunks using whatever tech stack we want so we went akka/akka-http

• We didn’t want to use akka clustering

• Recreating utils e.g. request retry

• Establishing new project = work upfront before creating any business value e.g. metrics

• Every project had it’s own structure - mental context switching

• REST is good but other protocols might be better

• Fighting the “spray DSL” and marshallers 😀

• Creating json is annoying and automatic derivation from classes was useless most of the time

• We had to think how to properly version our apis and hold backward compatibility

• We had situations where multiple services needed to be deployed for the solution to work (hidden monolith)

• How do I run this service locally?

What is it?• Protocol agnostic RPC system for JVM

• Based on Netty

• Allows for building servers and clients

• Core platform on which Twitter is build

• "Your Server as a Function" by Marius Eriksen

Foundations• Futures• Services• Filters

Future• NOT the Scala one unfortunately but very close

• Some api differences- import com.twitter.util.Future- Future.value == Future.successful- Future.exception == Future.failed- Future.collect == Future.sequence

• Usual monadic composition with flatMap and map

• There is also twitter’s Try, Duration and a bunch of others…

Service• Basic building block in Finagle

• Just a function that returns a Future

• Servers implement services to which Finagle dispatches incoming requests

• Finagle furnishes clients with instances of Service representing either virtual or concrete remote servers

abstract class Service[-Req, +Rep] extends (Req => Future[Rep])

Example local service (1)

import com.twitter.finagle.Serviceimport com.twitter.util.Future

val lengthService = Service.mk[String, Int] { req =>

Future.value(req.length)

}

lengthService("Hello TSUG").foreach(println)

import com.twitter.finagle.httpimport com.twitter.finagle.Serviceimport com.twitter.util.Futureimport com.twitter.finagle.Http

val service = new Service[http.Request, http.Response] { def apply(req: http.Request): Future[http.Response] = Future.value( http.Response(req.version, http.Status.Ok) )}val server = Http.serve(":8080", service)

Example http server (2)

import com.twitter.finagle.http.Method.Getimport com.twitter.finagle.{Http, Service}import com.twitter.finagle.http.{Request, Response}import com.twitter.finagle.service.FailFastFactory.FailFastimport com.twitter.util.Future

val productServiceHttpClient: Service[Request, Response] = Http.client.configured(FailFast(false)).newClient("127.0.0.1:8080", "productService").toService

val response: Future[Response] = productServiceHttpClient(Request(Get,"/products/22"))

response.foreach{ response => println(response.statusCode) println(response.contentString)}

Example http client (3)

Other protocols client examples

var memCache = ClientBuilder() .name("mc") .codec(Memcached()) .hostConnectionLimit(config.concurrency()) .hosts(config.hosts())

val mySqlClient = Mysql.client .withCredentials(username(), password()) .withDatabase(dbname()) .newRichClient("%s:%d".format(host().getHostName, host().getPort))

Services represents clients and servers

symmetrically

Please notice that:

Filter• Again a function that returns a Future

• Filters are always attached to Services altering their behavior

• Used for application agnostic concerns such as: timeouts, retry policies, service statistics, and authentication

abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn] extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut])

Filter• Filters just like any other functions

compose

• There is a andThen method that chains different filters with each other and in the end with service

Local service filter

import com.twitter.finagle.{Filter, Service}import com.twitter.util.Future

val someStringMetrics = Service.mk[String,Int] { req => Future.value(req.length)}

val evenMetricsFilter = new Filter[Float, Boolean, String, Int] { override def apply(input: Float, stringMetricsService: Service[String, Int]): Future[Boolean] = { stringMetricsService(input.toString).map(_ % 2 == 0) }}

(evenMetricsFilter andThen someStringMetrics)(1.234f)

import com.twitter.finagle.{Http,Service, Filter}import com.twitter.finagle.http.{Request,Response}import com.twitter.finagle.service.TimeoutFilterimport com.twitter.util.MockTimerimport com.twitter.conversions.time._

val httpClient: Service[Request, Response] = Http.client.newService("twitter.com")val timeoutFilter: Filter[Request, Response, Request, Response] = new TimeoutFilter[Request, Response](30.seconds, new MockTimer)val httpClientWithTimeout: Service[Request, Response] = timeoutFilter andThen httpClient

Filter example http client

WARNINGThis is only a showcase, timeouts for http clients are best configured by Http.client object

How to implement this?

Stacking filters

recordHandletime andThen traceRequest andThen collectJvmStats andThen parseRequest andThen logRequest andThen recordClientStats andThen sanitize andThen respondToHealthCheck andThen applyTrafficControl andThen virtualHostServer

Fronted webservers configuration at twitter:

Cool client filters (or wannabe-filters)

• MonitorFilter - handling exceptions

• StatsFilter - exposing metrics

• RetryFilter - retying calls with configured budget and a back-off policy

• TimeoutFilter - mentioned earlier, for different aspects like idle time, request time, connection time ect.

• LoadBalancing (not really a filter) - distributed load across nodes, just pass multiple host values

• FailFast (not really a filter) - mark node as dead for a certain period of time

• Unfortunately not all functionality can be expressed with Filters

• Some things are a bit hairy - e.g. request cancelation

• Beneath implementations is non-trivial but comprehensible

• But the good part is that it all works out to the box for free with almost none configuration!

There is more in Finagle to explore:

• Thrift protocol

• Mux - multiplexing RPC

• ZooKeeper support

• Network location naming

Twitter Server

Even more goodies

Twitter Server• A “template” for finagle-based

server

• Flags

• Logging

• Metrics

• Admin interface

Flags (1)• Cmd line arguments passed to the

application

• A remedy to “How do I start this thing and what are the options?”

• Smart with type inference and parsing capabilities

Flags (2)val addr = flag("bind", new InetSocketAddress(0), "Bind address")val durations = flag("alarms", (1.second, 5.second), "2 alarm durations")

$ java -jar target/myserver-1.0.0-SNAPSHOT.jar -helpAdvancedServer

-alarm_durations='1.seconds,5.seconds': 2 alarm durations -help='false': Show this help

-admin.port=':9990': Admin http server port -bind=':0': Network interface to use

-log.level='INFO': Log level -log.output='/dev/stderr': Output file

-what='hello': String to return

Option for fail fast when missing a flag

Logging• Twitter server provides a logging

trait

• It can be configured with standard flags

• Can be tuned on-the-fly

Admin interface loggers control

Metrics• Defining own statistics

• JVM stats

• Processing and network stats

Others• Linting your server for problems

• Looking at threads

• Profiling

Admin interface

Finatra

What is it?• Finatra is build atop Finagle and

Twitter Server

• Finatra does all the heavy-lifting that one needs to do when working only with Finagle

Finatra features• Dependency injection using Guice

• JSON handling with Jackson

• Mustache support

• Routing

• Integrates it all together nicely

package pl.tk.finagle.recommendation.controller

import javax.inject.{Inject, Singleton}

import com.twitter.finagle.http.Requestimport com.twitter.finagle.tracing.ClientTracingFilter.TracingFilterimport com.twitter.finatra.http.Controllerimport com.twitter.finatra.request._import com.twitter.inject.Loggingimport pl.tk.finagle.recommendation.engine.{RecommendationCmd, RecommendationEngine}

case class RecommendationFromCookieRequest(@Inject request: Request, @RouteParam cookieId: Int, @RouteParam eshopId: Int, @Header `x-auth`: String)

@Singletonclass RecommendationController @Inject()(recommendationEngine: RecommendationEngine) extends Controller with Logging {

get("/recommend/:eshop_id/:cookie_id") { r: RecommendationFromCookieRequest => infoResult("Hello") { TracingFilter("RecommendationEngine") andThen recommendationEngine apply RecommendationCmd(r.cookieId, r.eshopId) } }}

Unified error reporting

➜ finagle-spike git:(master) ✗ curl -v 127.0.0.1:2001/recommend/32/wsf* Trying 127.0.0.1...* Connected to 127.0.0.1 (127.0.0.1) port 2001 (#0)> GET /recommend/32/wsf HTTP/1.1> Host: 127.0.0.1:2001> User-Agent: curl/7.43.0> Accept: */*>< HTTP/1.1 400 Bad Request< Content-Type: application/json; charset=utf-8< Server: Finatra< Date: Wed, 30 Mar 2016 07:05:53 +00:00< Content-Length: 79<* Connection #0 to host 127.0.0.1 left intact{"errors":["cookie_id: 'wsf' is not a valid int","x-auth: header is required"]}%

JSON Support• Build on jackson-module-scala

• Support for case classes (de)serlialization

• Integrated with Joda

• Error accumulation (instead of fail-fast)

• Integration with routing

• Most of the cases just return object

Customizing Json

class Server extends HttpServer { override def jacksonModule = CustomJacksonModule ...}

object CustomJacksonModule extends FinatraJacksonModule { override val additionalJacksonModules = Seq( new SimpleModule { addSerializer(LocalDateParser) })

override val serializationInclusion = Include.NON_EMPTY

override val propertyNamingStrategy = CamelCasePropertyNamingStrategy

override def additionalMapperConfiguration(mapper: ObjectMapper) { mapper.configure(Feature.WRITE_NUMBERS_AS_STRINGS, true) }}

Validation

import com.twitter.finatra.validation._import org.joda.time.DateTimecase class GroupRequest(@NotEmpty name: String, description: Option[String], tweetIds: Set[Long], dates: Dates) { @MethodValidation def validateName = { ValidationResult.validate( name.startsWith("grp-"), "name must start with 'grp-'") }}

case class Dates(@PastTime start: DateTime, @PastTime end: DateTime)

Also checked at routing time

Testing (1)• Based on ScalaTest

• Feature testing (both black box and white box) - looking at external interface of our service

• Integration tests - only a subset of modules is instantiated and tested

• Finatra does not provide anything for unit testing

Testing (2)• Smart JSON diff

• Integration with DI

• Easy mocking

• Embedded http server with our service

last but not least…

What is it?• Distributed request tracing

• Based on Google Dapper paper

• Helps getting insight on how the services interact

Zipkin

Python pyramid_zipkinPyramid Http

(B3)Http (B3) Kafka | Scribe Yes

py2, py3

support.

Java brave

Jersey,

RestEASY,

JAXRS2,

Apache

HttpClient,

Mysql

Http (B3)Http, Kafka,

ScribeYes

Java 6 or

higher

Scala finagle-zipkin FinagleHttp (B3),

ThriftScribe Yes  

Ruby zipkin-tracer Rack Http (B3)Http, Kafka,

ScribeYes

lc support.

Ruby 2.0 or

higher

C# ZipkinTracerM

odule

OWIN,

HttpHandlerHttp (B3) Http Yes

lc support.

4.5.2 or higher

Go go-zipkin

Not tied to finagle

Zipkin Architecture

DEMO APP

Recommendation

ProductsService

UserProfile

1 GetProducts

2 GetPromotedProducts

GetUser

CookieId, Eshop

Thank you!