Logging in Scala
-
Upload
john-nestor -
Category
Software
-
view
974 -
download
1
Transcript of Logging in Scala
Logging in Scala
John Nestor 47 Degrees
www.47deg.com
August 9, 2016
147deg.com
47deg.com © Copyright 2016 47 Degrees
Outline
• Introduction
• Architecture
• Features
• Implementation
• Demos
• Cross Service Log Aggregation
2
Introduction
3
47deg.com © Copyright 2016 47 Degrees
Existing Scala Loggers
• Java loggers (used in Scala code)
• Log4j (used in Spark)
• SLF4J and Logback
• Scala loggers
• Abandoned: Configgy, Logula (Coda Hale)
• Log4s (wraps SLF4J)
• scala-logging (Heiko Seeberger) (wraps Log4j)
• Special Scala loggers
• Akka actor logging (can route to Logback)
• Play logging (uses Logback)
4
47deg.com © Copyright 2016 47 Degrees
Why a New Logger?
• Pure Scala. (No Java code)
• Richer messages (Json rather than text strings)
• Aggregate logs across multiple (micro-)services
• Clean design with rich features
5
47deg.com © Copyright 2016 47 Degrees
New Scala Logger History
• Whitepages Logger - part of a new Scala web service framework
• Persist Logger (work supported by 47 Degrees)
• clean up and improve API
• removed Whitepages specific stuff
• added complete documentation
• Full source and documentation on Github
• Maven central "com.persist" % "persist-logging_2.11" % "1.2.4"
• Apache 2 license
6
47deg.com © Copyright 2016 47 Degrees
Rich Logging Example
log.error(s”Too many values Size:${vals.size} Max:${vals.max}”)
log.error(map(“msg”->”Too many values”, ”Size”->vals.size, ”Max”->vals.max), id=currentId)
{“msg”:”Too Many Values”, “Size”: 10, “Max: “33}
7
Architecture
8
47deg.com © Copyright 2016 47 Degrees
Persist Logger Architecture
9
logging.conf
logback.xml
Scala Logger API Slf4j API
Akka Actor Logging API
Custom Logback
Appender
LogBack Custom Akka Logging Actor
Logging Actor
Stdout Appender
File Appender
Kafka Appender
Other Appenders
Logger State
Scala Logger
Control API
Features
10
47deg.com © Copyright 2016 47 Degrees
API Log Call
• Level
• Rich Message
• Id (optional)
• Per request level control
• Per request log aggregation
• Exception (optional)
• use RichException for a RichMessage
11
47deg.com © Copyright 2016 47 Degrees
Rich Messages
• String
• Int, Long, Float, Double, BigDecimal
• null, Boolean
• Seq[RichMessage]
• Map[String,RichMessage]
12
47deg.com © Copyright 2016 47 Degrees
Why Json?
• Easy to support rich metadata in addition to the message
• Easier to parse in downstream tools
• No need to manually escape strings
• Naturally supports multiline messages
• Standard text format
• But many internal possibilities
• This logger uses Persist Json (only internally)
• Rich messages use standard Scala types for in internal API
13
47deg.com © Copyright 2016 47 Degrees
Kind of Logs
• Error log
• Alternative logs
• GC
• Timing (for fine grain timing)
• Server (request-response, duration)
• Client (request, response, duration)
• User defined
14
47deg.com © Copyright 2016 47 Degrees
Standard Fields
• @category (error, gc, …)
• @host
• @service
• @severity (info, debug, warn, error, …)
• @timestamp (msec)
• class, file, line, actor
• msg
• trace (for exceptions, Json)
15
47deg.com © Copyright 2016 47 Degrees
Appenders
• Stdout
• no server/service specific fields
• color (optional)
• summary
• File (daily rotation)
• Kafka
• Custom, user defined
• Control destination
• Control contents and format
16
47deg.com © Copyright 2016 47 Degrees
Level Control
• logging.conf (default options), can override
• uses Typesafe config
• levels: API, SLF4J, Akka
• Change level via API
• Per request custom level
• Custom filters
• Based on level, and content
17
47deg.com © Copyright 2016 47 Degrees
Timing
• Fine grain timing
• Uses request ids to aggregate
• Time in microseconds
• Time log
• Can be used with concurrency (futures and actors)
18
47deg.com © Copyright 2016 47 Degrees
Other Loggers
• Captures messages from other legacy loggers that might be used in new project libraries
• SLF4J (captured via Logback appender)
• Akka Actor Logging (captured by custom Akka log handing actor)
19
47deg.com © Copyright 2016 47 Degrees
Look At Documentation
• logging.conf
• Overview
• API
20
Implementation
21
47deg.com © Copyright 2016 47 Degrees
Persist Logger Architecture
22
logging.conf
logback.xml
Scala Logger API Slf4j API
Akka Actor Logging API
Custom Logback
Appender
LogBack Custom Akka Logging Actor
Logging Actor
Stdout Appender
File Appender
Kafka Appender
Other Appenders
Logger State
Scala Logger
Control API
47deg.com © Copyright 2016 47 Degrees
Dealing with Concurrency
• The logging actor provides fully generalized sync
• Serializes output
• Handles filters
• But asking the actor on every message can be slow
• @volatile booleans for each log level associated with logger
• Note the logger itself is global state
• One copy for entire app, no need to pass a parameter everywhere: ClassLogging and ActorLogging traits.
• Must start before any messages and stop after any messages (but there is a development workaround)
• Contains log level booleans23
47deg.com © Copyright 2016 47 Degrees
Source Location
• Could find using exception stack trace at run-time
• Better solution is to use Scala macro and reflection at compile-time
24
Demos
25
47deg.com © Copyright 2016 47 Degrees
Demos
• Simple
• Actor
• Exceptions
• Request Id
• Alternative
• Other (Actor, SLF4J)
• Timing
• Filter
• Appender
26
Cross Service Log Aggregation
27
47deg.com © Copyright 2016 47 Degrees
The Problem
• Applications are being split into ever more micro-services
• Much harder to answer questions?
• What is the overall control flow for various requests?
• What was the root cause of an error?
• Where are the performance bottlenecks?
• In this talk we look at a prototype system that can be used to answer these kinds of questions
• Work in progress
28
47deg.com © Copyright 2016 47 Degrees
Related Work
• Google Dapper
• Zipkin
• Akka Tracing (wraps Zipkin)
29
47deg.com © Copyright 2016 47 Degrees
Approach
• Use Scala as much as possible
• Aggregate logs for each independent request across a set of Scala services
• Provide a basis for both
• Near real-time
• Alerts
• View aggregated logs for any request
• More comprehensive batch analysis
• Failure statistics
• Overall patterns
• Trends over time
30
47deg.com © Copyright 2016 47 Degrees
Overall Architecture
31
KafkaSpark
Streaming(Aggregation)
NoSql DB
REST Log API *
(Merge, Filter)
Services *
* Uses: Persist Service Framework
Persist Service Framework Uses:Akka HTTPPersist Json
Persist Logger
Spark Batch Analytics
47deg.com © Copyright 2016 47 Degrees
Scala REST Service
32
Server Log
Service(can be lots of these)
Error Log
Client Log
... To Other Services
Kafka(only one cluster/topic for all
logs and services)
... From Other Services
47deg.com © Copyright 2016 47 Degrees
Aggregation (via Spark Streaming)
• Pass from client to server
• Client name
• Tracking Id
• same for a single request across all micro services
• Span Id
• unique for each call for a given Client-Server pair
• Match Tracking Id and Span Id across
• Client log
• Server log
34
47deg.com © Copyright 2016 47 Degrees
Cross Service Matching
35
Id1Span1
X
AId1
Span1X
Id1Span2
A
Id1Span3
A
AId1
Span1Span2
B
BId1
Span2A
BId1
Span3A
AId1
Span1Span3
B
Match
Match
Match
Match
Service A Service B
47deg.com © Copyright 2016 47 Degrees
Look at Sample Output
• Array of log messages
• Tree of log messages
• Filtered tree with timings
36
Questions
37