LINE's messaging service architecture underlying more than 200 million monthly active users,

and core techs

Yuto KawamuraLINE Corp

Speaker introduction

• Yuto Kawamura

• Senior software engineer

• Apache Kafka contributor

• Joined: Apr, 2015(2 years 5 months)

About LINE•Messaging service

•More than 200 million active users1 in countries with top market share like Japan, Taiwan, Thailand and Indonesia

•Many family services

•News

•Music

• LIVE (Video streaming)

1 As of June 2017. Sum of 4 countries: Japan, Taiwan, Thailand and Indonesia.

Scale MetricsMessages delivered 25 billion per day

Peak time Messages sent(new year greetings) 420K per second

Total API requests per day 80 billion per day

Number of event records published > 150 billion per day

Accumulated analytics data 40PB

Messaging System Architecture

LINE Apps

LEGY JP

LEGY DE

LEGY SG

Thrift RPC/HTTP

talk-server

Distributed Data Store

Distributed async task processing

LEGY• LINE Event Delivery Gateway

• API Gateway/Reverse Proxy

• Written in Erlang

• Deployed to many data centers all over the world

• Features focused on needs of implementing a messaging service

• Zero latency code hot swapping w/o closing client connections

• Durability thanks to Erlang process and message passing

• Single instance holds 100K ~ connection per instance => huge impact by single instance failure

talk-server

• Java based web application server

• Implements most of messaging functionality + some other features

• Java8 + Spring + Thrift RPC + Tomcat8

Datastore with Redis and HBase

• LINE’s hybrid datastore = Redis(in-memory DB, home-brew clustering) + HBase(persistent distributed key-value store)

• Cascading failure handling

• Async write in app

• Async write from background task processor

• Data correction batch

Primary/Backup

talk-server

Cache/Primary

Dual write

When user sends a message

LEGY

LEGY

talk-server

Storage

1. Find nearest LEGY

2. sendMessage(“Bob”, “Hello!”)

3. Proxy request

4. Write to storage

talk-server

X. fetchOps()

6. Proxy request

7. Read message

8. Return fetchOps() with message

5. Notify message arrival

Alice

Bob

Communication between internal systems

• Communication for queuing, transactional updates:

• Query authentication/permission

• Synchronous updates

• Communication for data synchronization, update notification:

• Notify user’s relationship update

• Synchronize data update with another service

talk-server

Auth

Analytics

Another Service

HTTP/REST/RPC

Deeper look of some core components

• Asynchronous HTTP/2 RPC/REST client/server library

• Built on top of Java8, Netty and RPC frameworks

• Open sourced: https://github.com/line/armeria

• Find more details: https://speakerdeck.com/trustin/armeria-lines-next-generation-rpc-layer

Armeria

DocService• Quick API test call + Documentation

sb.serviceUnder("/docs", new DocServiceBuilder() .exampleHttpHeaders(HttpHeaders.of(AUTHORIZATION, "bearer b03c4fed1a")) .exampleRequest(new HelloService.hello_args("Armeria")) .build());

Other features• Client side load balancer

• Retrieve server list from pluggable centralized configuration directory and use for load balancing

• Structured logging

• Supports logging request/response including payload of RPC

• Supports external system as an output such as Apache Kafka

Armeria• Most of new services are built on top of Armeria

• and some old projects are/will be rewritten based on it

• Sophisticated yet minimum LINE’s “best practices”

• PRs of line/armeria would be worth visiting for you

• https://github.com/line/armeria/pulls

• You will see how LINE developers are working for development

Apache Kafka

• A distributed streaming platform

• (narrow sense) A distributed persistent message queue which supports Pub-Sub model

• Built-in load distribution

• Built-in fail-over on both server(broker) and client

How it worksProducer

Brokers

Consumer

Topic

TopicConsumer

Consumer

Producer

AuthEvent event = AuthEvent.newBuilder() .setUserId(123) .setEventType(AuthEventType.REGISTER) .build(); producer.send(new ProducerRecord(“events", userId, event));

consumer = new KafkaConsumer("group.id" -> "group-A"); consumer.subscribe("events"); consumer.poll(100)… // => Record(key=123, value=...)

Consumer GroupA

Pub-Sub

Brokers

Consumer

Topic

Topic

Consumer

Consumer GroupB

Consumer

ConsumerRecords[A, B, C…]

Records[A, B, C…]

• Multiple consumer “groups” can independently consume a single topic

Example: UserActivityEvent

Today’s Kafka usage

However…

• OSS usually isn’t a perfect solution

• We add/fix what we need

• Typically we need more than others, because of our scale and challenging usages

• https://issues.apache.org/jira/browse/KAFKA-4614

• https://issues.apache.org/jira/browse/KAFKA-4024

Sometimes we contribute

Sometimes we share

• Kafka Summit SF 2017

• One Day, One Data Hub, 100 Billion Messages: Kafka at LINE

• https://youtu.be/X1zwbmLYPZg

but we have more things to do

• Performance improvement to enable true stability

• Kafka based reliable async task processing system

• Multi-DC deployment

Summary• LINE’s server architecture is keep evolving

• Almost everything are “Distributed”

• Software we use:

• Java/Earlang/Redis/HBase/Kafka/Armeria

• We love OSS

• we Use/Understand/Develop/Contribute

• and it’s encouraged

• 6 years since we started our service, still have many challenges

• More stable and durable service architecture

• Dealing with a lot of new feature developments

• There are a lot of chances you make a great contribution to our systems and services, please come and talk to me!

End of presentation. Any questions?