Architecture of MessagePack
Transcript of Architecture of MessagePack
![Page 1: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/1.jpg)
Architecture of
MessagePackSadayuki Furuhashi
![Page 2: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/2.jpg)
What’s MessagePack?
• Efficient serialization library
• Rich data structures - compatible with JSON
• Dynamic typing
• Remote Procedure Call (RPC)
• Synchronous, Asynchronous and Callback style
• Concurrent calls with multiple servers
• Event-driven I/O
• Interface Definition Language (IDL) - compatible with Thrift
![Page 3: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/3.jpg)
Efficient Serialization
1. Compact 2. Fast
・ Zero-copy (C++)
・ Stream deserialization
・ Binary-based format
・ Embed type information
![Page 4: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/4.jpg)
Format of MessagePack
JSON MessagePack
null
Integer
Array
String
Map
null c0
10 0a
[20] 91 14
”30” a2 ‘3’ ‘0’
{“40”:null} 81 a1 ‘4’ ‘0’ c0
![Page 5: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/5.jpg)
Format of MessagePack
JSON MessagePack
null
Integer
Array
String
Map
null c0
10 0a
[20] 91 14
”30” a2 ‘3’ ‘0’
{“40”:null} 81 a1 ‘4’ ‘0’ c0
4 bytes 1 byte
2 bytes 1 byte
4 bytes 2 bytes
4 bytes 3 bytes
11 bytes 5 bytes
![Page 6: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/6.jpg)
Format of MessagePack
Fixed length types Variable length types
・ Raw bytes
・ Array
・ Map
・ Integer
・ Floating point
・ Boolean
・ Nil
type value type body...length
Type information
![Page 7: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/7.jpg)
Type information0x000xc20xc30xca0xcb0xcc0xcd0xce0xcf0xdf...
nil false true float double uint8 uint16 uint32 uint64 int8 ...
Types
0xc00xe0
Type information
![Page 8: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/8.jpg)
Embed value
Positive FixNum
Negative FixNum
FixMapFixArray
FixRaw
0x00
0xc0
0x800x900xa0
0xe0
0x000xc20xc30xca0xcb0xcc0xcd0xce0xcf0xdf...
nil false true float double uint8 uint16 uint32 uint64 int8 ...
Type information Types
![Page 9: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/9.jpg)
Zero-copy serialization
![Page 10: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/10.jpg)
Zero-copy deserialization
![Page 11: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/11.jpg)
Performance
It measured the elapsed time of serializing and deserializing 200,000 target objects. The target object consists of the three integers and 512 bytes string.
![Page 12: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/12.jpg)
Remote Procedure Call
MessagePack-RPC
Inter-process messaging library forclients, servers and cluster applications.
![Page 13: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/13.jpg)
Inter-process messaging library forclients, servers and cluster applications.
Remote Procedure Call
MessagePack-RPC
Concept of Future
Multithreadedevent-driven I/O
Communicates with multiple servers concurrently
![Page 14: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/14.jpg)
Synchronous call
require 'msgpack/rpc'
client = MessagePack::RPC::Client.new(host, port)
result = client.call(:method, arg1, arg2)
![Page 15: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/15.jpg)
Asynchronous callrequire 'msgpack/rpc'
client = MessagePack::RPC::Client.new(host, port)
future1 = client.call_async(:methodA, arg1, arg2)future2 = client.call_async(:methodB, arg1, arg2)
result1 = future1.getresult2 = future2.get
![Page 16: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/16.jpg)
Callbackrequire 'msgpack/rpc'
client = MessagePack::RPC::Client.new(host, port)
client.callback(:method, arg, arg2) do |future| result = future.getend
client.join
![Page 17: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/17.jpg)
Concurrent calls with multiple servers
require 'msgpack/rpc'
loop = MessagePack::RPC::Loop.new
client1 = MessagePack::RPC::Client.new(host1, port1, loop)client2 = MessagePack::RPC::Client.new(host2, port2, loop)
future1 = client1.call_async(:methodA, arg1, arg2)future2 = client2.call_async(:methodB, arg1, arg2)
result1 = future1.getresult2 = future2.get
![Page 18: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/18.jpg)
Connection Poolingrequire 'msgpack/rpc'
sp = MessagePack::RPC::SessionPool.new
session1 = sp.get_session(host1, port1)session2 = sp.get_session(host2, port2)
future1 = session1.call_async(:methodA, arg1, arg2)future2 = session2.call_async(:methodB, arg1, arg2)
result1 = future1.getresult2 = future2.get
![Page 19: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/19.jpg)
Concurrent calls with multiple servers
Client
Session Loop Server
Client
ServerSession Loop
![Page 20: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/20.jpg)
sharedevent loop
Client
Client
Concurrent calls with multiple servers
Server
Server
Loop
Session
Session
![Page 21: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/21.jpg)
Connection Pooling
Session PoolServer
Server
pools these connectionsLoop
Session
Session
![Page 22: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/22.jpg)
Server architecture
Dispatcher
ServerClient
Client
Loop
![Page 23: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/23.jpg)
Event-driven I/O
• Performance of the Loop is important.
• Java version uses Netty (JBoss’ I/O framework)
• Multithreaded
• Utilizes Java’s NIO
• C++ version uses mpio (Kumofs’ I/O architecture)
• Multithreaded
• Utilizes epoll or kqueue
• Ruby version uses Rev (libev for Ruby)
• Utilizes epoll or kqueue
![Page 24: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/24.jpg)
Multithreaded event-driven I/O
![Page 25: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/25.jpg)
![Page 26: Architecture of MessagePack](https://reader033.fdocuments.in/reader033/viewer/2022061303/54fc73c04a7959903c8b47a2/html5/thumbnails/26.jpg)
The MessagePack Project
http://msgpack.sourceforge.net/