Fluentd Meetup 2016 - ServerEngine Integration & Windows support
-
Upload
ritta-narita -
Category
Software
-
view
3.104 -
download
1
Transcript of Fluentd Meetup 2016 - ServerEngine Integration & Windows support
ServerEngine Integration & Windows support
Ritta Narita
Fluentd Meetup 2016
About me.
Ritta Narita
joined Treasure Data after internship in 2015
2
@naritta
@narittan
Fluentd v0.14
ServerEngine is integrated
What’s ServerEngine?
A framework to implement robust multiprocess servers like Unicorn
worker
worker
workersupervisor Server
Dynamic reconfiglive restart
Heartbeat via pipeauto restart
multi process/thread
how to use ServerEngine
workersupervisor Server
module MyServer def before_run @sock = TCPServer.new(
config[:bind], config[:port]) end
attr_reader :sockend
module MyWorker def run c = server.sock.accept c.write "Awesome work!" end
def stop c.close endend
1. write server module and worker module
how to use ServerEngine
workersupervisor Server
se = ServerEngine.create(MyServer, MyWorker, { daemonize: true, log: 'myserver.log', pid_path: 'myserver.pid', worker_type: 'process', workers: 4, bind: '0.0.0.0', port: 9071,})
se.run
2. write configuration and run
module MyWorker def spawn(process_manager) process_manager.spawn(env, config[:spawn_cmd]) endend
ServerEngine.create(MyServer, MyWorker, { worker_type: ‘spawn’, spawn_cmd: cmd})
ServerEngine.create(MyServer, MyWorker, { worker_type: ‘thread'})
workerServer
workerServer
workerServerspawn
thread
processServerEngine.create(MyServer, MyWorker, { worker_type: ‘process'})
worker type
What’s the advantages for integrating serverengine into fluentd?
serverengine advantages
Auto restartlive restart
socket managersignal handlerlog rotation
Auto restart live restart
socket managersignal handlerlog rotation
serverengine advantages
if worker died unexpectedly…
worker
workersupervisor Server
worker
if worker died unexpectedly…
worker
worker
workersupervisor Server
if worker died unexpectedly…
worker
workersupervisor Server
It’s detected by heartbeat and server launch worker again.
worker
Auto restartlive restart
socket managersignal handlerlog rotation
serverengine advantages
workersupervisor Server
tcp tcp
send SIGHUP
live restart
supervisor Server worker
tcp
kill the worker, leaving server’s condition
live restart
worker
supervisor Server worker
tcp
start new worker and read config again
live restart
worker
supervisor Server worker
tcp
tcpshare the TCP with living zero downtime
live restart
Signal and RPC server
api/config.reloadlive restart/config reload HUP
/api/processes.interruptWorkers
INT
/api/processes.killWorkers
/api/plugins.flushBuffers
system config reload USR2
TERM
USR1flush buffers
shutdown
workersupervisor Server
you can reload system config without restart
system config reload
send USR2
check mtime and previous load time,if there is new one, reload!
serverengineconfig cash
new config
Auto restartlive restart
socket manager signal handlerlog rotation
serverengine advantages
Socket Manager
workerSocket
Managerclient
SocketManagerserver
Server
26
①socket listen request②find or create listening socket
③share the socket FD(unix)/
duplicated socket info(windows)
UDS(unix)/TCP(windows)
④get socket and accept in worker
how to use Socket Manager
workerSocket
Managerclient
SocketManagerserver
Server
27
#generate path for UDS/TCP for server connection socket_manager_path = ServerEngine::SocketManager::Server.generate_path
#open socket manager server socket_manager_server = ServerEngine::SocketManager::Server.open(socket_manager_path)
In server
workerSocket
Managerclient
SocketManagerserver
Server
28
#get socket manager client from socket manager path socket_manager = ServerEngine::SocketManager::Client.new(socket_manager_path)
#get listening socket from socket manager lsock = socket_manager.listen_tcp(bind, port)
In worker
how to use Socket Manager
What’s the merit of Socket Manager?
worker
worker
worker
SocketManager
client
SocketManager
client
SocketManager
client
SocketManagerserver
supervisor
Server
29we can use multicore easily in future
easy to use multicore power fully 30
with SocketManagerwith in_multiprocess plugin<source> type multiprocess <process> cmdline -c /etc/td-agent/td-agent-child1.conf </process> <process> cmdline -c /etc/td-agent/td-agent-child2.conf </process></source>
#/etc/td-agent/td-agent-child1.conf<source> type forward port 24224</source>
#/etc/td-agent/td-agent-child2.conf<source> type forward port 24225</source>
<source> type forward port 24224</source>
setting when using 2 core
Auto restartlive restart
socket managersignal handler
log rotation
serverengine advantages
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
userUSR1
send USR1
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
userUSR1
USR1
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
userQUIT
USR1
send QUIT
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
QUIT
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
QUIT
run!
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
QUIT
Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
QUIT run!
Auto restartlive restart
socket managersignal handlerlog rotation
serverengine advantages
support new features(These are already ported to ruby core.)
・multiprocess aware log rotation
・reopening of log file
・'trace' level, which is lower level than ‘debug'
ServerEngine.create(MyServer, MyWorker, { log: 'myserver.log', log_level: 'debug', log_rotate_age: 5, log_rotate_size: 1*1024*1024,})
summary
windows・signal → rpc handler
・no socket FD and UDS
・fork based → spawn based
・live & auto restart
・multicore with socket manager
・sophisticated signal handler
・easier log rotation
Thank you!
appendix
unix→fork based damonwindows→spawn based damon
damonize
fluentdserverengine
①set windows_daemon_cmdline
in configuration
②run windows_daemon_cmdline
when damonize
③windows_daemon_cmdline
runs daemon.rb
④run_server with
servermodule and workermodule