© Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE...
Transcript of © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE...
![Page 1: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/1.jpg)
© Tubi, proprietary and confidential
![Page 2: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/2.jpg)
© Tubi, proprietary and confidential© Tubi, proprietary and confidential
LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP
Haofei WangDirector of Engineering @ Tubi
![Page 3: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/3.jpg)
© Tubi, proprietary and confidential 3
What is Tubi?
● VOD streaming service● 100% free● Over 25 different devices● In-stream advertising● Massive content library, over 20k movies and tv shows
![Page 4: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/4.jpg)
© Tubi, proprietary and confidential 4
Video Processing
● High quality sources. Minimally compressed source videos, ranging from 50GB to 600GB.● Massive amount of content ingestions.● Need to support over 25 devices.
![Page 5: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/5.jpg)
© Tubi, proprietary and confidential 5
Parallel Processing
● Process each small chunk in parallel.
● One video can be broken down to thousands of tasks to process in parallel
● CPU heavy tasks● IO heavy task● Network heavy tasks● Tasks have
dependencies.
![Page 6: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/6.jpg)
© Tubi, proprietary and confidential 6
Parallel Processing
![Page 7: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/7.jpg)
© Tubi, proprietary and confidential 7
Why Elixir/Erlang OTP?Supervisor: monitor the health of the child process, restart as needed
@spec start_link(node(), map()) :: GenServer.on_start()
def start_link(node_name, args) do
{:ok, pid} = :rpc.call(node_name, GenServer, :start, [__MODULE__, args])
Process.link(pid)
{:ok, pid}end
![Page 8: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/8.jpg)
© Tubi, proprietary and confidential 8
Why Elixir/Erlang OTP?Supervisor: monitor the health of the child process, restart as needed
def handle_info({:EXIT, pid, :normal}, state) do
if Models.Task.unfinished_count(state.job_id) == 0 do
{:stop, :normal, state}
else
{:noreply, %{state | tasks: Map.drop(state.tasks, [pid])}}
end
end
![Page 9: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/9.jpg)
© Tubi, proprietary and confidential 9
Why Elixir/Erlang OTP?Supervisor: monitor the health of the child process, restart as needed
def handle_info({:EXIT, pid, _error}, state) do
{task_id, rest_tasks} = Map.pop(state.tasks, pid)
with true <- not is_nil(task_id),
task = Models.Task.get_by_id(task_id),
true <- task.rest_restarts > 0 do
{:noreply, %{state | tasks: rest_tasks}}
else
false ->
{:stop, {:shutdown, :job_terminated}, %{state | tasks: rest_tasks}}
endend
![Page 10: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/10.jpg)
© Tubi, proprietary and confidential1
0
Why Elixir/Erlang OTP?Cluster is built-in: adding new node is simple, remote process is seemingless
@impl GenServer
def handle_info(:register, %{manager_node_name: mgr_node, quota: quota} = state) do
mgr_node
|> :rpc.call(Manager.Register, :register, [self(), Node.self(), quota])
|> case do
{:ok, pid} ->
_ = Logger.info("Connected to Manager node #{mgr_node}")
{:noreply, Map.put(state, :manager_pid, pid)}
_ ->
Process.send_after(self(), :register, 2000)
{:noreply, state}
endend
![Page 11: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/11.jpg)
© Tubi, proprietary and confidential1
1
Why Elixir/Erlang OTP?Cluster is built-in: adding new node is simple, remote process is seamless
@spec start_link(node(), map()) :: GenServer.on_start()
def start_link(node_name, args) do
{:ok, pid} = :rpc.call(node_name, GenServer, :start, [__MODULE__, args])
Process.link(pid)
{:ok, pid}end
![Page 12: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/12.jpg)
© Tubi, proprietary and confidential1
2
When a batch of content need to be processed
Processed 900+ videos in 28 hours. 700+ CPU cores running at full speed. It is 1.87 minutes for each video processing
![Page 13: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/13.jpg)
© Tubi, proprietary and confidential
Q&A
![Page 14: © Tubi, proprietary and confidential · © Tubi, proprietary and confidential LARGE SCALE DISTRIBUTED VIDEO PROCESSING WITH OTP Haofei Wang Director of Engineering @ Tubi](https://reader035.fdocuments.in/reader035/viewer/2022071022/5fd6191a5edb6d7521050a17/html5/thumbnails/14.jpg)
© Tubi, proprietary and confidential
Thank You.