Grid technology for next gen media processing
-
Upload
vrt-medialab -
Category
Technology
-
view
2.317 -
download
1
description
Transcript of Grid technology for next gen media processing
![Page 1: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/1.jpg)
Grid technologyfor next-gen media processing
Jens Buysse - Stijn De Smet - Koen Segers-Bruno Volckaert
![Page 2: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/2.jpg)
2
Overview
MediaGrid concept Distributed video transcoding Enabling technologies Setup overview Test results Simulation results Conclusions
![Page 3: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/3.jpg)
MEDIAGRID CONCEPT
3
![Page 4: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/4.jpg)
Originating problems
Tape-based media to file-based media
Multitude of file-based media transfers and processing Storage / retrieval / transfer of media Conforming Transcoding Upscaling Editing
Geographically disperse facilities / resources / media storage
4
![Page 5: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/5.jpg)
Grid technology as solution?
Grid technology
5
a Grid is a distributed processing architecture where heterogeneousresources are shared between different participating organizations, across
an interconnecting network
Resources Storage (media archive, temporary storage, etc.) Computational (rendering farm, work stations, etc.) Specialized (broadcasting, ingesting, etc.)
High speed interconnecting network (1-10 Gbit/s)
![Page 6: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/6.jpg)
MediaGrid
6
MMMhires
MhiresM
hires
![Page 7: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/7.jpg)
MediaGrid
7
MMMhires
MhiresM
hires
EDL
Grid Middleware
![Page 8: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/8.jpg)
MediaGrid: enabling virtual organisations
8
VO 1
VO 2
![Page 9: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/9.jpg)
DISTRIBUTED VIDEOTRANSCODING
9
![Page 10: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/10.jpg)
Grid technology proof-of-concept
Investigated the viability of Grid technology for processing tasksin media production / distribution companies Transcoding of media Upscaling of media
Video transcoding is a resource intense process I/O Processing needs
10
Video transcoding deals with converting a video signal intoanother one with different format, such as different bit rate,
frame rate, frame size, or even compression standard
![Page 11: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/11.jpg)
Need for transcoded / rescaled video
VRT online mediahttp://www.deredactie.be
YouTubehttp://www.youtube.com
11
![Page 12: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/12.jpg)
Distributed video transcoding
How can we accelerate this process?
12
00:00:00 00:51:53
00:13:15 00:26:30 00:26:30 00:39:45 00:39:45 00:51:5300:00:00 00:13:15
Server 1 Server 2 Server 3 Server 4
![Page 13: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/13.jpg)
ENABLING TECHNOLOGIES
13
![Page 14: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/14.jpg)
Enabling technologies
OS SuSe enterprise
Transcoding software Transcode library
Grid Middleware TORQUE (openPBS) Maui scheduler Grid distributed transcoder: custom Java application
Data retrieval / storage technology GPFS
14
![Page 15: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/15.jpg)
Enabling technologies: TORQUE
TORQUE : open PBS
15
Queue 1
MauiSheduler
Policy
pbs_mom
pbs_mom
TORQUE Server
User
Queue 2
Job
![Page 16: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/16.jpg)
Enabling technologies
Job / batch / workflow submitter Consider job dependencies
16
1
2
Stock 1 3 4
5 6
Stock 27
8
Stock 3
![Page 17: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/17.jpg)
Enabling technologies
Grid distributed transcoding application
17
![Page 18: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/18.jpg)
Enabling technologies
Grid distributed transcoding application
18
![Page 19: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/19.jpg)
SETUP OVERVIEW
19
![Page 20: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/20.jpg)
Setup overview
… TORQUE
… with GPFS cluster as media storage
… Java distributed transcoding front-end
… on each computational resource Transcode libraries
… the will to transcode in a distributed fashion
20
![Page 21: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/21.jpg)
00:13:15 00:26:3000:26:30 00:39:4500:39:45
00:51:5300:00:00 00:13:15
First distributed transcoding workflow
2121
User
TORQUE1. Split phase2. Transcoding3. Merge phase
Node 2
Node 4
Node 3
GPFS
Node 1
00:00:00 00:51:53
00:13:15 00:26:30 00:00:00 00:13:15
00:39:4500:51:53
00:26:30 00:39:45
00:00:00 00:51:53
![Page 22: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/22.jpg)
00:00:00 00:51:53
Current distributed transcoding workflow
22
User
TORQUE1. Preprocess phase2. Demux phase3. Transcoding4. Merge / multiplex
00:13:15 00:26:30
00:26:30 00:39:45
00:39:45 00:51:53
Nav.log
Audio.mp3
00:00:00 00:13:15
Node 2
Node 4
Node 3
GPFS
Node 1
00:00:00 00:51:53
![Page 23: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/23.jpg)
23
Future distributed transcoding workflow
23
User
TORQUE
1. Prefetch2. Preprocess3. Demux4. Transcode5. Merge /
multiplex
Node 2
GPFS Remote
Node 1
GPFS local
WAN
00:00:00 00:51:53
Node 3
00:00:00 00:51:53
![Page 24: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/24.jpg)
Discussion
Old version Video files were physically split Split / merge step could introduce artifacts
Current version File is inspected and navigation file created allowing for easy frame-
addressing Audio ripped and transcoded in separate step No artifacts Less media-transfers than in previous versions
Future version Pre-fetching / replication of media to remote sites
24
![Page 25: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/25.jpg)
TEST RESULTS
25
![Page 26: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/26.jpg)
Test topology
26GPFS node
Traffic Shaping
Torque Server
![Page 27: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/27.jpg)
Test results
Input media Vob file MPEG-2 video encoding AC3 audio encoding Size: 1,64 GB
Output media Avi file Xvid video encoding MP3 audio encoding Size: 700 MB
Currently no HD video input modules! Not the most optimized video transcoders
Focus on measuring benefits of distributing
27
![Page 28: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/28.jpg)
Results – Grid overhead
Grid Overhead
28
![Page 29: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/29.jpg)
Results – Preprocess phase
Preprocess
29
![Page 30: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/30.jpg)
Results – Audio ripping phase
Rip audio
30
![Page 31: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/31.jpg)
Results – Merging phase
Merging phase
31
![Page 32: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/32.jpg)
Results – 1Gbit/s WAN
32
![Page 33: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/33.jpg)
Results – Parameterised WAN interconnection
33
![Page 34: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/34.jpg)
Video (up)scaling
34
Video scaling is converting video signals from one size or resolution to another: usually"upscaling" or "upconverting" a video signal from a low resolution (e.g. standard definition)
to one of higher resolution (e.g. high definition television).
00:00:00 00:51:53
00:00:00 00:51:53
720X576 984x752
![Page 35: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/35.jpg)
Video (up)scaling results – 52Mbit/s WAN
35
![Page 36: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/36.jpg)
Video (up)scaling results – 52Mbit/s WAN
36
![Page 37: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/37.jpg)
SIMULATION RESULTS
37
![Page 38: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/38.jpg)
Simulation results
We introduced a WAN connection to a remotecomputational resource provider
38
User
Node 4GPFS
Node 1
Node 3
Node 2
TORQUE
• 1 Gbit/s• 100 Mbit/s• 52 Mbit/s• 35 Mbit/s
![Page 39: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/39.jpg)
Simulation results – total job turnaround time
39
![Page 40: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/40.jpg)
Comparison with measured results
40
![Page 41: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/41.jpg)
Comparison with measured results
41
![Page 42: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/42.jpg)
Simulation results
Simulations provide very accurate total job turnaround times Real-life transcoding behaves erroneously when
interconnecting GPFS with computational resource provider bymeans of WAN link lower than 35Mbit/s
Simulation results show what would happen to job turnaroundtime for lower WAN interconnections
42
Click Router
Control Traffic
Data
Control Traffic
Data
GPFS
![Page 43: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/43.jpg)
Simulation results – low-speed WANinterconnection
43
![Page 44: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/44.jpg)
Simulation results – 10 chunks
44
![Page 45: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/45.jpg)
CONCLUSIONS
45
![Page 46: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/46.jpg)
Conclusions
Grid technology is a viable technology for dealing with mediaproduction / distribution tasks Inherent support for parallelism can seriously decrease the total
processing time Need for adaptation of media tasks Grid overhead is no issue
Outsourcing task processing to remote resource providers Viable when interconnection is sufficient Technical limitations (e.g. GPFS time-outs)
MediaGrid simulator can provide accurate performancepredictions
46
![Page 47: Grid technology for next gen media processing](https://reader034.fdocuments.in/reader034/viewer/2022052618/5487e55ab47959d80c8b5579/html5/thumbnails/47.jpg)
Questions ?Feel free to e-mail: [email protected]