Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)

download Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)

of 56

  • date post

    19-Dec-2015
  • Category

    Documents

  • view

    213
  • download

    0

Embed Size (px)

Transcript of Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)

  • Slide 1
  • Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)
  • Slide 2
  • Motivations zWe know how to encode, transmit and store multimedia data today zPeople start looking into ways to process multimedia data
  • Slide 3
  • Video Editing zConcat, cross-fade, overlay, cut and paste
  • Slide 4
  • Gateway zConvert from one format to another zChange compression rate transcoder (format A) format B input video output video
  • Slide 5
  • DBMS Find all movies in the database that contain this image
  • Slide 6
  • Set-top Boxes zSubtitling and close captioning zMixing of incoming movies
  • Slide 7
  • Lecture Browser zMatch video to slide zSwitch between video streams
  • Slide 8
  • Current Solutions zBlack box C code (mpeg_play) yHard to reuse/adapt/break apart yUnpredictable performance zStandard library (PPM, IJG) yLeast common denominator (RGB for all) xVideo frames --> RGB --> gray
  • Slide 9
  • History zRivl - high level scripting language for multimedia processing zStill suffers from black box problem
  • Slide 10
  • Experience zReal cost of processing yInherently complex operations xProjective transform yLayered operations xVideo decoding ySimple operations but lots of data xCopy
  • Slide 11
  • Dali Architecture Dali Code Dali Compiler DVM Code Dali Virtual Machine User Expert User MMXTriMedia C
  • Slide 12
  • Todays Talk DVM Code Dali Virtual Machine
  • Slide 13
  • DVM Code : Design Goal zSmall number of composable abstractions zSimple, predictable opcodes zHigh performance zTarget for compiler zEasy to extend
  • Slide 14
  • Abstractions zByte image zBitstream zMPEG zJPEG zBit masks zPCM
  • Slide 15
  • Abstractions zByte image zBitstream zMPEG zJPEG zBit masks zPCM
  • Slide 16
  • Byte Image zTwo dimensional array of byte zCan be either physical or virtual physical image virtual image
  • Slide 17
  • Byte Image Can Represent zGray scale image zRGB image zYUV image zAlpha channel
  • Slide 18
  • Example Code : Fractal
  • Slide 19
  • smaller = byte_new (neww, newh); byte_shrink_2x2(image,smaller); image smaller neww newh
  • Slide 20
  • image target = byte_clip(image,0,0,dx,2*dy); byte_set(target,0); target = byte_clip(image,3*dx,0,dx,2*dy); byte_set(target,0); target dx 2dy
  • Slide 21
  • target = byte_clip(image, dx, 0, neww, newh); byte_copy(smaller, target); target image smaller newh neww dx
  • Slide 22
  • target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target); image smaller neww newh dx 2dy
  • Slide 23
  • DVM Code Dx = 0.25 * byte_width(image); dy = 0.25 * byte_height(image); neww = 0.5 * byte_width(image); newh = 0.5 * byte_height(image); smaller = byte_new (neww,newh); for (i = 0; i < n; i++) { byte_shrink_2x2(image,smaller); Target = byte_clip(image,0,0,dx,2*dy); byte_set(target,0); target = byte_clip(image,3*dx,0,dx,2*dy); byte_set(target,0); target = byte_clip(image, dx, 0, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target); }
  • Slide 24
  • General Dali Strategies zSpecific instruction ybyte_shrink_2x2, byte_shrink_4x4, etc zExplicit memory allocation ybyte_new zReduce data ybyte_clip zComposable abstraction ybyte image
  • Slide 25
  • Performance run fractal with n = 4 on 800x600 gray scale image about 21 frames/sec for 320x240 video
  • Slide 26
  • Abstractions zByte image zBitstream zMPEG zJPEG zBit masks zPCM
  • Slide 27
  • MPEG zGetting important and pervasive zComplex format zComplex code (mpeg_play) zMost decoders provides GetNextFrame() interface zRandom access, direct transfer difficult
  • Slide 28
  • Abstraction for MPEG Video... seq hdr gop hdr gop hdr gop seq end
  • Slide 29
  • Abstraction for MPEG Video... seq hdr gop hdr gop hdr gop seq end... pic hdr pic hdr pic
  • Slide 30
  • DVM Code Examples Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
  • Slide 31
  • DVM Code Examples Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
  • Slide 32
  • DVM Code Examples Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
  • Slide 33
  • DVM Code Examples Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
  • Slide 34
  • inbs1 inbs2 outbs hdr gop Concat 2 Video Sequences
  • Slide 35
  • DVM Code zConcat two sequences While not end_of(inbs1) { gop_hdr_dump(inbs1, outbs); gop_dump(inbs1, outbs); } While not end_of(inbs2) { gop_hdr_dump(inbs2, outbs); gop_dump(inbs2, outbs); }
  • Slide 36
  • Performance Analysis zDecode MPEG sequences to PPM zNo output zRun on 3 different sequences
  • Slide 37
  • Performance
  • Slide 38
  • Slide 39
  • MPEG Strategies zExpose structure yGop header, gop instead of just frames zBreak up layer yParse, decode, color space conversions
  • Slide 40
  • Show Time ! zDecode a MPEG sequence zFor each frame scale it to stamp size zCreate a contact sheet
  • Slide 41
  • Abstractions zByte image zBitstream zMPEG zJPEG zBit masks zPCM
  • Slide 42
  • What Is Bitstream ? Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
  • Slide 43
  • Bitstream parser 1011001 chunk of memory
  • Slide 44
  • Mode of Operations zFile i/o static memory fread/fwrite parser
  • Slide 45
  • Mode of Operations zNetwork static memory socket parser
  • Slide 46
  • Mode of Operations zMemory map parser entire file
  • Slide 47
  • inbsvideo audio mpeg system stream
  • Slide 48
  • filter videobs inbsvideo audio mpeg system stream
  • Slide 49
  • filter videobs inbsvideo audio parser mpeg system stream
  • Slide 50
  • Bitstream Strategies zPredictable performance yExplicit I/O
  • Slide 51
  • Recap: Current Solutions zBlack box C code (mpeg_play) yHard to reuse/adapt/break apart yUnpredictable performance
  • Slide 52
  • Recap: Cost of Processing zInherently complex operations zLayered operations zSimple operations but lots of data
  • Slide 53
  • How We Solve the Problems zDifficult to reuse/adapt/break apart yMinimum set of abstractions yAbstractions are simple ySimple reusable opcodes zUnpredictable performance yExplicit I/O yExplicit memory allocations
  • Slide 54
  • How We Solve the Problems zInherently complex operations ySpecial case yClipping and masking zLayered operations yExpose structure zSimple operations but lots of data ySpecial case yClipping and masking
  • Slide 55
  • zExtending with new formats : JPEG, WAV, AVI etc zTriMedia chip implementation zApplication : lecture browser Work in Progress
  • Slide 56
  • Future Work zMultithreading zMMX implementation zCompiler