CS-3013 & CS-502, Summer 2006 Multimedia topics1 Multimedia CS-3013 & CS-502 Operating Systems.

CS-3013 & CS-502, Summer 2006

Multimedia topics 1

Multimedia

CS-3013 & CS-502Operating Systems

CS-3013 & CS-502, Summer 2006

Multimedia topics 2

Outline

• CD devices and formats• Requirements and challenges for audio

and video in computer systems• Systems for multimedia• Compression and bandwidth• Processor scheduling• File, disk, and network management

Tanenbaum, Chapter 7Silbershatz, Chapter 20

CS-3013 & CS-502, Summer 2006

Multimedia topics 3

CD-ROMs

• See Tanenbaum, pp. 306-310• Audio CD

– Molded polycarbonate– 120 mm diameter with 15 mm hole– One single spiral track

• Starts in center, spirals outward• 22,188 revolutions, approx 5.6 kilometers long

– Constant linear velocity under read head• Audio playback:– 120 cm/sec• Variable speed motor:– 200 – 530 rpm

• ISO standard IS 10149, aka the Red Book

CS-3013 & CS-502, Summer 2006

Multimedia topics 4

CD-ROM (continued)

• Problem for adapting to data usage– No bad block recovery capability!

• ISO standard for data: Yellow Book– Three levels of error-correcting schemes: – Symbol,

Frame, Sector• ~7200 bytes to record 2048 byte payload per sector

– Mode 2: less error correction in exchange for more data rate

• Audio and video data– Sectors linearly numbered from center to edge

• Read speed– 1x ~ 153,000 bytes/sec– 40x ~ 5.9 megabytes/sec

• ISO standard for multi-media: Green Book– Interleaved audio, video, data in same sector

CS-3013 & CS-502, Summer 2006

Multimedia topics 5

CD-ROM File SystemISO 9660 — High Sierra

• Write once contiguous file allocation• Variable length directories• Variable length directory entries

• Points to first sector of file• File size and metadata stored in directory entry• Variable length names

• Several extensions to standard for additional features

• See Tanenbaum, pp. 310-313

CS-3013 & CS-502, Summer 2006

Multimedia topics 6

DVD devices

• Similar in concept, different in details• More data, higher bandwidth• Details not provided here

– Tanenbaum, pp. 313-315

CS-3013 & CS-502, Summer 2006

Multimedia topics 7

What do we mean by “multimedia”

• Audio and video within a computer system– CD’s & DVD’s– Computer hard drive

• Live broadcast & web casts– Webcams, Skype, …

• Video on demand– Pause, fast forward, reverse, etc.

• Interactive meetings– Presentations with 2-way audio– Teleconferencing

• Interactive gaming• …

CS-3013 & CS-502, Summer 2006

Multimedia topics 8

Requirements

• “Smooth” audio and video– Deterioration in quality >> jerky playback– Note: human is more sensitive to jitter in

audio than to jitter in video!• Audio/video on PC’s doing something

else• Multiple concurrent streams

– Video & multimedia servers– TiVo, etc.

• Wide range of network bandwidths

CS-3013 & CS-502, Summer 2006

Multimedia topics 9

System and OS Challenges

• Bandwidths and Compression

• Jitter

• Processor Scheduling

• Disk Scheduling

• Network Streaming

CS-3013 & CS-502, Summer 2006

Multimedia topics 10

Some System Architectures

• Simple: – Data paths for audio/video that are

separate from computational data paths

• Modern– Fast system bus, CPU, devices

• Video server– Disk farm and multiple streams

CS-3013 & CS-502, Summer 2006


audio stream

CD-ROM drive

System Organization (simple)

• Separate data path for audio stream• Main system bus and CPU were too

busy/slow to handle real-time audio

CPU

Memory

memory bus

Device Sound card

CS-3013 & CS-502, Summer 2006


video stream viaISA & bridge to

graphics card

audio stream viaISA bridge to

sound card

System Organization (typ. Pentium)

ISA bridge

IDE disk

Main Memory

CPULevel

2 cache

Bridge Moni-tor

Graphics card

USB

Key-board

Mouse

Ether-net

SCSI

ModemSoundcard

Printer

PCI bus

ISA bus

AGP Port

CS-3013 & CS-502, Summer 2006


Video Server

• Multiple CPUs• Disk farm

– 1000s of disks

• Multiple high-bandwidth network links– Cable TV– Video on demand– Internet

CS-3013 & CS-502, Summer 2006


CS-3013 & CS-502, Summer 2006


Challenges

• Compression– to keep bandwidth manageable

• “Quality of Service”– to deliver sounds and images on time

for acceptable quality reproduction

CS-3013 & CS-502, Summer 2006


Why Compression? – CD-quality audio

• 22,050 Hz 44,100 samples/sec• 16 bits per sample

• Two channels 176,400 bytes/sec 1.4 mbits/sec

– Okay for a modern PC– Not okay for 56 kb/sec modem (speed) or iPod

(space)!

• MP-3 0.14 mbits/sec (10:1)– Same audio quality!– Compression ratio varies with type of music

CS-3013 & CS-502, Summer 2006


Why Compression? – Video

• “Standard” TV frame = 640 480 pixels @ 25-30 frames/sec (fps) 9,216,000 pixels/sec = 27,648,000 bytes/sec

• Typical movie 133 minutes approx. 210 gigabytes (standard resolution)!

• HDTV = 1280 720 pixels @ 30 fps 82,944,000 bytes/sec (and rising!)

• DVD holds ~ 4.7 gigabytes average of ~ 620 kilobytes/sec!

• “Standard” movie of 133 minutes requires serious compression just to fit onto DVD

CS-3013 & CS-502, Summer 2006


Video Compression Requirements

• Compression ratio > 50:1• i.e., 210 gigabytes:4.7 gigabytes

• Visually indistinguishable from original

• Even when paused

• Fast, cheap decoder• Slow encoder is okay

• VCR controls• Pause, fast forward, reverse

CS-3013 & CS-502, Summer 2006


Video Compression Standards

• MPEG (Motion Picture Experts Group)– Based on JPEG (Joint Photographic Experts Group)– Multi-layer

• Layer 1 = system and timing information• Layer 2 = video stream• Layer 3 = audio and text streams

• Three standards– MPEG-1 – 352240 frames; < 1.5 mb/sec ( < VHS

quality)• Layer 3 = MP3 Audio standard

– MPEG-2 – standard TV & HDTV; 1.5-15 mb/sec• DVD encoding

– MPEG-4 – combined audio, video, graphics• 2D & 3D animations

CS-3013 & CS-502, Summer 2006


JPEG compression (single frame)

1. Convert RGB into YIQ• Y = luminance (i.e., brightness) ~ black-white TV• I, Q = chrominance (similar to saturation and hue)

Reason: Human eye is more sensitive to luminance than to color (rods vs. cones)

2. Down-sample I, Q channels• i.e., average over 22 pixels to reduce resolution• lossy compression, but barely noticeable to eye

3. Partition each channel into 88 blocks• 4800 Y blocks, 1200 each I & Q blocks

CS-3013 & CS-502, Summer 2006


JPEG (continued)

CS-3013 & CS-502, Summer 2006


JPEG (continued)

4. Calculate Discrete Cosine Transform (DCT) of each 88 block

• What is a Discrete Cosine Transform?

5. Divide 88 block of DCT values by 88 quantization table

• Effectively throwing away higher frequencies

6. Linearize 88 block, run-length encode, and apply a Huffman code to reduce to a small fraction of original size (in bytes)

CS-3013 & CS-502, Summer 2006


JPEG (concluded)

7. Store or transmit 88 quantization table followed by list of compressed blocks

• Achieves 20:1 compression with good visual characteristics

• Higher compression ratios possible with visible degradation

• JPEG algorithm executed backwards to recover image

• Visually indistinguishable from original @ 20:1

• JPEG algorithm is symmetric• Same speed forwards and backwards

CS-3013 & CS-502, Summer 2006


MPEG

• JPEG-like encoding of each frame• Takes advantage of temporal locality• I.e., each frame usually shares

similarities with previous frame encode and transmit only differences

• Sometimes an object moves relative to background find object in previous frame, calculate

difference, apply motion vector

CS-3013 & CS-502, Summer 2006


Temporal Locality (example)

Consecutive Video Frames

CS-3013 & CS-502, Summer 2006


MPEG organization

• Three types of frames– I-frame: Intracoded or Independent.

• Full JPEG-encoded frame• Occurs at intervals of a second or so• Also at start of every scene

– P-frame: Predictive frame• Difference from previous frame

– B-frame: Bidirectional frame• Like p-frame but difference from both previous and

next frame

I B BB P B BB P B BB P B BB I B BB P B BB P

CS-3013 & CS-502, Summer 2006


MPEG Characteristics

• Non-uniform data rate!• Compression ratios of 50:1 – 80:1 are

readily obtainable• Asymmetric algorithm

– Fast decode (like JPEG)– Encode requires image search and analysis to

get high quality differences

• Cheap decoding chips available for – graphics cards– DVD players, etc.

CS-3013 & CS-502, Summer 2006


MPEG Problem – Fast Forward/Reverse

• Cannot simply skip frames– Next desired frame might be B or P derived

from a skipped frame

• Options:– Separate fast forward and fast reverse files

• MPEG encoding of every nth frame• See Tanenbaum §7.5.1 – video-on-demand server

– Display only I and P frames• If B frame is needed, derive from nearest I or P

CS-3013 & CS-502, Summer 2006


“Movie” File Organization

• One MPEG-2 video stream• Multiple audio streams

• Multiple languages

• Multiple text streams• Subtitles in multiple languages

• All interleaved• Possibly in multiple files!

CS-3013 & CS-502, Summer 2006


CS-3013 & CS-502, Summer 2006


Operating System Challenge

• How to get the contents of a movie file from disk or DVD drive to video screen and speakers.– Fixed frame rate (25 or 30 fps)– Steady audio rate– Bounded jitter

• Classical problem in real-time scheduling– Obscure niche become mainstream!– See Silbershatz, Chapter 19

CS-3013 & CS-502, Summer 2006


Processor Scheduling for Real-Time

Rate Monotonic Scheduling (RMS)• Assume m periodic processes

– Process i requires Ci msec of processing time every Pi msec.

– Equal processing every interval — like clockwork!

• Assume

• Let priority of process i be– Statically assigned

• Let priority of non-real-time processes be 0

11

m

i i

i

P

C

iP

1

CS-3013 & CS-502, Summer 2006


Rate Monotonic Scheduling (continued)

• Scheduler simply runs highest priority process that is ready– May pre-empt other real-time processes– Real-time processes become ready in

time for each frame or sound interval

CS-3013 & CS-502, Summer 2006


Rate Monotonic Scheduling (continued)

• Theorem: using these priorities, scheduler can guarantee the needed Quality of Service (QoS), provided that

)12(1

1

mm

i i

i mP

C

• Asymtotically approaches ln 2 as m • I.e., must maintain some slack in scheduling• Assumes fixed amount of processing per

periodic task– Not MPEG!

CS-3013 & CS-502, Summer 2006


Processor Scheduling for Real-Time

Earliest Deadline First (EDF)

• When each process i become ready, it announces deadline Di for its next task.

• Scheduler always assigns processor to process with earliest deadline.

• May pre-empt other real-time processes

CS-3013 & CS-502, Summer 2006


Earliest Deadline First Scheduling (continued)

• No assumption of periodicity• No assumption of uniform processing times

• Theorem: If any scheduling policy can satisfy QoS requirement for a sequence of real time tasks, then EDF can also satisfy it.– Proof: If i scheduled before i+1, but Di+1 < Di ,

then i and i+1 can be interchanged without affecting QoS guarantee to either one.

CS-3013 & CS-502, Summer 2006


Earliest Deadline First Scheduling (continued)

• EDF is more complex scheduling algorithm

• Priorities are dynamically calculated• Processes must know deadlines for tasks

• EDF can make higher use of processor than RMS

• Up to 100%

• However, it is usually a good idea to build in some slack

CS-3013 & CS-502, Summer 2006


Questions?

CS-3013 & CS-502, Summer 2006


Multimedia File & Disk Management

• Single movie or multimedia file on PC disk– Interleave audio, video, etc.

• So temporally equivalent blocks are near each other

– Attempt contiguous allocation• Avoid seeks within a frame

TextFrame

AudioFrame

CS-3013 & CS-502, Summer 2006


File organization – Frame vs. Block

• Frame organization• Small disk blocks (4-16 Kbytes)• Frame index entries point to starting block for each

frame• Frames vary in size (MPEG)• Advantage: very little storage fragmentation• Disadvantage: large frame table in RAM

• Block organization• Large disk block (256 Kbytes)• Block index entries point to first I-frame of a sequence• Multiple frames per block• Advantage: much smaller block table in RAM• Disadvantage: large storage fragmentation on disk

CS-3013 & CS-502, Summer 2006


Frame vs. Block organization

smaller larger

CS-3013 & CS-502, Summer 2006


File Placement on Server

• Random• Striped• “Organ pipe” allocation

– Most popular video in center of disk– Next most popular on either side of it– Etc.– Least popular at edges of disk– Minimizes seek distance

CS-3013 & CS-502, Summer 2006


Placing Multiple files on Single Disk

Zipf’s Law• Rank items by “popularity”• Zif’s Law says: Probability that next

customer will choose item k is approximately C/k– Where C is a normalization constant such that

11

N

kk

C

N = 10 C = 0.341N = 100 C = 0.193N = 1000 C = 0.134N = 10000 C = 0.102

CS-3013 & CS-502, Summer 2006


Zipf’s Law applied to US cities

• Points represent populations 20 largest cities– sorted on rank order

CS-3013 & CS-502, Summer 2006


Application of Zipf’s Law tomovie file placement

• Organ-pipe distribution of files on server– most popular movie in middle of disk– next most popular either on either side, etc.

CS-3013 & CS-502, Summer 2006


Disk Scheduling (server)

• Scheduling disk activity is just as important as scheduling processor activity

• Advantage:– Predictability• Unlike disk activity of ordinary computing

• In server, there will be multiple disk requests in each frame interval

• One request per frame for each concurrent video stream

CS-3013 & CS-502, Summer 2006


Disk Scheduling (continued)

• SCAN (Elevator) algorithm for each frame interval

• Sort by cylinder #• Complete in time for start of next frame interval

• Variation – SCAN–EDF:• Sort requests by deadline• Group similar deadlines together, apply SCAN to

group• Particularly useful for non-uniform block sizes

and frame intervals

CS-3013 & CS-502, Summer 2006


Network Streaming

• Traditional HTTP• Stateless• Server responds to each request

independently

• Real-Time Streaming Protocol (RTSP)• Client initiates a “push” request for stream• Server provides media stream at frame rate

CS-3013 & CS-502, Summer 2006


Push vs. Pull server

CS-3013 & CS-502, Summer 2006


Bandwidth Negotiation

• Client (or application) provides feedback to server to adjust bandwidth

• E.g., – Windows Media Player– RealPlayer– Quicktime

CS-3013 & CS-502, Summer 2006


Conclusion

• Multimedia computing is challenging• Possible with modern computers

• Compression is essential, especially for video

• Real-time computing techniques move into mainstream

• Processor and disk scheduling

• There is much more to this subject than fits into one class

CS-3013 & CS-502, Summer 2006


Reading Assignments

• Tanenbaum, pp. 306-315– CD and DVD devices

• Tanenbaum, Chapter 7– Multimedia systems

CS-3013 & CS-502, Summer 2006


CS-3013 & CS-502, Summer 2006


Digression on TransformsFourier’s Theorem

• Every continuous, periodic function can be reduced to the sum of a series of sine waves

• The Fourier transform is a representation of that function in terms of the frequencies of those sine waves

• Original function can be recovered from its Fourier transform

• Fourier transforms occur frequently in nature!

CS-3013 & CS-502, Summer 2006


Nyquist’s Theorem (1924)

• If a continuous function is sampled at a frequency 2f, then the function can be recovered from those samples provided that its maximum Fourier frequency is f.

CS-3013 & CS-502, Summer 2006


Discrete Cosine Transform

• A form of the Fourier Transform• When applied to an 88 block of

samples (i.e. pixel values) yields an 88 block of spatial frequencies

• Original 88 block of samples can be recovered from its DCT.– Assuming infinite arithmetic precision

CS-3013 & CS-502, Summer 2006


More on Nyquist• If arithmetic precision is not infinite, we

get quantization error during sampling• Recovered signal has quantization noise

• i.e., a lossy transform

CS-3013 & CS-502, Summer 2006 Multimedia topics1 Multimedia CS-3013 & CS-502 Operating Systems.

Documents

Transcript of CS-3013 & CS-502, Summer 2006 Multimedia topics1 Multimedia CS-3013 & CS-502 Operating Systems.