3. AV compression - More about video compression & MPEG - Dec06
Transcript of 3. AV compression - More about video compression & MPEG - Dec06
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
1/39
Audio/Video compressionMore about video compression and MPEG
Alain Bouffioux
December, 20, 2006
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
2/39
December, 20, 2006AV Compression / Alain Bouffioux
2
Agenda
About video formatsVideo compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage
context
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
3/39
December, 20, 2006AV Compression / Alain Bouffioux
3
Agenda
About video formatsVideo compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage
context
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
4/39
December, 20, 2006AV Compression / Alain Bouffioux
4
Some video formats (1)
Max. component video signal bandwidth: 6 MHz.
CCIR601 (CCIR is now ITU-R): Video sampling frequency: 13.5 MHz
for 525 & 625 line standards
(Shannon requirement)
Synchronous with line (& image) sampling frequency
Fsampling= 864*Fh for 625 line system (50Hz countries)
Fsampling= 858*Fh for 525 line system (60Hz countries)
Why synchronous? Points at the same place
RGB format
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
5/39
December, 20, 2006AV Compression / Alain Bouffioux
5
Some video formats (2)
YCbCr format
Cb = B-Y, Cr = R-Y
Eye is more sensitive to luminance than to chrominance (lower
resolution needed for chrominance)
Red
Blue
Green
MatrixLPF
Y
Cb
Cr
ModulatorSubC
+CompositeVideo
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
6/39
December, 20, 2006AV Compression / Alain Bouffioux
6
Some video formats (3)
The 4:2:2 format Y sampling @ 13.5 MHz
C sampling @ 6.75 MHz
8 bits per pixel
720 active points per line
576 lines active lines per image (2 fields) (625 lines)
and 480 active lines (525 lines) Pixels are not square (e.g. for 480 lines, only 640 active points are needed
- VGA format)
Image size 720*576 or 720*480
The 4:2:0 format
Vertical chrominance resolution reduced by a factor 2(average on two successive lines)
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
7/39December, 20, 2006AV Compression / Alain Bouffioux
7
Some video formats (4)
SIF format (Source Intermediate Format)
Half the vertical & horizontal resolution of 4:2:0
For 50Hz countries: Luminance: 360*288
Chrominance: 180*120
CIF format (Common Intermediate Format) Intermediate format used in videoconferencing
(communication between US & Europe)
resolution: 360*288
Sampling frequency: 30 Hz
QCIF (Quarter CIF) Half the vertical & horizontal resolution of CIF.
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
8/39December, 20, 2006AV Compression / Alain Bouffioux
8
Agenda
About video formatsVideo compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage
context
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
9/39December, 20, 2006AV Compression / Alain Bouffioux
9
Video compression in MPEG-1&2 (1/6)
Principles removal of intra-picture redundancy :Image is decomposed in 8*8 pixels sub-images.
Each sub-image contains redundant information
DCT transformation (in frequency domain) de-correlates the input
signal.( most energy in low spatial frequencies)
removal of inter-picture redundancy :
coding of difference with an interpolated picture (moving vectors)
high frequent spatial frequencies quantized with lower resolution
than low ones(remove irrelevancy)
Zig-zag scan and VLC (remove redundancy)
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
10/39December, 20, 2006AV Compression / Alain Bouffioux
10
Video compression in MPEG-1&2 (2/6)
Result 4:2:2 CCIR 601 resolution : 166 Mbps
(=25images/sec *576lines* 720pixels* 2(lum & chrom) *8bits)
3-4 Mbps (mean) in MPEG2
4:2:0 SIF resolution : 30 Mbps
(=25 images/sec *288 lines *352pixels* 1.5(lum & chrom)*8bits)
1.2 Mbps (CBR) in video CD (MPEG1)
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
11/39December, 20, 2006AV Compression / Alain Bouffioux
11
Video compression in MPEG-1&2 (3/6)
Spatial redundancy reduction (DCT example)
158 0 -1 0 0 0 0 0
-1 -1 0 0 0 0 0 0
-1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
139 144 149 153 155 155 155 155
144 151 153 156 159 156 156 156
150 155 160 163 158 156 156 156
159 161 162 160 160 159 159 159
159 160 161 162 162 155 155 155
161 161 161 161 160 157 157 157
162 162 161 163 162 157 157 157
162 162 161 161 163 158 158 158
158 0 -1 -1 -1 -1 EOB
zig-zag scan
1260 -1 -12 -5 2 -2 -3 1
-23 -17 -6 -3 -3 0 0 -1
-11 -9 -2 2 0 -1 -1 0
-7 -2 0 1 1 0 0 0
-1 -1 1 2 0 -1 1 1
2 0 2 0 -1 1 1 -1
-1 0 0 -1 0 2 1 -1
-3 2 -4 -2 2 1 -1 0
DCT
Quantisation
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
12/39December, 20, 2006AV Compression / Alain Bouffioux
12
Video compression in MPEG-1&2 (4/6)
Temporal redundancy reduction
B
5
Bi-direction al p rediction
I : Intra-co ded pictureP: Pred icted pictureB: Bi-directionally interpolated picture
4
B
rde r ofrese nta tion
rde r ofransm ission
BI P
0 3
B P
1 2 6
B
Prediction
I B P B
nc reas e ofom pressionate
0 1 2 3 4
7
B P B
5 9
B I P
8
P B B P I B
86 7 9
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
13/39December, 20, 2006AV Compression / Alain Bouffioux
13
Video compression in MPEG-1&2 (5/6)
Model of a possible encoder
Buffer
Data
Regulator
Motionestimation
Motionvectors
andmodes
Formatconversionand picturereorder
putictures
Picturestore and
prediction
1/DCT
DCT VLC
1/Q
Q Multi-plex
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
14/39December, 20, 2006AV Compression / Alain Bouffioux
14
Video compression in MPEG-1&2 (6/6)
MPEG1 en MPEG2 video features MPEG1 sequential picture
resolution : SIF format 288(240)*356*24,25 or 30 Hz
MPEG2 sequential or interlaced
various levels : low level (SIF: 288*356), main level (CCIR601: 576 *720), high 1440 level (HDTV: 1152*1440), high level (EQTV:
1152*1920)
various profiles (toolboxes) : simple profile (No B picture), main profile
(=MPEG1+interlaced), SNR scalable profile (allows graceful degradation
(noise improvement at same resolution), spatial scalable profile
(hierarchical coding : improvement at higher resolution), high profile.
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
15/39
December, 20, 2006AV Compression / Alain Bouffioux
15
MPEG-4 AVC or H264
Defined about 10 years after MPEG-2
Similar structure used as for MPEG-2, but with some improvements
And two additional functional blocks: Intra-frame prediction: exploit the redundancy between neighboring blocks within a
frame. A block is predicted using previously decoded neighboring blocks.
Deblocking filter: filter applied at the block edges (except picture boundaries) in order to
improved perceptual quality
Cost/benefits Higher complexity
Compression efficiency: factor 2
More than compression
Network adaptation layer defining packets adapted for each transport network
Uses HD-DVD (MPEG2 & H264) & HD-camcorder
Video services over IP networks
Mobile television
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
16/39
December, 20, 2006AV Compression / Alain Bouffioux
16
Agenda
About video formatsVideo compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage
context
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
17/39
December, 20, 2006AV Compression / Alain Bouffioux17
Synchronisation
Synchronisation in the multimedia context
refers to the mechanism that ensures a temporal
consistent presentation of the audio-visual
information to the user
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
18/39
December, 20, 2006AV Compression / Alain Bouffioux18
Intramedia synchronisation
T between capture & presentation = ConstantSame clock frequency & Data on time
Need for corresponding tools
Network
T1T2 =
T1
Capture time
Audiosignal
Encoder
T = Constant
Decoder
Presentation time
Audiosignal
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
19/39
December, 20, 2006AV Compression / Alain Bouffioux19
Intermedia synchronisation
T_Audio = T_Video
Sampled at the same time Presented at the same time)Possible tools : common time base and presentation control (media
synchronisation with the common time base)
Ex.: Lip_sync (requirement: |delay_difference| < 80msec)
Network
T_audio = Constant
T_video = Constant =Capture time
ideoignal
udioignal
Encoder
Capture time
T_AudioPresentation time
Decoder
Videosignal
Audiosignal
Presentation time
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
20/39
December, 20, 2006AV Compression / Alain Bouffioux20
Recovery of clock in CBR
CBR = Constant Bit Rate
if the clock to recover is synchronous with transport clock Recovery ofclock but not of common time base
Remark : possibility to slave stream from DSM (Digital Storage Media) to
local clock
Filter
BR stream
Phaseerror
Filling level
50%
ime Information carriedy each sample
VCO
Processing
Recoveredclock
time
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
21/39
December, 20, 2006AV Compression / Alain Bouffioux21
Recovery of clock and time base in VBR
VBR = Variable Bit Rate
Need for insertion of time stamps (OUTPUT TIME)Output time stamp says for example : It is now 16h25
Receiver adjusts its own horloge to the received time stamp
Recovery of clock & of common time base
Time information carriedonly by time stamps
Time stampextraction
Time counter
Recovery of clock & time
First time stamp
Others
lock
Data stream
Data stream
Time stamps
Counter
Recovered clock
Error Filter VCO
VBR Stream
Counter sample(=Time stamps)
Channel
Time stamps
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
22/39
December, 20, 2006AV Compression / Alain Bouffioux22
Synchronisation with common time base
Insertion of time stamp (=INPUT TIME)Input time stamp says : Input has been sampled at 16h29.Receiver presents the sample at (its input time stamp + maximum encodingand decoding delay).Alternative: transmission of presentation time stamp (input time+delay)
Buffering
Med iaoutp ut
Med ia input
Time c lock(Rec ov ered )
Compa rison of time clockwith sam pled time clock
Samp le "T ime clock "
Ass em ble frame
Time c lock
Timestamp
Proce ss ing
Proce ss ing
Chann el
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
23/39
December, 20, 2006AV Compression / Alain Bouffioux23
Getting data on time
On time Not too late, not too earlyNo buffer over- or underflow
Flow control : not applicable in broadcasting
Common time base and
Definition of a standard target decoder that describes the data
consumption pattern of the receiver.
Remark: Direct MPEG (Microsoft) does not use time information for
clock recovery but relies on flow control
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
24/39
December, 20, 2006AV Compression / Alain Bouffioux24
Streams
Idea of continuity (pipelining)Carry time information for clock recovery
No flow control (allows broadcasting)
The emitter must have a precise knowledge of the receiver data
consumption pattern (explicit in MPEG STD)
Just-in-time
Shorter delay and smaller buffer size than with flow control
Two aspects in synchronisation :
Clock recovery & timing control (model & buffering)
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
25/39
December, 20, 2006AV Compression / Alain Bouffioux25
Requirement on for stream transport
Data information BER (Bit Error Rate) requirement
No repetition of frame possible FEC (Forward Error Correction)
Time information No jitter
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
26/39
December, 20, 2006AV Compression / Alain Bouffioux26
Agenda
About video formats
Video compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storagecontext
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
27/39
December, 20, 2006AV Compression / Alain Bouffioux27
What is MPEG ? (1/2)
Moving Picture Expert Group
International standard (ISO/IEC)
Interoperability & economy of scale
Various standard developed/under development
Not only audio/video compression
MPEG-1/2/4 - Compression of audio and video and multiplexing in
a single stream
Definition of the interface not of the codecs
room for improvement
MPEG-1 : until 1.5 Mbps, for DSM
Progressive picture, stereo (Dolby surround)
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
28/39
December, 20, 2006AV Compression / Alain Bouffioux28
What is MPEG ? (2/2)
MPEG-2 : Various bit rates (CBR & VBR)
Program stream for DSM, transport stream for network
Interlaced picture, 5.1 audio channels
Definition of various video levels (e.g. CCIR601 resolution: 4-9 Mbps,
HDTV:15-25 Mbps) and profiles
MPEG-3 : Cancelled, integrated in MPEG-2(Initially : for HDTV)
MPEG-4 : standard for audio, video and graphics in interactive 2D
and 3D multimedia communication.
(Initially : low bit rate for real-time personal communication)
MPEG-7 : Multimedia contents description interface
MPEG-21 : Focus on multimedia distribution and on DRM aspects.
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
29/39
December, 20, 2006AV Compression / Alain Bouffioux29
The MPEG model (1/2)
Audiodecoder
Audiosignal
Videosignal
Presented signals
Multiplexer
Videodecoder
Captured signals
Audioencoder
Videoencoder
udioignal
ideoignal Digital storage medium
orNetwork
Transmissionchannel
Demulti-plexer
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
30/39
December, 20, 2006AV Compression / Alain Bouffioux30
The MPEG model (2/2)
Compression of audio & video and multiplexing in a single stream
Guarantees intramedia and intermedia synchronisation.
MPEG defines an interface bitstream syntax
timing of the bitstream STD specifying timing requirement (ideal
model)
Consequences: Decoder should compensate deviations from STD
Network should correct jitter introduced by the channel (RTD-LJ)
MPEG stream must be adapted to transmission channel formatting,error correction, channel coding (b.v.video-CD)
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
31/39
December, 20, 2006AV Compression / Alain Bouffioux31
Components of the MPEG standard
The MPEG standard is composed of 3 main parts : Audio : Specifies the compression of audio signals
Video : Specifies the compression of video signals
System : specifies how the compressed audio and video signals are
combined in the multiplexed stream (program stream or transport
stream).
Each part specifies : The bitstream syntax
The timing requirement and the related information (bit rate, buffer
needs)
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
32/39
December, 20, 2006AV Compression / Alain Bouffioux32
Synchronisation Mechanism (1/2)
Syste m decoderand
demu ltiplexe r
Mu ltiplexerand systemencoder
Audiodecoder
Vide odecoder
Comp ar isonPTS and STCandpresenta tion
Vide ooutpu t
Comp ar isonPTS and STCandpresenta tion
Audioou tpu t
Extr actionof PCR(SCR )
STC
Audioencoder
Vide oencoder
Assemblepictu res,Samp le STCfor PTS
ideopu t
Assembleaudio fra me s,Samp le STCfor PTS
ud iopu t
Samp le STCfor PC R(SCR )
STC
Transm issionchannel
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
33/39
December, 20, 2006AV Compression / Alain Bouffioux33
Synchronisation Mechanism (2/2)
PCR for TS & SCR for PS (but same concept)
Clock & time base recovery: Time-stamping at OUTPUT (PCR included in
TS multiplex, SCR in pack header)
Audio & video clock locked to STC easy recovery (see next slide)
Synchronisation of audio & video to common time base (Time stamping at
Input)
STD is defined (because of the absence of flow control)
streams are such that STD buffers never over- or underflow
In TS, many program in a single stream but unique clock per program.
Time information No Jitter requirement for transport
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
34/39
December, 20, 2006AV Compression / Alain Bouffioux34
Clock recovery in receiver
VCOAudio clock
divider
Video clockdivider Video
clock
Audioclock
Audiooutput
STC
Comparis onPTS and STCand
PresentationDecodedaudio
PTS
STC(Counter)
CR
STC
ErrorLow Pass Filter(Integrator)
Load firs t PCR
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
35/39
December, 20, 2006AV Compression / Alain Bouffioux35
MPEG-2 program & transport streams
Program streams: Relatively error free environment
program stream packet may have variable and great length
Single time base
Transport streams: environment where errors are likely
many programs (independent time base)
Transport stream packet : fixed, 188 bytes
Contains tables
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
36/39
December, 20, 2006AV Compression / Alain Bouffioux36
Channelencode
Channeldecode
Bit-stream
Sourcedecode
formationink
Digitaloutput
Format Decrypt
Sourceencode
Format
formationource
Digitalinput
Encrypt
Synchro-nisation
Digitalwaveform
Otherdestination
Demodu-lateDemulti-plex Multipleaccess
Modu-late
Multi-plex
Channelbits
Othersources
Multipleaccess
Channel
MPEG in a communication context (1)
Typical communication system
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
37/39
December, 20, 2006AV Compression / Alain Bouffioux37
MPEG in a communication context (2)
MPEG : Source coding only (bit rate reduction) + multiplexing
The MPEG stream must be adapted to the channel in what concern its physical
characteristics and in order to get the required QoS (Quality of Service) &
Security Encryption
Channel coding (forward error correction, interleaving, modulation codes)
multiplexing & formatting modulation (frequency allocation)
multiple access method
Some channels : CD/DVD - satellite - cable - ATM - 1394
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
38/39
December, 20, 2006AV Compression / Alain Bouffioux38
VideoEncoder
PEG2 compression layer
Audioencoder
udio,ideo
ources
ES
(ElementaryStream)
Adap-tationto the
channel
PS
(1 pro-gram)
MPEG2 system layer
PSMulti-plexing
Adap-
tationto thechannel
DVB, DVD ...
Disc
Satellite
TSMulti-plexing
TS(n pro-
grams)
Adap-tationto the
channelCable
TS (Transport Stream)
orPS (Program Stream)
MPEG-2 in a communication context (3)
A simple view of MPEG-2 in the communication context
-
8/4/2019 3. AV compression - More about video compression & MPEG - Dec06
39/39
39