CS 4763 Fundamentals of Multimedia Systems - Introduction to Image Processing

Qi Tian

Computer Science Department University of Texas at San Antonio

qitian@cs.utsa.eduhttp://www.cs.utsa.edu/~qitian/

Image Processing

Manipulation of multidimensional signals− image (photo)− video− CT, MRI− Fluid flow

),( yxf),,( tyxf),,,( tzyxf),,,( tzyxv

A Typical Image Processing System

object observe digitize store process Refresh/store

Record

output

DisplayImaging systems

Sample and quantize

Digital storage (disk)

Digital computer

On-line buffer

X-ary, radar imaging, infrared imaging, ultrasound imaging, medical imaging, geophysical imaging

A/D

Fundamentals of Image Processing

Representation– acquisition, digitization, and display to mathematical

characterization of images for subsequent processing– a prerequisite for an efficient processing techniques such as

enhancement, filtering, and restoration.

Processing Techniques– Image compression, image restoration, and image reconstruction– Statistical image processing techniques

Communications

Multimedia Processing Techniques

– Coding/compressionStorage and communications

JPEG, JPEG2000MPEG-1 (CD, mp3), MPEG-2 (HDTV, DVD) H.261, H.263

– Enhancement, restoration, reconstructionfeature extraction for image analysis and visual information displayremoval of degradation in an image, LS, ML, Max entropy, MAP2D -> 3D image MRI, CT, Radon transform

– Analysis, detection, recognition, understandingquantitative measurements from an image to produce a description on it

– Visualization

Advanced Processing Techniques

Statistical processing techniques– Hidden Markov model (HMM)

– Probabilistic graphical models

– Bayesian networks

– Markov random field

Many applications to speech recognition, pattern classification, data

compression, and channel coding, etc.

History of Image/Video Coding

1950

1960

1970

1980

1990

2000+

Math PR, CV, CG

Fractal 3-D Model based coding

Signal ProcessingBased

PCM DPCM

Transform CodingVQ

Subband Coding Wavelets

Reference:– F. Nebeker, Signal Processing: The Emergency of a Discipline,

1948-1998– IEEE History Center, 1998

Broadband TV (NTSC)500 × 500 × 8 × 3 × 30 bits/sec≈100 Mb/sec (compression is necessary!)Modem: 56Kb/sec

Picture Element– Pixel West coast people in USC– Pel East people in MIT

Image/Video Compression

Signal-Processing Based:Encoder

),( yxfH

),( yxgSignal Proc.

Representation ),( yxg

Decoder1H− ),(ˆ yxf),( yxg

Image/Video Compression

3D Model-Based:Encoder

Representation P

Decoder

),( yxfH

Analysis Model Parameter P

Model

),(ˆ yxfP 3D

Model

Image/Video Compression

Fractal-Based:Encoder

Representation S

Decoder

),( yxf System S ),( yxf

Find S for which is an Attractor.),( yxf

SAny signal

),(ˆ yxf

Iteration

Image/Video Compression Standard

Facsimile: Fax Group 1, 2, 3, 4JBIG (Joint Bi-level Image Expert Group)

Images: JPEG (http://www.jpeg.org/)

JPEG2000

Video: H.261, H.263 P × 64 Kb/s (P =1 ~ 30)MPEG 1 1.2 Mb/s Video, CD, MP3MPEG 2 1.2 – 20 Mb/s, sports, HDTV, DVDMPEG 4 1 kb/s → 1Mb/s, very low speed video

coding, MultimediaMPEG 7 Multimedia description, audio/video

MPEG 21 Multimedia framework

Based on Wavelet Transform

A de facto image for the past three decade for its rich texture

Lena

What are Challenging Problems in Multimedia Processing?

Multimedia Processing is taken in a broad sense, including:Image/Video compression, enhancement, restoration, reconstruction, analysis, recognition, understanding, visualization, and synthesis/animation.

Examples

Face modeling, detection, and recognitionEmotion recognitionGesture recognitionGender/age/ethnicity recognitionAudio-visual speech recognitionImage/video superresolutionImage/video browsing, indexing, and retrievalBiometrics

Face Related Research

Face modelingFace detectionFace recognitionFacial expression recognition

Generic Face Model

Texture mapping

Face model morphing

Generic Face Model

The generic face model is generated from a MRI data set

Customize A Genetic Face on An Individual

Polygon Mesh: 2240 Vertices + 3946 Triangles.Polygon Mesh: 2240 Vertices + 3946 Triangles.NonNon--Uniform Rational BUniform Rational B--Splines (NURBS): 63 control points.Splines (NURBS): 63 control points.

The iFACE system in a distributed collaborative environment. (a) Avatar in the head mounted display, (b) avatar in the desk screen of

MIC3E, (c) avatar in the main screen of MIC3E

Avatar – talking head

University of Illinois at Urbana-Champaign

Text-Driven Face Animation

“We strive to make the meter on animation production, and are always looking for new technology that will enable faster, more appealing character creation,”

said Joel Kransove, Digital Director of Nickelodeon. (Source: Digital Producer)

Speech-Driven Face Animation

“Game characters have become synthetic actors and dialogue is an essential element of the effect we create. The quality of the lip-synching can make or break the sense of reality,”

said Scott Cronce, vice president and CTO at Electronic Art (Source: Gamepro)

Video-Driven Face Animation

Emotion Recognition

Emotion Recognition

Emotion Recognition

Face Detection Techniques

Face Detection Techniques

Face Recognition: Why it is easy?

Face Recognition: Why it is hard?

Beauty Check

What Are the Causes and Consequences of Human Facial Attractiveness?

Babyfaceness

Symmetry

Social perception

Universities of Regensburg, Germany

Which is more attractive?

Universities of Regensburg, Germany

BabyfacenessLarge head

Large curved forehead

Facial elements (eyes,

nose, mouth) located

relatively low

Large, round eyes

Small, short nose

Round cheeks

Small chinKate Moss4-year old girl

Include mature female features: high, prominent cheek bones and concave cheeks

Which one is cuter?

Miss Germany (2002)

A selection of the 22 contestants of the final round of the contest

Real vs. Virtual Miss Germany

Image Analysis

Texture synthesis and transferImage Super-resolutionImage RepairsIllumination/Lighting changes and transfer

Texture Synthesis and Transfer

+

SIGGRAPH’01 Effros & Freeman, MIT, 2001

synthesis

transfer

Texture Synthesis and Transfer

Image Superresolution

True Sub-sampled

Intelligent guess about details of texture

Image Superresolution

Gaussian filter Bicubic interpolation

Image Superresolution

Median filter Wiener filter

Image Superresolution

Dynamic resolution enhancement Amos Storkey

True

Image Repairs

Image Repairs

Original Image

Result

Segmentation

Image synthesis based on Tensor Voting

Curve connection

Image Repairs

Illumination Effects on Images

Relighting – Basic Algorithm

Step 2: Approximate radiance environment map

Step 3: Synthesize novel appearance by adjusting the 9 spherical harmonic coefficients

Step 1: Align image with generic 3D face model

Lighting Transfer

input target results

Image/Video Retrieval

Image database

CBIR based on color, texture, shape/structure

MARS: Multimedia Analysis and Retrieval System

metadata

User Interface

Similarity ranking

memory

Feature weighting

Visual C++

Feature Extraction

C/C++ Color

Texture

structure

State-of-the-artCBIR Systems

QBIC (IBM), PhotoBook (Media Lab), Netra (UCSB), VisualSeek (Columbia), PicHunter (NEC-NJ), Amore (NEC-CA), EI Niňo (Praja), MARS (UIUC), Virage (Virage Inc.), CORE, PictoSeek, Piction, InfoScope …

Research CommunitiesComputer Vision, Image/Video Processing, Library and Information Science, Database and Management Systems

Leading Journals & StandardPAMI, ACM Multimedia, IJCV, CVIU

MPEG-7

MARS using global features

Biometrics

Security Threats:We now live in a global society of increasing desperate and dangerous people whom we can no longer trust based on identification documentswhich may have been compromised.

A challenging Pattern Recognition ProblemEnabling technology to make our society safer, reduce fraud and offer user convenience.

Too many passwords to remember

Identification Problems

Identity Theft: Identity thieves steal PIN (e.g., date of birth) to open credit card account, withdraw money from accounts and take out loans

3.3 million identity thefts in U.S. in 2002; 6.7 million victims of credit card fraud

Surrogate representations of identity such as password and ID cards no longer suffice

Biometrics

Automatic recognition of people on their distinctive anatomical (e.g., face, fingerprint, iris, retina, hand geometry) and behavioral (e.g., signature, gait) characteristics.

Person identification is now an integral part of the infrastructure needed for diverse business sectors such as banking, border control, law enforcement…

Biometric Applications

Biometric Applications

There are ~500 million border crossing/year (each way) in the US

Want to charge it?

Biometric Characteristics

Biometric Market Growth

International Biometric Group

“State-of-the-art” Error Rate

False accept rate (FAR):Proportion of imposters accepted

False reject rate (FRR):Proportions of genuine users rejected

Multibiometrics

Soft Biometrics

Privacy Concerns

Tracking