Video and Image Analytics for Business, Consumer and ... · Video and Image Analytics for Business,...

Video and Image Analytics for

Business, Consumer and Social

Insights

Jialie Shen

School of Information Systems, SMU

Outline

• Part I: General introduction

– What is image and video mining (analytics)?

– Why they are so important to business?

• Part II: Novel business applications

– Online advertisement

• Rich media advertising

– Retail customer analytics

• Visual tracking for monitoring customers in-house behaviours

• Part III: Conclusion & QA

– Proactive thoughts, open issues and future directions

2

General Introduction

• Modern ITs have led to fast growth of large video &

image collections

• Analyzing image & video to discover useful knowledge is

becoming important

– manual processing is important due to huge volume

• Goal of image and video analytics (IVA)

– developing intelligent computer program to automatically gain

• comprehensive understandings of data content

• effective/scalable data management

3

General Introduction (cont.)

• IVA research is a hot area ……

• Technologies developed are generally used to

perform two kinds of tasks

– Descriptive mining – find out the general properties of

the data in the database (e.g., image clustering)

– Predictive Mining – perform inference on the current

data in order to make predictions (e.g., age and emotion

estimation based on face)

4


• IVM research is a hot area ……




the data in the database (e.g., image clustering), and



estimation based on face)

5

Descriptive Mining – Image Clustering

6

Assuming we have a

set of images without

any structure


7

Raw image set

Clustering

Algorithm

After passing through clustering algorithm, ........


8

Raw image set

Waterfall cluster

Sun cluster

Clustering

Algorithm

After passing through clustering algorithm, the

images are grouped into two clusters


• IVA research is a hot area ……




the data in the database (e.g., image clustering), and



estimation based on face image)

9

Predictive Mining - Age Estimation

10

Face Images

Assuming we have a set of face images as inputs

to age estimation system ........


11

Age estimation system

extracts visual features

from images

Face Images


12

Machine

Learning

Algorithm

AGE

(25, 18, ?)

Supervised or Semi-Supervised

Predictor

Feature Extraction

(Gabor Filtering)

Face Images

Using visual

features, ML

algorithm

predicts age


• IVM research is a hot area ……



– Descriptive mining – find out the general properties of the

data in the database (e.g., scene analysis), and

– Predictive Mining – perform inference on the current data

in order to make predictions (e.g., age estimation based on

visual appearance)

13

Why images and videos

are important to business?

General Introduction (Cont.)

– “One picture worth a thousand words”

• a complex business idea can be conveyed with just a single still

image or video sequence

• Image and video can bring your customer more information

14

15

What is this image???

Message: ????

Who is this person????

16

Uncle Sam (initials U.S.) is a

common national personification

of the American government

Who:

Message: Join us! Buddy! We need u!....

What?

Very classic advertisement used

for promoting US army

recruitment in WWI and WW II.


– “One picture worth a thousand words”

• a complex business idea can be conveyed with just a single

still image and video sequence

• Image and video can bring your customer more information

– Image and video are very helpful for people to retain

business information

• People can remember 10% of the information they read

20% of the information they hear, but

50% of the information they see.

17

18

Perhaps no one knows accurate number!


19

6 billion (Aug 2011)

• 192 years to view all of them (1s per image)

• 3000+ uploads/minute

• 2% Internet users visit (2009)

• Daily time on site: 4.7 minutes (2009)

690 million (Mar 2012)

• 3,450 years to see all of them

• 48 hours uploaded/minute (2012)

• 20% Internet users visit (2009)

• Daily time on site: 23 minutes (2009)

100 billion (Middle of 2011 )

• 3,200 years to view all of them (1s per image)

• ~200M uploads/day; ~ 6B/month (2012)

• 800+M users (Dec 2011)

• Daily time on site: 30 minutes (2009)


20

We have millions online images and

videos

They could be wonderful carriers for

business messages

if we know how to use them effectively

???

Outline



– Why they are so important to business people?





• Visual tracking for monitoring customer in-house behaviours



22

Online Advertising

• A sub-area of advertising/marketing

• Power the revenue of a large number of Internet

service companies

• Provide financial support for free content/service

• Many online ads formats

– Text ads

– Rich media ads

– …

23

Online Advertising

• A sub-area of advertising/marketing

• Power the revenue of a large number of Internet

service companies

• Provide financial support for free content/service

• Many online ads formats

– Text ads

– Rich media ads

– …

24

Online Rich Media Ads

25

• No formal definition yet!

• A range of interactive digital media including video, audio and

associated text for online marketing campaigns.

• An emerging online monetization strategy to achieve various goals

Importance of Rich Media Ads

• Reshaping for online marketing

• Evolution of Internet has brought fundamental changes in

the way people obtain and generate biz info.

– Online media contents surged to an unprecedented level

due to the Web 2.0

• Rich media becomes the primary force to drive marketing

campaign

• Richmedia ads possess many unique advantages

– Better capability to convey business information

– More attractive and salient than plain text ads

– Prefect for driving purchase intent

26

An Exemplary System

27

• A contextual in-image advertizing platform

– Aims to embed “right” visual-based ads into online images

• To achieve the goal, the system can

– match ads to image content and related text information

automatically

• “contextual relevance” – which image ads to be selected?

– find “right” insertion positions

• “optimal position” - where to insert ads?

An Exemplary System (cont.)

28

• Three major components

Relevance

Optimization

Multimodal

Information

Extraction &

Analysis

Vision based

Web Page

Analysis


29

Vision-based Web

Page Analysis

Web Pages

Image: Position:100,200

Size: 500x250 URL:http://www.nytimes.

com/imagepages/

2010/01/23/sports/

Crawler

Surrouding Text: Kobe, NBA, LA Lakers,

New York, Basketball,

Sports, ……

• Vision-based web page analysis

has two functionalities.

• Given a web page with images, it • selects the images suitable for

advertising, and

• extracts surrounding text


30

Vision-based Web

Page Analysis

Multimodal Information

Extraction & Analysis

Web Pages



com/imagepages/

2010/01/23/sports/

Crawler



Sports, ……

Local + Gobal Textual Information

High-level

Concept Detector

Insertion Point

Detection

Text Data

Filtering

Visual Information Analysis

High level

concept

detection using

surrounding text

Image analysis for

determining ads

insertion point

• Two sub-modules • Text analysis module

• Image analysis module


31

Vision-based Web

Page Analysis

Multimodal Information

Extraction & Analysis Relevance

Optimization

Text

Relevance

Match

Ad Rank List Ads

Ads

Reranking

Module

Web Pages



com/imagepages/

2010/01/23/sports/

Crawler



Sports, ……


High-level

Concept Detector

Insertion Point

Detection

Text Data

Filtering


Relevance optimization module uses the high level

concept gained via surrounding text analysis to get

“right” ad via ranking ads based on relevance.


32

Vision-based Web

Page Analysis Multimodal Information

Extraction & Analysis

Relevance

Optimization

Text

Relevance

Match

Ad Rank List Ads

Ads

Reranking

Module

Web Pages



com/imagepages/

2010/01/23/sports/

Crawler



Sports, ……


High-level

Concept Detector

Insertion Point

Detection

Text Data

Filtering


The optimal insertion point determined by

visual information analysis module tells us

where is the ‘”right” place to insert “right” ad

Experience Gained

• Effective solution = media analysis + text mining +

business domain knowledge

• A hugh unexplored domain:

– How to achieve the best trade-off advertizing strategy making

different users happy at the same time?

– How to maximize its business or economic impacts under

different metrics?

What is best business model (e.g. Cost per Impression)

• Industrial partner’s involvement is IMPORTANT!

33

Outline





– Online Advertisement



• Visual tracking for monitoring customers in-house behaviours



34

Visual Tracking (VT) – Customer Traffic

35

• The goal of every retailer: sales profit maximization

• Understanding customers is essential.

• Tracking customers can provide a lot off

information about customer in-store movement

patterns & appearance

Visual Tracking – Customer Traffic

• Labour based security is expensive

• Cameras could be very helpful to reduce cost

– But, watching the video feeds is tedious

• Labour based security is ineffective

– dedicated and well motivated personnel can't stay focused on

monitoring tasks for more than 20 minutes

• VT techniques are very useful to increase the business

performance via analyzing customer s’ in-store

behaviors automatically

36

Visual Tracking – Customer Traffic

37

• With VT technology, we can automatically

– Count/aggregate the number of people passing

through certain spots

• valuable to improve customer service by adapting

business resources to match customer traffic

– Track the number of customers during different days

and hours

• allowing optimal displaying configuration for in-store

promotions and events

Visual Tracking – Customer Behavior

38

• With VA technique, we also can

– Analyze store traffic to identify the major customer

trajectories

• determine general strategies for product placement

– Identify the regions that attract less customers

• optimize store layout to best satisfy customers and drive sales

Outline








• Visual tracking for monitoring customer in-house behaviours



39

Conclusion and Q&A

• We have reviewed

– what is image and video analysis

– why they are so important to us

– two business applications

• Analysing image and video is one of the most

appealing research directions for business

intelligence

40

Conclusion and Q&A

• Why this is an interesting domain for scholars in

management science/CS or CEO?

• Before answering the question, we need to have a

look another set of questions

– Who should be real god for a firm?

– Over long run, where do firm’s cash flows come from? (how should we measure business value of a firm)

• For sure, from your customers

• Where /how do you estimate cash flow from customer & predict

customer’s life time value?

41

Acknowledgement

42

• The research study is kindly supported by

Thank you very much for your kind

attentions and valuable time!

Video and Image Analytics for Business, Consumer and ... · Video and Image Analytics for Business,...

Documents

Transcript of Video and Image Analytics for Business, Consumer and ... · Video and Image Analytics for Business,...