Dhruv Batra Talk

Beyond MAP: Making Multiple Predictions: Diversity, DPPs and more.

Dhruv BatraVirginia TechAlex KuleszaUniv. of MichiganDennis ParkUC IrvineDeva RamananUC Irvine

CVPR 2013 Diversity Tutorial1ScheduleTimeTopicPresenter9:00 9:30Opening Remarks + Need for Multiple Diverse SolutionsDhruv9:30 10:15Multiple Solutions via M-Best MAPDennis10:15 10:45Coffee Break10:45 11:30Multiple Solutions via Diverse M-BestDhruv12:00 1:30Lunch(C) Dhruv Batra 2CVPR 2013 Diversity TutorialTimeTopicPresenter9:00 9:30Opening Remarks + Need for Multiple Diverse SolutionsDhruv9:30 10:15Multiple Solutions via M-Best MAPDennis10:15 10:45Coffee Break10:45 11:30Multiple Solutions via Diverse M-BestDhruv12:00 1:30Lunch1:30 2:00Multiple Solutions via SamplingDeva2:00 3:15Multiple Solutions via DPPsAlex3:15 3:45Coffee Break3:45 4:30DPPs (Continued)Alex4:30 5:00Closing Remarks + What can we do with diverse solutions?DhruvSchedule(C) Dhruv Batra 31. Please interrupt & ask questions!2. All slides available online.CVPR 2013 Diversity TutorialLocal Ambiguity(C) Dhruv Batra 4

slide credit: Fei-Fei Li, Rob Fergus & Antonio Torralba

CVPR 2013 Diversity TutorialGraphical Models to the Rescue!(C) Dhruv Batra 5MAPInferenceMost Likely Assignment

y1y2xn

PersonTablePlate

CVPR 2013 Diversity TutorialThe visual perception problem have a lot of local ambiguity. If I ask you to recognize this patch, you probably wont be able to.

Graphical models are useful tools that allow us to model all variables of interest, write down local beliefs in terms of node & edge energies, and then use these to make a global prediction by jointly minimizing this energy function, also called MAP inference.

5Vision in 2000s(C) Dhruv Batra 6

CVPR 2013 Diversity TutorialGraphical Models in Vision(C) Dhruv Batra 7

SegmentationLeft imageRight imageDisparity map

Stereo

Geometric Labelling

DenoisingMotion Flow

Object Recognition / Pose Estimation

CVPR 2013 Diversity Tutorial(C) Dhruv Batra 8

Alpha-ExpansionSimulated AnnealingCVPR 2013 Diversity Tutorial

Dollar et al., BMVC 20099CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 10CVPR 2013 Diversity TutorialProblems(C) Dhruv Batra 11Model-Class is Wrong!-- Approximation Error

Human Body TreeFigure Courtesy: [Yang & Ramanan ICCV 11]CVPR 2013 Diversity TutorialUnfortunately, we often run into a number of problems with MAP.

Most often, our model is simply wrong. So even if we predict the most probable state from our model, it could be very far from ground-truth.

For example, a tree model assumes that we walk around like this, with our limbs always un-occluded. 11Problems(C) Dhruv Batra 12Model-Class is Wrong!-- Approximation ErrorNot Enough Training Data!-- Estimation ErrorCVPR 2013 Diversity TutorialEven if your model is rich enough, you may not have enough training data to learn the correct parameters. 12Problems(C) Dhruv Batra 13Model-Class is Wrong!-- Approximation ErrorNot Enough Training Data!-- Estimation ErrorMAP is NP-Hard-- Optimization Error

CVPR 2013 Diversity TutorialEven if you have enough training data, actually computing MAP may be NP-hard. 13Biggest Problem(C) Dhruv Batra 14Model-Class is Wrong!-- Approximation ErrorNot Enough Training Data!-- Estimation ErrorMAP is NP-Hard-- Optimization ErrorInherent Ambiguity-- Bayes Error

??Old Lady looking left / Young woman looking away?Rotating clockwise / anti-clockwise?One instance / Two instances?

CVPR 2013 Diversity TutorialEven if you can compute MAP, there may simply be multiple acceptable answers.

For example, this woman could be rotating left or rotating right. This could be a young woman looking away or an old lady looking left. When we have a user-in-the-loop, different users may expect different outputs from the same input. 14Problems(C) Dhruv Batra 15Model-Class is Wrong!-- Approximation ErrorNot Enough Training Data!-- Estimation ErrorMAP is NP-Hard-- Optimization ErrorInherent Ambiguity-- Bayes ErrorMake Multiple Predictions!Single Prediction = Uncertainty MismanagementCVPR 2013 Diversity TutorialSo where does that leave us?

When your model is wrong, your data is insufficient, you cant compute the optimal MAP and youre not sure which answer the user wanted anyway, making a single best prediction is simply inadequate. What we need to do is to make multiple plausible predictions! 15Multiple Predictions(C) Dhruv Batra 16Flerova et al., 2011Fromer et al., 2009Yanover et al., 2003(Diverse) M-Best MAP

Dhruv10:45 11:30

Dennis9:30 10:15CVPR 2013 Diversity Tutorial16Multiple Predictions(C) Dhruv Batra 17

Porway & Zhu, 2011TU & Zhu, 2002Rich HistorySamplingxxxxxxxxxxxxx

Deva1:30 2:00-pmCVPR 2013 Diversity TutorialSo how can we make multiple predictions.

Well, its a probabilistic model. We could sample from the distribution. Unfortunately, sampling is rather wasteful since we observe the same modes of the distribution over and over again. And if there is a low-probability mode, we will have to wait a long time to observe a sample from it. 17Multiple Predictions(C) Dhruv Batra 18Build a new model over sets that prefers diverse setDeterminentalPoint Process

Alex2:00 3:153:45 4:30

CVPR 2013 Diversity Tutorial18Multiple Predictions(C) Dhruv Batra 19Build a new model over sets that prefers diverse setDeterminentalPoint Process

Alex2:00 3:153:45 4:30

Porway & Zhu, 2011TU & Zhu, 2002Rich HistorySamplingFlerova et al., 2011Fromer et al., 2009Yanover et al., 2003(Diverse) M-Best MAP

CVPR 2013 Diversity Tutorial19Multiple Predictions(C) Dhruv Batra 20Build a new model over sets that prefers diverse setDeterminentalPoint ProcessPorway & Zhu, 2011TU & Zhu, 2002Rich HistorySamplingFlerova et al., 2011Fromer et al., 2009Yanover et al., 2003(Diverse) M-Best MAP

CVPR 2013 Diversity Tutorial20ScheduleTimeTopicPresenter9:00 9:30Opening Remarks + Need for Multiple Diverse SolutionsDhruv9:30 10:15Multiple Solutions via M-Best MAPDennis10:15 10:45Coffee Break10:45 11:30Multiple Solutions via Diverse M-BestDhruv12:00 1:30Lunch1:30 2:00Multiple Solutions via SamplingDeva2:00 3:15Multiple Solutions via DPPsAlex3:15 3:45Coffee Break3:45 4:30DPPs (Continued)Alex4:30 5:00Closing Remarks + What can we do with diverse solutions?Dhruv(C) Dhruv Batra 21All slides available online.CVPR 2013 Diversity TutorialNotation andReview of CRFs(C) Dhruv Batra 22CVPR 2013 Diversity TutorialConditional Random FieldsDiscrete random variables

Factorized Model

23(C) Dhruv Batra Node Energies / Local CostsEdge Energies / Distributed Prior

X1X2XnXikx11 1 10 0kxk1010101000

CVPR 2013 Diversity Tutorial23MAP InferenceIn general NP-hard [Shimony 94](C) Dhruv Batra 24Approximate Inference Heuristics: Loopy BP[Pearl, 88] Greedy: -Expansion [Boykov 01, Komodakis 05] LP Relaxations:[Schlesinger 76, Wainwright 05, Sontag 08, Batra 10] QP/SDP Relaxations: [Ravikumar 06, Kumar 09]

CVPR 2013 Diversity TutorialMAP Integer Program(C) Dhruv Batra 25

kx1

CVPR 2013 Diversity TutorialLet me begin by writing the MAP problem, which involves minimizing an energy composed of node terms & edge terms.

Instead of each variable being an integer between 1 and k, we will represent them as indicator vectors of length k.

25MAP Integer Program(C) Dhruv Batra 26

kx11000

CVPR 2013 Diversity TutorialSo this is xi = 1.

26MAP Integer Program(C) Dhruv Batra 27

kx10100

CVPR 2013 Diversity TutorialSo this is xi = 2.

27MAP Integer Program(C) Dhruv Batra 28

kx10010

CVPR 2013 Diversity TutorialThis is xi = 3.

28MAP Integer Program(C) Dhruv Batra 29

kx10001

CVPR 2013 Diversity TutorialAnd this is xi = 4.

29MAP Integer Program(C) Dhruv Batra 30

kx10001

k2x1

CVPR 2013 Diversity TutorialWe can do the same thing at edges, where the indicator vector is of length k^230MAP Integer Program(C) Dhruv Batra 31

kx10001

k2x1

CVPR 2013 Diversity TutorialSo this is the Linear Integer program whose solution in the MAP state. 31MAP Integer Program(C) Dhruv Batra 32Graphcuts, BP, Expansion, etc

CVPR 2013 Diversity TutorialWe typically solve this with algorithms like graph-cuts, BP, alpha-expansion. Etc.

32MAP Integer ProgramLP view(C) Dhruv Batra 33

MAPGraphcuts, BP, Expansion, etc

CVPR 2013 Diversity TutorialScheduleTimeTopicPresenter9:00 9:30Opening Remarks + Need for Multiple Diverse SolutionsDhruv9:30 10:15Multiple Solutions via M-Best MAPDennis10:15 10:45Coffee Break10:45 11:30Multiple Solutions via Diverse M-BestDhruv12:00 1:30Lunch1:30 2:00Multiple Solutions via SamplingDeva2:00 3:15Multiple Solutions via DPPsAlex3:15 3:45Coffee Break3:45 4:30DPPs (Continued)Alex4:30 5:00Closing Remarks + What can we do with diverse solutions?Dhruv(C) Dhruv Batra 34All slides available online.CVPR 2013 Diversity Tutorial