Generative Models

64
Generative Models

description

Generative Models. Announcements. Probability Review (Friday, 1:15 Gates B03). Late days…. To be fair…. double late days. Start the p-set early. Where we are. Search. Machine Learning. CS221. Variable Based. Search. Machine Learning. CS221. Variable Based. Search. Machine Learning. - PowerPoint PPT Presentation

Transcript of Generative Models

Page 1: Generative Models

Generative Models

Page 2: Generative Models
Page 3: Generative Models

Announcements

• Probability Review (Friday, 1:15 Gates B03)

• Late days…

• To be fair…

• Start the p-set early

double late days.

Page 4: Generative Models

Where we are

Page 5: Generative Models

Machine LearningVariable

Based

Search

CS221

Page 6: Generative Models

Machine LearningVariable

Based

Search

CS221

Page 7: Generative Models

Machine Learning

Search

Variable Based

CS221

Page 8: Generative Models
Page 9: Generative Models
Page 10: Generative Models

Where We Left Off

Page 11: Generative Models

Where We Left Off

Loopy Not loopy

Purple Not Purple Purple Not Purple

Drugged 0.108 0.012 0.072 0.008

Not Drugged 0.016 0.064 0.144 0.576

Page 12: Generative Models

Key IdeaIf we have a joint distribution over all variables, then given evidence (which could be multiple variables) E = e, we can find the probability of any query variable X = x.

Page 13: Generative Models

These are values in our table!

Y is all variables that aren’t in X or E

Y is all variables that aren’t in E

Key IdeaIf we have a joint distribution over all variables, then given evidence (which could be multiple variables) E = e, we can find the probability of any query variable X = x.

Page 14: Generative Models

Key IdeaIf we have a joint distribution over all variables, then given evidence (which could be multiple variables) E = e, we can find the probability of any query variable X = x.

Since we know that p(x | e)’s must sum to 1

Page 15: Generative Models

Loopy Not loopy

Purple Not Purple Purple Not Purple

Drugged 0.108 0.012 0.072 0.008

Not Drugged 0.016 0.064 0.144 0.576

Key Idea

Page 16: Generative Models

Loopy Not loopy

Purple Not Purple Purple Not Purple

Drugged 0.108 0.012 0.072 0.008

Not Drugged 0.016 0.064 0.144 0.576

Key Idea

Page 17: Generative Models

Loopy Not loopy

Purple Not Purple Purple Not Purple

Drugged 0.108 0.012 0.072 0.008

Not Drugged 0.016 0.064 0.144 0.576

Key Idea

Page 18: Generative Models

Loopy Not loopy

Purple Not Purple Purple Not Purple

Drugged 0.108 0.012 0.072 0.008

Not Drugged 0.016 0.064 0.144 0.576

Key Idea

Page 19: Generative Models

Loopy Not loopy

Purple Not Purple Purple Not Purple

Drugged 0.108 0.012 0.072 0.008

Not Drugged 0.016 0.064 0.144 0.576

Key Idea

Page 20: Generative Models

Loopy Not loopy

Purple Not Purple Purple Not Purple

Drugged 0.108 0.012 0.072 0.008

Not Drugged 0.016 0.064 0.144 0.576

Key Idea

Page 21: Generative Models

Key Idea

Page 22: Generative Models

Our joint gets too big

Page 23: Generative Models

Where We Left OffLoopy Not loopy

Purple Not Purple Purple Not Purple

Drugged 0.108 0.012 0.072 0.008

Not Drugged 0.016 0.064 0.144 0.576

Add variable Snowden location: { Hong Kong, Sao Paulo, Moscow, Nairobi, Caracas, Guantanamo}

Size of the table is now 2*2*2*6 = 48

But what does Snowden have to do with drugged out rockstars?

Really are independent…

Joint is exponential in size.

Page 24: Generative Models

Independence

l = loopyp = purpled = druggeds = snowden

If we have two tables, one over l, p, d and one for s, we could recreate the joint.

Page 25: Generative Models

What else is independent?

SnowdenDrugged

Purple Loopy

Page 26: Generative Models

What else is independent?

SnowdenDrugged

Purple Loopy

Purple and loopy?

Page 27: Generative Models

What else is independent?

SnowdenDrugged

Purple Loopy

Both caused by drugged

Page 28: Generative Models

What else is independent?

SnowdenDrugged

Purple Loopy

If you know drugged, purple and loopy are

independent!

Page 29: Generative Models

Conditional Independence

If you know drugged, purple and loopy are

independent!

Page 30: Generative Models

If you know drugged, purple and loopy are

independent!

Conditional Independence

Joint

Page 31: Generative Models

This is important!

Page 32: Generative Models

If you know drugged, purple and loopy are

independent!

𝑃 (𝑙 ,𝑝 ,𝑑)=𝑃 (𝑙 ,𝑝|𝑑 )𝑃 (𝑑)

Conditional Independence

Joint

Page 33: Generative Models

If you know drugged, purple and loopy are

independent!

Conditional Independence

Joint

Page 34: Generative Models

Drugged

Purple Loopy

No longer need the full joint.

Conditional Independence

Page 35: Generative Models

We only need p(var | causes) for each var.

Page 36: Generative Models

Model the world with variables

Page 37: Generative Models

And what causes what

Page 38: Generative Models

Bayesian Network

Page 39: Generative Models

Bayesian Network

Page 40: Generative Models

Bayesian Network

CoughFeverVomit

FluStomach

Bug

Page 41: Generative Models

Bayesian Network

CoughFeverVomit

FluStomach

Bug

Page 42: Generative Models

Bayesian Network

Cough (c)Fever (t)Vomit (v)

Flu (f)Stomach bug (s)

Page 43: Generative Models

Bayesian Network

Cough (c)Vomit (v)

Flu (f)Stomach bug (s)

Joint

Fever (t)

Page 44: Generative Models

Bayesian Network

Joint

Page 45: Generative Models

Bayesian Network

Cough (c)Fever (t)Vomit (v)

Flu (f)Stomach bug (s)

Joint

Page 46: Generative Models

Definition: Bayes Net = DAGDAG: directed acyclic graph (BN’s structure)

• Nodes: random variables (typically discrete, but methods also exist to handle continuous variables)

• Arcs: indicate probabilistic dependencies between nodes. Go from cause to effect.

• CPDs: conditional probability distribution (BN’s parameters) Conditional probabilities at each node, usually stored as a table (conditional probability table, or CPT)

Root nodes are a special case – no parents, so just use priors in CPD:

iiii xxP of nodesparent all ofset theis where)|(

)()|( so , iiii xPxP

Formally

Page 47: Generative Models
Page 48: Generative Models

What does NSA do with our data?

Page 49: Generative Models

Real World Problem

Formal Problem

Solution

Model the problem

Apply an Algorithm

Evaluate

The AI Pipeline

Page 50: Generative Models
Page 51: Generative Models

Live Research

Page 52: Generative Models

Research Project

g3

t1 t2 t3

e1 e2 e3

g1 g2 b

i

?

Page 53: Generative Models

Research Project

g3

t1 t2 t3

e1 e2 e3

g1 g2 b

i

?

Page 54: Generative Models

Research Project

g1 g1*≃?

Page 55: Generative Models

Modeling Surprise

g1 g1*≃?

Page 56: Generative Models

Competition

Chose top 5

Test how well they predict grades

Select a finalist (gets +)

TA Review

Actually re-grade

Publish?

Page 57: Generative Models

On worst pset question

Prize

+Due Tuesday before class (email staff. Subject:

Modeling Regrades)

Page 58: Generative Models

Novel Science

Page 59: Generative Models

http://vimeo.com/60381274

http://vimeo.com/60381274
Page 60: Generative Models

What does NSA do with our data?

Page 61: Generative Models
Page 62: Generative Models

Research Project

g3

t1 t2 t3

e1 e2 e3

g1 g2 b

i

?

Page 63: Generative Models

Can someone fix this?

Page 64: Generative Models

Peer Graders


Learning Deep Generative Models - MIT

Learning Deep Generative Models - MIT

Deep Generative Image Models using a Laplacian …papers.nips.cc/paper/5773-deep-generative-image-models...Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Deep Generative Image Models using a Laplacian …papers.nips.cc/paper/5773-deep-generative-image-models...Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Blackboard models, Generative Communication, Inference ...old.disco.unimib.it/simone/ldp_com/Linguaggi-april-2006.pdfBlackboard models, Generative Communication, Rule-base Inference

Blackboard models, Generative Communication, Inference ...old.disco.unimib.it/simone/ldp_com/Linguaggi-april-2006.pdfBlackboard models, Generative Communication, Rule-base Inference

Generative Models for Graph-Based Protein Design...Generative models for protein sequence and structure A number of works have explored the use of generative models for protein engineering

Generative Models for Graph-Based Protein Design...Generative models for protein sequence and structure A number of works have explored the use of generative models for protein engineering

Generative Models of Images of Objects

Generative Models of Images of Objects

One-Shot Generalization in Deep Generative Models · One-shot Generalization in Deep Generative Models aware generative models that make use of spatial trans-formers in this way.

One-Shot Generalization in Deep Generative Models · One-shot Generalization in Deep Generative Models aware generative models that make use of spatial trans-formers in this way.

Generative Adversarial Networks, and Applications Adversarial Networks, and Applications Outline: •Generative Models vs Discriminative Models (Background) •Generative Adversarial

Generative Adversarial Networks, and Applications Adversarial Networks, and Applications Outline: •Generative Models vs Discriminative Models (Background) •Generative Adversarial

Taxonomy of generative models - TUMTaxonomy of generative models Prof. Leal-Taixé and Prof. Niessner Figure from Ian Goodfellow, Tutorial on Generative Adversarial /networks, 2017

Taxonomy of generative models - TUMTaxonomy of generative models Prof. Leal-Taixé and Prof. Niessner Figure from Ian Goodfellow, Tutorial on Generative Adversarial /networks, 2017

Probabilistic generative models for machine vision

Probabilistic generative models for machine vision

Generative Models; Naive Bayes

Generative Models; Naive Bayes

Generative 3D Models

Generative 3D Models

Generative Models (other than GANs)alinlab.kaist.ac.kr/resource/AI602_Lec09_Explicit_generative_models.pdf · Generative Models (other than GANs) Algorithmic Intelligence Lab 1. Introduction

Generative Models (other than GANs)alinlab.kaist.ac.kr/resource/AI602_Lec09_Explicit_generative_models.pdf · Generative Models (other than GANs) Algorithmic Intelligence Lab 1. Introduction

Deep Generative Models

Deep Generative Models

Tutorial on Deep Generative Models - shakirm.com · Abstract This tutorial will be a review of recent advances in deep generative models. Generative models have a long history at

Tutorial on Deep Generative Models - shakirm.com · Abstract This tutorial will be a review of recent advances in deep generative models. Generative models have a long history at

IMPROVISATION: METHODS AND MODELS in: Generative ...

IMPROVISATION: METHODS AND MODELS in: Generative ...

Exploiting Generative Models in Discriminative Classifierspapers.nips.cc/paper/1520-exploiting-generative-models-in... · Exploiting Generative Models in Discriminative Classifiers

Exploiting Generative Models in Discriminative Classifierspapers.nips.cc/paper/1520-exploiting-generative-models-in... · Exploiting Generative Models in Discriminative Classifiers

Bayesian Inference, Generative Models, and …web.stanford.edu/.../BayesianInfGenModsProbCompsNNets.pdfBayesian Inference, Generative Models, and Probabilistic Computations in Interactive

Bayesian Inference, Generative Models, and …web.stanford.edu/.../BayesianInfGenModsProbCompsNNets.pdfBayesian Inference, Generative Models, and Probabilistic Computations in Interactive

Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Introduction to generative models - Mihaela Roscaelarosca.net/intro_generative_models.pdf · Introduction to generative models— Mihaela Rosca Generate new dataWhat do intelligent

Introduction to generative models - Mihaela Roscaelarosca.net/intro_generative_models.pdf · Introduction to generative models— Mihaela Rosca Generate new dataWhat do intelligent

Fingerprint Inpainting with Generative ModelsFingerprint Inpainting with Generative Models s1739461 s1313429 s1771851 (Group 107) Abstract We propose and test two generative models

Fingerprint Inpainting with Generative ModelsFingerprint Inpainting with Generative Models s1739461 s1313429 s1771851 (Group 107) Abstract We propose and test two generative models