Person: Pick up a big red block.

Computer: OK.

Person: Grasp the pyramid.

Computer: I don't understand which pyramid you mean.

Person (changing their mind): Find a block which is taller than the one

you are holding and put it into the box.

Computer: By "it", I assume you mean the block which is taller than

the one I am holding.

Computer: OK.

Person: What does the box contain?

Computer: The blue pyramid and the blue block.

Person: What is the pyramid supported by?

Computer: The box.

color(red) There is a room with a table and a cake.

There is a red chair to the right of the table.

a) Explicit Constraints

right(o2,o1)

o3

cake

o0

room

o1

table

o2

chair

Parsecolor(red)

b) Inferred Scene Template

supports(o0,o1) supports(o0,o2)

right(o2,o1)

o3

cake

o0

room

o1

table

o2

chair

supports(o1,o4)

supports(o4,o3)o4

plate

Infer

color(red) There is a room with a table and a cake.

There is a red chair to the right of the table.

a) Explicit Constraints

right(o2,o1)

o3

cake

o0

room

o1

table

o2

chair

Parsecolor(red)

b) Inferred Scene Template

supports(o0,o1) supports(o0,o2)

right(o2,o1)

o3

cake

o0

room

o1

table

o2

chair

supports(o1,o4)

supports(o4,o3)o4

plate

Infer

c) 3D Scene

Ground

Layout

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

supports(o1,o3)

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

o2

room

o3

plate

supports(o1,o3)

supports(o3,o0)

o4

fork

supports(o1,o4)

o1

tablematerial(wood)

o0

cake

supports(o1,o3)

𝐶𝒐 𝐶𝒔

𝐶𝒐 𝐶𝒔

𝐶𝑝 𝐶𝑐

𝑷𝒄𝒉𝒂𝒊𝒓|𝒅𝒊𝒏𝒊𝒏𝒈 𝒕𝒂𝒃𝒍𝒆(𝒙, 𝒚, θ)

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

o2

room

o3

plate

supports(o1,o3)

supports(o3,o0)

o4

fork

supports(o1,o4)

o1

tablematerial(wood)

o0

cake

supports(o1,o3)

SceneType =𝑪𝒔𝑃(𝐶𝑠|𝑪𝒂𝒌𝒆, 𝑻𝒂𝒃𝒍𝒆)

SceneType =𝑪𝒔𝑃(𝐶𝑠|𝑪𝒂𝒌𝒆, 𝑻𝒂𝒃𝒍𝒆)

Sample 𝑃(𝐶𝒐|𝑫𝒊𝒏𝒊𝒏𝒈𝑻𝒂𝒃𝒍𝒆)

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

supports(o1,o3)

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

o2

room

o3

plate

supports(o1,o3)

o4

fork

𝑃𝑠𝑢𝑝𝑝𝑜𝑟𝑡(𝐶𝑝|𝐶𝑐)

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

o2

room

o3

plate

supports(o1,o3)

o4

fork

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

o2

room

o3

plate

supports(o1,o3)

supports(o3,o0)

o4

fork

supports(o1,o4)

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

o2

room

o3

plate

supports(o1,o3)

supports(o3,o0)

o4

fork

supports(o1,o4)

There is a piece of cake on the wooden table.

There is a piece of cake on the wooden table.

o1

tablematerial(wood)

o0

cake

o2

room

o3

plate

supports(o1,o3)

supports(o3,o0)

o4

fork

supports(o1,o4)

There is a piece of cake on the wooden table.

Room

Desk Chair

Lamp

Room

Desk Chair

Computer

http://nlp.stanford.edu/projects/text2scene.shtml