Geometry-Aware Neural Rendering

Josh Tobin, OpenAI Robotics, Pieter Abbeel

Josh Tobin Geometry-Aware Neural Rendering

How to model complex robots scenes?

!2

From… …To

Josh Tobin Geometry-Aware Neural Rendering

Model the state of all objects?

!3

• Scales with scene complexity

• How to deal with complex internal state?

• How to deal with out-of-distribution?

Josh Tobin Geometry-Aware Neural Rendering

Only use the state implicitly?

!4

• Can be data inefficient

• May require learning from scratch (which can be dangerous)

• Often lacks reusability

Josh Tobin Geometry-Aware Neural Rendering

Model the 3D structure of the scene?

!5

• High-dimensional representations - scale poorly with scene detail

• No notion of semantics

Josh Tobin Geometry-Aware Neural Rendering

Model the 3D structure implicitly?

!6

Random viewpoints Model

Arbitrary “query”

viewpoint

The Neural Rendering problem

Josh Tobin Geometry-Aware Neural Rendering

Motivation: Generative Query Nets

!7

Eslami, SM Ali, et al. "Neural scene representation and rendering." Science 360.6394 (2018): 1204-1210.

Josh Tobin Geometry-Aware Neural Rendering

Motivation: Generative Query Nets

!8

Eslami, SM Ali, et al. "Neural scene representation and rendering." Science 360.6394 (2018): 1204-1210.

Josh Tobin Geometry-Aware Neural Rendering

Key questions

!9

• Can it scale to high(er)-dimensional images (GQN is 64x64)?

• Does it work for objects with complex state?

• Does it work for a wide range of realistic objects?

• Is it useful for downstream robotics tasks?

Josh Tobin Geometry-Aware Neural Rendering

Potential limitations

!10

• Scene representation is a sum — each feature contains only local information

• Rendering process cannot interact with the full representations (except through backprop)

Josh Tobin Geometry-Aware Neural Rendering

Background: Epipolar Geometry

!11

Josh Tobin Geometry-Aware Neural Rendering

Our Approach

!12

attn attnattnattn attn attnepi

r1

v1

f

e1epi

r1

v1

f

e1

vq

epi

r1

v1

f

e1

attn

clstm h 1h 0 h L

z1

attn

⋯ clstm

zL

attn ⋯

Our ModelPrevious Method Ground Truth

r

1

23

4Representation network

Attention mechanism

Generation network

loss

clstm

z2

Josh Tobin Geometry-Aware Neural Rendering

Epipolar extraction

!13

hl rk

ek

vqvk

Josh Tobin Geometry-Aware Neural Rendering

Attention mechanism

!14

akl

h l−1

1 × 1 × 1 1 × 11 × 1 × 1

ek

h ′� × w′� × d′�h ′� × w′� × h ′� × d′�

h ′� × w′� × h ′� × dv h ′� × w′� × h ′� × dk h ′� × w′� × dk

h ′�w′� × h ′� × dk h ′�w′� × dk × 1

softmax

h ′�w′� × 1 × h ′�h ′�w′� × h ′� × dv

h ′�w′� × dv

h ′�w′� × h ′�

h ′� × w′� × dv

1 × 1

akl

QlKk

Vk

Josh Tobin Geometry-Aware Neural Rendering

Does it help?

!15

Josh Tobin Geometry-Aware Neural Rendering

Examples

!16

Context TargetGQN E-GQN

Josh Tobin Geometry-Aware Neural Rendering

Examples

!17

Context GQN E-GQN Target

Josh Tobin Geometry-Aware Neural Rendering

Examples

!18

Context GQN E-GQN Target

Josh Tobin Geometry-Aware Neural Rendering

Conclusion

!19

• Geometrically-inspired neural network primitives improve implicit 3D understanding

• Forcing the model to understand geometry can improve downstream robotic tasks

• How to go from this to general 3D understanding?