cmu - University of Wisconsin–Madison

11/27/2011

1

from Art and Perceptionto Visualization and Video

(and Virtual Reality)

Michael Gleicher

Dept of Computer SciencesUniversity of Wisconsin ‐Madison

Acknowledgements• All of this work is done in collaboration with a great group of students and collaborators

• This talk is work done with students:

(who didn’t want their pictures shown)

Feng Liu – multimedia, video(work supported by NSF, Adobe)

Now at Portland State Univ

Greg Cipriano– molecules, vis(work supported by DOE,NIH)

Now at Solidworks

Aaron Bryden– molecule motion(work supported by NIH)Now at DOE Ames Lab

Danielle Albers – genomics(work supported by NSF, DOE)

Adrian Mayorga– vis(work supported by NIH)

Michael Correll – humanities(work supported by NSF)

Kevin Ponto– Virtual Environments(work supported by NIH)

Sean Andrist, Tomislav Pejsa– agents(work supported by NSF)

Michael Gleicher




Michael Gleicher




Michael Gleicher




Stuff I do,and why you might be interested

What do these have in common?

Analysis of Proteins

Scientific DataDisplay

Motion Synthesisfor Characters

Video Authoring

Image and VideoRetargeting

Non‐Verbal Cues forCommunicative Agents

11/27/2011

2


It’s all stuff I’ve done in the past few years

It involves large amounts of data

It involves creating effective presentations

It requires some understanding of the datain order to simplify it











Does this all tie together?

How can we use our understanding of human perception and artistic traditions

to improve our tools for communicating and data understanding?

11/27/2011

3


to improve our tools for communicating and data understanding.

Talk Roadmap

Molecular Surface Abstraction

Virtual Environment Replay Video Stabilization

Molecular Motions

Sequence Comparison



Molecular Motions

Sequence Comparison

Prelude: Art or Perception? A Protein Surface

Work with Greg Cipriano and George Phillips

An aside…

How do scientists look at proteins?

Stick and Ball Model (internals)

An aside…



Ribbon Diagram (internals)

11/27/2011

4

An aside…



Molecular Surface (externals)

A Protein Surface


Work with Greg Cipriano and George Phillips. TVCG 2007, NAR 2010.

What’s Happening?

Simplification

Stylized Display

Surface Indications

Art

• Abstraction

• Good Lighting

• Line Drawings

• Non‐Photorealism

Visual Cognitive Science

• Cue reduction

• Provide Depth Cues

• Enhance Contours

• Tolerance of Shading

Why does B fight cancer?

11/27/2011

5



Molecular Motions

Sequence Comparison

Inspired by Art (and a math trick) …

From Cutting, Representing motion in a static image, 2002

Molecular Motions

Bryden, Phillips, Gleicher. TVCG Jan ‘12

Molecular Motions

Coarse‐grained models

Normal‐mode Analysis (NMA)

Motion Illustration Motion Illustration

Artistic Inspirations:

Comic Books / Diagrams

Abstract

Model

Illustrate

11/27/2011

6

The Problem:

• Vector per point

Abstract

Group to fit models

Model

Illustrate

Abstract

Group to fit models

Model

Affine model per group

Illustrate x = A x

x = A x Abstract

Group to fit models

Model

Affine model per group

Illustrate

Affine Exponentials

Glyph Design

x = A x

x = A x

p(t) = eAt

p(t) = eAt

x = A x

x = A x

p(t) = eAt

p(t) = eAt

11/27/2011

7



Molecular Motions

Sequence Comparison

Learning from perception…

How do we interpret massive amounts of sequence data?

A tough problem…

TACTAGCTAGTAGCTAGCATCGACTACGACTGAC TACTAGCTAGTAGCTAGCATCGACTACGACTGAC

GCTAGCTGTTGCTAGCCGACTCATCGACTAAGCT

TCGACTAGCTAGATCGACTTATCGACTCACACTA

CTGGCTAGTTACACTATCTACCGACTGATCGACT

11/27/2011

8

TACTAGCTAGTAGCTAGCATCGACTACGACTGAC
























11/27/2011

9

Mauve sequence alignment and visualization. Perna Lab.

How to see anything in this sea of data?

11/27/2011

10

How to see anything in this sea of data?

1. Be Realistic2. New Designs based on Perception

Scalable Overviews of Whole Genome Sequence Alignments

Sequence Surveyor

Albers, Dewey, Gleicher. IEEE TVCG (InfoVis 2011)

Number of Genomes Length of Genomes Types of Inquiry

Scalable

Mauve (Perna lab) Sequence Surveyor

Visual Search

Visual Clutter Summarization

Pre-Attentive Phenomena

Perceptual Principles

Visual Search



11/27/2011

11

Visual Search


Pre-Attentive Phenomena Visual Search



Visual Search


Pre-Attentive Phenomena What are you searching for?

What textures form? What statistics do you get?

What pops out?

What are you looking for?

MappingColor Mapping Color Schemes Position Mapping

Index Membership Freq Grouped Freq Pos in Reference

Index

Grouped Freq

Pos in Reference

Combinations of different color and position mappings reveal interesting things in the data

11/27/2011

12

Blocking

Group (relatively) continuous sets of neighboring genes into a single unit

roftilS yaeQ phnA tadG

Aggregate Encodings

Average

Aggregate Encodings

Average Robust Average

Color Weaving Event Striping

Anecdotes: 100 Bacteria

Conservation relationships between different families of genomes

Color by position in reference (arrow), order by relative ordering

Anecdotes: Buchnera

Color by position in reference (arrow), order by set of genomes containing each gene

Anecdotes: Buchnera

Averaging:

No significant trend

Color Weaving: Overall distribution

11/27/2011

13

Anecdotes: Fungi

Bioinformatics applications allow users to test algorithms using visual checks

Color by overall frequency, order by relative ordering

Anecdotes: Fungi

Bioinformatics applications allow users to test algorithms using visual checks

Color by position in a reference, order by relative ordering

Another Application

Since I can’t pronounce the names of the Fungi …

A Different Kind of Evolution . . .

Google Books Word Count DataDecades ‐>

Word Rank ‐> Color by position in reference

A Different Kind of Evolution . . .

Google Books Word Count Data

Decades ‐>

Word Rank ‐> Color by occurrence count

11/27/2011

14

Visualizing English Printfrom 1470 to 1800

With Mike Witmore, Robin Valenza, Michael Correll, …

Domain Specific Tools

Correll, Witmore, and Gleicher. Eurovis 2011.

Back to the details that cause patterns

Lots of fun domains! Visual ComparisonsA Common Thread?

11/27/2011

15

There are general principles that apply across domains, data types, …

Maybe provocative?

And if we can figure it out, it’ll be easier to crank out the comparison tools/techniques quickly

Mike’s theory of visual comparison 0.2*

* This is a work in progress.Comments welcomed!

The 3 3s

• 3 Axes of Scalability / Hardness

• 3 Strategies for Scalability

• 3 Basic Designs

Comparison is easy. Until it gets hard.

3 Axes of Hardness

• Number of things to compare

• Size/complexity of things to compare

• Complexity of the relationships

People can only compare a few things at a time*

3 Strategies for scaling

• Select Subset

• Summarize Statistically

• Scan Serially

My cognitive scientist friends say the magic number 7+/‐2 is an over‐simplification, but …

All designs appear to fall into 3 categories

3 Basic Designs*

* Each has its pros and cons

11/27/2011

16

Sequence Comparison


Virtual Environment Replay

Molecular Motions

Video Stabilization

Using Perception and Art …

3D or not 3D, that is the question

From Shakespeare to Video Processing

Video Stabilization

Problem:

Shaky video in, less shaky (good?) video out

Art (and Perception)

What is good camera movement?

Perception (and Art)

How can we avoid impossible computer vision?

Three Projects

Re‐Cinematography

What can you do beyond removing jitter?

Stabilization by 3D Warping

How can you make bigger changes?

Stabilization by Subspace Constraints

How do you make it practical?

Work with Feng Liu and Adobe

Lessons from Filmmaking 101

Direct the viewer’s attention

Use a tripod! (a damped one)

Avoid distracting the viewer (unless…)

Camera movements should be motivated

Smooth movements to make connections

Pittsburgh Filmakerscirca 1991

Sam’s First Steps, July 6th, 2006

Skip

11/27/2011

17

The rule of cinematography

Camera movement should be intentional

Re‐Cinematography:1. Figure out intent2. Make it be so

The key insight:Translate cinematography to implementation

Motion should be intentional

Static shots should be static

Moving shots are goal directed

Constant velocity with ease in/out

Re‐Cinematography “Works”Velocity profiles meet goals

Frame number

Magnitude of apparent velocity

Source video

Result video

Static segments are static

Moving segments have piecewise constant velocity

Ease in and out

11/27/2011

18

Problems

Show source images when motion estimation fails

Visual Artifacts from bad inpainting

2D Stabilization limits how much change

Video Stabilization

Existing approach: 2D stabilization

Track a bunch of pointsFit full‐frame warps that best smooth point motion

Limited!

Doesn’t model parallax

Can’t move the camera very far

To Swing or Not to Swing(big changes require inpainting)

Source Footage Re‐Cinematography Result

Temporal Artifacts

Where did she come from?

Input: Output:

Beyond 2D Stabilization

11/27/2011

19

3D Video Stabilization: Pipeline

3D reconstruction viaStructure from motion

Novel view synthesis

Camera path planningLine Parabola

Low‐pass filter

3D Reconstruction from VideoStructure from Motion

Voodoo Camera Tracker: http://www.digilab.uni-hannover.de/docs/manual.htm

Camera Path PlanningMove the camera to a new place

Line Parabola

Low‐pass filter3D experienceswithout 3D models

How do you move the camera to a different place?

Liu et al. SIGGRAPH 2009.

Novel viewpoint rendering

How to make a video‐quality image?

Build a really high‐quality 3D model?Too hardNot enough input data

Image‐Based Rendering using other frames?Violates temporal constraintsComputationally expensive

Avoid Temporal Artifacts:Need Novel view from One Frame

Where did she come from?

Input: Output:

11/27/2011

20

Novel view from one frame

Impossible?

Incomplete geometric model (sparse)

Occlusions / Dis‐occlusions

Novel view from one frame

Impossible?

Incomplete geometric model (sparse)

Occlusions / Dis‐occlusions

Impossible! So Fake it!

Just need visually plausible, not accuracy

Viewpoint shifts will be small

Avoid artifacts

3D Stabilization by Image Warping

• Structure from motion gives sparse points

• 3D camera planning gives motion of points

• Use sparse points to warp image

Content‐Preserving Warps

Warp each input frame to create the output frame by least‐squares minimization Data term: Soft, sparse displacement constraint

Smoothness term: Local similarity transformation constraint

Weightings: Higher penalties in salient regions

119

Results & ComparisonsIs this a good reconstruction? NO!

Camera position Output points

11/27/2011

21

One frame isn’t so bad…

Especially if you crop

Video Stabilization withSubspace Constraints

Act III

Limitations Limitations

11/27/2011

22

What do we need 3D for?

Plausibility Constraints

Low‐rank constraint:

[Tomasi and Kanade1992, Irani 2002]

The trajectory matrix of a rigid scene imaged by amoving camera over a short period of time shouldapproximately lie in a low‐dimensional subspace.

11/27/2011

23

• Incomplete Moving Matrix Factorization

• Filter the “Eigen‐Trajectories”


Video Stabilization

Molecular Motions

Sequence Comparison

Virtual Environment Replay

Learning from perception…

Cinematography in our tools?

Re‐Cinematography

Model of how camera should move

(couldn’t move camera enough)

3D Stabilization

Potential to reason about camera movments

(limited to simple clips)

Subspace Stabilization

Practical!

(only filtering)

Camera Stabilization forVirtual Worlds

Back to 3D…

11/27/2011

24

Why? The Home is where Health Happens

Increasingly

Homes are different!

Traditional Laboratories may not be adequate

11/27/2011

25

Visual Simulation?

How to review Virtual Environment Experiences?

How do we use Visual Simulation as a Behavioral Research Tool?

Ponto, Kohlmann, and Gleicher. Effective Replays and Summarization of Virtual Experiences. IEEE TVCG (VR 2012, accepted, to appear).

Head Tracked Virtual Experiences

What did they see?

Where were they “looking”?

A (3D) Video Stabilization Problem?

Easier than 3D Stabilization:

Have camera path

Have world geometry (to render novel views)

Higher Expectations!

More power for analysis

More flexibility for re‐synthesis

11/27/2011

26

Key Ideas

Cinematography Modelfrom Re‐Cinematography

stable points + interpolations

Content‐Dependent Metriccameras see the same things?

efficient GPU implementation

Segmentationfind key “fixations”

serves as compact summary

Path Generationcinematic smooth arcs

exp. Coord interpolation

Reproject Original Viewpoint

Key Ideas










Key Ideas










Key Ideas









Reproject Original Viewport

Key Ideas









Reproject Original Viewport

11/27/2011

27

EvaluationThree Initial Studies

Objective: VR Participant Viewed Objects

Subjective: Viewer Prefers

Objective: Viewer Comprehends



Molecular Motions

Sequence Comparison


to improve our tools for communicating and data understanding?

Thanks!

• To you for listening

• To my students and collaborators

• To the folks who pay the bills

(NSF, NIH, DoE, …)

Michael [email protected] of Computer SciencesUniversity of Wisconsin ‐Madison

cmu - University of Wisconsin–Madison

Documents

Transcript of cmu - University of Wisconsin–Madison