Creating new classes of objects with deep generative neural nets

28
Akın Kazakçı MINES ParisTech PSL Research University, CGS-I3 UMR 9217 [email protected] Mehdi Cherti, Balázs Kégl CNRS, Université Paris-Saclay Centre for Data Science {mehdi.cherti, balazskegl}@gmail.com

Transcript of Creating new classes of objects with deep generative neural nets

Akın Kazakçı

MINES ParisTech PSL Research University, CGS-I3 UMR 9217

[email protected]

Mehdi Cherti, Balázs Kégl

CNRS, Université Paris-SaclayCentre for Data Science

{mehdi.cherti, balazskegl}@gmail.com

*Runco, M. A., and Jaeger, G. J. 2012. The standard definition of creativity. Creativity Research Journal 24(1):92–96.

Creativity is a process by which novel and valuable ideas are

produced*

This does not tell us anything about the relationship between

value and novelty

Relationship between value and novelty is critical

Empirical evidence suggests that, for genuinely novel objects, it’s hard to determine the value.

Especially, if the object is not there yet.

So, how to guide the generative process?

Relationship between value and novelty is critical - yet understudied

For instance, no design theory mentions explicitly “value” as a chief construct of its ontology (FBS, PSI, GDT, CK, CDP…)

The field of economics which deals explicitly with “value” does not consider generative processes.

In computational or psychometric studies of creativity either the value or the objects are known beforehand.

The “value of novelty” is a hard problem

- and it’s the blindspot of creativity research.*

* Kazakci, A. "Conceptive artificial intelligence: Insights from design theory.” International Design Conference, Dubrovnik 2014

Kazakci, A. Cherti, M, Kégl, B., “Digits that are not: Generating new types through deep generative nets” ICCC, Paris 2016

The Unknown Value Problem (UVP)

Assume you have a set of (existing) objects. Assume you know their value.

0

1

V(x)

x ∈ Sx1 x3x2 x4

The Unknown Value Problem (UVP)

0

1

V(x)

x ∈ Sx1 x3x2 x4

y* = V(x*)

Let S° = {x1, x2, x3, x4} ∈ S. How can we build an x* such that:Problem 1: y* = V(x*) is different than V(xi), for xi ∈ S°?Problem 2: y* = V(x*) is bigger than V(xi), for xi ∈ S°?

Bounding-box and Out-of-the-box

0

1

V(x)

x ∈ℝn

S1 ∈ S

Let S1 be the “bounding-box” that can be generated from S° by some generative process G: S1 = G(S°).

If we can model V(x) explicitly (that is, we have a knowledge model about value), then, for any xi ∈ S1, we know that the value will be within [Vmin, Vmax] with some distribution p(V(S1)) of value.

p(V(S1))

Vmax

Vmin

The magnitude of novelty can be defined as the shift in the posterior distribution (after generation). Note that, as we keep generating points, this magnitude vanishes, since we’ll approximate uniform distribution.

Bounding-box and Out-of-the-box

0

1

V(x)

x ∈S

If we want to go out-of-the-box, we have two problems:- We have to be able to generate x*- We no longer have a value function that can guide the generation of x*

x*

The difficulty in generating valuable novelty

64 pixels

64 pixels

In a 64x64 pixel space, the probability of generating this (or any) image by random selections of 0 or 1 for each pixel is:

2-4096 ≈ 0

In this space, images with any structure (under any meaningful definition of structure) is a very small subset of all the possible images.

An example: GAs without fitness (value) function

Successive panels in the sequence show populations of images generated by cross-over and mutation operators from the previous population.

Without a value function guiding the exploration of the unknown, everything become noise in just a few iterations.

Value function barrier• In most cases, the value function is an exogenous factor and

its extension over the novelty is not considered.

• For example, in evolutionary computational creativity systems, the value function is fixed and predetermined

• These fitness functions reflects system designer’s preference for novelty - not the machine’s.

• We want to try & get rid of the hard-coded value functions; let the system develop its own (Kazakçı, 2016)

A program to go beyond the value barrier

Any theory of creativity or design that does not explain how an agent can build its own value function for novelty

is incomplete

A program to go beyond the value barrier

We adopt a computational approach to study mechanisms by which such value functions may be built.

- Our main tools are deep generative neural nets

- DGNNs learn a hierarchy of transformations, given a set of objects

- These transformations are robust with respect to novelty

A program to go beyond the value barrier

1. Given a set of objects, can we generate new objects that are not white noise? Digits that are not: Generating new types through deep generative nets, ICCC, Paris 2016

2. What is a good generative model for novelty generation? Out-of-class novelty generation: an experimental foundation. ICLR 2017 (submitted, see openreview.net)

3. How does an agent can develop its own value function?(work in progress)

We adopt a computational approach to study mechanisms by which such value functions may be built.

Lear

nFo

rce

Fixa

te

Can we generate new images that are significantly different than existing images - yet still not white noise?

Auto-associative neural netsa.k.a auto-encoders

Learning to disassemble

Learning to build

- Auto-encoders have existed for long time (Kramer 1991)

- Deep variants are more recent (Hinton, Salakhutdinov, 2006; Bengio 2009)

- A deep auto-encoder learns successive transformations that decompose and then recompose a set of training objects

- The depth allows learning a hierarchy of transformations

The experimental setup

- We trained sparse convolutional auto-encoder (3c1d) on 70000 images of handwritten digits (MNIST) of size 28x28

- Training objective is to minimize the reconstruction error

Generating new symbols

- We use an iterative method to build images the system has never seen:- Start with a random image x0 = r, - and force the network to construct (i.e. interpret)- xk = f(xk-1), until convergence

- Our method is inspired by Bengio et al. (2013)- By contrast to them, we do not constrain the net to generate only

known types (we do not consider unknown symbols as spurious)

A map of the unknown

Coloured clusters are original MNIST digits (classes from 0 to 9)

The gray dots are newly generated objects

New objects form new clusters

Using a clustering algorithm, we recover coherent sets of new symbols

A distance preserving projection of digits to a two-dimensional space

(van der Maaten and Hinton 2008)

Suppose we ask a thousand children, who only know how to write digits, to write something different.

How would you evaluate these children on their ability to come up with coherent, and even meaningful symbols?

Please replace “children” in the above two sentences with “generative models”.

What is a good generative model for novelty generation?

Idea: Simulate the unknown

Unknown can be simulated by held-out classes, just like future data is simulated by held-out data in traditional machine learning

Train on known objects and types, test on types known to the experimenter but unknown to the model

Train on digits, test on letters

~1000 DGNNs: Autoencoders (sparse, contractive, denoising) and GANs, with a wide variety of hyperparameter values

+ a referee model: a discriminator between letters and digits (36 classes)

What are the models that are better at generating novelty?

Two measures: - Out of class count: The number of times a model has generated an image - evaluated as a letter by the discriminator

- Objectness (Salimans, 2016): Posterior entropy to distinguish noise from structure

Combining the two objectives

Yields images that are not digits, not random noise and convincing a specialist discriminator that they are letters.

27

PANGRAMS

hand-picked letters

top models found automatically

Reviewer: “For the "novel" samples in Fig. 3, they are clearly digits.”

Us: We kindly ask the reviewer to classify the symbols on the following panel into one of the digit categories 0-9. They are from Figure 3.

Thank you

Mehdi Cherti, Balázs Kégl {mehdi.cherti, balazskegl}@gmail.com

Akın Kazakçı [email protected]