Learning spline-based curve models (Laure Amate)

Learning spline-basedcurves models

L. Amate

Some definitions

Goal: learning curvesmodel

Goal: “simple”representation

Collective splinemodeling

Pb statement

Criterion

EM approaches

Some definitions

Monte-Carlo onlineEM

Results

Conclusion

Learning spline-based curves models

Laure Amate

MISTIS(INRIA-LJK Grenoble)& LIG

Seminaire BigMC – 27 mai 2010

1 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Overview

1 Some definitionsGoal: learning curves modelGoal: “simple” representation

2 Collective spline modelingproblem statementCriterion

3 EM approachesSome definitionsMonte-Carlo online EM

4 Results

5 Conclusion

2 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Concept of class for curves

learning a model from available objects

3 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Concept of class for curves

learning a model from available objects

Characterizing a group

C = {cj(t)}Mj=1, set of contours

probabilistic approach : cj ∼ p(c), unknown

determination of an estimate p(c)

4 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

”simple” representation

sampling

segments + arcs

ellipsoids

5 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Spline curves

adaptivity to the data

sparse representations (a few parameters)

6 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Spline curves



piecewise continuouspolynomials of order m

s(t) : [0, 1] → R2

0 0.2 0.4 0.6 0.8 10

0.5

1

1.5

2

2.5

3

3.5

4

knots (limits of pieces)

∀ξ ∃ B-spline basis {bmi (t; ξ)}ki=1: s(t) =

k∑

i=1

βibmi (t; ξ)

7 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Spline curves



ξ ↔ Mk probabilistic simplexβi ∈ R2 ↔ C ⇒ β1:k ∈ Ck

θ = (k , β1:k , ξ1:k) ∈ K × Ck ×Mk

︸︷︷︸

Θk

s(ti )Ni=1 → θ

2N → 3k + 1

8 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Choice of ξ

c(t) is not a spline → approximative representation

1) Quality ր with k

9 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Choice of ξ



50 100 150 200 250 300

0

50

100

150

200

Spline subspace

of dimension 10

10 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Choice of ξ



50 100 150 200 250 300

0

50

100

150

200

Spline subspace

of dimension 25

Uniform knots

11 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Choice of ξ



50 100 150 200 250 300

0

50

100

150

200

Spline subspace

of dimension 25

Uniform knots

⇒ we need to adapt k to the complexity of c(t)to capture the relevant morphological features of c(t) 12 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Choice of ξ



2) Quality ր with well-chosen ξ

50 100 150 200 250 300

0

50

100

150

200

Spline subspace

of dimension 25

Uniform knots

13 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Choice of ξ




50 100 150 200 250 300

0

50

100

150

200

Spline subspace

of dimension 25

Free-knots

14 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Choice of ξ




50 100 150 200 250 300

0

50

100

150

200

Spline subspace

of dimension 25

Free-knots

⇒ we need to adapt ξ to c(t) (for same k)

15 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Representation space = varying complexity free-knotssplines space

s(t) =

k∑

i=1

βibmi (t; ξ)

Θ =⋃

k∈K Θk

→ Θ is not a vector space→ Nested models family

· · · ⊂ Sk1 ⊂ Sk1+1 ⊂ Sk1+2 ⊂ · · ·

Sk , family of free-knots splines models with fixed k

16 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Overview




4 Results

5 Conclusion

17 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Collective spline modeling





18 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion






c(t) = s(t) + ε =⇒ c |θ ∼ N (s, σ2I)

p(c) =

∫

Θp(c |θ)p(θ)dθ

19 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion






c(t) = s(t) + ε =⇒ c |θ ∼ N (s, σ2I)

p(c) =

∫

Θp(c |θ)p(θ)dθ

k fixedParametric model: p(θ) = p(θ|γ)βj |ξj , σ

2 ∼ N (µ0,Σ(ξj , σ2))

ξj ∼ Dir(α)

}

⇒ γ = (µ0, α, σ2)

20 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Model structure

σ2

µ0

α ξj

βj sj cj

21 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Model structure

σ2

µ0

α ξj

βj sj cj

Problem

From {cj}Mj=1, estimating γ

22 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Problem


1) ”Decoupled” approach:{cj}Mj=1 →

{

θj

}M

j=1→ γ

23 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Problem



{

θj

}M

j=1→ γ

cj → (βj , ξj) non linear estimation pb=⇒ MCMC methods (Metropolis-Hastings)

γ = argmaxγ∈G

p({

θj

}M

j=1|γ)

In general, θ not sufficient statistics →information loss

24 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Problem



{

θj

}M

j=1→ γ

cj → (βj , ξj) non linear estimation pb=⇒ MCMC methods (Metropolis-Hastings)

γ = argmaxγ∈G

p({

θj

}M

j=1|γ)

In general, θ not sufficient statistics →information loss

2) {θj}Mj=1: unobserved variables

25 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Criterion

Marginal Max. likelihood criterion

γ = argmaxγ∈G

p({cj}Mj=1 |γ)

= argmaxγ∈G

∫

· · ·

∫

p({cj}Mj=1 , {θj}

Mj=1 |γ)dθ1 · · · dθM

26 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Criterion

Marginal Max. likelihood criterion

γ = argmaxγ∈G

p({cj}Mj=1 |γ)

= argmaxγ∈G

∫

· · ·

∫

p({cj}Mj=1 , {θj}

Mj=1 |γ)dθ1 · · · dθM

no analytical solution → numerical method

⇒ Expectation-Maximization algorithm

27 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Overview




4 Results

5 Conclusion

28 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

EM algorithm

2-steps iterative method:

Expected value of complete data likelihood:Q(γ|γ(t)) = Eθ

[log p(c , θ|γ)|c , γ(t)

]

Maximization of the complete data likelihood:γ(t+1) = argmaxγ∈G Q(γ|γ(t))

local convergence

”hill climbing” algorithm

29 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Exponential family

Case of exponential family:

p(c , θ|γ) = h(c , θ) exp (ℓ(S(c , θ), γ))

ℓ(s, γ) = −Ψ(γ) + 〈s,Φ(γ)〉

(E)-step: s(c , γ(t−1)) = Eθ

[S(c , θ)|c , γ(t−1)

]

(M)-step: γ(t) = argmaxγ∈G

ℓ(s(c , γ(t−1)), γ)

30 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Monte-Carlo EM algorithm

No anaytical expression for Q(γ|γ(t))

Stochastic approximation:{θj}M

j=1∼ p(θ|c , γ(t−1)),

Q(γ|γ(t)) ≈ 1M

∑M

j=1 log p(c , θ(j)|γ)

M ր with iteration: M(i) = ip, p > 1

Convergence: established for curved exponential families[Fort & Moulines, 2003]

31 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Online EM algorithm

Sequential process of data: [Cappe & Moulines, 2009]

1 iteration ↔ 1 observation (1 curve)

(E)-step: s(ci , γ(i−1)) = Eθi

[S(ci , θi )|ci , γ

(i−1)]

(online)-step: si = si−1 + ηi(s(ci , γ

(i−1))− si−1

)

(M)-step: γ(i) = argmaxγ∈G

ℓ(si , γ)

ηi ց with iteration: ηi = η0i−κ, κ ∈]1/2, 1[, η0 ∈ [0, 1]

Convergence: established for exponential families [Cappe &

Moulines, 2009]

γ1 γ2 · · · γ

c1 c2 · · ·

s1 s2 · · ·

s1 s2 · · ·

32 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Monte-Carlo online EM algorithm

γ1 γ2 · · · γ

c1 c2 · · ·

MC MCs1 s2 · · ·

s1 s2 · · ·

33 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Monte-Carlo online EM algorithm

i − th iteration:1 MC approximation:

{

θji

}M i

j=1∼ p(θi |ci , γ

(i−1)),

s(ci , γ(i−1)) ≈ 1

M i

∑M i

j=1 S(ci , θji )

2 Online step: si = si−1 + ηi(s(ci , γ

(i−1))− si−1

)

3 Maximization step: γi = argmaxγ∈G

ℓ(si , γ)

Numerical method (gradient)

34 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Overview




4 Results

5 Conclusion

35 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Results : simulated data

−4 −3 −2 −1 0 1 2 3 4 5−6

−4

−2

0

2

4

6

8

36 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Different proposals for MC sampler:

−4 −3 −2 −1 0 1 2 3 4 5−6

−4

−2

0

2

4

6

8

0 2 4 6 8 10 120

10

20

30

40

50

60

red: simulated datablue: Dir(α)green: Dir(1)magenta: 1 rand knot + triangular distribution between neighbours

37 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


With different initializations:

−4 −3 −2 −1 0 1 2 3 4 5−6

−4

−2

0

2

4

6

8

0 2 4 6 8 10 120

5

10

15

20

25

30

red: simulated datablue: good convergencegreen: local convergence → identifiability pb.

38 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion


Identified models samples

−4 −3 −2 −1 0 1 2 3 4 5−6

−4

−2

0

2

4

6

8

−4 −3 −2 −1 0 1 2 3 4 5−6

−4

−2

0

2

4

6

8

−4 −3 −2 −1 0 1 2 3 4 5−6

−4

−2

0

2

4

6

8

−4 −3 −2 −1 0 1 2 3 4 5−6

−4

−2

0

2

4

6

8

−4 −3 −2 −1 0 1 2 3 4−6

−4

−2

0

2

4

6

8

−4 −3 −2 −1 0 1 2 3 4 5−6

−4

−2

0

2

4

6

8

39 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Real data: leaves

Model selection criterion to identify the complexity:M1 : k = 30 M2 : k = 15

Learning sets: L1 with 66 leaves, L2 with 33 leaves

Test set: 51 leaves with 33 from C1 and 18 from C2

Classification (likelihood) for curves from the test setk1 = 15 & k2 = 30

HHH

HHHH

Modelclass.

Realclass

C1 C2

M1 33 0

M2 0 18

k1 = k2 = 15

HHHHH

HH

Modelclass.

Realclass

C1 C2

M1 2 31

M2 0 18

40 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Overview




4 Results

5 Conclusion

41 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Conclusion & future work

Conclusion

probabilistic model for a set of curves

new variant: MC online EM

Future works

Solve the identifiability issue

Compare results with another method. Which one ?

Establish convergence properties of MC online EM

Introduce the complexity of the model in the collectivemodeling problem

Develop links with “Shape” theory

42 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

THANK YOU !ANY QUESTIONS ?

43 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Details: curved exponential family

p(c , θ|γ) = h(c , θ) exp (ℓ(S(c , θ), γ))

ℓ(s, γ) = −Ψ(γ) + 〈s,Φ(γ)〉

h(c , θ) = N|BTB |

Ψ(γ) = (N + k) log(2πσ2) + log(B(α))

S(c , θ) =

log(N)

(C − Bβ)H(C − Bβ) + βHBTBβN

ℜ

(

BTBβN

)

ℑ

(

BTBβN

)

Vec(

BTB/N)

Φ(γ) =

α− 1

− 12σ2

ℜ

(

µ0σ2

)

ℑ

(

µ0σ2

)

−Vec

(

µ∗0 µ

T0

)

2σ2

44 / 45


L. Amate

Some definitions




Pb statement

Criterion

EM approaches

Some definitions


Results

Conclusion

Details: computation of s

s(c, γ) =

∫

log(∆)q(∆|c, γ)d∆

CHC + kσ2 − 2ℜ(

∫

CHβϕq(∆|c, γ)d∆)

+ N+1N

∫

ϕHBTBϕq(∆|c, γ)d∆

ℜ

(

∫

BT BN

ϕq(∆|c, γ)d∆

)

ℑ

(

∫

BT BN

ϕq(∆|c, γ)d∆

)

∫

Vec(BTB)q(∆|c, γ)d∆

ϕ = NN+1

(BTB)−1BT(

C +Bµ0N

)

q(∆|c, γ) =exp

[

−T (c,∆,γ)

2σ2

]

p(∆|γ)

∫

exp

[

−T (c,∆,γ)

2σ2

]

p(∆|γ)d∆

T (c,∆, γ) = CHC + 1Nµ0B

TBµ0 − NN+1

(

C +Bµ0N

)HB(BTB)−1BT

(

C +Bµ0N

)

45 / 45

Learning spline-based curve models (Laure Amate)

Technology

Transcript of Learning spline-based curve models (Laure Amate)