Wasserstein distances in the analysis of time series and...

Universiteit Utrecht, February 29, 2016

Wasserstein distances in the analysis of time series and

dynamical systems

Sjoerd Verduyn Lunel (Utrecht University)

[email protected]

– Typeset by FoilTEX – 1

Abstract

A new approach based on Wasserstein distances, which are numerical costs of anoptimal transportation problem, allows to analyze nonlinear phenomena in arobust manner.

The long-term behavior is reconstructed from time series, resulting in a probabilitydistribution over phase space. Each pair of probability distributions is thenassigned a numerical distance that quantifies the differences in their dynamicalproperties.

From the totality of all these distances a low-dimensional representation in aEuclidean space is derived. This representation shows the functional relationshipsbetween the dynamical systems under study. It allows to assess synchronizationproperties and also offers a new way of numerical bifurcation analysis.


Discrete Dynamical Systems

Let X ⊂ Rd en f : X → X and consider the discrete dynamical system

xn+1 = f(xn)

starting from a given initial point x0.

A set A is called attracting with respect to U if for every neighborhood V of A,there exists a K = K(V ) such that fk(U) ⊂V for all k ≥ K.

The basin of attraction of A is defined by

B(A) = ∪k≥0f−k(U).

If B(A) = X, then we call A a (global) attractor.

A trajectory x = (x1, . . . , xN) generated by the dynamical system modeled by fcan be viewed as a time series.

In this talk we consider time series generated by f as well as generated directlyfrom measurements.


Intermezzo

Consider the iterative process

xn+1 =xn2

+1

xn.

This process more than 4000 years old and nowadays is known as an application ofNewton’s method.

Suppose that we start with x0 = 1 then

x1 =x02

+1

x0=

3

2.

And the next step

x2 =x12

+1

x1=

3

4+

2

3=

17

12.

Continuing this procedure yields the following table


Intermezzo II

n xn approximation0 1 11 3/2 1.5000000002 17/12 1.4166666673 577/408 1.4142156864 665857/470832 1.4142135625 1.414213562

Notice that according to your calculator:√

2 = 1.414213562.

n xn approximation0 51 511 2603/102 25.519607842 12.798989483 6.4776259074 3.3931905015 1.9913031376 1.4978352807 1.4165477958 1.4142154869 1.414213562


Intermezzo III

Let a a number greater than zero and consider the iterative process

xn+1 =xna

+a− 1

xn.

Choose your favorite a, start with x0 = 1, and compute x2, x3 and x4.

There are also other interesting iterating systems. Let r be a number between 1en 4 and consider the iterative process

xn+1 = rxn(1− xn).

Choose your favorite r, start with x0 between 0 and 1, for example, x0 = 1/2, andcompute x2, x3 and x4.


The Henon map

The this talk the running example in case the underlying dynamical system isknown will be the Henon map.

The Henon map is defined by(xn+1

yn+1

)= H

(xnyn

), n ≥ 0,

where x0 en y0 are given initial conditions and H : R2 → R2 is given by(xy

)7→(

1 + y − ax2bx

).

Here a en b are parameters. This map was introduced in 1976 by Henon to study(strange) attractors. If a = 1.4 and b = 0.3 the system exhibits chaotic behavior.

Time series data from a real world system implicitly defines a dynamical systemgiven by the information contained in repeated measurements, and delaycoordinate reconstruction allows to represent these trajectories in Euclidean space.


Intermezzo IV

The aim is to apply our algorithm directly to data, i.e., measurements as timeseries.

But we can also use a model to produce a time series. Such a time series we call asynthetic time series, and allows us to test our algorithm theoretical.

Consider the Henon map H. Start with x0 = 1 en y0 = 1 and a = 1.4 andb = 0.3. Compute x1 and y1 by(

x1y1

)=

(1 + 1− 1.4(1)2

0.3

)=

(0.60.3

)and x2 en y2 by (

x2y2

)=

(1 + 0.3− 1.4(0.6)2

0.18

)=

(0.5040.18

)We can plot the points (

x0y0

),

(x1y1

),

(x2y2

), . . .

in the plane. As in the√

2 example, this sequence approaches a geometricalobject: the attractor.


Intermezzo V

We can plot the attractor by plotting the tails of a long time series.

From the theory, it however follows that we can reconstruct the attractor alsofrom a piece of the tail of time series given by only by the x coordinate of thepoints (x, y).

This method, plotting blocks of the tail of a time series, we can also do if we onlyhave time series from measurements.

This approach will be explained next.


Attractor reconstruction I

Given a time seriesx = (x1, . . . , xN)

of N measurements of a single observable X, a trajectory of a dynamical systemis reconstructed by mapping each consecutive block

x[i] = (xi, xi+q, . . . , xi+(k−1)q)

of k values, sampled at discrete time intervals q, into a single point x[i] in aEuclidean space Ω = Rk. This map defines the delay-coordinate map F into thereconstructed phase space Ω.

Sauer, Yorke and Casdagli (1991) extending work by Takens (1981) showed, undermild assumptions, that almost every delay-coordinate map F : RN → Rk isone-to-one on A provided that the embedding dimension k is larger than twice thebox counting dimension of A. Also, any manifold structure within A will bepreserved in F (A).

The optimal value of the lag q and of the embedding dimension k is a prioriunknown but can be estimated from the data.


Attractor reconstruction II

This methodology of attractor reconstruction by delay embedding


Attractor reconstruction III

The result of the embedding process is a discrete trajectory in reconstructed phasespace Ω = Rk.

This trajectory can be interpreted as a probability measure µ on Ω, where

µ[A] =1

N ′

N ′∑i=1

δx[i][A], A ⊆ Ω,

is the time average of the characteristic function of the points in phase spacevisited.

In the limit N ′ →∞ the measure µ is invariant under the dynamics and undermild assumptions the invariant measure is unique and its support contains anattractor (Lasota and Mackey, 2004).

If a dynamical system underlying the time series is available, subdivision methodsallow to approximate the attractor and its natural measure with arbitrary precision.

We shall illustrate this for the Henon map.


The Henon map - a time series

For the parameters a = 1.0 en b = 0.54 and the initial condition x0 = 0.1 eny0 = 0.0 the time series for the x-coordinate is given by

0 500 1000 1500 2000 2500 3000

−2

−1

01

2

Time

x−Value

Henon system: Time series #1

0 20 40 60 80 100

−2

−1

01

2

Time

x−Value

Henon system: Time series #1 (detail)

Figuur 1: Time series of the x-coordinate with x0 = 0.1 en y0 = 0.0

The time series consists of 3000 iterations and is relatively short - this is thesituation one often encounters in practice.


The Henon map – reconstruction of the attractor

Reconstruction of the attractor using the generated time series

−2 −1 0 1 2

−2

−1

01

2

x[n]

x[n+1]

Henon system: Reconstructed attractor #1

Figuur 2: The attractor corresponding to the parameters a = 1.0 en b = 0.54.

The first 1000 entries of the time series are considered transient and the delayreconstruction is based on the entries [1000-3000].

Zie Kant/Schreiber, Nonlinear time series analysis, Cambridge.


The Henon map - reconstruction of the attractor

Here are some reconstructed attractors for different parameter values.

−2 −1 0 1 2

−2

−1

01

2

x[n]

x[n+1]


−2 −1 0 1 2

−2

−1

01

2

x[n]

x[n+1]


Figuur 3: The reconstructed attractor for, respectively, a = 1.0,b = 0.52 en a = 1.0,b = 0.5065.

Note that the chaotic behavior disappears for last set of parameter values and thatwe have periodic behavior.


The Henon map - the subdivision algorithm

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−3

−2

−1

0

1

2

3

x

y

Subdivision #4 of Henon attractor

Student Version of MATLAB

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−3

−2

−1

0

1

2

3Subdivision #6 of Henon attractor

yx


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−3

−2

−1

0

1

2

3

x

y

Subdivision #8 of Henon attractor


−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−3

−2

−1

0

1

2


y

x


−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−3

−2

−1

0

1

2


x

y


−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−3

−2

−1

0

1

2


y

x


−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−3

−2

−1

0

1

2


y

x


−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−3

−2

−1

0

1

2


y

x



Calculation of Wasserstein distances in the discrete case

An example of an optimal transportation problem in the discrete case.

0.2

0.2

0.2

0.2

0.25

0.20.25

0.25

0.25

0.2

0.05

0.1

0.05

0.15

0.2

0.2

0.05

Figuur 4: Open circles correspond to the first measure, filled circles correspondto the second measure. Left panel: Initial configuration. Numbers indicateprobability mass at each point. Right panel: An optimal transportation plan withWasserstein distance W ≈ 3.122. The numbers next to the arrows indicate howmuch probability mass is transported from the first measure to the second measure.

Since the algorithms for the transportation problem have at least a quadraticdependence on sample size. A practical solution is to resample smaller subseriesfrom the reconstructed trajectory and to estimate the Wasserstein distancesmultiple times, bootstrapping its expected value. This reduces the computationalload.


Distance matrices

We assume that the distance information is presented in the form of a singlematrix M whose entries

Mij = W (µi, µj)

represent the distances between the family of dynamical systems underconsideration.

Multidimensional scaling (MDS) techniques are used to model distance data aspoints in a geometric (usually Euclidean) space.

In our present application to dynamical systems, each point in this spacerepresents a single dynamical system and the space can be interpreted as thespace of (the totality of)their possible dynamical behavior.

We call this space the behavior space.

Principle 1. The reconstructed behavior space, i.e., the MDS coordinatesderived from a distance matrix, is the object at which all (statistical) analysisstarts.


Methodology

A summary of the methodology

Dynamical systems

Time series

Delay reconstruction

Distance matrix

Behavior representation

Measurement / Projection

Probability distributions

Wasserstein distances

Multidimensional scaling

Statistical analysis Parameter changes?

Coupling strength?Classification

Accuracy?

Clustering

Significance?

Permutation tests Cross-validation


Visualization of parameter changes

One of the most interesting aspects of the distance based analysis that we haveintroduced is the possibility to visualize changes in dynamical behavior withrespect to parameter changes, similar to a bifurcation analysis.

The parameters a, b of the Henon map were varied, with a ranging from 1.4 to 0.7in fourteen steps of 0.05, and b ranging from 0.3 to 0.02 in steps of 0.02. Theinvariant measures of the x-variable, corresponding to a trivial embeddingdimension k = 1, are shown

0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4

−2

−1

01

2

b = 0.3a

dens

ity o

f x

14 12 10 8 6 4 2 0

0.05 0.10 0.15 0.20 0.25 0.30

−2

−1

01

2

a = 1.4b

dens

ity o

f x

−14 −11 −9 −7 −5 −3 −1

Figuur 5: Left panel: Variation in parameter a with constant b = 0.3. Right panel:Variation in parameter b with constant a = 1.4. Darker shade indicates large timeaverages, and lighter shade smaller time averages.


Henon map: Visualization of parameter changes II

The left panel shows a two-dimensional projection of behavior space, i.e., of theWasserstein distances of the respective dynamical systems. The distinct behaviorof these systems, with respect to parameter changes, is clearly visible.

Larger deviations of the parameters from (a0, b0) = (1.4, 0.3) result in points thatare farther away from the point 0

−0.6 −0.4 −0.2 0.0 0.2 0.4

−0.

4−

0.2

0.0

0.2

0.4

14131211109 8 76

5 4

32

10

−14 −13−12−11−10−9

−8−7

−6−5−4−3−2

−1

−0.3 −0.2 −0.1 0.0 0.1

−0.

15−

0.05

0.05

0.15

14

13

1211

10 9

8

7

6

5 4

32

10

−14−13

−12−11

−10

−9−8−7−6

−5−4

−3−2

−1

Figuur 6: Two-dimensional MDS representation of Wasserstein distances. Squarescorrespond to variation in the first parameter a, Triangles to variation in the secondparameter b. The circles around the points corresponding to a = 1.4, b = 0.3 haveradius 0.091 and 0.118 which are the mean self-distances.


Lung diseases - Background

Both asthma and the condition known as chronic obstructive pulmonary disease(COPD) are obstructive lung diseases that in the early stages they show similarsymptoms, rendering correct diagnosis difficult. As different treatments areneeded, this is of considerable concern.

An important diagnostics tool is the forced oscillation technique (FOT), as itallows to assess lung function non-invasive.

By superimposing a range of pressure oscillations on the ambient air and analyzingthe response of the airway systems, a number of parameters can be estimated thatdescribe the mechanical properties of airway tissue.

In particular, for each forcing frequency ω, a complex input impedance Z(ω) isavailable as time series, discretely sampled during a short period of tidal breathing.

The real part R(ω) represents the airway resistance and the imaginary part Xω)the airway reactance.


Lung diseases - The data

Clinicians usually only use the mean values R(ω) and X(ω) of these parameters,averaged over the measurement period.

Clearly there is a lot more (dynamical) information contained in the time series asthe example time series of R(ω) and X(ω) show

0 5 10 15 20 25 30

−1

01

23

45

time [s]

cmH

2OL

s

0 5 10 15 20 25 30−

10

12

34

5

time [s]

cmH

2OL

s

Figuur 7: Example time series of respiratory resistance R(8) (upper curves) andrespiratory reactance X(8) (lower curves) by forced oscillation technique duringthirty seconds of tidal breathing. Left panel: A patient with mild asthma. Rightpanel: A patient with mild to severe COPD. The horizontal lines indicate the meanvalues used routinely in clinical assessment.


Lung diseases - Discrimination by Wasserstein distances I

The motivation for the application of Wasserstein distances to this dataset is theassumption that the two lung diseases affect the temporal dynamics of inputimpedance in distinct ways.

Considering asthma and COPD as dynamical diseases, we assume an underlyingdynamical systems with different parameters for the different diseases. Althoughthese parameters are not accessible, it is then possible to discriminate the twodiseases using the Wasserstein distances between the data to quantify thedifferences in dynamics.

For simplicity, we only consider here a two-dimensional reconstructing here, wherethe time series of R(8) and X(8) were combined into a series of two-dimensionalvectors with trivial embedding dimension k = 1, trivial lag q = 1, and a length ofabout 12000 values (recorded at 16 Hz, the Nyquist frequency for the 8 Hz forcedoscillation, concatenating all 12 measurements into one long series per patient).

We consider the distribution of these points in Ω = R2 as an approximation of theinvariant measure of the underlying dynamical system.


Lung diseases - Discrimination by Wasserstein distances II

For reference we show the results for the squared sum of differences

dij =((Xi(8)− Xj(8))2 + (Ri(8)− Rj(8))2

)1/2in means (not the Wasserstein distances).

−3 −2 −1 0 1 2 3 4

−3

−2

−1

01

23

4

1 2

3

45

678

9

1011

121314

15

16

17

18

1920

2122

23

24 25

A

5 10 15

0.00

00.

010

0.02

0

Dimension

B

1.8 1.9 2.0 2.1 2.2

05

1015 P=0.0321

C

Figuur 8: Panel A: Two dimensional MDS reconstruction for patients suffering fromasthma (open circles) and COPD (filled squares). Panel B: Strain values againstreconstruction dimension. Panel C: Some statistics for the two classes.


Lung diseases - Discrimination by Wasserstein distances III

The results using the Wasserstein distances are shown below

−0.4 −0.2 0.0 0.2 0.4

−0.

4−

0.2

0.0

0.2

0.4

1

2

3

4

5

6

7

8

9

10

11

12 13 14

15

16

1718

19

20

21

22

23

24

25

A

5 10 15

0.0

0.4

0.8

Dimension

B

0.42 0.44 0.46 0.48

040

80

P=3e−04

C

Figuur 9: Panel A: Two dimensional MDS reconstruction for patients suffering fromasthma (open circles) and COPD (filled squares). Panel B: Strain values againstreconstruction dimension. Panel C: Some statistics for the two classes.

The classification by Wasserstein distances is noticeable better.

From this we conclude that the dynamical information contained in thefluctuations of respiratory impedance contains valuable clinical information.


Lung diseases - Summary

The results presented for asthma and COPD patients show that the Wassersteindistances are able to capture differences of shape of the long-term behavior ofreal-world time series.

Even for the trivial embedding shown and a low number of bootstrap samples, theWasserstein distances allow to classify a large proportion of the lung diseasescorrectly.

In fact, these result are the best known classification results (at single FOTfrequency) of these two lung diseases known in the literature.


Conclusion

Uit Mackey, M.C.; Milton, J.G.: ”Dynamical Diseases”, Ann. N.Y. Acad. Sci. 504(1987), 16-32:

A dynamical disease is defined as a disease that occurs in an intactphysiological control system operating in a range of control parameters thatleads to abnormal dynamics. The signature of a dynamical disease is achange in the qualitative dynamics of some observable nature as one or moreparameters are changed.

This is precisely the situation that we discussed in this talk and showed thatWasserstein distances can potentially be effectively used even in noisy environmentto detect the signature of a dynamical disease.

Muskulus-VL, J Applied Physiology (2010)

Muskulus-VL, Physica D (2011)

Muskulus-VL, In progress

Please contact me for further information.


Nationale Wetenschapsagenda


Wasserstein distances in the analysis of time series and...

Documents

Transcript of Wasserstein distances in the analysis of time series and...