Lecture series: Data analysis

Lectures: Each Tuesday at 16:00

(First lecture: May 21, last lecture: June 25)

Thomas Kreuz, ISC, CNR thomas.kreuz@cnr.it

http://www.fi.isc.cnr.it/users/thomas.kreuz/

• Lecture 1: Example (Epilepsy & spike train synchrony), Data acquisition, Dynamical systems

• Lecture 2: Linear measures, Introduction to non-linear dynamics

• Lecture 3: Non-linear measures

• Lecture 4: Measures of continuous synchronization (EEG)

• Lecture 5: Application to non-linear model systems and to epileptic seizure prediction, Surrogates

• Lecture 6: Measures of (multi-neuron) spike train synchrony

(Very preliminary) Schedule

• Example: Epileptic seizure prediction

• Data acquisition

• Introduction to dynamical systems

Last lecture

Epileptic seizure prediction

Epilepsy results from abnormal, hypersynchronous neuronal activity in the brain

Accessible brain time series:EEG (standard) and neuronal spike trains (recent)

Does a pre-ictal state exist (ictus = seizure)?

Do characterizing measures allow a reliable detection of this state?

Specific example for prediction of extreme events

Data acquisition

Sensor

System / Object

Amplifier AD-Converter

Computer

Filter

Sampling

Dynamical system

• Described by time-dependent states

• Evolution of state

- continuous (flow)

- discrete (map)

can be both be linear or non-linear

• Example: sufficient sampling of sine wave (2 sampling values per cycle)

Control parameter

Non-linear model systems

Linear measures

Introduction to non-linear dynamics

Non-linear measures

- Introduction to phase space reconstruction

- Lyapunov exponent

Today’s lecture

[Acknowledgement: K. Lehnertz, University of Bonn, Germany]

Non-linear model systems

Non-linear model systems

Continuous Flows

• Rössler system

• Lorenz system

Discrete maps

• Logistic map

• Hénon map

Logistic map

r - Control parameter

• Model of population dynamics • Classical example of how complex, chaotic behaviour can

arise from very simple non-linear dynamical equations

[R. M. May. Simple mathematical models with very complicated dynamics. Nature, 261:459, 1976]

𝑟=4

Hénon map

• Introduced by Michel Hénon as a simplified model of the Poincaré section of the Lorenz model

• One of the most studied examples of dynamical systems that exhibit chaotic behavior

[M. Hénon. A two-dimensional mapping with a strange attractor. Commun. Math. Phys., 50:69, 1976]

Rössler system

• designed in 1976, for purely theoretical reasons• later found to be useful in modeling equilibrium in

chemical reactions

[O. E. Rössler. An equation for continuous chaos. Phys. Lett. A, 57:397, 1976]

Lorenz system

• Developed in 1963 as a simplified mathematical model for atmospheric convection

• Arise in simplified models for lasers, dynamos, electric circuits, and chemical reactions

[E. N. Lorenz. Deterministic non-periodic flow. J. Atmos. Sci., 20:130, 1963]

Linear measures

Linearity

Dynamic of system (and thus of any time series measured from the system) is linear if:

H describes the dynamics and two state vectors

• Superposition:

• Homogeneity: scalar

Linearity:

Overview

• Static measures

- Moments of amplitude distribution (1st – 4th)

• Dynamic measures

- Autocorrelation- Fourier spectrum- Wavelet spectrum

Static measures

• Based on analysis of distributions (e.g. amplitudes)

• Do not contain any information about dynamics

• Example: Moments of a distribution - First moment: Mean - Second moment: Variance - Third moment: Skewness - Fourth moment: Kurtosis

First moment: Mean

Average of distribution

Second moment: Variance

Width of distribution(Variability, dispersion)

Standard deviation

Third moment: Skewness

Degree of asymmetry of distribution(relative to normal distribution)

< 0 - asymmetric, more negative tailsSkewness = 0 - symmetric > 0 - asymmetric, more positive tails

Fourth moment: Kurtosis

Degree of flatness / steepness of distribution(relative to normal distribution)

< 0 - platykurtic (flat)Kurtosis = 0 - mesokurtic (normal) > 0 - leptokurtic (peaked)

Dynamic measures

Autocorrelation Fourier spectrum

[ Cross correlation Covariance ]

Time domain Frequency domain

x (t)Amplitude

Fx()

Frequency amplitudeComplex number Phase

Physical phenomenon

Time series

Autocorrelation

)()0( XXXX CC

Time domain: Dependence on time lag

One signal with

(Normalized to zero mean and unit variance)

0)(

0 1)( 1

''

XX

N

nnn

XX

C

xxNC

Autocorrelation: Examples

periodic stochastic memory

𝜏 𝜏 𝜏

Discrete Fourier transform

Condition:

Fourier series (sines and cosines):

Fourier coefficients:

Fourier series (complex exponentials):

Fourier coefficients:

Power spectrum

Parseval’s theorem:

𝑃 (𝜔)=∫−∞

∞

|𝑥 (𝑡 )|2𝑑𝑡=∫−∞

∞

|𝐹 𝑋 (𝜔 )|2𝑑𝜔Overall power:

=

Wiener-Khinchin theorem:

𝐶𝑋𝑋 (𝜏 )=∫−∞

∞

𝑃 (𝜔)𝑒𝑖 𝜏𝜔𝑑𝜔𝑃 (𝜔)=∫−∞

∞

𝐶𝑋𝑋(𝜏 )𝑒− 𝑖𝜏𝜔𝑑𝜏

Tapering: Window functionsFourier transform assumes periodicity Edge effectSolution: Tapering (zeros at the edges)

EEG frequency bands

[Buzsáki. Rhythms of the brain. Oxford University Press, 2006]

Description of brain rhythms

• Delta: 0.5 – 4 Hz

• Theta: 4 – 8 Hz

• Alpha: 8 – 12 Hz

• Beta: 12 – 30 Hz

• Gamma: > 30 Hz

Example: White noise

Example: Rössler system

Example: Lorenz system

Example: Hénon map

Example: Inter-ictal EEG

Example: Ictal EEG

Time-frequency representation

Wavelet analysisBasis functions with finite support

Example: complex Morlet wavelet

– scaling; – shift / translation(Mother wavelet: , )

Implementation via filter banks (cascaded lowpass & highpass):

– lowpass(approximation) – highpass(detail)

Wavelet analysis: Example

[Latka et al. Wavelet mapping of sleep splindles in epilepsy, JPP, 2005]

Advantages:

- Localized in both frequency and time

- Mother wavelet canbe selected accordingto the feature of interest

Further applications:- Filtering- Denoising- Compression

Pow

er

Introduction tonon-linear dynamics

Linear systems

• Weak causality

identical causes have the same effect (strong idealization, not realistic in experimental situations)

• Strong causality

similar causes have similar effects (includes weak causality applicable to experimental situations, small deviations in initial conditions; external disturbances)

Non-linear systems

Violation of strong causality

Similar causes can have different effects

Sensitive dependence on initial conditions

(Deterministic chaos)

Linearity / Non-linearity

Non-linear systems- can have complicated solutions- Changes of parameters and initial conditions lead to non-

proportional effects

Non-linear systems are the rule, linear system is special case!

Linear systems- have simple solutions- Changes of parameters and initial

conditions lead to proportional effects

Phase space example: Pendulum

Velocity v(t)

Position x(t)

t

State space:

Time series:

Phase space example: Pendulum

Ideal world: Real world:

Phase space

Phase space: space in which all possible states of a system are represented, with each possible system state corresponding to one unique point in a d dimensional cartesian space (d - number of system variables)

Pendulum: d = 2 (position, velocity)

Trajectory: time-ordered set of states of a dynamical system, movement in phase space (continuous for flows, discrete for maps)

Vector fields in phase space Dynamical system described by time-dependent states

– d-dimensional phase space

– Vector field (assignment of a vector to each point in a subset of Euclidean space)

Examples:- Speed and direction of a moving fluid- Strength and direction of a magnetic force

Here: Flow in phase space Initial condition Trajectory (t)

Divergence

Rate of change of an infinitesimal volume around a given point of a vector field:

- Source: outgoing flow ( with , expansion)

- Sink: incoming flow ( with , contraction)

System classification via divergence

Liouville’s theorem:

Temporal evolution of an infinitesimal volume:

conservative (Hamiltonian) systems

dissipative systems

instable systems

Dynamical systems in the real world

• In the real world internal and external friction leads to dissipation

• Impossibility of perpetuum mobile (without continuous driving / energy input, the motion stops)

• When disturbed, a system, after some initial transients, settles on its typical behavior (stationary dynamics)

• Attractor: Part of the phase space of the dynamical system corresponding to the typical behavior.

Attractor

Subset X of phase space which satisfies three conditions:

• X is forward invariant under f: If x is an element of X, then so is f(t,x) for all t > 0.

• There exists a neighborhood of X, called the basin of attraction B(X), which consists of all points b that "enter X in the limit t → ∞".

• There is no proper subset of X having the first two properties.

Attractor classification

Fixed point: point that is mapped to itself

Limit cycle: periodic orbit of the system that is isolated (i.e., has its own basin of attraction)

Limit torus: quasi-periodic motion defined by n incommensurate frequencies (n-torus)

Strange attractor: Attractor with a fractal structure

(2-torus)

Introduction tophase space reconstruction

Phase space reconstruction• Dynamical equations known (e.g. Lorenz, Rössler):

System variables span d-dimensional phase space

• Real world: Information incomplete

Typical situation: - Measurement of just one or a few system variables (observables) - Dimension (number of system variables, degrees of freedom) unknown - Noise - Limited recording time - Limited precision

Reconstruction of phase space possible?

Taken’s embedding theoremTrajectory of a dynamical system in - dimensional phase space .

One observable measured via some measurement function :

; M:

It is possible to reconstruct a topologically equivalent attractor via time delay embedding:

- time lag, delay; – embedding dimension

[F. Takens. Detecting strange attractors in turbulence. Springer, Berlin, 1980]

Taken’s embedding theoremMain idea: Inaccessible degrees of freedoms are coupled into the observable variable via the system dynamics.

Mathematical assumptions:

- Observation function and its derivative must be differentiable - Derivative must be of full rank (no symmetries in components) - Whitney’s theorem: Embedding dimension

Some generalizations:Embedding theorem by Sauer, Yorke, Casdagli

[Whitney. Differentiable manifolds. Ann Math,1936; Sauer et al. Embeddology. J Stat Phys, 1991.]

Topological equivalence

original reconstructed

Example: White noise

Example: Rössler system

Example: Lorenz system

Example: Hénon map

Example: Inter-ictal EEG

Example: Ictal EEG

Real time seriesPhase space reconstruction / Embedding:First step for many non-linear measures

Choice of parameters:

Window length T - Not too long (Stationarity, control parameters constant) - Not too short (sufficient density of phase space required)

Embedding parameters - Time delay - Embedding dimension

Influence of time delaySelection of time delay (given optimal embedding dimension)

• too small: - Correlation in time dominate - No visible structure - Attractor not unfolded

• too large: - Overlay of attractor regions that are rather separated in the original attractor - Attractor overfolded

• optimal: Attractor unfolded

Influence of time delay

too small too large

optimal:

Criterion: Selection of time delay

Aim: Independence of successive values

• First zero crossing of autocorrelation function (only linear correlations)

• First minimum of mutual information function (also takes into account non-linear correlations)

[Mutual information: how much does knowledge of tell you about ]

Criterion: Selection of embedding dimensionAim: Unfolding of attractor (no projections)

• Attractor dimension known: Whitney’s theorem:

• Attractor dimension unknown (typical for real time series):

Method of false nearest neighbors: Trajectory crossings, phase space neighbors close: Increase of distance between phase space neighbors

Procedure: - For given m count neighbors with distance - Check if count decreases for larger (if yes some were false nearest neighbors) - Repeat until number of nearest neighbors constant

[Kennel & Abarbanel, Phys Rev A 1992]

Non-linear measures

Non-linear deterministic systems

• No analytic solution of non-linear differential equations

• Superposition of solutions not necessarily a solution

• Behavior of system qualitatively rich e.g. change of dynamics in dependence of control parameter (bifurcations)

• Sensitive dependence on initial conditions

Deterministic chaos

Bifurcation diagram: Logistic mapBifurcation: Dynamic change in dependence of control parameter

Fixed point Period doubling Chaos

Deterministic chaos

• Chaos (every-day use): - State of disorder and irregularity

• Deterministic chaos - irregular (non-periodic) evolution of state variables - unpredictable (or only short-time predictability) - described by deterministic state equations (in contrast to stochastic systems) - shows instabilities and recurrences

Deterministic chaosregular chaotic random

deterministic deterministic stochastic

Long-time predictions possible

Rather un-predictable

unpredictable

Strong causality No strong causalityNon-linearity

Uncontrolled (external) influences

Characterization of non-linear systems

Linear meaures:

• Static measures (e.g. moments of amplitude distribution):

- Some hints on non-linearity- No information about dynamics

• Dynamic measures (autocorrelation and Fourier spectrum)

Autocorrelation Fourier

Fast decay, no memory Typically broadband

Distinction from noise?

Wiener-Khinchin-Theorem

Characterizition of a dynamic in phase space

Predictability

(Information / Entropy)Density

Self-similarityLinearity / Non-linearity

Determinism /Stochasticity

(Dimension)

Stability (sensitivityto initial conditions)

Lyapunov-exponent

Stability

Analysis of long-term behavior () of a dynamic system

• Unlimited growth (unrealistic)• Limited dynamics - Fixed point / Some kind of equilibrium - periodic or quasi-periodic motion - chaotic motion (expansion and folding)

How stable is the dynamics? - when the control parameter changes - when disturbed (push to neighboring points in phase space)

Stability of equilibrium pointsDynamical system described by time-dependent states

Suppose has an equilibrium .

• The equilibrium of the above system is Lyapunov stable, if, for every , there exists a such that if , then , for every .

• It is asymptotically stable, if it is Lyapunov stable and if there exists such that if , then .

• It is exponentially stable, if it is asymptotically stable and if there exists such that if , then, for every .

Stability of equilibrium points

• Lyapunov stability: Tube of diameter Solutions starting "close enough" to the equilibrium (within a distance from it) remain "close enough" forever (within a distance from it). Must be true for any that one may want to choose.

• Asymptotic stability:Solutions that start close enough not only remain close enough but also eventually converge to the equilibrium.

• Exponential stability:Solutions not only converge, but in fact converge faster than or at least as fast as a particular known rate.

Divergence and convergenceChaotic trajectories are Lyapunov-instable:

• Divergence:Neighboring trajectories expandSuch that their distance increasesexponentially (Expansion)

• Convergence:Expansion of trajectories to theattractor limits is followed by adecrease of distance (Folding).

Sensitive dependence on initial conditions

Quantification: Lyapunov-exponent

Lyapunov-exponentCalculated via perturbation theory:

infintesimal perturbation in the initial conditions

Local linearization:

Solution:

𝐷 𝑓 (𝑥) - Jacobi-Matrix

Taylor series

𝜆 - Lyapunov exponent

Lyapunov-exponentIn m-dimensional phase space:

Lyapunov-spectrum: , (expansion rates for different dimensions)

Relation to divergence:

Dissipative system:

Largest Lyapunov exponent (LLE) (often ):

Regular dynamics Chaotic dynamics Stochastic dynamics Stable fixed point

Example: Logistic map

Bifurcation diagram

Fixed point

Period doubling

Chaos

Largest Lyapunov exponent

Dependence of the control parameter

Lyapunov-exponent

- Dynamic characterization of attractors (Stability properties)- Classification of attractors via the signs of the Lyapunov- spectrum- Average loss of information regarding the initial conditions

Average prediction time:

( – localization precision of initial condition, j+ – index of last positive Lyapunov exponent)

Largest Lyapunov-exponent: Estimation- Reference trajectory: - Neighboring trajectory: - Initial distance: - Distance after T time steps: - Expansion factor:- New neighboring trajectory to , to etc.- Calculate times:

Largest Lyapunov exponent (LLE):

𝑇

𝜆

ln𝛬

(𝑖)𝜆 (𝑇 )= 1

𝑇 ∑𝑖=1

𝑙

Λ (𝑖)

[Wolf et al. Determining Lyapunov exponents from a time series, Physica D 1985]

Non-linear model systems

Linear measures

Introduction to non-linear dynamics

Non-linear measures

- Introduction to phase space reconstruction

- Lyapunov exponent

Today’s lecture

Non-linear measures

- Dimension

- Entropies

- Determinism

- Tests for Non-linearity, Time series surrogates

Next lecture