• ICA is a technique to separate linearly mixed sources, also known as blind source/signal separation.

• The goal is to find a linear

representation of non-gaussian data so that the components are statistically independent, or as independent as possible.

Supposes:

– non-Gaussian source signals – statistical independence of the

signals – Note: in theory, ICA can only

extract sources that are combined linearly.

1

What is ICA?

EEL 683

6

ICA Overview

2

EEL 683

6

ICA

3

Blind Signal Separation (BSS) or Independent

Component Analysis (ICA) is the identification

& separation of mixtures of sources with little

prior information.

ICA seeks directions that are most independent

from each other.

Given a set of observations of random variables x1(t), x2(t)…xn(t), where t is the time or

sample index, assume that they are generated as a linear mixture of independent

components: y=Wx, where W is some unknown matrix. Independent component

analysis now consists of estimating both the matrix W and the yi(t), when we only

observe the xi(t).

1) Eye movements and eye blinks project mainly to

frontal sites.

2) Temporal muscle activity should project to the

temporal sites .

3) Occipital (rear head) movements project to the

back. EEL 683

6

Rationale of ICA • Find the components Si that are as independent as possible

in the sense of maximizing or minimizing some function F(s1,s2,.,sk) that measures independence.

• The variance of all ICs is 1. • There is no hierarchy between Ics.

4

EEL 683

6

The idea behind ICA

EEL 683

6

The ICA density model

• Assume the data is obtained by linearly mixing the sources

• The filter matrix is the inverse of the mixing matrix.

• The sources are independent non-Gaussian. The density of the data is a product of the marginal densities of the sources

)()(

,

i

i

i spp

s

AWxWs

Asx

1TT

Mixing matrix

Source vector

W=A-1

EEL 683

6

ICA Example: Two Time Signals

• Suppose we have two different time signals A and B as shown:

• Let’s create two new signals C

and D, such that: • C = A – 2 B • D = 1.73 A – 3.41 B

• Purpose of the ICA: Given C and D, find A and B 7

EEL 683

6

Another ICA Example: The Cocktail Party Problem

• A number of people are talking simultaneously in a room

• One is trying to follow one of the discussions or sounds

Find a nice demo at: http://www.cis.hut.fi/projects/ica/cocktail/cocktail_en.cgi

Human brain can handle this sort of auditory source separation problem, but it is a very tricky problem in digital signal processing

8

EEL 683

6

Modeling the Cocktail Party Problem

• Example of 2 microphones and 2 speakers

Microphone 1: x1(t) = a11s1(t) + a12s2(t) Microphone 2: x2(t) = a21s1(t) + a22s2(t)

Goal:

• estimate s1(t) and s2(t) using only x1(t) and x2(t)

9

EEL 683

6

Motivation

Two Independent Sources Mixture at two Mics

aIJ ... Depend on the distances of the microphones from the speakers

2221212

2121111

)(

)(

sasatx

sasatx

Motivation: Get the Independent Signals out of the Mixture 10

EEL 683

6

Definitions • Assume that we observe n linear mixtures x1, ...,xn of n

independent components.

xj = aj1s1+aj2s2+...+ajnsn, for all j.

where:

xj: mixture

sn: independent components

11

EEL 683

6

ICA Model

• xj = aj1s1 + aj2s2 + .. + ajnsn, is the sum of the independent components sn and

weighted by the mixing weights ajn

x = As

• IC‘s s are latent variables & are unknown AND Mixing matrix A is also

unknown

• Task: estimate A and s using only the observeable random vector x

• Lets assume that no. of IC‘s = no of observable mixtures

and A is square and invertible

• So after estimating A, we can compute W=A-1 and hence

s = Wx = A-1x

*latent variables are not directly observed but are rather inferred (through a mathematical model).

12

EEL 683

6

Definitions • Using vector-matrix notation, the mixing model is written as: x = As where A the matrix with elements ai j

• Model can also be written as:

i

n

i

i sax

1

• Assume that the components si are statistically independent.

• After estimating the matrix A, we can compute its inverse, say W, and obtain the independent component simply by: s =Wx.

13

EEL 683

6

ICA

14

2221212

2121111

)(

)(

sasatx

sasatx

xj = aj1s1 + aj2s2 + .. + ajnsn, for all j

x = As (statistical mixing model or ICA model)

X—Colunms vectors whose elements are the mixures x1, ...xn

S—Colunms vectors whose elements are s1, ... sn

A—matrix with elements ajn

i

n

i

isax

1

ICA model: It describes how a set of data is generated by a process of mixing

the components sn .

• The ICA components are latent variables (not visually or directly observed)

• All we observed if the vector x and we must estimate A and s.

• For this, you need as general assumptions as possible

EEL 683

6

ICA

15

Assumptions:

• Components (sn ) are statiscally independent

• Components must have nongaussian distributions

• Unknown mixing matrix is square

Then,

If we estimate matrix A, we can compute its inverse W and obtain

s = A-1x = Wx

Ambiguities:

• We cannot determine the variance of the Ics: A and s are unknown, so we assume

that each has unit variance (variance =1).

• We cannot determine the order of the Ics: A and s are unknown, and we can change

the order of the terms in the sum. So we can call any IC as the first one.

EEL 683

6

ICA Principal (Non-Gaussian is Independent)

• Key estimating A is non-gaussianity

• The distribution of a sum of independent random variables tends toward a Gaussian distribution.

f(s1) f(s2) f(x1) = f(s1 +s2)

• Where w is one of the rows of matrix W.

• y is a linear combination of si, with weights given by zi.

• Since sum of two indep r.v. is more gaussian than individual r.v., so zTs is more gaussian than either of si. AND becomes least gaussian when its equal to one of si.

• So we could take w as a vector which maximizes the non-gaussianity of wTx.

szAswxwy TTT

16

EEL 683

6

ICA

17

Illustration:

Consider 2 ICs with uniform distribution ( mean zero and variance=1)

The joint density of 𝑠1 and 𝑠2 is then uniform ( since the joint density of 2 independent

variables is the product of their marginal densities

Maximizing the nongaussianity of wTx gives us one of the ICs. s = wTx

Finding these local maxima, we find the ICs.

To use nongaussianity in ICA, we must have a quantitative measure of nongaussianity

of a random variable (y).

The classical measure of nongaussianity is Kurtosis. Kurtosis is equal to zero for a

gaussian random variable and for nongaussian is nonzero.

EEL 683

6

ICA

18

• To find projections that maximize the non-Gaussianity of components by their kurtosis (the fourth-order cumulant given to a random variable).

• As kurtosis is identically zero for Gaussian distributed signals, the aim is to maximize the magnitude of the kurtosis to make the estimated sources as non-Gaussian (and hence as independent) as possible.

• ICA problem is posed as an optimization problem with the sources as its solution.

EEL 683

6

ICA

19

Kurtosis:

If 𝑥1 and 𝑥2 are two independent random variables: Kurt(𝑥1 + 𝑥2)=kurt(𝑥1)+kurt(𝑥2) Kurt(α 𝑥1)= α4kurt(𝑥1) Where α is a scalar Example:

Assume that the independent components 𝒔𝟏 𝒂𝒏𝒅 𝒔𝟐 have Kurtosis values: krt(𝑠1 ) and krt(𝑠2 ) different from zero and unit variance we seek for one of the independent components as y=w𝑻𝒙

EEL 683

6

ICA

20

Kurtosis:

If we seek for one of the independent components as y=w𝑻𝒙

z=A𝑻w

y=w𝑇𝑥 = w𝑇As= z𝑇s= 𝑠1𝑧1+𝑠2𝑧2

𝑘𝑢𝑟𝑡 𝑦 = 𝑘𝑢𝑟𝑡(𝑠1𝑧1 ) + 𝑘𝑢𝑟𝑡 𝑠2𝑧2

Abs(𝑘𝑢𝑟𝑡 𝑦 ) = 𝑎𝑏𝑠(𝑘𝑢𝑟𝑡(𝑠1𝑧1 ) + 𝑘𝑢𝑟𝑡 𝑠2𝑧2 ) maxima of the function on the

unit circle

We start from some weight vector w and compute the direction in which the kustosis of

y=w𝑻𝒙 is growing most strongly and find a new vector w.

Kurtosis can be used as an optimization criterion for the ICA problem

EEL 683

6

21

• Typically nongaussianity is measured by the absolute value of kurtosis. The square of kurtosis can also be used. These are zero for a gaussian variable, and greater than zero for most nongaussian random variables.

• Independent components could be found by kurtosis minimization or maximization.

Kurtosis

In practice we would start from some weight vector w , compute the direction in which the kurtosis of y is growing most strongly (if kurtosis is positive) or decreasing most strongly (if kurtosis is negative) based on the available sample of mixture vector x, and use a gradient method or one of their extensions for finding a new vector w . EEL 6

836

Definitions

• Key to estimating the ICA model is nongaussianity.

• Goal:

Find vector w that maximizes the nongaussianity

of Wx.

• Measures of nongaussianity: kurtosis,maximum likelihood estimation, entropy, etc

22

EEL 683

6

Kurtosis

23

• Kurtosis measures the degree of peakedness (spikiness) of a distribution and it is zero only for Gaussian distribution.

• kurtosis is either positive if it is supergaussian (spikier than Gaussian) or negative if it is subgaussian (flatter than Gaussian).

EEL 683

6

Definitions

ICA components = the matrix that allows projecting the data in the initial space to one of the axis found by ICA. The weight matrix is the full transformation from the original space. Forward transform: S = W X where X is the data in the original space (data space), and W is the weight matrix to go from the S space (the source activity) to the X space Back-transform: X = W-1 S

24

EEL 683

6

The Algorithm

• Recall: objective of ICA = maximization of nongaussianity.

• FastICA is a very efficient method of maximization of nongaussianity.

• Assumed that the data is preprocessed by centering and whitening.

25

EEL 683

6

PCA vs ICA

• PCA

– Focus on uncorrelated and Gaussian components – Second-order statistics – Orthogonal transformation

• ICA

– Focus on independent and non-Gaussian components – Higher-order statistics – Non-orthogonal transformation

26

EEL 683

6

ICA Applications

• Noise reduction in signals and images. • Separation of artifacts in EEG. • Finding hidden factors in data. • Telecommunications (blind separation: real signal from interference).

27

EEL 683

6

Application in EEG • Severe contamination of EEG activity by eye movements, eye blinks, muscle, heart activity

and line noise is a serious problem for EEG interpretation and analysis.

• Rejecting contaminated EEG epochs results in a considerable loss of collected information.

• EOG records also contain brain signals, so regressing out EOG activity inevitably involves

subtracting a portion of the relevant EEG signal from each recording as well.

• One alternative is to apply ICA to multichannel EEG recordings and remove a wide variety of

artifacts from EEG records by eliminating the contributions of artifactual sources onto the

scalp sensors.

28

EEL 683

6

Application in EEG The ICA method is based on the assumptions that the time series recorded on the scalp:

• Are spatially stable mixtures of the activities

of temporally independent cerebral and artifactual sources.

• Summation of potentials arising from different

parts of the brain, scalp, and body is linear at the electrodes.

• Propagation delays from the sources to the

electrodes are negligible.

29

EEL 683

6

Application in EEG

• Rows of the input matrix, are EEG signals recorded at different electrodes and the columns are measurements recorded at different time points.

• ICA finds an `unmixing' matrix, W, which decomposes or linearly unmixes the multi-channel scalp data into a sum of temporally independent and spatially fixed components. 30

Rows of the output data matrix, S = WX, are time courses of activation of the ICA components. The columns of the inverse matrix, inv(W), give the relative projection strengths of the respective components at each of the scalp sensors. EEL 6

836

Application in EEG

Scalp weights give the scalp topography of each component, and provide evidence for the components' physiological origins. For example:

Eye movements should project mainly to frontal sites (Fp1 and Fp2). Eye blinks should project to frontal sites and have large activations(Fp1 and Fp2). Temporal muscle activity should project to temporal sites (T3 and T4). 31

EEL 683

6

Application in EEG

In EEG, the recordings are expressed by:

These are time points

After whitening the data, ICA will "rotate all the axis" in order to minimize the Gaussianity of the projection on all axis (note that unlike PCA the axis do not have to remain orthogonal).

32

EEL 683

6

Application in EEG (Cont)

For example:

To compute the activity of the second source or second independent component (in a matrix multiplication format), you may simply multiply matrix X by the row vector:

33

EEL 683

6

Now you have the activity of the second component, but the activity is unitless.

Each dipole has an activity (which project linearly to all electrodes).

The activity of the Brain source (dipole) is unitless unless it is projected to the electrodes. So each dipole creates a contribution at each electrode site. ICA components are just the same. How to reproject one component to the electrode space.

X = W-1 S

W-1 is the inverse matrix to go from the source space S to the data space X.

W-1

Application in EEG (Cont)

34

EEL 683

6

If S is a row vector (for instance the activity of component 2 computed above) and we multiply it by the following column vector from the inverse matrix W-1 :

Application in EEG (Cont)

we will obtain the projected activity of component 2 (the inverse weights for component 2 (column vector; bottom left below) multiplied by the activity for component 2 (row vector; top right below) leads to the component projection (matrix; bottom right).

(above is the projection of one component activity on all the electrodes . This matrix will be denoted XC2.

35 X = W-1 S

S= EEL 683

6

If one want to remove component number 2 from the data (for instance if component number 2 proved to be an artifact), one can simply subtract the matrix XC2 from the original data X.

Note that in the matrix computed XC2 all the columns are proportional, which means that the scalp activity is simply scaled.

The columns of the W-1 matrix are the scalp topography of the components. Each column of this matrix is the topography of one component which is scaled in time by the activity of the component.

The scalp topography of each component can be used to estimate the equivalent dipole location for this component (assuming the component is not an artifact).

Application in EEG (Cont)

36

EEL 683

6

ICA • X = W-1 S

• Rows of the S matrix: time

course of the component activity.

• Columns of the W-1 matrix: scalp projection of the components

37 Note: The component scalp maps are the columns of the mixing matrix (the inverse of the unmixing matrix W)

EEL 683

6