Neuronal Computation Using High Order Statisticsfeng/teaching/dcsp_2018_15_and_r… · Digital...
Transcript of Neuronal Computation Using High Order Statisticsfeng/teaching/dcsp_2018_15_and_r… · Digital...
DCSP-15: Wiener Filter and Kalman Filter*
Jianfeng Feng
Department of Computer Science Warwick Univ., UK
http://www.dcs.warwick.ac.uk/~feng/dcsp.html
Father of cybernetics (control theory)
Norbert Wiener: Harvard awarded Wiener a PhD, when he was 17.
The RGB format stores three color values, R,
G and B, for each pixel.
RGB = imread(‘bush.png');
size(RGB)
ans =
1500 1200 3
imshow(RGB)
Example
Example
SetupRecorded signal x(n,m) = s(n,m) +x(n,m)
for example, an image of 1500 X1200
True Signal noise
RGB = imread(‘bush.png');
I = rgb2gray(RGB);
J =
imnoise(I,'gaussian',0,0.0
05);
figure, imshow(I),
figure, imshow(J)
Setup
To find a constant a(m,n) such that
E ( a(m,n) x(m,n) – s(m,n) ) 2
as small as possible
y(m,n) = a(m,n) x(m,n)
Different from the filter before, a(m,n) depends on (m,n) :
Adaptive filter
Setup
Minimum mean square error which is one of the most commonly used methods
The way to solve it is the gradient descent method (Newton method)
For a given error function c(a), we simply update a according to
a prop - c’(a)
C(a) is usually called cost function
Setup
C(a)
Luckly
Gradient Descent For Machine Learningb
C(a)
Gradient Descent For Machine Learningb
C(a)
Gradient Descent For Machine Learningb
Hard to reach the global minima
2 2 2 2
2 2 2
( ) ( 2 )
2
E ax s E a x axs s
a Ex aExs Es
The quanttity above is minimized if the derivative
of it with respect to a is zero
2aEx2 - 2Exs = 0
a =Exs
Ex2=E(s+x )s
Es2 + Ex 2=
Es2
Es2 + Ex 2=
s 2 - Ex 2
s 2
Setup
(optimal Wiener gain)
• Without loss of generality, we assume that
Es=0 Ex=0
for simplicity of formulation .
• S, x are independent
Setup
y(n1,n
2) = m(n
1,n
2)+
s 2(n1,n
2)-Ex 2
s 2(n1,n
2)
(x(n1,n
2)-m(n
1,n
2))
The filtered output is given by
Note that the coefficient a depends on the position
s is the local variance, Ex2 is the global variance (noise)
Algorithm
where summation is over an area of N and M
1 2
1 2
1 2
,
2 2 2
1 2
,
1( , )
1( , )
n n
n n
x n nNM
x n nNM
s
3X 3 area
(N=3, M=3)
Algorithm
Algorithm in Matlab
• wiener2 lowpass - filters an intensity
image that has been degraded by constant
power additive noise.
• wiener2 uses a pixelwise adaptive Wiener
method based on statistics estimated from
a local neighborhood of each pixel.
RGB = imread(‘bush.png');
I = rgb2gray(RGB);
J =
imnoise(I,'gaussian',0,
0.005);
K = wiener2(J,[5 5]);
figure, imshow(J),
figure, imshow(K)
Example
• Large M and N reduce noise, but blur the photo
• small M and N will not affect noise
• optimal size should be used
Further issues
N=M=50 N=M=2
•A more general version of Wiener: Kalman filter
•Kaman filter is widely used in many areas
(still a very hot research topic)
• As an introductory course, we only give you
a taste of its flavour
Further issues
Kalman Filter
Kalman Filter: find out xk
• Have two variables state variable x which is
not observable
xk = Fk xk-1 + Bkuk + wk
• Observable variable z which we can observe,
but is contaminated by noise
zk = Hk xk +vk
where wk ~ N(0,Qk) and vk ~ N(0,Rk) are noise, Fk and Hk are constants.
Xk-1Input uk
Bk
Output xk
Kalman FilterFor given xk-1|k-1 and pK-1|K-1
Predict
•Predicted (a priori) state estimate xk|k-1 = Fk-1 xk-1|k-1+Bkuk
•Predicted (a priori) estimate covariance pk|k-1 = Fk-1 pk-1 |k-1 FK-1 +QK
Update
•measurement residual yk = zk –Hk X k|k-1
•measurement residual covariance Sk = Hk Pk|k-1HK+Rk
•optimal Kalman gain Kk = P k|k-1 Hk (SK)-1
•updated (a posteriori) state estimate X k|k = x k|k-1 + Kk y k
•updated (a posteriori) estimate covariance pK|K = (I-K kH k) P k|k-1
Kalman Filter
• It is one of the most useful filters in the literature, as shown below
• We do not have time to go further, but do read around
clear all
close all
N=1000;
q=1;
r=1;
f=0.1;
h=0.2;
x(1)=0.5;
xsignal(1)=0.5;
xhat=zeros(1,N+1);
phat=zeros(1,N+1);
xhat(1)=x(1);
phat(1)=x(1);
p(1)=1;
z(1)=0.1;
y(1)=1;
for i=1:N
xsignal(i+1)=f*xsignal(i)+randn(1,1)*sqrt(q)+1*cos(2*pi*0.01*i);
z(i+1)=h*xsignal(i)+randn(1,1)*sqrt(r);
end
for i=1:N
x(i+1)=f*xhat(i)+1*cos(2*pi*0.01*i);
p(i+1)=f*phat(i)*f+q;
y(i+1)=z(i+1)-h*x(i+1);
s(i+1)=h*p(i+1)*h+r;
k(i+1)=p(i+1)*h/s(i+1);
atemp=x(i+1)+k*y(i+1);
xhat(1,i+1)=atemp(1,i+1);
atemp=(1-k*h)*p(i+1);
phat(i+1)=atemp(1,i+1);
clear atemp
end
plot(xsignal);
hold on
plot(xhat,'r')
plot(z,'g')
plot(xhat,'r')
Applications• Attitude and Heading Reference Systems
• Autopilot
• Battery state of charge (SoC) estimation [1][2]
• Brain-computer interface
• Chaotic signals
• Tracking and Vertex Fitting of charged particles in Particle Detectors [35]
• Tracking of objects in computer vision
• Dynamic positioning
• Economics, in particular macroeconomics, time series, and econometrics
• Inertial guidance system
• Orbit Determination
• Power system state estimation
• Radar tracker
• Satellite navigation systems
• Seismology [3]
• Sensorless control of AC motor variable-frequency drives
• Simultaneous localization and mapping
• Speech enhancement
• Weather forecasting
• Navigation system
• 3D modeling
• Structural health monitoring
• Human sensorimotor processing[36]
Flight control unit of an Airbus A340
modern weather forecasting.
DCSP-Revision
Jianfeng Feng
Department of Computer Science Warwick Univ., UK
http://www.dcs.warwick.ac.uk/~feng/dcsp.html
Next week
• Guest lecture next Monday:
Prof. Edmund Rolls
Digital Computation in Your Brain
323× 397Images may be subject to copyright
Next week
• Guest lecture next Monday:
Prof. Edmund Rolls
Digital Computation in Your Brain
• Assignment help Tuesday
(same time and venue as lecture)
B13. Rolls,E.T. (2018) The Brain, Emotion, and Depression. Oxford University Press: Oxford. (In press.)
B12. Rolls,E.T. (2016) Cerebral Cortex: Principles of Operation. Oxford University Press: Oxford. Paperback published November 2017.
B11. Rolls,E.T. (2014) Emotion and Decision-Making Explained. Oxford University Press: Oxford. Virtual Special Issue of Cortex on Emotion and
Decision-Making Explained.
B10.
Rolls,E.T. (2012) Neuroculture: On the Implications of Brain Science. Oxford University Press: Oxford.
B9.
Rolls,E.T. and Deco,G. (2010) The Noisy Brain: Stochastic Dynamics as a Principle of Brain Function. Oxford University Press: Oxford.
B8. Rolls,E.T. (2008) Memory, Attention, and Decision-Making: A Unifying Computational Neuroscience Approach. Oxford University Press:
Oxford.
B7. Rolls,E.T. (2005) Emotion Explained. Oxford University Press: Oxford.
B6. Rolls,E.T. and Deco,G. (2002) Computational Neuroscience of Vision. Oxford University Press: Oxford.
B5. Rolls,E.T. (1999) The Brain and Emotion. Oxford University Press: Oxford.
B4. McLeod,P., Plunkett,K. and Rolls,E.T. (1998) Introduction to Connectionist Modelling of Cognitive Processes. Oxford University Press:
Oxford.
B3. Rolls,E.T. and Treves,A. (1998) Neural Networks and Brain Function. Oxford University Press: Oxford.
B2. Rolls,B.J. and Rolls,E.T. (1982) Thirst. Cambridge University Press: Cambridge.
B1. Rolls, E.T. (1975) The Brain and Reward. Pergamon Press Oxford.
Next week
• Guest lecture next Monday:
Prof. Edmund Rolls
Digital Computation in Your Brai
• Assignment help on Tuesday
(same time and venue as lecture)
Revision class (second week, next term)
Fourier Thm.
Any signal x(t) of period T can be
represented as the sum of a set of
cosinusoidal and sinusoidal waves of
different frequencies and phases
x(t) = A0 + An cos(nwt)+n=1
¥
å Bn sin(nwt)n=1
¥
å
A0 =<1, x(t) >=1
Tx(t)dt
-T /2
T /2
ò
An =< cos(nwt), x(t) >=2
Tx(t)cos(nwt)dt
-T /2
T /2
ò
Bn =< sin(nwt), x(t) >=2
Tx(t)sin(nwt)dt
-T /2
T /2
ò
w =2p
T
ì
í
ïïïïï
î
ïïïïï
Frequency domain
Example I
2/p
|X(F)|
Time domain
=
If the periodic signal is replace with an aperiodic signal (general case)
FT is given by
FT in complex exponential
X(F) = FT (x) = x(t)exp(-2p jFt)dt-¥
¥
ò
x(t) = FT -1(X) = X(F)exp(2p jFt)dF-¥
¥
ò
ì
íï
îï
Is that deadly simple?
Defined the information contained in an outcome xi in x={x1, x2,…,xn}
I(xi) = - log2 p(xi)
H(X)= Si P(xi) I(xi) = - Si P(xi) log2 P(xi)
Information
Entropy
An instantaneous code can be found that
encodes a source of entropy H(X) with an
average number Ls (average length)
such that
Ls >= H(X)
Shannon's first theorem
• Ls = 0.729*1+0.081*3*3+0.009*5*3+0.001* 5
= 1.5980 (remember entropy is 1.4)
Huffman coding
code length
The DTFT gives the frequency representation of
a discrete time sequence with infinite length.
X(w): frequency domain x [n]: time domain
Definition and properties:
X(w) = DTFT (x) = x[n]exp(- jwn)n=-¥
¥
å
x[n] = IDTFT (X) =1
2pX(w)exp( jwn)dw
-p
p
ò
ì
í
ïï
î
ïï
DFT IIIDefinition: The discrete Fourier transform (DFT) and its own
inverse DFT (IDFT) associated with a vector
X = ( X[0], X[1], … , X[N-1] )
to a vector
x = ( x[0], x[1], …, x[N-1] )
of N data points, is given by
X[k] = DFT{x[n]} = x[n]exp - j2pk
Nn
æ
èç
ö
ø÷
n=0
N-1
å
x[n] = IDFT{X[k]} =1
NX[k]exp j
2pk
Nn
æ
èç
ö
ø÷
k=0
N-1
å
ì
í
ïï
î
ïï
clear all
close all
sampling_rate=100;%Hz
omega=30; %signal frequecy Hz
N=20000; %total number of samples
for i=1:N
x_sound(i)=cos(2*pi*omega*i/sampling_rate); %signal
x(i)=x_sound(i)+2*randn(1,1); %signal+noise
axis(i)=sampling_rate*i/N; % for psd
time(i)=i/sampling_rate; % for time trace
end
subplot(1,2,1)
plot(time,x); %signal + noise, time trace
xlabel('second')
ylabel('x')
subplot(1,2,2)
plot(axis,abs(fft(x)),'r'); % magnitude of signal
xlabel('Hz');
ylabel('Amplitude')
sound(x_sound)
Mysterious sound
Simple Matlab program is left on slides
Fig. 2
Spectrogram • t=0:0.001:2; % 2
secs @ 1kHz sample rate
• y=chirp(t,100,1,200,'q'); %
Start @ 100Hz, cross 200Hz at
t=1sec
•
spectrogram(y,128,120,128,1E3);
% Display the spectrogram
• title('Quadratic Chirp: start at
100Hz and cross 200Hz at
t=1sec');
• sound(y)
•• y=chirp(t,100,1,200,'q');
% Start @
• Fs=5000;
• filename = 'h.wav';
• audiowrite(filename,y,Fs);
Time (sec)
F
r
e
q
u
e
n
c
y
DFT for image
• DFT for x
• IDFT for X
X[k1,k2 ]=
n2=0
N2-1
å x[n1,n2 ]exp(-2p jk1N1
n1)n1=0
N1-1
å exp(-2p jk2
N2
n2 )
x[n1,n2 ]=1
N1N2 k2=0
N2-1
å X[k1,k2 ]exp(2p jk1N1
n1)k1=0
N1-1
å exp(2p jk2
N2
n2 )
Original picture Gray scale picture
clear all
close all
figure(1)
I = imread('peppers.png');
imshow(I);
K=rgb2gray(I);
figure(2)
F = fft2(K);
figure;
imagesc(100*log(1+abs(fftshift(F)))); colormap(gray);
title('magnitude spectrum');
figure;
imagesc(angle(F)); colormap(gray)
Example I
Example I
• Notice two facts:
• The signal has a frequency spectrum within the interval 0 to Fs/2 kHz.
• The disturbance is at frequency F0 (1.5) kHz.
• Design and implement a filter that rejects
the disturbance without affecting the signal excessively.
• We will follow three steps:
simple filter design
Step 1: Frequency domain specifications
• Reject the signal at the frequency of the disturbance.
• Ideally, we would like to have the following frequency response:
where w0=2p (F0/Fs) = p/4 radians, the digital frequency
of the disturbance.
Step 2: Determine poles and zeros
We need to place two zeros on the unit circle
z1 =exp(jw0)=exp(jp/4)
and
z2=exp(-jw0)=exp(-jp/4)
z2H (z) = k(z - z1)(z - z
2)
= k[z2 - (z1+ z
2)z + z
1z
2]
= k[z2 - (1.414)z +1]
p/4
Step 2: Determine poles and zeros
• We need to place two zeros on the unit circle
z1 =exp(jw0)=exp(jp/4)
and
z2=exp(-jw0)=exp(-jp/4)
• If we choose, say K=1, the frequency response is shown in the figure.
z2H (z) = k(z - z1)(z - z
2)
= k[z2 - (z1+ z
2)z + z
1z
2]
= k[z2 - (1.414)z +1]
PSD and Phase
• As we can see, it rejects the desired frequency
• As expected, but greatly distorts the signal
Result
• A better choice would be to select the poles close to the zeros, within the unit circle, for
stability.
• For example, let the poles be p1= r exp(jw0) and p2= r exp(-jw0).
• With r =0.95, for example we obtain the transfer function
Recursive Filter
H (z) = k(z- z1)(z- z2 )
(z- p1)(z- p2 )
Recursive Filter
H(z) = k(z- z1)(z- z2 )
(z- p1)(z- p2 )= k
(z- p1 +e)(z- z2 )
(z- p1)(z- p2 )= k 1+
e
(z- p1)
æ
èç
ö
ø÷
(z- z2 )
(z- p2 )
e= z1 - p1 = (1-r) z1 ~ 0. So we have H(w0) = 0, but close to 1 everywhere else
• We choose the overall gain k so that H(z)|z=1=1.
• This yields the frequency response function as shown in the figures below.
H(z) = k(z- z1)(z- z2 )
(z- p1)(z- p2 )= 0.9543
z2 -1.414z+1
z2 -1.343z+0.9025
Recursive Filter
Step 3: Determine the difference
equation in the time
domain
• From the transfer function, the difference
equation is determined by inspection:
y(n)= 0.954 x(n) -1.3495x(n-1)+0.9543 x(n-2)
+1.343 y(n-1)-0.9025 y(n-2)
• Easily implemented as a recursion in a high-level language.
• The final signal y(n) with the frequency spectrum shown in the
following Fig.,
• We notice the absence of the disturbance.
Step 3: difference equation
Step 3: Outcomes
• Define ai = sN-i (reversing the order)
y(n) = a0 x(n) + a1 x(n-1) + … + aN x(n-N)
So when the signal arrives we have
y(n) = SN x(n) + SN-1 x(n-1) + … + S1 x(n-N)
= SN SN + SN-1 SN-1 + … + S1 S1
Matched filter
y(n1,n
2) = m(n
1,n
2)+
s 2(n1,n
2)-Ex 2
s 2(n1,n
2)
(x(n1,n
2)-m(n
1,n
2))
The filtered output is given by
Note that the coefficient a depends on the position
s is the local variance, Ex2 is the global variance (noise)
Wiener Filter
Wiener Filter
past papers before the exam