Advanced Signals and Systems Part 2: Discrete Signals … · ... Advanced Signals and Systems|...

Digital Signal Processing and System Theory| Advanced Signals and Systems| Discrete Signals and Random Processes

Gerhard Schmidt Christian-Albrechts-Universität zu Kiel Faculty of Engineering Institute of Electrical Engineering and Information Engineering Digital Signal Processing and System Theory

Advanced Signals and Systems –

Part 2: Discrete Signals and Random Processes

Digital Signal Processing and System Theory| Advanced Signals and Systems| Discrete Signals and Random Processes Slide II-2 Digital Signal Processing and System Theory| Advanced Signals and Systems| Discrete Signals and Random Processes

Entire Semester:

Contents of the Lecture

Introduction

Discrete signals and random processes

Spectra

Discrete systems

Idealized linear, shift-invariant systems

Hilbert transform

State-space description and system realizations

Generalizations for signals, systems, and spectra


Contents of this Part

Discrete Signals and Random Processes

Definitions

Description of stochastic processes

Examples

Simple operations on stochastic processes


A signal is called periodic if the following conditions holds: If there is no repetition, (i.e. ), then, the signal is non-periodic. The concept is easily extended to – dimensional signals with multiple periods with .


Definitions – Part 1

Periodic and Non-Periodic Signals:

Remark: In the continuous-signal domain, the function is the prototype of a periodic signal, with period . The corresponding sequence is not a-priori periodic – see conditions contained in the equation above!




Energy and power signals:

For closely modeling the physically defined values of continuous signals, we define

the instantaneous/local power:

the instantaneous/local energy:

the total signal energy:

the average power:




Energy and power signals (continued):

For periodic signals the definition of the average power turn into This is the average in any period. Signals for which the sums in the energy definitions do exist ( ), are termed signals of finite energy or, sloppily, energy signals. Signals for which the energy definitions cannot, but the power definitions can be evaluated, are termed signals of finite power or, sloppily, power signals.




Deterministic and stochastic signals:

If is determined by (possibly non-uniquely) and if the relation can be “evaluated”, then is deterministic. If is is of random character or if the relation cannot be “evaluated” despite of a non-random character, then is treated as stochastic. A typical example of a stochastically treated, though actually deterministic signal is speech: Contents are fixed by thoughts, the generation is determined by the “data transmission” and the physiological apparatus – but cannot be described or evaluated by any means. Another example is the error resulting from quantization of a deterministic signal (even for a sinusoid). Assume, e.g. a linear quantizer with four bits. This means: bit quantizer, leading to quantization intervals of a constant size (linear quantizer = straight line through the "stair-case“ characteristic).




Deterministic and stochastic signals (continued):

(Sign-magnitude characteristic)

With limitation (might also be different)

The quantization error is determined as If is known, i.e. deterministic, also the quantization error is deterministic!

But: Difficult evaluation even for a sinusoid signal (Bessel functions, …), rather impossible evaluation for more general signals! Therefore : the quantization error is very often modeled as a random process, termed quantization noise. Within a quantization interval, all amplitudes are assumed to be “equally probable”.



Description of Stochastic Processes – Part 1

Probability and probability-density function:

Definition of the probability function

where is called probability and as well as must be real! For reasons of probability definitions (see theory/theorems of statistics) we have

which must be monotonically non-decreasing over . The probability density function (pdf) is defined as

This is a non-negative function.




Relations and extensions:

The relation between the probability function and the probability density function can be modified (inverted) according to

In addition we get

If is a multidimensional or complex variable or process, e.g. with the definitions “above” (see slides before) need to be extended.




Relations and extensions (continued):

A bi-variate probability function is defined as If the two processes can assumed to be stationary, the probability should only depend on the difference between the two sampling points : A bi-variate probability density function is defined as Again this is a non-negative function:





This can be easily extended to multi-variate probability (density) functions. Please have a look in the corresponding literature. We will continue with bi-variate functions. The relation between probability function and the probability density function can be inverted according to

As in the one-dimensional case we get also

(volume under ).





One might ask how to connect uni-variate quantities with bi- or multi-variate functions :

From 2-D quantities to 1-D descriptions: marginal probability (density) (for all or only for special , correspondingly).

From 1-D quantities to 2-D descriptions:

Only for special cases (will be treated soon)

As a second question the dependency on is of interest. To clarify this we will introduce the concept of stationarity:

If the statistical properties of a one-dimension process do not depend on the (absolute) time index, the process is called stationary. This means that the probability density function does not depend on the time index .





Concept of stationarity (continued):

For an instationary uni-variate process we get

In the bivariate case, the descriptions of an instationary process do not only depend on the distance between two points

but, if the process is instationary, on both and (or on and ):

In our definitions mentioned before, stationarity was assumed a priori. This will hold mostly in the following.

stationary … … instationary

stationary … … instationary




Expectation, moments, correlation, and covariance:

Often only the statistics up to a certain “order” are required. These moments often can be estimated from measured process realizations. For this we introduce the general definition of the expectation operation (or operator) for stationary processes for …

… uni-variate functions:

… bi-variate functions:

Note that is a linear operator in !




Expectation, moments, correlation, and covariance (continued):

Special expectation values:

Mean value, also called first moment :

Mean-square value, also called second moment :

Variance, also called second central moment :

Please note the similarity to AC/DC circuits: represents the DC component, is the DC power, can be interpreted as the AC power, and is the total power. Also we have





Special expectation values (continued):

Please show that the statement

is really true!

Please try first in groups of two students … afterwards solution on the blackboard!






Characteristic function :

Some remarks:

Obviously, is a (modified) continuous Fourier transform of . Due to the unique inversion of this transform, carries all information about .

The Taylor-series coefficients of , via expansion of , are moments of ( , , and higher). Thus, knowledge of all moments is equivalent to the knowledge of , therefore, of .






Cross correlation :

Cross covariance :

Both and have the same meaning: except for the product of the means , they are deterministic sequences describing, according to the definition above, in an identical manner the average similarity of and as a function of the shift (distance) .






Matlab example for an application (localization) of cross correlation …






Autocorrelation function : Note, that this is a special case of the cross correlation with and The probability density function can be obtained from the probability function as

… to be continued on the next slide.






Autocorrelation function (continued) :

The probability density function can be obtained from the probability function as The probability function is defined as

The same process but at

different instances/points!






Auto-covariance function : The auto-correlation sequence/function (ACF) and the auto-covariance sequence/function describe the average similarity of with itself after a shift by .






Additional considerations concerning the autocorrelation and the auto-covariance function:

For a shift of zero we obtain for the autocorrelation function

This is also the “maximum similarity” of and . Thus we have

For the auto-covariance function we get for a shift of zero






Additional considerations concerning the autocorrelation and the auto-covariance function (continued):

By replacing with in the definitions, we obtain the symmetry conditions and are hermitean functions of with their maximum absolute values and being real and non-negative. The symmetry of the cross-correlation and cross-covariance function is less simple, and no maximum statement can be given, in general.





Matlab example for an application (pitch estimation) of autocorrelation …





Independent, uncorrelated, and orthogonal processes:

Statistical independence is defined as the separability of the joint probability density function (for specific values of or for all ). For the bi-variate case we get as a definition Less strict conditions which are often sufficient for cases that focus on minimizing the mean-squared error is given by the uncorrelatedness and by orthogonality. They are defined as follows:

Uncorrelated processes:

Orthogonal processes:

According to the definitions above both are identical if and/or , but in general two processes can be uncorrelated but not orthogonal and vice versa.

Derived on the next slide!



Some Questions …

Independent, uncorrelated, and orthogonal processes:

Partner work – Please think about the following questions and try to find answers (first group discussions, afterwards broad discussion in the whole group).

Assume you have two uncorrelated random processes. The first process has a mean of one. What is required from the second process for being orthogonal with the first process?

……………………………………………………………………………………………………………………………..

……………………………………………………………………………………………………………………………..

How would you estimate the mean of a process?

………………………………………………………………………………………………………………………………

………………………………………………………………………………………………………………………………




Independent, uncorrelated, and orthogonal processes (continued):

Cross-correlation function in case of statistical independence:

… inserting the general definition of the cross-correlation function …

… inserting the definition of independence …

… splitting the integral into single-variable integrals …

… using the definition of the mean value …





For the cross-covariance function (in case of statistical independence) we obtain in the same way Thus we can conclude:

Independence includes uncorrelatedness!

The inversion, however, is not true in general.





The concepts of independence, uncorrelatedness, and orthogonality may be applied also to a single process with independent, uncorrelated, or orthogonal values and with . Thus, for a signal, that is “orthogonal to itself ” we have:

and for a process that is “uncorrelated with itself ” we get

In the following we will call a stationary, zero-mean process with autocorrelation

a white noise process. Note, that some definitions of white noise allow a mean value different from zero.

Unit impulse (defined on the next slide) …



Examples – Part 1

Deterministic signals:

Some basic (deterministic) signals (this should be a repetition of known things):

Unit impulse:

Unit step:

“Double impulse”:

… for corresponding graphs, see next slide …



Examples – Part 2

Deterministic signals (continued):

Graphs for some basic (deterministic) signals:

Unit impulse sequence

Double impulse sequence

Unit step sequence



Examples – Part 3


Some basic (deterministic) signals (this should be a repetition of known things):

Unit ramp:

Rectangle (of length ):

Some remarks:

A unit ramp can also be expressed as a sum over shifted unit step sequences

“-1” is necessary!



Examples – Part 4

Some remarks (continued):

In general the basic signals introduced before (unit impulse, unit step, double impulse, and unit ramp) can be transformed into each other. We can see the following relations:

Intuitively, we can observe the inversion of the difference is the summation. That is very much related with inverting a differentiation by integration.

The summation leads also to the summation property of the unit impulse:


Difference after shift

Summation after shift



Examples – Part 5

Some remarks (continued):

Another property (sampling property) of the unit impulse can be of help in some derivations. It is described as

Multiplication of the sequence by leaves a sequence with only one value at and zeros elsewhere. Sampling of a sequence can be described in this way.

Some interesting parallelisms from the continuous signal domain:




Examples – Part 6


Some basic (deterministic) signals (continued):

(Real) exponential sequence:

Sinusoidal sequence: with : amplitude ( ), : frequency ( ), : phase/angle ( ).



Examples – Part 7


Some basic (deterministic) signals (continued):

Complex harmonic sequence

with Obviously, this includes the sinusoidal sequences with

(General) complex exponential sequence: with as described above and the complex “frequency variable"



Examples – Part 8


The complex exponential sequence can be used to generate all discrete sequences that have been defined before. To understand this sequence in more detail we will start with a continuous version and split it into a complex harmonic and an real exponential term:

determines the frequency, for an exponential (or a constant) sequence can be generated determines that decay (or growth), for

a constant or a sinusoid can be generated)



Examples – Part 9


By sampling we can go from the continuous domain to the discrete domain. Note, that not for all a periodic sequence (assuming ) is generated. The signals , , , and are termed often elementary signals – they are the basis of a very general treatment of signals and systems.



Some Questions …

Partner work – Please think about the following questions and try to find answers (first group discussions, afterwards broad discussion in the whole group).

What is meant by “being an eigenfunction or eigensequence” of a linear, shift-invariant system?

……………………………………………………………………………………………………………………………..

……………………………………………………………………………………………………………………………..

Why is the sequence not necessarily periodic?

………………………………………………………………………………………………………………………………

………………………………………………………………………………………………………………………………




Examples – Part 10

Stochastic signals:

A stochastic signal is described by its properties, mainly by a description of its amplitude distribution and by a description of its correlation, which gives also information about its average spectrum (e.g. power spectral density). We will focus now on a few probability density functions.

Uniform distribution:

is a rectangle between and . The “incremental probabilities” are constant, meaning that we obtain for the probability density function

The factor follows from the rule of the total probability


For the mean and the standard deviation we get:

Mean:



Stochastic signals (continued):

… inserting the uniform distribution …

… inserting the limits …

… using (a² - b²) = (a-b)(a+b) …

… simplifying …





For the mean and the standard deviation we get (continued):

Standard deviation (analog to the mean deviation):

The uniform distribution is the probability density function suitable to describe quantization noise/errors (as briefly discussed before).





Univariate probability density functions (continued):

Gaussian or Normal distribution:

(“bell-shaped” curve with a symmetry point at and a width proportional to ). It is parametric distribution with very nice properties (see references …). Some further densities that can be generated out of Gaussians will be shown on the next slide.





Processes that can be generated by combining several independent zero-main Gaussian processes that are mutually uncorrelated and have standard deviation :






Graphs of the before mentioned densities


… taken from D. Wolf: Signaltheorie – Modelle und Strukturen, Springer, 1999





Finally we discuss two bivariate distributions:

Bivariate uniform distribution in the case of independence of the two processes and :

„Height“:





Bivariate uniform distribution (continued): For the cross correlation function we get For the cross covariance function we obtain

… in both definitions it was assumed that both processes are real (not complex) …





Bivariate distributions (continued):

Bivariate Gaussian distribution in the case of independence of the two real processes and : The formula above describes a “bell-shaped” volume with ellipsoidal level lines. The ellipses have …

… a center which is determined by ,

… width of which are determined by and ,

… an “angle” towards the axes that depends on , with

Cross covariance at lag





Bi-variate distributions (continued): Two examples for bi-variate Gaussian distributions.

Note for the bi-variate density can be separated into two uni-variate densities (statistical independence).



Simple Operations on Stochastic Processes – Part 1

Summation:

The summation of random processes (possibly after weighting) is a frequent operation. We will consider here the simplest case: the summation of two processes:

We assume that and are stationary processes with known statistics (we have knowledge about ). What we want to know are the statistics of , namely .

Mean of :

Please try to prove that result on your own!

… afterwards presentation on the blackboard ….

Independent of !




Summation (continued):

Second moment and variance of :

For the variance we obtain in an analog manner:

… inserting …

… splitting the brackets in three parts …

… inserting the definitions of the individual expectations …

Note, that stationary, real processes were assumed!




Summation (continued):

Probability density function of : The density function can only be computed if statistical independence of and is assumed. In this case we get a quite simple result: This means that the density function of the sum of two statistically independent processes can be obtained by convolving their individual densities. To prove this result we will use the characteristic function …

… derivation on the blackboard!




General mapping:

The probability density function of a process can be derived from its probability function . However, in many cases it easier to derive the density function directly out of and . For that “direct” method it is necessary that does not contain a Dirac distribution!

For the transformation of the density function we assume that the characteristic has for and for with an amount of solutions:

If contains distributions, a separate treatment can be done: is transformed to !




General mapping (continued):

Example for a quadratic characteristic, leading to solutions.





Now we can define the following „events“ (probabilities):

For sufficiently small both events are disjoint and we obtain for their probabilities

In addition we can make the following approximations:





Using these approximation we get

If we assume that is differentiable, we can rewrite the approximation from above for With

we obtain finally This relation can be utilized

for system analysis purposes. However, also systems for manipulation

of densities (processes) can be realized in such a way.




Graphic close to E. Hänsler: Statistische Signale, 2. edition, Springer, 1996




Summary and Outlook

This part:

Definitions

Description of stochastic processes

Examples

Simple operations on stochastic processes

Next part:

Spectra

Advanced Signals and Systems Part 2: Discrete Signals … · ... Advanced Signals and Systems|...

Documents

Transcript of Advanced Signals and Systems Part 2: Discrete Signals … · ... Advanced Signals and Systems|...