Chapter 4: DISCRETE PROBABILITIES AND RANDOM ...bazuinb/ECE3800/B_Notes04.pdfChapter 4: DISCRETE...
Transcript of Chapter 4: DISCRETE PROBABILITIES AND RANDOM ...bazuinb/ECE3800/B_Notes04.pdfChapter 4: DISCRETE...
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 1 of 51 ECE 3800
Charles Boncelet, βProbability, Statistics, and Random Signals," Oxford University Press, 2016. ISBN: 978-0-19-020051-0
Chapter 4: DISCRETE PROBABILITIES AND RANDOM VARIABLES
Sections 4.1 Discrete Random Variable and Probability Mass Functions 4.2 Cumulative Distribution Functions 4.3 Expected Values 4.4 Moment Generating Functions 4.5 Several Important Discrete PMFs 4.5.1 Uniform PMF 4.5.2 Geometric PMF 4.5.3 The Poisson Distribution 4.6 Gambling and Financial Decision Making Summary Problems
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 2 of 51 ECE 3800
Probability Mass Functions (pmf)
From: http://en.wikipedia.org/wiki/Probability_mass_function
In probability theory, a probability mass function (abbreviated pmf) gives the probability that a discrete random variable is exactly equal to some value.
Cumulative Distribution Function (CDF)
Probability mass functions are related discrete countable outcomes of an experiment and the probability each outcomes has at the discrete values.
A pmf for rolling a six sided dice would be
1 2 3 4 5 6
xf X1.0
0.0x
1/6
The probability of each discrete value is 1 in 6. Therefore, you would define βdelta functionsβ of magnitude 1/6 at each of the six discrete values.
In addition, there is function based on performing a summation from β infinite to +infinite that would be a βcumulative functionβ of the probability called the Cumulative Distribution Function.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 3 of 51 ECE 3800
Properties of the pmf include
1. π π ππ π π₯ 0 (all probabilities are positive)
2. The summation of the pmf for all k is equal to 1.
π π 1.0
The pmf function may be defined based on a table or other form. For example, a possible pmf would be
π π
0.4, π 0 0.3, π 10.2, π 20.1, π 3
where
0 π π 1
π π 0.4 0.3 0.2 0.1 1.0
A pmf for flipping a coin
π π1 π, π 0π, π 1
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 4 of 51 ECE 3800
This is also referred to as the Bernoulli Distribution, see
https://en.wikipedia.org/wiki/Bernoulli_distribution
The Bernoulli Distribution is a special case of the Binomial Distribution for (n=1) to be discussed later.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 5 of 51 ECE 3800
Generalized properties of probability mass function (pmf) (from Stark and Wood)
Properties of the pmf include
1. xforxf X ,0
2. 1
u
X uf
3.
x
u
XX ufxF The CDF is the sum of the pmf from ββ to x
4.
2
1
21Prx
xuX ufxXx
The probability is the pmf sum of the region(s) of interest
Generalized Properties of CDF (from Stark and Wood)
Cumulative Distribution Function (CDF):The probability of the event that the observed random variable X is less than or equal to the allowed value x.
xXxFX Pr
The defined function can be discrete or continuous along the x-axis. Constraints on the cumulative distribution function are:
xforxFX ,10
0XF and 1XF (property #1 and #3 in the textbook)
XF is non-decreasing as x increases (property #2 in the textbook)
1221Pr xFxFxXx XX Notice that the βinequalitiesβ are important for discrete random variables!
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 6 of 51 ECE 3800
For discrete events, the cumulative density function, on the x-axis, consists of discrete steps βclimbingβ towards 1 at the appropriate points.
For a six-sided die,
6
161,Pr intint egereger aaX
The cumulative density function can be defined as:
For discrete events, 061,Pr intint egereger aaX or
061,Pr intintintint egerXegerXegereger aFaFaaX
There will be a difference for continuous events β¦ coming soon.
Examples (watch the definition of the inequalities):
6
111Pr XFX
2
133Pr XFX
6
555Pr XFX
0.177Pr XFX
6
2
6
41414Pr XFX
6
3
6
2
6
52552Pr XX FFX
From: Probabilistic Methods of Signal and System Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 7 of 51 ECE 3800
Important Discrete Random Variables
The Uniform Random Variable
The Bernoulli Random Variable
The Binomial Random Variable
The Geometric Random Variable
The Poisson Random Variable
The Zipf Random Variable
Definitions and examples available on homework solution/password web site.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 8 of 51 ECE 3800
UpdatingPreviousExamples
Experiment: Flip two Coins and count the number of heads
HHHTTHTTSPair ,,, 2,1,0S
For xXxFX Pr
xfor
xfor
xfor
xfor
xFX
2,1
21,43
10,41
0,0
And the probability mass function, xXxf X Pr , is then
else
xfor
xfor
xfor
xf X
,0
2,41
1,42
0,41
1 2 3 4
xFX
1.0
0.0x
0-1 1 2 3 4
xf X
1.0
0.0x1/4
0-1
1/2
Cumulative Distribution Function (CDF)
Probability Mass Function (pmf)
Note: The pmf corresponds to multiple Bernoulli trials resulting in a Binomial Probability of n=2 trials with a probability of p=50%.
ππ π΄ ππππ’ππππ π π‘ππππ ππ π π‘πππππ π πππ β π β 1 π
π 0 20
β π β 1 π π 1 21
β π β 1 π π 2 22
β π β 1 π
π 0 1 β 0.5 π 1 2 β 0.5 π 2 1 β 0.5
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 9 of 51 ECE 3800
4.3 Expected Values, Moments, Central Moments and Variance
For random variables, the expected value operation produces a βprobabilistic averageβ of the particular probability based function of interest.
πΈ π π π π₯ β π π
https://en.wikipedia.org/wiki/Expected_value
where g(X) is a function of the random variable X and p(k) is the pmf. (Chapter 3 ROI calculations are expected values!)
Meanor1stMoment
For example, the 1st moment or mean value of the random variable is defined by
π β‘ πΈ π π π₯ β π π
Themeansquarevalueorsecondmomentis
πΈ π π π₯ β π π
Otherβmomentsβ(thenthmoment)aredefinedas
πΈ π π π₯ β π π
CentralMoments
πΈ π π π π β π π
The2ndcentralmomentorvariance
π β‘ πΈ π π π π β π π
The standard deviation is defined in terms of the 2nd central moment (or variance) as
π π
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 10 of 51 ECE 3800
Defined PMF functions: mean and standard deviation
Importance of mean and standard deviation
Often when we talk about values we say the βmean +/- 1 standard deviationβ
π π
That is to say that we expect the experimental result x to be
π π π₯ π π
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 11 of 51 ECE 3800
A useful variance formula β moments proof (Theorem 4.1)
π πΈ π π π π β π π
π πΈ π π π 2 β π β π π β π π
π πΈ π π π β π π 2 β π β π β π π π β π π
π πΈ π π πΈ π 2 β π π
π πΈ π π πΈ π π
or
πΈ π π πΈ π πΈ π
You need only compute the 1st and 2nd moment to derive the variance.
Alternate expected value operator proof
π πΈ π π πΈ π 2 β π β π π
π πΈ π 2 β π β πΈ π πΈ π
π πΈ π 2 β π β π π
π πΈ π π
If you know the mean and standard deviation, you can compute the 2nd moment. Note that the second moment is related to signal power/energy!
πΈ π π π
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 12 of 51 ECE 3800
More basics math related to βthe expected value operatorβ
πΈ π π π π₯ β π π
A constant, non-random variable
πΈ π π β π π π β π π π
A constant multiplication
πΈ π β π π π β π π β π π π β π π β π π π β πΈ π π
Therefore
πΈ π β π π π β π π β π π π π β π β π π π β π π
Note: πΈ π πΈ π β π π π β π π π
Summations (superposition)
πΈ π π π π π π π π β π π π π β π π π π β π π
πΈ π π π π πΈ π π πΈ π π
Note that multiplication of two functions generally does not work!
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 13 of 51 ECE 3800
The expected value operator is a linear operator β¦ therefore: the integration and differentiation of a function of random variables can be performed as.
πππ£
πΈ π π, π£πππ£
π π, π£ β π ππππ£
π π, π£ β π π πΈπππ£
π π, π£
πΈ π π, π£ β ππ£ π π, π£ β π π β ππ£ π π, π£ β ππ£ β π π
πΈ π π, π£ β ππ£ πΈ π π, π£ β ππ£
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 14 of 51 ECE 3800
Some Common Discrete Random Variables
1. Bernoulli β (flipping coins, one of two results)
1,0XS
qpp 10 and pp 1 , for 10 p
else
kp
kq
kpmfkPB
,0
1,
0,
1 kpkqkpmfkPB
k
kq
k
kFB
1,1
10,
0,0
1 kupkuqkpmfkFB
Mean, 2nd moment and variance
π β‘ πΈ π π π β π π
π β‘ πΈ π π 0 β π 1 β π π
πΈ π π π β π π
πΈ π π 0 β π 1 β π π
π πΈ π π
π π π π β 1 π π β π
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 15 of 51 ECE 3800
2. Binomial β (S a sequence of Bernoulli trials)
nS X ,,2,1,0
knkk pp
k
np
1 , for nk ,,2,1,0
else
nkppk
n
kpmfkPknk
B
,0
,,1,0,1
kn
nkppj
n
k
kFk
j
jnjB
,1
0,1
0,0
0
Mean, 2nd moment and variance
See Example 4.3-1
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 16 of 51 ECE 3800
3. Geometric
First Version
,2,1,0XS
π π πππ π π β 1 π , π 0,1,β― ,β0, πππ π
πΉ π
0,π 0
π β 1 π , 0 π β
Math Tricks β¦.
1,1
11
1
00
qfor
q
qpqppp
kk
j
jk
j
j
1
1
0
111
11
k
kk
j
j qp
qppp
Therefore, it is commonly stated as
πΉ π0,π 0
π β 1 π
1 π, 0 π β
Alternate version xaaxpmfxP xXX 0,1
Second Version
,2,1XS
π π πππ π π β 1 π , π 1,2,β― ,β0, πππ π
πΉ π0,π 1
π β 1 π1 π
, 1 π β
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 17 of 51 ECE 3800
Determine the expected value of the 1st version (caution different notations!)
0x
X nPxXE
πΈ π π₯ β 1 π β π
β
1 π β π₯ β π
β
Note: 1,1
1
0
afora
ax
x
and 2
0
1
0 1
1
1
1
aada
daxa
da
d
x
x
x
x
2
0
1
1
111
aaaaxaaXE
x
x
πΈ π ππ
1 π
This allows the mean value to be quickly found once βaβ is known. Determine the 2nd moment
0
22
xX nPxXE
πΈ π π₯ β 1 π β π
β
Note: 32
2
0
2
02
2
1
2
1
11
aada
daxxa
da
d
x
x
x
x
0
2222 11x
xx axaxxaaXE
0
1
0
222 111x
x
x
x axaaaxxaaXE
23
22
1
11
1
21
aaa
aaaXE
πΈ π2 β π
1 ππ
1 π2 β π π
Determine the variance
πΈ π π πΈ π πΈ π
2
2
22
111
2
a
a
a
a
a
aXE
πΈ π ππ
1 ππ
1 ππ π
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 18 of 51 ECE 3800
A six-sided die example (uniform pmf)
π π
0, π 116
, π 1,2,3,4,5,6
0, 6 π
Mean value
π β‘ πΈ π π₯ β π π π β16
16β π
16β
7 β 62
72
3.5
and
π πΈ π π π π β π π π π β16
π π 2 β π β π π β16
16β π 2 β π β
16β π
16β π 1
π16β
13 β 7 β 66
2 β π β π π916
π
π916
494
182 14712
3512
2.927
The standard deviation is
π π =1.708
Note that a generalized form for a uniform discrete random variable exists!
*Discrete Math Hints
ππ 1 β π
2
π2 β π 1 β π 1 β π
6
ππ 1 β π
4
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 19 of 51 ECE 3800
Bernoulli Trials
A repeated trial can take the form of:
1. Repeated experiments where the relative frequency of occurrence is of interest
2. The creation of a new experiment that consists of a defined number of elementary events
Bernoulli Trials: Determining the probability that an event occurs k times in n independent trials of an experiment.
For some experiment let: pA Pr and qA Pr
where 1 qp
Then for an experiment where we get 2 event βAβs followed by 2 βnot Aβ (i.e., AAAAB ) β¦
knk qpAAAAB PrPrPrPrPr
But what about the other ways to have 2 event Aβs in 4 trials? Note that for each instance, the probability of occurring will be the same as just defined β¦ so how many of them are there?
AAAAAAAAAAAAAAAAAAAAAAAA ,,,,,
The number of occurrences can be defined using binomial coefficients and the Binomial Theorem.
The number of instances is defined by the binomial coefficient, kn C or
k
n.
the number of ways to select k elements out of a set of n elements ...
Where !!
!
knk
n
k
n
Therefore, to describe the desired outcome of 2 Aβs in 4 trials, the probability is
242242
4 !24!2
4
2
4242Pr
qpqpptrialsintimesoccuringA
Therefore β¦
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 20 of 51 ECE 3800
Binomial Probability
The probability that an event occurs k times in n independent trials of an experiment can be defined as
knkn qp
k
nkptrialsnintimeskoccuringA
Pr
ExampleFlippingCoins
The probability for each outcome of flipping a coin 4 times, where Pr(H)= p and Pr(T)=q with
2
1 qp
4 H : 16
1
16
11
2
1
2
1
4
4
4
44Pr
04444
4
qppHHHH
3 H & 1 T: 16
4
16
14
2
1
2
1
3
4
3
43Pr
13343
4
qppHHHT
2 H & 2 T: 16
6
16
16
2
1
2
1
2
4
2
42Pr
22242
4
qppHHTT
1 H & 3 T: 16
4
16
14
2
1
2
1
1
4
1
41Pr
31141
4
qppHTTT
4 T: 16
1
16
11
2
1
2
1
0
4
0
40Pr
40040
4
qppTTTT
What if p = 0.6 and q=0.4? An βunfair coinβ!!
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 21 of 51 ECE 3800
ExampleBinaryCommunications
Example 1: For a bit-error-rate (BER) of 310 in a binary data stream, what is the probability of exactly 1 error in a 32-bit word?
3131332 10110
1
321
p
313332 10110321 p
0310.09695.010321 332 p
Example 2: For a bit-error-rate (BER) of 310 in a binary data stream, what is the probability of 0 errors in a 32-bit word?
3230332 10110
0
320
p
32332 101110 p
π 0 0.9685
Example 3: What is that probability of having one or more errors in 32 bits?
32
1
323332
132 10110
32
i
ii
i iip
or 0315.09685.0101 32
32
132
pipi
Notice that 1 bit error dominates the computation β¦
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 22 of 51 ECE 3800
ExampleBaseball/SoftballStatistics
Example 1: A batter has a 0.250 batting average. What is the probability that the batter gets 1 hit in 4 at bats?
knkn qp
k
nkptrialsnintimeskoccuringA
Pr
314 25.0125.0
1
41
p
422.064
27
444
333
4
1475.025.0
!14!1
!41 31
4
p
Example 2: A batter has a 0.250 batting average. What is the probability that the batter gets 2 hit in 4 at bats?
224 25.0125.0
2
42
p
211.0128
27
44
33
44
11
2
3475.025.0
!24!2
!42 22
4
p
Example 3: A batter has a 0.250 batting average. What is the probability that the batter gets at least 1 hit in 4 at bats?
014321 44444 ppppp
684.0256
175
256
811
4444
33331175.025.0
!04!0
!4101 40
4
p
Example 4: A batter has a 0.250 batting average. What is the probability that the batter gets at most 1 hit in 4 at bats?
314044 75.025.0
!14!1
!475.025.0
!04!0
!410
pp
738.0256
189
256
108
256
81
444
333
4
14
4444
3333110 44
pp
Defining a player having a hitting slump β¦ how many at bats until it is a slump?
How many at bats would the batter need to take β¦ to have a 90% (or 99%) probability of getting at least one hit.
Miguel Cabreraβs career BA β¦.was 0.315 ? (see Excel Spread Sheet
1 π 0 0.900 or 1 π 0 0.990
1 π 0 0.8949 or 1 π 0 0.9889
See Baseball Spread Sheet
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 23 of 51 ECE 3800
Example 1-10.2 from Cooper-McGillem
In playing an opponent of equal ability, which is more probable:
knkn qp
k
nkptrialsnintimeskoccuringA
Pr
a) To win 4 games out of 7, or to win 5 games out of 9?
3744747 5.05.0
!3!4
!7
4
74
qpp
2734.0128
1
6
2105.0
23
5674 7
7
p
5955959 5.05.0
!4!5
!9
5
95
qpp
2461.0512
1
24
30245.0
234
67895 9
9
p
Therefore, winning 4 out of 7 is more probable.
b) To win at least 4 games out of 7, or to win at least 5 games out of 9.
77777 5.0
7
7
6
7
5
7
4
77654
pppp
128
1
1
1
1
7
2
42
6
2107654 7777
pppp
50.0128
64
128
11721357654 7777 pppp
999999 5.09
9
8
9
7
9
6
9
5
998765
ppppp
512
11
1
9
2
72
6
504
24
302498765 99999
ppppp
50.0512
256
512
119368412698765 99999 ppppp
The probabilities are the same! (You should have a 50-50 chance of winning or losing) !
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 24 of 51 ECE 3800
ECE Applications of Bernoulli Trials
(1) Bit errors in binary transmissions:
Degree of error detection and correction needed. The theoretical validation of performance of the system after βextra bitsβ for error correction have been added.
bit-error-rate may also increase if a greater bandwidth is needed because of the βextra bitsβ
(2) Radar (or similar) signal detection:
After setting a signal detection threshold, the expected signal should be above the threshold when being received for a fixed number of sample times. If the signal is above the threshold for m (or more) of n sample periods, one may also say the signal has been detected.
n
mk
kns
ks
n
mks pp
k
nknpDetection 1,Pr
One can also define a noise threshold where the noise should not be above a particularly level more than m (or more) of n time samples.
n
mk
kna
ka
n
mka pp
k
nknpAlarmFalse 1,_Pr
(3) System reliability improvement using redundancy.
If a unit has a known failure rate, by incorporating redundant units, the system will have a longer expected lifetime.
Important when dealing with systems that cannot be serviced, systems that may be very expensive to service, systems that require very high reliability, system with components with high failure rates, etc. . (e.g. satellites, computer hard-disk farms, internet order entry servers).
Defining the probability that one of the redundant elements is still working β¦
FailedAllFunctional _Pr(1Pr
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 25 of 51 ECE 3800
Moments of Random Variables
Example4.3β1Binomial/BernoulliR.V.StarkandWoods
xnxXX pp
x
nxpmfxP
1
Determine the expected value
n
xX nPxXE
0
n
x
xnx ppx
nxXE
0
1
pnXE
Proof based on Wikipedia https://en.wikipedia.org/wiki/Binomial_distribution
n
x
xnxn
x
xnx ppxxn
nxpp
xxn
nxXE
10
1!!
!1
!!
!
Subtle change since you are multiplying by x, the x=0 term is always zero. As the math trick, can the βcombinatorialβ and probability be restructure to sum to one based on the βn-1β term summation that remains?
n
x
xnx ppxxn
nXE
1
1!1!
!
n
x
xnx pppxxn
nnXE
1
111 1!1!11
!1
n
x
xnx ppxxn
npnXE
1
111 1!1!11
!1
Let 1 xy
1
0
11!!1
!1n
y
yny ppyyn
npnXE
But the 1.0 desired is 11!!
!
0
m
y
ymy ppyym
m
Therefore pnXE
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 26 of 51 ECE 3800
Determine the 2nd moment
0
22
xX nPxXE
pnpnnXE 22 1
Proof
πΈ π π₯ βπ!
π π₯ ! β π₯!β π β 1 π π₯ β
π!π π₯ ! β π₯!
β π β 1 π
Cancelling one of the x and adjusting the summation. Now make two terms β¦
πΈ π π₯ 1 1 βπ!
π π₯ ! β π₯ 1 !β π β 1 π
n
x
xnxn
x
xnx ppxxn
npp
xxn
nxXE
11
2 1!1!
!1
!1!
!1
The second term was previously computed (math trick β¦ make summation = 1.0). The first term can now βcancelβ an (x-1) β¦ and look for a summation in x-2 terms.
pnppxxn
nXE
n
x
xnx
2
2 1!2!
!
n
x
xnx pppxxn
nnnpnXE
2
22222 1!2!22
1!2
n
x
xnx ppxxn
npnnpnXE
2
22222 1!2!22
!21
11 22 pnnpnXE
pnpnnXE 22 1
Next determine the variance
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 27 of 51 ECE 3800
Determine the variance
222 XEXEXE
222 1 pnpnpnnXE
pnpnpnpnpnpnXE 2222222
qpnppnXE 12
We can evaluate the variance form
πΈ π ππ
ππ
π β π
The maximum variance occurs for p=0.5 with minimal variances near p=0 or p=1.0.
Figure 4.3-1 Variance of a binomial RV versus p.
The variances is the largest when the individual event probability is p=q=0.5 !
If you were building a communications system, this can be used to define what the desired bit-wise probability should be to send the most information per bit !
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 28 of 51 ECE 3800
Example Geometric Distribution (not identical to textbook index is offset by 1!)
Geometric Distribution:
In probability theory and statistics, the geometric distribution is either of two discrete probability distributions:
The probability distribution of the number Y of Bernoulli trials needed to get one success, supported on the set { 1, 2, 3, ...}
The probability distribution of the number X = Y β 1 of failures before the first success, supported on the set { 0, 1, 2, 3, ... }
https://en.wikipedia.org/wiki/Geometric_distribution
The number of heads before the first tail or the number of failures before the first success is:
π π₯ πππ π₯ π β 1 π , 0 π₯
π π₯ πππ π₯ 1 π β π, 0 π₯
Determine the expected value
0x
X nPxXE
00
11x
x
x
x axaaaxXE
Note: 1,1
1
0
afora
ax
x
and 2
0
1
0 1
1
1
1
aada
daxa
da
d
x
x
x
x
2
0
1
1
111
aaaaxaaXE
x
x
πΈ π ππ
1 π
πΈ π π1 ππ
This allows the mean value to be quickly found once βaβ is known.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 29 of 51 ECE 3800
Determine the 2nd moment
0
22
xX nPxXE
0
22 1x
xaaxXE
Note: 32
2
0
2
02
2
1
2
1
11
aada
daxxa
da
d
x
x
x
x
0
2222 11x
xx axaxxaaXE
0
1
0
222 111x
x
x
x axaaaxxaaXE
23
22
1
11
1
21
aaa
aaaXE
22
22 2
11
2
a
a
a
aXE
Determine the variance
222 XEXEXE
2
2
22
111
2
a
a
a
a
a
aXE
22
22
11 a
a
a
aXE
πΈ π ππ
1 ππ
1 ππ π
πΈ π π1 ππ
1 ππ
1 π π β 1 ππ
1 ππ
πΈ π π π1 ππ
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 30 of 51 ECE 3800
Now can you repeat the computations for the other form of the geometric probability?
π π πππ π 1 π β π β, 1 π
π π πππ π π β 1 π , 1 π
resulting in (Assuming Y=X+1 based on the π πππ 0 π₯ and π πππ 1 π¦ )
πΈ π π πΈ π 1 π1 ππ
11π
πΈ π π π πΈ π π1 ππ
Note that the shape of the CDF is the same for the two distributions. Therefore, the means are different (shifted), but the variances must be identical.
[class derivation of equivalence of variance ?]
The geometric random variable arises in applications where one is interested in the time (i.e., number of trials) that elapses between the occurrences of events in a sequence of independent experiments. Examples where the modified geometric random variable arises are:
number of customers awaiting service in a queueing system (line at grocery store or DMV);
number of white dots between successive black dots in a scan of a black-and-white document.
https://en.wikipedia.org/wiki/Geometric_distribution
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 31 of 51 ECE 3800
Chebyshev Inequality (Stark and Woods β continuous derivation)
There are a number of probability relationships that bound aspects of engineering problems. They are typically based on moments, particularly the mean and variance. This is the first.
The Chebyshev inequality furnishes a bound on the probability of how much an R.V. can deviate from its mean value.
ChebyshevinequalityTheorem4.4β1
Let X be an arbitrary R.V. with known mean and variance. Then for any 0
2
2
XXXP
Derivation
dxxfXxXXEXX X
2222
Then
Xx
XX dxxfXxdxxfXx222
and
XxPdxxfdxxfXxXx
X
Xx
X2222
Results #1:
XxP2
2
If we also consider the complement of the probability described,
1 XxPXxP
and using the complement
XxP12
2
Therefore
Results #2: 2
2
1 XxP
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 32 of 51 ECE 3800
It may be convenient to define the delta function in terms of a multiple of the standard deviation.
k
Then the Chebyshev inequality becomes
2
1
kkXxP
2
11
kkXxP
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 33 of 51 ECE 3800
Chebyshev Inequality Textbook description
Defining a bound based on the variance of a random variable.
ππ |π π | ππππ ππ
Define an indicator factor
πΌ π₯1, π₯ β π΄0, π₯ β π΄
Taking the expected value
πΈ πΌ π₯ πΌ π₯ β π π πΌ π₯ β ππ π π₯
πΈ πΌ π₯ 1 β ππ π π₯: β
ππ π β π΄
For the Chebyshev Inequality the set A is defined based on the equation
π΄ |π π | π
which can be interpreted as
πΌ| |π ππ
which can be graphically depicted as
Notice that the two functions are equal for
π ππ
1
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 34 of 51 ECE 3800
or
π π π
Based on previous derivations, we know that
πΈπ ππ
πΈ π ππ
πππ ππ
The taking an expected value of the previous interpreted set
πΈ πΌ| | πΈπ ππ
πππ ππ
or based on the set
πΈ πΌ| | ππ π β π΄ ππ |π π | π πΈπ ππ
πππ ππ
Note that the Chebyshev Inequality is a bound that applies in all cases. There is no judgment or determination if it is a good or even useful bound!
Note that for any R.V. where the variance tends to zero, you would have
ππ |π π | π β 0
and the βzero variance random variable must equal the mean value!
Extension, relating epsilon to sigma β¦
π π β π with πππ π π
ππ |π π | π β πππ β π
ππ |π π | π β π1π
or considering the negative of the probability function
11π
1 ππ |π π | π β π
or
11π
ππ |π π | π β π
Note that this is developed here, but will be used and discussed much later in the course.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 35 of 51 ECE 3800
Moment-Generating Functions (Stark & Woods)
The text is now moving into some advanced concepts that support mathematical derivation of higher order moments.
I have been exposed to problems where the 4th moment of a R.V. is required as part of a solution. If you really like and are comfortable with Laplace and Fourier Transforms these approach provide solutions faster and more easily than more brute force summation or integral approaches.
The moment generation function (MGF) is the two sided Laplace transform of the probability mass function (pmf) or the probability density function (pdf). If the MGF exists, there is a forward and inverse relationship between the MGF and the pmf/pdf. The MGF is defined based on the expected value as
π π’ πΈ ππ₯π π’ β π
π π’ ππ₯π π’ β π β π π
For continuous pdf, we would have
dxxtxftM XX exp
If you like s better than t in your Laplace transforms β¦
dxxsxfsM XX exp
For discrete R.V. we perform a discrete Laplace transform
i
iiXi
iiXX xsxPxsxpmfsM expexp
Why do we do this?
1. It enables a convenient computation of the higher order moments
2. It can be used to estimate fx(x) from experimental measurements of the moments
3. It can be used to solve problems involving the computation of the sums of R.V.
4. It is an important analytical instrument that can be used to demonstrate results and establish additional bounds (the Chernoff Bound and the Central Limit Theorem).
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 36 of 51 ECE 3800
TheMGFenablesaconvenientcomputationofthehigherordermoments
Based on the definition
XtEtM X exp
Perform the Taylor series expansion of the exponential
ππ₯π π₯ 1π₯1!
π₯2!
β―π₯π!
β―
π π‘ πΈ ππ₯π π‘ β π πΈ 1π‘ β π
1!π‘ β π
2!β―
π‘ β ππ!
β―
or
π π‘ πΈ ππ₯π π‘ β π 1π‘ β π
1!π‘ β π
2!β―
π‘ β ππ!
β―
The mi are the ith moments of the density function!
So how would we solve for the moments? By taking the derivatives and setting t=0!
Taking the 1st derivative β¦
πππ‘
π π‘πππ‘πΈ ππ₯π π‘ β π 0
π1!
2 β π‘ β π2!
β―π β π‘ β π
π!β―
Setting t=0
πππ‘
π π‘ 0π1!
0 β― 0 β―π1!
π
Taking the nth derivative β¦
πππ‘
π π‘ 0 β― 0π! β ππ!
π! β π‘ β ππ 1!
β―
Setting t=0
πππ‘
π π‘ 0 β― 0π! β ππ!
0 β―π! β ππ!
π
Therefore, all moments can be determined if the moment generation function exists!
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 37 of 51 ECE 3800
Additionalusefulexamples:
Example 4.5-2 MGF of Binomial
knkX qp
k
nkpmf
MGF:
dxxtxftM XX exp
n
n
knkX ktqp
k
ntM
0
exp
n
n
knkX qtp
k
ntM
0
exp
Magical math steps β¦ not really, but I havenβt done the derivation myself β¦
nX qtptM exp
The 1st Moment
0
1
0
expexpexp
t
nn
tX tpqtpnqtp
ttM
t
pnpqpntMt
n
tX
1
0
The 2nd Moment
0
1
02
2
expexp
t
n
t
X qtptpnt
tMt
0
2
0
1
02
2
expexp1exp
expexp
t
n
t
n
t
X
tpqtpntpn
qtptpntMt
221
02
2
1
nn
t
X qpnpnqppntMt
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 38 of 51 ECE 3800
ppnpnnpnpntMt
t
X
11 22
02
2
qpnpntMt
t
X
2
02
2
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 39 of 51 ECE 3800
Example 4.5-3 MGF of Geometric Distribution
10,1 fornpmf nX
MGF:
dxxtxftM XX exp
0
exp1n
nX nttM
Note infinite sum: 1,1
1
0
afora
ax
x
ttM X exp1
1
The 1st Moment
0
20
expexp1
1
exp1
1
ttX t
ttttM
t
111
11
exp1
exp12
0
20 tt
Xt
ttM
t
The 2nd Moment
2
02
2
exp1
exp1
t
t
ttM
tt
X
tt
t
t
ttM
tt
X expexp1
exp12
exp1
exp132
02
2
0
3
2
0
20
2
2
exp1
2exp12
exp1
exp1
ttt
Xt
t
t
ttM
t
2
2
02
2
1
2
1
t
X tMt
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 40 of 51 ECE 3800
Discrete Uniform pmf
The same probability for m different values from 1 to m.
ππ π π
0, π 11π
, 1 π π
0, π π
Computing the mean or first moment
πΈ π π π β1π
πΈ π π1πβ π
1πβπ β π 1
2π 1
2
Computing the second moment
πΈ π π β1π
πΈ π1πβ π
1πβ
2 β π 1 β π 1 β π3 β 2
2 β π 1 β π 16
πΈ π2 β π 3 β π 1
6
Computing the variance or second central moment
πΈ π π πππ π π πΈ π π
πππ π π2 β π 3 β π 1
6π 1
2
πππ π π2 β π 3 β π 1
6π 2 β π 1
4
πππ π π4 β π 6 β π 2 3 β π 6 β π 3
12π 1
12
πππ π ππ 1
12π 1 β π 1
12
Computing the Moment Generating Function
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 41 of 51 ECE 3800
π π’ ππ₯π π’ β π β π π ππ₯π π’ β π β1π
π π’1πβ ππ₯π π’
1πβ ππ₯π π’ β ππ₯π π’
π π’1πβ ππ₯π π’ β
1 ππ₯π π’ β π1 ππ₯π π’
1πβππ₯π π’ β π 1 ππ₯π π’
ππ₯π π’ 1
Or example 4.5 on p. 86-87
Note that the computation of the moments is not straight forward and requires LβHospitalβs rule instead of direct derivation! (Note that at u=0 you always get β0/0β.)
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 42 of 51 ECE 3800
Binomial pmf review
xnxXX pp
x
nxpmfxP
1
Determine the expected value
n
xX nPxXE
0
n
x
xnx ppx
nxXE
0
1
n
x
xnxn
x
xnx ppxxn
nxpp
xxn
nxXE
10
1!!
!1
!!
!
n
x
xnx ppxxn
nXE
1
1!1!
!
n
x
xnx pppxxn
nnXE
1
111 1!1!11
!1
n
x
xnx ppxxn
npnXE
1
111 1!1!11
!1
Let 1 xy
1
0
11!!1
!1n
y
yny ppyyn
npnXE
Therefore pnXE
Determine the 2nd moment
0
22
xX nPxXE
πΈ π π₯ βπ!
π π₯ ! β π₯!β π β 1 π π₯ β
π!π π₯ ! β π₯!
β π β 1 π
Cancelling one of the x and adjusting the summation. Now make two terms β¦
πΈ π π₯ 1 1 βπ!
π π₯ ! β π₯ 1 !β π β 1 π
n
x
xnxn
x
xnx ppxxn
npp
xxn
nxXE
11
2 1!1!
!1
!1!
!1
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 43 of 51 ECE 3800
The second term was previously computed (math trick β¦ make summation = 1.0). The first term can now βcancelβ an (x-1) β¦ and look for a summation in x-2 terms.
pnppxxn
nXE
n
x
xnx
2
2 1!2!
!
n
x
xnx pppxxn
nnnpnXE
2
22222 1!2!22
1!2
n
x
xnx ppxxn
npnnpnXE
2
22222 1!2!22
!21
11 22 pnnpnXE
pnpnnXE 22 1
Determine the variance
222 XEXEXE
222 1 pnpnpnnXE
pnpnpnpnpnpnXE 2222222
qpnppnXE 12
Determine the Moment Generating Function
n
n
knkX ktqp
k
ntM
0
exp
n
n
knkX qtp
k
ntM
0
exp
Which can be determined to be β¦
nX qtptM exp
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 44 of 51 ECE 3800
Textbook Geometric Distribution (4.5.2)
ππ π π0, π 0
π β 1 π , π 1,2,β―
The 1st Moment
πΈ π π π β π β 1 π
πΈ π π π β π β 1 π
using
20
1
0 1
1
1
1
aada
daxa
da
d
x
x
x
x
πΈ π π π β1
1 1 ππ β
1π
1π
The 2nd Moment
πΈ π π β π β 1 π
πΈ π π β π β 1 π
using
32
2
0
2
02
2
1
2
1
11
aada
daxxa
da
d
x
x
x
x
πΈ π π β π β π 1 1 β 1 π
πΈ π π β 1 π β π β π 1 β 1 π π β π β 1 π
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 45 of 51 ECE 3800
πΈ π π β 1 π β2
1 1 ππΈ π
πΈ π2 β π β 1 π
π1π
πΈ π2 2 β π
π1π
2 ππ
The Variance
πΈ π π πππ π π πΈ π π
πππ π π2 ππ
1π
1 ππ
The Moment Generating Function (MGF)
π π’ ππ₯π π’ β π β π π ππ₯π π’ β π β π β 1 π
π π’ π β ππ₯π π’ β ππ₯π π’ β 1 π
using
1,1
1
0
afora
ax
x
π π’ π β ππ₯π π’ β ππ₯π π’ β 1 π
π π’ π β ππ₯π π’ β1
1 ππ₯π π’ β 1 π
π π’π β ππ₯π π’
1 ππ₯π π’ β 1 ππ
ππ₯π π’ 1 π
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 46 of 51 ECE 3800
Textbook Poisson Distribution (4.5.2)
In probability theory and statistics, the Poisson distribution, named after French mathematician SimΓ©on Denis Poisson, is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant rate and independently of the time since the last event.[1] The Poisson distribution can also be used for the number of events in other specified intervals such as distance, area or volume.
https://en.wikipedia.org/wiki/Poisson_distribution
ππ π π0, π 0
ππ!β ππ₯π π , π 0,1,2,β―
Examples:
Wikipedia β pieces of mail received each day arrival time of new individuals in a line the number of photons striking a pixel of a camera
The 1st Moment
πΈ π π π βππ!β ππ₯π π
πΈ π π π β ππ₯π πππ 1 !
β π β ππ₯π πππ!β
πΈ π π πππ!β ππ₯π π π
The 2nd Moment
πΈ π π βππ!β ππ₯π π
πΈ π π β π 1 1 βππ!β ππ₯π π
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 47 of 51 ECE 3800
πΈ π π β π 1 βππ!β ππ₯π π π β
ππ!β ππ₯π π
πΈ π π βππ 2!
β ππ₯π π πΈ π
πΈ π π π
The Variance
πΈ π π πππ π π πΈ π π
πππ π π π π π π
The mean and the variance are both equal to lambda!
The Moment Generating Function (MGF)
π π’ ππ₯π π’ β π β π π ππ₯π π’ β π βππ!β ππ₯π π
π π’ ππ₯π π βπ β ππ₯π π’
π!
Recognizing an exponential sequence
ππ₯π π₯ 1π₯1!
π₯2!
β―π₯π!
β―
π π’ ππ₯π π β ππ₯π π β ππ₯π π’ ππ₯π π β ππ₯π π’ 1
Interesting factors
π ππ π 1
ππ! β ππ₯π π
ππ 1 ! β ππ₯π π
π ππ π 1
ππ
This implies that the distribution rises to a maximum at π π.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 48 of 51 ECE 3800
Notes and figures are based on or taken from materials in the textbook: Alberto Leon-Garcia, βProbability, Statistics, and Random Processes For Electrical Engineering, 3rd ed.β, Pearson Prentice Hall, 2008, ISBN:
013-147122-8.
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 49 of 51 ECE 3800
Textbook Risk Taking Decision Making (4.6)
π π π1, ππ π 0 1 ππ€, ππ π 1 π
Examples:
Win-lose propositions (X=1 win, X=0 lose) Gambling bets with odds Buying a stock as an investment
The 1st Moment
πΈ π π π π β ππ π
πΈ π π 1 β 1 π π€ β π
πΈ π π π€ β π π 1
The 2nd Moment
πΈ π π π β ππ π
πΈ π π€ β π 1 π
The Variance
πΈ π π πππ π π πΈ π π
πππ π π π β 1 π β π€ 1
To have a chance at winning β¦.
πΈ π π 0
π€ or π
Notice that the variance goes up by the square of the amount wagered!
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 50 of 51 ECE 3800
The Variance β alternate derivation
πΈ π π πππ π π πΈ π π
πππ π π π€ β π 1 π π€ β π π 1
πππ π π π€ β π 1 π π€ β π 2 β π€ β π β π 1 π 2 β π 1
πππ π π π€ β π π€ β π 2 β π€ β π 2 β π€ β π π π
πππ π π π β π€ β 1 π 2 β π€ β 1 π 1 π
πππ π π π β 1 π β π€ 2 β π€ 1
πππ π π π β 1 π β π€ 1
To have a chance at winning β¦.
πΈ π π 0
π€ β π π 1 0
π€1 ππ
or
π1
π€ 1
Notice that the variance goes up by the square of the amount wagered!
Playing Dice β¦.
https://en.wikipedia.org/wiki/Craps
House percentages in winning, estimated income per hour!
Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet,
Probability, Statistics, and Random Signals, Oxford University Press, February 2016.
B.J. Bazuin, Fall 2020 51 of 51 ECE 3800
Matlab Sims
Sec4_5_Coins
Geometric_Example
Binomial_hist
bon_nchoosek
Uniform_hist