1

Review of Probability Theory

[Source: Stanford University]

2

A random experiment with set of outcomes

Random variable is a function from set of outcomes to real numbers

Random Variable

3

Example

Indicator random variable:

A : A subset of is called an event

4

CDF and PDF Discrete random variable:

The possible values are discrete (countable)

Continuous random variable: The rv can take a range of values in R

Cumulative Distribution Function (CDF):

PDF and PMF:

5

Expectation and higher moments

Expectation (mean):

if X>0 :

Variance:

6

Two or more random variables

Joint CDF:

Covariance:

7

Independence

For two events A and B:

Two random variables

IID : Independent and Identically Distributed

8

Useful Distributions

9

Bernoulli Distribution

The same as indicator rv:

IID Bernoulli rvs (e.g. sequence of coin flips)

10

Binomial Distribution Repeated Trials:

Repeat the same random experiment n times. (Experiments are independent of each other)

Number of times an event A happens among n trials has Binomial distribution (e.g., number of heads in n coin tosses, number of arrivals in n time

slots,…)

Binomial is sum of n IID Bernoulli rvs

11

Mean of Binomial

Note that:

12

1 2 3 4 5 6 7 8 9 10 11 120

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

Binomial - Example

n=4

n=40

n=10

n=20

p=0.2

13

Binomial – Example (ball-bin)

There are B bins, n balls are randomly dropped into bins.

: Probability that a ball goes to bin i : Number of balls in bin i after n drops

14

Multinomial Distribution Generalization of Binomial Repeated Trails (we are interested in

more than just one event A) A partition of into A1,A2,…,Al

Xi shows the number of times Ai occurs

among n trials.

15

Poisson Distribution

Used to model number of arrivals

16

0 5 10 150

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Poisson Graphs

=10

=.5

=1

=4

17

Poisson as limit of Binomial

Poisson is the limit of Binomial(n,p) as

Let

18

0 1 2 3 4 5 6 7 8 9 100

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Poisson and Binomial

Poisson(4)

n=5,p=4/5

n=20, p=.2

n=10,p=.4

n=50,p=.08

19

Geometric Distribution

Repeated Trials: Number of trials till some event occurs

20

Exponential Distribution

Continuous random variable Models lifetime, inter-arrivals,…

21

Minimum of Independent Exponential rvs

: Independent Exponentials

22

Memoryless property

True for Geometric and Exponential Dist.:

The coin does not remember that it came up tails l times

Root cause of Markov Property.

23

Proof for Geometric

24

Moment Generating Function (MGF) For continuous rvs (similar to Laplace

transform)

For Discrete rvs (similar to Z-transform):

Characteristic Function

25

Characteristic Function

Can be used to compute mean or higher moments:

If X and Y are independent and T=X+Y

26

Useful CFs

Bernoulli(p) :

Binomial(n,p) :

Multinomial:

Poisson: