Probability Theory

Part 1: Basic Concepts

Sample Space - Events Sample Point

The outcome of a random experiment Sample Space S

The set of all possible outcomes Discrete and Continuous

Events A set of outcomes, thus a subset of S Certain, Impossible and Elementary

Set Operations Union Intersection Complement

Properties Commutation

Associativity

Distribution

De Morgan’s Rule

A BA B

A B

CA

CA

A B B A

A B C A B C

A B C A B A C

C C CA B A B

S

A B

Axioms and Corollaries Axioms If

If A1, A2, … are pairwise exclusive

Corollaries

A B P A B P A P B

11

k kkk

P A P A

0 P A

1P S 1CP A P A

1P A 0P

P A B

P A P B P A B

Computing Probabilities Using Counting Methods Sampling With Replacement and Ordering

Sampling Without Replacement and With Ordering

Permutations of n Distinct Objects

Sampling Without Replacement and Ordering

Sampling With Replacement and Without Ordering

kn

1 ... 1n n n k

!k

!

! !

n n n

k n k k n k

1 1

1

n k n k

k n

Conditional Probability Conditional Probability of

event A given that event B has occurred

If B1, B2,…,Bn a partition of S, then

(Law of Total Probability)

A B

CA

S

A B

|

P A BP A B

P B

B1

B3

B2

A

1 1| ...

| j j

P A P A B P B

P A B P B

Bayes’ Rule If B1, …, Bn a partition of

S then

1

|

|

|

j

j

j j

n

k kk

P A BP B A

P A

P A B P B

P A B P B

likelihood priorposterior

evidence

0 11-p p

1010

1-ε ε 1-εε

input

output

Example

Which input is more probable if the output is 1? A priori, both input symbols are equally likely.

Event Independence Events A and B are

independent if

If two events have non-zero probability and are mutually exclusive, then they cannot be independent

P A B P A P B

C

A B

½

½

½

½

½ 1 1

1

1

1 1

P A B P A P B

P B C P B P C

P A C P A P C

P A B C P

P A P B P C

Sequential Experiments Sequences of Independent

Experiments E1, E2, …, Ej experiments A1, A2, …, Aj respective

events Independent if

Bernoulli Trials Test whether an event A

occurs (success – failure) What is the probability of k

successes in n independent repetitions of a Bernoulli trial?

Transmission over a channel with ε = 10-3 and with 3-bit majority vote

1 2

1 2

...

...

n

n

P A A A

P A P A P A

1

!

! !

n kkn

np k p p

k

n n

k k n k