C HAPTER 2 Basic Concepts of Probability Theory

INHA UniversityINHA Universityhttp://multinet.inha.ac.krhttp://multinet.inha.ac.kr

Multimedia Network Lab.Multimedia Network Lab.

CHAPTER 2

Basic Concepts of Probability Theory

Prof. Sang-Jo [email protected]

http://multinet.inha.ac.kr

INHA UniversityINHA University

http://multinet.inha.ac.kr Multimedia Network LabMultimedia Network Lab..

Random Experiments Random Experiments

An experiment in which the outcome varies in an unpredictable fashion when the experiment is repeated under the same conditions

The specification of a random experiment must include an unambiguous statement of exactly what is measured or observed.

Example 2.1 E1: Select a ball from an urn containing balls numbered 1 to 50 E2: Select a ball from an urn containing balls numbered 1 to 4. Balls 1 & 2 are black and balls 3 & 4 are white. Note number

and color. E3: Toss a coin three times and note the sequence of heads and tails E4: Toss a coin three times and note the number of heads E7: Pick a number at random between zero and one E12: Pick two numbers at random between zero and one E13: Pick a number X at random between zero and one, then pick a

number Y at random between zero and X



Sample Space Sample space of a random experiment

The set of all possible outcomes Outcomes(or sample points) are mutually exclusive in the sense that they cannot

occur simultaneously. When we perform a random experiment, one and only one outcome occurs.

A sample space can be finite, countably infinite or uncountably infinite. Example 2.2

E1: Select a ball from an urn containing balls numbered 1 to 50 E2: Select a ball from an urn containing balls numbered 1 to 4. Balls 1 & 2 are black and balls 3 & 4 are white. Note the number and color. E3: Toss a coin three times and note the sequence of heads and tails E4: Toss a coin three times and note the number of heads E7: Pick a number at random between zero and one E12: Pick two numbers at random between zero and one E13: Pick a number X at random between zero and one, then pick a number Y at

random between zero and X



Sample Space Toss a dice until a ‘six’ appears and count the number of times

the dice was tossed. S = {1, 2, 3, …}; S is discrete and countably infinite (one-to-one correspondence

with positive integers)

Pick a number X at random between zero and one, then pick a number Y at random between zero and X.

S = {(x, y): 0 ≤ y ≤ x ≤ 1}; S is a continuous sample space.

4



Events Events

Outcomes that satisfies certain conditions A subset of S Certain event consists of all outcomes, always occurs Null event contains no outcomes, never occurs Event class is the events of interests: set of sets

Example 2.3 E1: “An even-numbered ball is selected,” A1= ? E2: “The ball is white and even-numbered,” A2= ? E3: “The three tosses give the same outcome,” A3= ? E4: “The number of heads equals the number of tails,” A4= ? E7: “The number selected is non-negative,” A5= ? E12: “The two numbers differ by less than 1/10,” A6= ?



Review of Set Theory Terminologies

U: universal set consists of all possible objects Objects are called the elements or points of the set Subset of B : A B Empty set: Equal: A=B Union: AB={x: xA or x B} Intersection: AB={x: xA and x B} Disjoint (mutually exclusive): if AB= Complement: Ac={x: xA} Relative complement (difference): A-B={x: xA and xB}

Notations

6

n

n

kkn

n

kk AAAAandAAAA

121

121



Set OperationsUnion

A B

A B(a)

A B

A B(b)

Intersection

Complement

Ac(c)

A B

A B = (d)

Mutuallyexclusive

A

Ac

A

B

A B (e)



Properties of Set Operations Commutative properties

Associative properties

Distributive properties

DeMorgan’s Rules

A B B A and A B B A

and ( ) ( )A B C A B C

( ) ( ) ( )A B C A B A C

( )C C CA B A B ( )C C CA B A B

( ) ( )A B C A B C

and



Properties of Set Operations De Morgan’s rules

(A ∩ B)c = Ac ∪ Bc and (A ∪ B)c = Ac ∩ Bc

Proof of the second rule: Suppose x ∈ (A ∪ B)c

⇔ x is not contained in any ofthe events A and B⇔ x is contained in Ac and Bc

⇔ x ∈ Ac ∩ Bc.

Proof of the first rule: Based on the second rule, take A → Ac and B → Bc, we then have

(Ac ∪ Bc)c = A ∩ B. Taking complement on both sides, we obtain the first rule.

9



The Axioms of Probability Axioms 1 :

Axioms 2 :

Axioms 3 :

Axioms 3’ :

0 [ ]P A

[ ] 1P S

If , then [ ] [ ] [ ]A B P A B P A P B

1 2If , , is a sequenceof eventssuch thatA A

for all , theni jA A i j

11

[ ]k kkk

P A P A



Properties of Probability Corollary 1 : Corollary 2 : Corollary 3 :

Corollary 4 :

Corollary 5 :

Corollary 6 :

Corollary 7 :

[ ] 1 [ ]cP A P A

[ ] 1P A

1 2If , , , are pairwise mutuallyexclusive,nA A A

11

then [ ]for 2n n

k kkk

P A P A n

[ ] [ ]P A B P A P B P A B

If A B, then [ ] [ ]P A P B

[ ] 0P

kjn

nkj

n

jj

n

kk AAPAAPAPAP ].[)1(][][ 1

1

10



Discrete Sample Space Example 2.9

An urn contains identical balls numbered 0.1,…,9. Randomly select a ball from the urn

A = “number of ball selected is odd,”

B = “number of ball selected is a multiple of 2,”

C = “number of ball selected is less than 5,”

Find [ ], [ ], [ ], [ ], and [ ].P A P B P C P A B P A B C



Continuous Sample Space Continuous sample space

Outcome are numbers. The events consist of intervals of the real line or rectangular

regions in the plane.

Example 2.12: Pick a number between zero and one Sample space S = ?

The probability that the outcome falls in a subinterval of A = the length of the subinterval

A = the event when the outcome is at least 0.3 away from the center of the unit interval. P [A] = ?



Conditional Probability Conditional probability

To determine whether two events A and B are related in the sense that knowledge about the occurrence of B alters the likelihood of occurrence of A.

[ ]| for [ ] 0[ ]

/ [ ]/ [ ]

A B A B

B B

P A BP A B P BP B

n n n P A Bn n n P B

A

B

A B

S

[ ] | [ ]

[ ] | [ ]

P A B P A B P B

P A B P B A P A



Example: Conditional Probability Example 2.24

A ball is selected from an urn containing two black balls (1, 2) and two white balls (3, 4). The sample space is S ={(1, b), (2, b), (3, w), (4, w)}. Assuming that the outcome are equally likely, find P [A|B], P [A|C], where A, B, and C are the following events:

A = {(1, b), (2, b), (3, w), (4, w)}, “black ball selected” B = {(1, b), (2, b), (3, w), (4, w)}, “even-numbered ball selected’” C = {(1, b), (2, b), (3, w), (4, w)}, “number of ball is greater than 2”

Example Tossing 2 dice Suppose the first die is ‘3’; given this information, what is the probability

that the sum of the 2 dice equals 8? There are 6 possible outcomes, given the first dice is ‘3’:

(3, 1), (3, 2), (3, 3), (3, 4), (3, 5), and (3, 6). The outcomes are equally probable. Hence, the probability that the sum is

8, given the first dice is ‘3’, is 1/6 .“” Conditional Probability Note



Theorem On Total Probability Let B1 , B2 , …, Bn be mutually exclusive events whose union

equals the sample space S. Any event can be represented as the union of mutually exclusive events:

1 2 1

1 2

1 1 2 2

( ) ( ) ( )[ ] [ ] [ ] [ ][ ] [ | ] [ ] [ | ] [ ] [ | ] [ ]

n n

n

n n

A A S A B B B A B A BP A P A B P A B P A BP A P A B P B P A B P B P A B P B

B1B3

B2

A

Bn-1

Bn…

…



Example: Total Probability Example 2.28

A manufacturing process produces a mix of “good” memory chips and “bad” memory chips. The lifetime of good chips follows the exponential law with a rate of α. The life time of bad chips also follows the exponential law, but the rate of failure is 1000α. Suppose that the fraction of good chips is 1-p and of bad chips p. Find the probability that a randomly selected chip is still functioning after t seconds.

C = the event “chip still functioning after t seconds”

G = the event “chip is good”

B = the event “chip is bad”



Example: Total Probability Example

In a trial, the judge is 65% sure that Susan has committed a crime. Person F (friend) and Person E (enemy) are two witnesses who know whether Susan is innocent or guilty.

Person F is Susan’s friend and will lie with probability 0.25 if Susan is guilty. He will tell the true if Susan is innocent.

Person E is Susan’s enemy and will lie with probability 0.30 if Susan is innocent. Person E will tell the truth if Susan is guilty.

What is the probability that Person F and Person E will give conflicting testimony?

Solution Let I and G be the two mutually exclusive events that Susan is

innocent and guilty, respectively. Let C be the event that the two witnesses will give conflicting testimony. Find P[C] based on P[C|I] and P[C|G].

By the law of total probabilities

18

.2675.065.025.035.030.0][]|[][]|[][

GPGCPIPICPCP



Bayes’ Rule Let B1 , B2 , …, Bn be a partition of a sample space S.

Suppose that event A occurs. What is the probability of event Bj ?

1

[ ] [ | ] [ ]|

[ ] [ | ] [ ]

j j jj n

k kk

P A B P A B P BP B A

P A P A B P B



Example: Bayes’ Rule Example 2.30

A manufacturing process produces a mix of “good” memory chips and “bad” memory chips. Suppose that in order to “weed out” the bad chips, every chip is tested for t seconds prior to leaving the factory.

Find the value of t for which 99% of chips sent out to customers are good.

C = the event “chip still functioning after t seconds” G = the event “chip is good” B = the event “chip is bad”

We want to find the value of t for which

Example 2.26 & 2.29 (Note)

[ ] 0.99

[ | ] [ ][ ] [ ] [ ] [ ]

P G C

P C G P GP C G P G P C B P B



Example: Bayes’ Rule Example

A judge is 65% sure that a suspect has committed a crime. During the course of the trial, a witness convinces the judge that there is an 85% chance that the criminal is left-handed. If 23% of the population is left-handed and the suspect is also left-handed. With this new information, how certain should the judge be of the guilt of the suspect?

SolutionG = event that the suspect is guiltyI = event that the suspect is innocentL = event that the suspect is left-handed

Since {G, I} forms a partition of the sample space, by Bayes’ Theorem:

21

.87.035.023.065.085.0

65.085.0][]|[][]|[

][]|[]|[

IPILPGPGLPGPGLPLGP



Independence of Events Event A is independent of B if knowledge of the occurrence of B

does not alter the probability of A

Three events A, B, and C are independent if

[ ] [ ]

| and |

P A B P A P B

P A B P A P B A P B

[ ] [ ] [ ]P A B C P A P B P C



Properties of independence and mutual exclusiveness

(1) If A ∩ B = φ, then P[A|B] = 0. A can never occur if B has occurred.

(2) If A ≠ φ, B ≠ φ and A and B are independent, then A and B

are not mutually exclusive. This is becauseP[A ∩ B] = P[A]P[B] ≠ 0.

Remarks we have seen that if A and B are mutually exclusive, then A and B cannot be independent.

(3) If A and B are independent and A ∩ B = φ, then either A = φ or B = φ or both. This is because 0 = P[A ∩ B] = P[A] P[B] so that either P[A] = 0, P[B] = 0 or both are equal.

23



Example: Independent Events Example 2.31

A ball is selected from an urn containing two black balls (1, 2) and two white balls (3, 4). The sample space is S = {(1, b), (2, b), (3, w), (4, w)}. Let events A, B, and C are defined as follows:

A = {(1, b), (2, b)}, “black ball selected” B = {(2, b), (4, w)}, “even-numbered ball selected’” C = {(3, w), (4, w)}, “number of ball is greater than 2”

Are events A and B independent? Are events A and C independent? In general if two events have nonzero probability and are mutually

exclusive, then they cannot be independent.



Example: Independent Events Example Two numbers x and y are selected at random

between zero and one. Let events A, B and C be defined by

However,

25

21yA

21xB

21,

21

21,

21 yxyxC

][][41][ BPAPBAP

][][41][ CPAPCAP

][][41][ CPBPCBP

81][][][0][][

,

CPBPAPPCBAP

soCBA



Sequential Experiment Sequential of independent experiment

Suppose that a random experiment consists of performing experiments E1, E2, …, En.

Outcome of the experiment

Sample space of the experiment

Let A1 , A2 , …, An be events such that Ak concerns only the outcome of the kth sub-experiment. If the sub-experiments are independent, then

1 2( , , , )ns s s s

1 2 nS S S

1 2 1 2[ ] [ ] [ ]n nP A A A P A P A P A



Example: Sequential Events Example 2.36

Suppose that 10 numbers are selected at random from the interval [0, 1]. Find the probability that the first 5 numbers are less than ¼ and the last 5 numbers are greater than ½.

Let x1, x2, …, x10 be the sequence of 10 numbers.



Binomial Probability Law Bernoulli trial

Performance an experiment once and nothing whether a particular event A occurs.

Outcome = “success” if A occurs, outcome = “fail” otherwise

Binomial probability law Probability of k successes in n independent repetitions of a

Bernoulli trial

(1 )

!! !

k n kn

np k p p

k

n nk k n k



Example: Binomial Probability Law Example 2.37

Suppose that a coin is tossed three times. Assume that the tosses are independent and the probability of heads is p

P [{HHH}] = P [{HHT}] = P [{HTH}] = P [{THH}] =P [{TTH}] = P [{THT}] =P [{HTT}] = P [{TTT}] =

Let k be the number of heads in three trials

P [k=0] =P [k=1] =P [k=2] =P[ k=3] =



Example: Binomial Probability Law Example 2.39 Voice Communication System

Let k be the number of active speakers in a group of eight independent speakers. Suppose that a speaker is active with probability 1/3.

Find the probability that the number of active speakers is greater than six.

Example 2.40: Error Correction Coding A communication system transmits binary information over a

channel that introduces random bit errors with probability =10-3. the transmitter transmits each information bit three times and a decoder takes a majority vote of the received bits to decide on what the transmitted bit was.

Find the probability that the receiver will make an incorrect decision.



Multinomial Probability Law Let B1,B2, …, BM be a partition of the sample space S of some

random experiment and P[Bj]=Pj. The events are mutually exclusive P1+P2+…+PM=1

Suppose the n independent repetitions of the experiment are performed. Let kj be the number of time event Bj occurs, then the vector (k1,k2,

…,kM) specifies the number of times each of the event Bj occurs

Multinomial probability law M=2: Binomial probability law 1 2

1 2 1 21 2

![ , , , ]! ! !

Mk k kM M

M

nP k k k p p pk k k



Geometric Probability Law A sequential experiment in which we repeat independent

Bernoulli trials until the occurrence of the first success. Let the outcome of this experiment be m.

Geometric probability law The probability p(m), that m trials are required.

1

1 0

[ ]

1 =1

m K j

m K j

K K

P m K p q pq q

pq qq

11 1,2,....mP m p p m

C HAPTER 2 Basic Concepts of Probability Theory

Documents

Transcript of C HAPTER 2 Basic Concepts of Probability Theory