SDS 321: Introduction to Probability and Statistics ...psarkar/sds321/lecture6-ps.pdf · SDS 321:...

SDS 321: Introduction to Probability andStatistics

Lecture 6: Conditional Independence and countingintroduction

Purnamrita SarkarDepartment of Statistics and Data Science

The University of Texas at Austin

www.cs.cmu.edu/∼psarkar/teaching.html

Conditional Independence

I Recall, we said two events A and B were independent ifI P(A|B) = P(A) - knowing B tells us nothing about the probability

of A.I This means that P(A ∩ B) = P(A)P(B).

I We can extend this definition to conditional probabilities. We saytwo events A and B are conditionally independent given some eventC if

I P(A|B ∩ C) = P(A|C).I We write this as A ⊥⊥ B|C .I Like before, this boils down to: P(A ∩ B|C) = P(A|C)P(B|C)I Can you prove it?

Conditional Independence: Urn example

I Consider two urns, each containing 100 balls.

I The first urn contains all red balls.

I The second urn contains all blue balls.

I We select an urn at random. Let A be the event that the first urn ischosen.

I We select a ball from the urn, note it’s color, and put it back. Wethen select another ball from the urn, note it’s color, and put it back.

I Let A be the event that the first urn was chosen, let R1 be the eventthat the first ball was red, and let R2 be the event that the secondball was red.

I Are R1 and R2 independent?

Think about it intuitively first.

I I tell you that the first ball is red.

I Then you know for sure that the first urn is picked.

I So the second ball has to be red as well.

I Knowing about the first ball tells you a lot about the color of thesecond ball.

I Clearly they are not independent!

I We need to compare P(R1 ∩ R2) with P(R1)P(R2)

I By the law of total probability,

P(R1) =P(R1|A)P(A) + P(R1|Ac )P(Ac )

=1× 0.5 + 0× 0.5 = 0.5

I What is P(R2)? Its also 0.5.

I Again, by the law of total probability,

P(R1 ∩ R2) =P(R1 ∩ R2|A)P(A) + P(R1 ∩ R2|Ac)P(Ac)

=1× 1× 0.5 + 0× 0× 0.5 = 0.5

I Let’s compare: P(R1)P(R2) = 0.5× 0.5 = 0.25I So, P(R1 ∩ R2) 6= P(R1)P(R2) – i.e. R1 6⊥⊥ R2.

P(R1) =P(R1|A)P(A) + P(R1|Ac )P(Ac )

=1× 0.5 + 0× 0.5 = 0.5

=1× 1× 0.5 + 0× 0× 0.5 = 0.5

P(R1) =P(R1|A)P(A) + P(R1|Ac )P(Ac )

=1× 0.5 + 0× 0.5 = 0.5

=1× 1× 0.5 + 0× 0× 0.5 = 0.5

P(R1) =P(R1|A)P(A) + P(R1|Ac )P(Ac )

=1× 0.5 + 0× 0.5 = 0.5

=1× 1× 0.5 + 0× 0× 0.5 = 0.5

P(R1) =P(R1|A)P(A) + P(R1|Ac )P(Ac )

=1× 0.5 + 0× 0.5 = 0.5

I What is P(R2)? Its also 0.5.I Again, by the law of total probability,

=1× 1× 0.5 + 0× 0× 0.5 = 0.5

I Let’s compare: P(R1)P(R2) = 0.5× 0.5 = 0.25

I So, P(R1 ∩ R2) 6= P(R1)P(R2) – i.e. R1 6⊥⊥ R2.

P(R1) =P(R1|A)P(A) + P(R1|Ac )P(Ac )

=1× 0.5 + 0× 0.5 = 0.5

I What is P(R2)? Its also 0.5.I Again, by the law of total probability,

=1× 1× 0.5 + 0× 0× 0.5 = 0.5

I What if we condition on the chosen urn? Are the two colors nowindependent?

I Our gut says yes... let’s just double check the math.

P(R1|A) = P(R2|A) = 1

P(R1 ∩ R2|A) = 1× 1 = 1 = P(R1|A)P(R2|A)

I So, R1 ⊥⊥ R2|AI Knowing which urn was used tells us something about how likely it

is that they are both red!

I Conditional independence does not imply independence!

P(R1|A) = P(R2|A) = 1

P(R1 ∩ R2|A) = 1× 1 = 1 = P(R1|A)P(R2|A)

P(R1|A) = P(R2|A) = 1

P(R1 ∩ R2|A) = 1× 1 = 1 = P(R1|A)P(R2|A)

Summing up

So far we have done–

I Sets, sample spaces

I Axioms of probability

I Conditional probability and Bayes rule

I Independence and conditional independence

I Today we will start counting. Reading-Ross chapter 1.

Ordered Samples – Sampling with replacement

There are total n different elements in a population or set. You want tocreate an (ordered) arrangement of r elements.

I You pick an element, note what it is, and put it back.

I An element can be repeated in your arrangement.

I The first choice can be one of n elements.

I The second choice can again be one of n elements.

I Total number of arrangements is nr .

Ordered Samples – Sampling without replacement

I You pick an element, note what it is, but do not put it back.

I Any element appears at most once in your arrangement.

I The first element can be any of n elements.

I The second can be any of the remaining n − 1 elements.

I The third can be any of the n − 2 elements.

I So the total number of arrangements is n(n− 1)(n− 2) . . . (n− r + 1).This is also denoted by (n)r (or P(n, r)) and called n permute r .

Counting example I: balls and bins

I r balls and n bins:

I Each ball can be placed in any of the n bins independently.

I So r balls can be placed in n bins in how many ways?

I First ball can be placed in one of the n bins.

I Second ball can be placed in one of the n bins. One bin can havemultiple balls.

I Total n × n × · · · × n repeated r times. So nr different ways.

I Sometimes you will hear words like distinguishable orindistinguishable balls/bins.

I Distinguishable basically means that each have a uniqueidentifier/color/number on them. Here often the ordering matters.

I In this case the balls and bins are both distinguishable.

Permutations

I You want to arrange n elements of a set. How many arrangementsare there?

I First element can be picked in n ways.

I Second element can be picked in n − 1 ways.

I There are n × (n − 1)× · · · × 1 different permutations.

I This is simply (n)n. We also write this as n!.

I n permute r is none other than n(n − 1) . . . (n − r + 1) =n!

(n − r)!.

I How many three digit numbers are there with digits from {1, 2, 3}and no repeated digit?

I 123,132,213,231,312,321

I 3! = 3× 2 = 6.

Permutations

(n − r)!.

I 123,132,213,231,312,321

I 3! = 3× 2 = 6.

Permutations

(n − r)!.

I 123,132,213,231,312,321

I 3! = 3× 2 = 6.

Permutations

(n − r)!.

I 123,132,213,231,312,321

I 3! = 3× 2 = 6.

Permutations

(n − r)!.

I 123,132,213,231,312,321

I 3! = 3× 2 = 6.

Permutations

(n − r)!.

I 123,132,213,231,312,321

I 3! = 3× 2 = 6.

Permutations

(n − r)!.

I 123,132,213,231,312,321

I 3! = 3× 2 = 6.

Permutations

(n − r)!.

I 123,132,213,231,312,321

I 3! = 3× 2 = 6.

Practice problem

Find out the number of ways:

1. 3 boys and 3 girls can sit in a row?

2. 3 boys and 3 girls can sit in a row if the boys and girls are each tosit together?

3. 3 boys and 3 girls can sit in a row if only the boys must sit together?

4. 3 boys and 3 girls can sit in a row if no two people of the same sexare allowed to sit together?

Combinations

So far we have been talking only about ordered samples. For manyapplications we only need an unordered sample. In particular we want tofigure out how many ways one can form groups of size r from a set ofsize n.

I Q: How many groups of size r are there of a given population of sizen?

I In other words: how many ways can I choose r unordered elementswithout replacement from a population of size n?

I We denote this by(nr

). We say n choose r .

I How many ways can I choose two digits without replacement from{1, 2, 3}?

I 12,23,13

Combinations

I 12,23,13

Combinations

I 12,23,13

Combinations

I 12,23,13

Combinations

I 12,23,13. So(32

Subpopulations and partitions

I Earlier we learned about n permute r . This is how we choose r

elements without replacement, but the order matters.

I Let consider all ordered samples of size 2 picked withoutreplacement from 3 numbers.

I Now we want the same. But order does not matter. So {1, 2} and{2, 1} are the same.

I (1, 2), (2, 1)︸︷︷︸2times

, (1, 3), (3, 1)︸︷︷︸2times

, (2, 3), (3, 2)︸︷︷︸2times

I Say we are picking 3 out of 4 numbers. Consider all orderedarrangements.123, 132, 231, 213, 312, 321,︸︷︷︸(1,2,3) appears 6 times

143, 134, 431, 413, 314, 341︸︷︷︸(1,3,4) appears 6 times

, . . .

I (1, 2), (2, 1)︸︷︷︸2times

, (1, 3), (3, 1)︸︷︷︸2times

, (2, 3), (3, 2)︸︷︷︸2times

143, 134, 431, 413, 314, 341︸︷︷︸(1,3,4) appears 6 times

, . . .

I (1, 2), (2, 1)︸︷︷︸2times

, (1, 3), (3, 1)︸︷︷︸2times

, (2, 3), (3, 2)︸︷︷︸2times

143, 134, 431, 413, 314, 341︸︷︷︸(1,3,4) appears 6 times

, . . .

SDS 321: Introduction to Probability and Statistics ...psarkar/sds321/lecture6-ps.pdf · SDS 321:...

Documents

Transcript of SDS 321: Introduction to Probability and Statistics ...psarkar/sds321/lecture6-ps.pdf · SDS 321:...