Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 1

Sections from Text and MIT Video Lecture: Sections 1.2, 1.3, 1.4, and 4.1, 4.2

http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-041-probabilistic-

systems-analysis-and-applied-probability-fall-2010/video-lectures/lecture-3-independence/

http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-041-probabilistic-

systems-analysis-and-applied-probability-fall-2010/video-lectures/lecture-4-counting/

Topics from Syllabus: Counting, Random Variables: Discrete and Continuous, Probability mass

functions and probability densities

A (Ridiculously Brief) Introduction to Discrete Random Variables

Obviously we will deal with this topic in come complexity and depth, but for a start let’s see

what the simplest idea in probability might be. If I am going to give you the most meager bit of

information, I could tell you the result of a dichotomy, i.e. a Yes or a No. Can you think of

anything simpler?

We model this easiest of all situations with a “Bernoulli Random Variable”. For example,

consider a coin toss with a “fair” coin. Our sample space is given as

𝑆 = {𝐻, 𝑇}

You can immediately generalize to a polling situation where we are trying to determine the

proportion of “Lefties” amongst mathematicians. (I am thinking right brain/left brain, not

politics!) We poll for dichotomies all the time.

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 2

We already know from the Classical Notion of Probability that 𝑃(𝐻) =1

2= 𝑃(𝑇) . We can also

create a bookkeeping function that “counts” the number of Heads on this single toss. This may

be overkill in the current situation, but it will generalize nicely. Our function 𝑋 maps the sample

space 𝑆 into the real numbers ℝ.

𝑋: 𝑆 → ℝ

Definitions

Given a sample space S, we define a random variable X, to be a (measurable) function which

maps elements of the sample space, s ∈ S , onto real numbers, X(s).

Notice that a random variable induces probabilities on subsets of the real line.

Given a random variable 𝑋, define 𝑋 to be discrete if X may assume a finite or a countably

infinite number of values 𝑥 = 𝑋(𝑠). Also, define a function

𝑝(𝑥) ≡ 𝑃(𝑋 = 𝑥)

We call 𝑝(𝑥) the probability function of X or the probability mass function of X. We immediately

have the following (note the similarities with relative frequencies):

𝑝(𝑥) ≥ 0 ∀ 𝑥 ∈ ℝ

∑ 𝑝(𝑥)𝑥 = 1

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 3

We can make a table like this.

𝑠 ∈ 𝑆 𝑋(𝑠)

H 1

T 0

We can also make a table showing each of the possible real number outcomes and their

associated probabilities. Such a table represents a “probability mass function”, or pmf.

𝑘 𝑃(𝑋 = 𝑘) ≡ 𝑝(𝑘)

0 1

2

1 1

2

∑ 𝑝(𝑘)1

𝑘=0= 1

This is not bad so far. Just keep track of our notation- we let 𝑘 represent a possible real number

outcome from our experiment and call its probability mass function 𝑝(𝑘) . Can you build the

next case- tossing two fair coins? Here are our tables for 𝑛 ≡ number of tosses = 2.

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 4

𝑠 ∈ 𝑆 𝑋(𝑠) ≡ number of heads

HH 2

HT 1

TH 1

TT 0

𝑘 𝑃(𝑋 = 𝑘) ≡ 𝑝(𝑘)

0 1

4

1 1

4+

1

4=

1

2

2 1

4

∑ 𝑝(𝑘)2

𝑘=0= 1

It is not a bad practice to begin plotting our pmf’s.

Here is the code for the previous picture.

k=seq(0,2)

p=c(1/4, 1/2, 1/4)

plot(k, p,main="Probability Mass Function, n=2",xlim=c(-1,3),ylim=c(0,1.1))

lines(k,p, type="h")

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 5

Now, what happens when we consider larger values for 𝑛, ≡ the number of tosses. I can stand

to do the case for n=3 by hand, but this becomes tedious for larger values. A little theory can

help automate our results.

𝑠 ∈ 𝑆 𝑋(𝑠) ≡ number of heads

HHH 3

HHT 2

HTH 2

HTT 1

THH 2

THT 1

TTH 1

TTT 0

𝑘 𝑃(𝑋 = 𝑘) ≡ 𝑝(𝑘)

0 1

8

1 1

8+

1

8+

1

8=

3

8

2 1

8+

1

8+

1

8=

3

8

3 1

8

∑ 𝑝(𝑘)2

𝑘=0= 1

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 6

Also, we can’t help but notice that, if we watch the values of the numerators in our probabilities

a famous result becomes apparent. We will spend the rest of this lecture trying to understand the

two tables shown below, courtesy of Wikipedia:

Counting

We are now able to calculate probabilities in fairly simple situations. We would like to be able to

move towards more complex situations. For example, what is the probability of exactly 10 heads

when tossing a coin 20 times?

Here are a few situations where we need to learn how to “count”.

A set of 𝑛 simple coin tosses (also called a Binomial Random Variable) with a possibly

unfair or biased coin. In this case we try to describe the probability of obtaining 𝑟

successes on 𝑛 independent trials. This is modeled with a Binomial Random Variable.

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 7

A waiting time scenario where we keep tossing a (possibly unfair or biased) coin until

our first Heads/Success. (For example, when playing “Trouble”, we keep rolling a die

until our first "Six"). This is modeled by a Geometric Random Variable.

A waiting time scenario where we keep tossing a (possibly unfair or biased) coin until we

have obtained exactly 𝑟 Heads/Successes. We try to describe the probability that this will

take 𝑟 trials, 𝑟 + 1 trials, etc. This is modeled with a Negative Binomial Random

Variable.

Simple Counting Arguments

Remember back to when you were a kid playing with Mr. Potato Head.

Suppose you are putting together a Mr. Potato Head and you have in front of you 4 mouths, 3

noses and 2 sets of eyes. How many different faces can you make?

We can think about the problem this way: We can choose a mouth in any of 4 different ways:

Start

Mouth 1

Mouth 2

Mouth 3

Mouth 4

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 8

Now we will select a nose. We can select a different nose with any of the 4 mouths:

It’s already looking pretty busy. The point is that with 4 Mouths and 3 Noses we can put together

4 times 3 = 12 different faces. If we include the eyes we can do 4 ∙ 3 ∙ 2 = 24 different faces.

Multistage Process: Suppose a process will be performed in N stages. For example, N = 4

when we toss a fair coin 4 times. Each of these N = 4 stages can be done in two ways (HEADS

or TAILS) so there are 2 ∙ 2 ∙ 2 ∙ 2 = 16 ways to toss a coin (fair or biased) 4 times.

In the Mr. Potato Head example, there are 3 stages: Select a mouth, select a nose, select some

eyes. The first stage may be done in 4 ways, the second in 3 ways and the last in 2 ways.

Key Example: Here’s an example that is solved every day many times all around the world.

We’ll present it in a simplified way. Suppose you wish to visit 5 cities: Albany, Utica,

Binghamton, Rochester, and Syracuse. You will visit each one once and only once. How many

ways can you do this?

By simply using the multistage process approach we see that we can choose the

first city in 5 ways. Say that we start from Utica. Once that’s chosen there are 4

Start

Mouth 1

Nose 1

Nose 2

Nose 3

Mouth 2

Nose 1

Nose 2

Nose 3

Mouth 3

Nose 1

Nose 2

Nose 3

Mouth 4

Nose 1

Nose 2

Nose 3

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 9

ways to select from the remaining cities (Albany, Binghamton, Rochester, and

Syracuse). If you were only visiting 2 out of 5 cities there would be 5 ∙ 4 = 20 .

We’re visiting all 5, so if we choose, say Utica for the first and Rochester for the

second then selecting the third from Albany, Binghamton, and Syracuse means

there are 5 ∙ 4 ∙ 3 = 60 ways to choose 3 cities. The fourth comes from the

remaining 2, then the last is determined for us. All told: 5 ∙ 4 ∙ 3 ∙ 2 ∙ 1 =

120 ways to visit 5 cities.

Let’s push this a little further. Suppose you want to visit each state capitol once and only once.

There are 50 states, so there are 50 ways to choose the first. If we were only visiting 2 of them

there would be 50 ∙ 49 = 2450 ways to plan our trip. For 3 cities we would have 50 ∙ 49 ∙ 48 =

117600. You should be starting to get a little worried. Build a table:

Number of Cities Number of ways to plan trip

1 50

2 50∙49 = 2450

3 50∙49∙48 = 117600

4 50∙49∙48∙47 = 5527200

5 50∙49∙48∙47∙46 = 254251200

6 50∙49∙48∙47∙46∙45 11441304000

By the time we get to 50 cities there are 3.041409320171338e+064 ways. This number is so

large as to be incomprehensible. I can barely understand one million, but it helps to think about a

screen door.

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 10

Some Notation: Factorials When we multiply numbers like “3 times 2 times 1” we will

usually write 3 ∙ 2 ∙ 1 = 3! which is read as “3 factorial”. From the above we have

50 ! ≈ 3.041409320171338e + 064

At this point we can probably use some refreshments.

Suppose you have 3 candy bars in front of you, Snickers, a Three Musketeers, and a Baby

Ruth. Call them S, T, and B. In how many ways can you arrange these objects in order?

3! = 3 ∙ 2 ∙ 1 = 6

Here they are: STB, SBT, TSB, TBS, BST, BTS.

(Just as a side note: I taught High School some time ago in Cameroon. Some students needed

help in combinatorics or counting arguments, (which I had never looked at myself) and I

foolishly agreed to help them. I wasn’t able to really “click” with the material until I physically

placed objects in front of me on the table top and started manipulating them. You are strongly

encouraged to do the same in order to make this all less abstract.)

Suppose now you add a Hershey bar, call it H. How many arrangements are possible now?

4! = 4 ∙ 3 ∙ 2 ∙ 1 = 24

Write them down on a piece of paper!

How many when you add a Reese’s, R?

5! = 5 ∙ 4 ∙ 3 ∙ 2 ∙ 1 = 120

You can be forgiven if you don’t write these down, but make sure you understand where the

numbers are coming from. Stay with the 5 bars. Suppose you are going to eat three of them, and

it matters to you in what order you eat them. In how many ways can this be done?

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 11

5 ∙ 4 ∙ 3 = 60

Did you notice that we started out in the form of 5! But lost the last two numbers? We can write:

5 ∙ 4 ∙ 3 =5 ∙ 4 ∙ 3 ∙ 2 ∙ 1

2 ∙ 1=

5!

2!= 60

Permutations:

The number of possible sequences (ordered arrangements) of n objects is

𝑛! = 𝑛 · (𝑛 − 1) · (𝑛 − 2) · . . . 3 · 2 · 1

If the arrangement is to contain only r of these n objects, then the number of ways to form

sequences is called the permutation of n things taken r at a time and we have

𝑃𝑛,𝑟 ≡ 𝑛!

(𝑛 − 𝑟)!

As another example, suppose you are dealing out 5 cards from a normal 52 card deck. If the

order in which the cards come to you is important (like when you bet on each card before

moving on), then the number of possible “hands” is

52 ∙ 51 ∙ 50 ∙ 49 ∙ 48 = 𝑃52,5 =52!

(52−5)!=

52!

(47)!= 311875200 .

That’s a lot!!!

Combinations

This is the last idea from Chapter 4. Consider again the 5 candy bars: S, T, B, H, and R.

Sometimes we don’t care about the order in which we select items. How many ways may you

select 3 of these candy bars if the order of the selection is unimportant to you? That is, STH is

the same TSH?

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 12

Let’s take this from the top. If you are very patient, you can write out all

𝑃5,3 ≡ 5!

(5 − 3)!=

120

2= 60

ways to select a candy bar if order is important and then start crossing off the sequences that are

really equivalent, or just rearrangements.

Luckily, I am this patient.

STH SHT TSH THS HST HTS

STR SRT TSR TRS RST RTS

STB SBT TSB TBS BST BTS

SHR SRH HSR HRS RSH RHS

SHB SBH HSB HBS BSH BHS

SRB SBR RSB RBS BSR BRS

THR TRH HTR HRT RTH RHT

THB TBH HTB HBT BTH BHT

TRB TBR RTB RBT BTR BRT

HRB HBR RHB RBH BHR BRH

BCD BDC CBD CDB DBC DCB

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 13

If you have less patience, reason as follows: Within the 60 possible sequences, how many are

really just TSH rearranged? Evidently that would be 3 ∙ 2 ∙ 1 = 6 If we “divide these out” we

come up with

𝑃5,3

3!≡

5!

(5 − 3)! 3!=

120

2 ∙ 6= 10

These we should all be happy to list by hand: S, T, B, H, R select 3, order unimportant

STB SBH SHR TBH THR BHR

STH SBR TBR

STR

More formally, the number of possible unordered groups which contain r out of n objects is

simply the number of ordered groups, i.e. the number of possible permutations, 𝑃𝑛,𝑟 divided by

the number of ways to arrange r objects (i.e. we eliminate the redundant groups). Therefore

𝐶𝑛,𝑟 ≡ 𝑃𝑛,𝑟

𝑟! =

𝑛!

(𝑛 − 𝑟)! 𝑟!

Another Key Example Suppose you toss a biased coin 6 times. We know that there are 2 ∙ 2 ∙

2 ∙ 2 ∙ 2 ∙ 2 = 26 = 64 possible sequences:

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 14

HHHHHH HHTHHH HTHHHH HTTHHH THHHHH THTHHH TTHHHH TTTHHH

HHHHHT HHTHHT HTHHHT HTTHHT THHHHT THTHHT TTHHHT TTTHHT

HHHHTH HHTHTH HTHHTH HTTHTH THHHTH THTHTH TTHHTH TTTHTH

HHHHTT HHTHTT HTHHTT HTTHTT THHHTT THTHTT TTHHTT TTTHTT

HHHTHH HHTTHH HTHTHH HTTTHH THHTHH THTTHH TTHTHH TTTTHH

HHHTHT HHTTHT HTHTHT HTTTHT THHTHT THTTHT TTHTHT TTTTHT

HHHTTH HHHTTTH HTHTTH HTTTTH THHTTH THHTTTH TTHTTH TTTTTH

HHHTTT HHTTTT HTHTTT HTTTTT THHTTT THTTTT TTHTTT TTTTTT

o How many of these have exactly 0 HEADS:

That’s easy, just TTTTTT

o How many of these have exactly 1 HEADS:

That’s not too bad, just look through the list to see HTTTTT, THTTTT, TTHTTT,

TTTHTT, TTTTHT, TTTTTH

o How many of these have exactly 2 HEADS:

This is starting to become work. Here’s how we think about it: envision 6 places,

like 6 cities, candy bars, etc. Out of these 6, select 2 of them. This is a

combination because selecting the third slot for a H and then the fifth gives the

same outcome TTHTHT as selecting the fifth and then the third. Evidently we

should find

𝐶6,2 ≡ 𝑃𝑛,𝑟

𝑟! =

6!

(6 − 2)! 2!=

720

24 ∙ 2= 15

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 15

Of these, take a second and with a highlighter indicate these 15 slots.

o How many of these have exactly 3 HEADS:

𝐶6,2 ≡ 𝑃𝑛,𝑟

𝑟! =

6!

(6 − 3)! 3!=

720

6 ∙ 6= 20

o You should do the same for 4, 5, and 6 HEADS.

o

More Examples

How many (5 card) poker hands are possible if the order of the cards is unimportant?

o How many of these are full houses?

o How many are straight flushes, etc?

In how many distinct ways can you arrange the letters: a, b, c, d?

In how many distinct ways can you arrange the letters: b, o, o, k?

The letters: m, i, s, s, i, s, s, i, p, p, i?

There are 10 math majors, 15 engineering majors, and 5 students from other majors in a

probability class. In how many ways can you select two students from this class in such a

way that they are from different majors?

In how many ways can a set of 𝑛 elements be arranged if there are 𝑛1 objects of one type, 𝑛2

objects of another type, and so on till 𝑛𝑘 objects of the 𝑘𝑡ℎ type, where

∑ 𝑛𝑖 𝑘

𝑖=1= 𝑛

Prof. Thistleton MAT 505 Introduction to Probability Lecture 6

SUNY POLY Page 16

How many different binary numbers of length 7 are possible if 3 of the digits are 1’s?

In how many ways can a group of 8 people be divided into working groups of 2 people each?

In how many ways can a set of 𝑛 elements be partitioned into 𝑘 subsets with 𝑛1,

𝑛2, … , 𝑛𝑘 elements in each subset?

An excellent reference for combinatorics is the text Applied Combinatorics by Alan Tucker.