Permutations & Combinations and Distributions 1 Krishna.V.Palem Kenneth and Audrey Kennedy Professor...
-
Upload
dulcie-morrison -
Category
Documents
-
view
217 -
download
0
Transcript of Permutations & Combinations and Distributions 1 Krishna.V.Palem Kenneth and Audrey Kennedy Professor...
Permutations & Combinations and Distributions
1
Krishna.V.PalemKenneth and Audrey Kennedy Professor of ComputingDepartment of Computer Science, Rice University
Take Home II - Generalizing the sum of expectations
result (hint)
2. Prove that the expectation of sum of n random variables is equal to the sum of expectation of the n random variables.Let x1, x2, x3…. xn be n random variablesLet z = x1 + x2 + x3…. + xn
To prove
n
iixEzE
1
)()(
Hint for the proof
3
Use the result E(X+Y)=E(X)+E(Y) to generalize for n random variables
Consider E(X1 + X2 + X3…. + Xn )
Let X2 + X3…. + Xn = Y1
Then E(X1 + Y1) = E(X1) + E(Y1)
Now consider X3…. + Xn = Y2
and repeat the same procedure
)....()()...( 32121 nn XXXEXEXXXE
ContentsPermutations and CombinationsCalculating probabilities using
combinationsDistributionProof of Law of Large NumbersBinomial DistributionNormal Distribution
4
5
Permutations vs. CombinationsBoth are ways to count the possibilitiesThe difference between them is whether
order matters or notConsider a 5-card hand:
A♦, 5♥, 7♣, 10♠, K♠Is that the same hand as:
K♠, 10♠, 7♣, 5♥, A♦Does the order the cards are handed out
matter?If yes, then we are dealing with permutationsIf no, then we are dealing with combinations
6
PermutationsA permutation is an ordered arrangement of
the elements of some set SLet S = {a, b, c}c, b, a is a permutation of Sb, c, a is a different permutation of S
An r-permutation is an ordered arrangement of r elements of the setA♦, 5♥, 7♣, 10♠, K♠ is a 5-permutation of the
set of cardsThe notation for the number of r-
permutations: P(n,r)For example, poker hand is one of P(52,5)
permutations
7
PermutationsNumber of poker hands (5 cards):
P(52,5) = 52*51*50*49*48 = 311,875,200r-permutation notation: P(n,r)
The poker hand is one of P(52,5) permutations
)1)...(2)(1(),( rnnnnrnP
)!(
!
rn
n
n
rni
i1
8
Deriving the formula of PermutationsThere are n ways to choose the first
elementn-1 ways to choose the secondn-2 ways to choose the third…n-r+1 ways to choose the rth element
By the product rule, that gives us:P(n,r) = n(n-1)(n-2)…(n-r+1)
9
CombinationsWhat if order doesn’t matter?In poker, the following two hands are
equivalent:A♦, 5♥, 7♣, 10♠, K♠K♠, 10♠, 7♣, 5♥, A♦
The number of r-combinations of a set with n elements, where n is non-negative and 0≤r≤n is:
)!(!
!),(
rnr
nrnC
10
Deriving the formula for CombinationsLet C(n,r) be the number of ways to generate
unordered combinationsThe number of ordered combinations (i.e. r-
permutations) is P(n,r)The number of ways to order a single one of those
r-permutations P(r,r) The total number of unordered combinations is
the total number of ordered combinations (i.e. r-permutations) divided by the number of ways to order each combination
Thus,
C(n,r) = P(n,r)/P(r,r) (1)
11
Deriving the formula for CombinationsBut from the derivation of permutation formula,
we know that
Hence, substituting n=r, we get
Replacing (2) and (3) in (1), we get
(since, 0! = 1)
)!/(!
)!/(!
),(
),(),(
rrr
rnn
rrP
rnPrnC
)!(
!
rr
r
),( rrP
)!(
!
rn
n
)1)...(2)(1(),( rnnnnrnP (2)
(3)
)!(!
!),(
rnr
nrnC
In-class Exercise - 1Card Terminology:
face value – same number cards (2-10, J, Q, K, A) has 4 cards of same face value
suite – set of cards with same symbol four suites – diamond, heart, spade, clubs each suite has 13 cards
Q) In a standard deck of cards, compute the number of ways you can deal each of the following five-card hands in poker.
1. Total number of different possible hands (five cards in a hand)2. Number of distinct Flush (all 5 cards have the same suite) 3. Number of distinct Four of a kind (4 same face value cards)
A) 1. C (52,5)
2. C (13,5) * C (4,1)
3. C (13,1) * C (48,1)
12
13
ContentsPermutations and CombinationsCalculating probabilities using
combinationsDistributionProof of Law of Large NumbersBinomial DistributionNormal Distribution
14
In-Class Exercise -2
Q) Now, compute the probability of getting a flush in a five-card poker game?
A)
Number of favorable events = C (13,5) * C (4,1)Total no. of events = C (52,5)Hence, Probability = C (13,5) * C (4, 1)/ C ( 52,5)
15
Probabilityof outcome
No. of favorable events
Total no. of events
In-Class Exercise - 3Consider an example: In an experiment of
20 coin tosses, we want to calculate the probability of heads falling exactly 5 times. How do we do this?
Solution:Probability of heads in 1 coin toss = ½Probability of heads falling in 5 of the coin tosses =
½* ½* ½* ½* ½ = (1/2)5 (Method of intersection of events)
Probability of heads not falling in 1 coin toss = ½Probability of heads not falling in the rest (20-5=15)
coin tosses = (1/2)15
16
Use of Combinations to Calculate Probabilities
Hence, the probability of getting exactly 5 heads out of 20 tosses = (1/2)5 *(1/2)15 =(1/2)20
Q) Did we account for which of the coin tosses had an event HEAD?
A) No
Q) How do we account for it? Permutations or Combinations?
A) Combinations, as the order of selection is not important
17
Is this correct?
Use of Combinations to Calculate Probabilities
18
Q) How do we select 5 tosses out of 20 tosses with heads outcome using combinations?
1 2 3 4 5 6 7 8 9 10
11
12
13
14
15
16
17
18
19
20
H H H H H T T T T T T T T T T T T T T T
T T T T T T T T T T T T T T T H H H H H
H H T T H H T T H T T T T T T T T T T T
…
Let us make a table of all possible outcomes of 20 coins which have 5 HEADs
We can see that this table can be generated by choosing 5 places out of the 20 places where H canoccur.
Thus the total number of such combinations would be C(20,5)
Use of Combinations to Calculate Probabilities
Hence, the probability of getting exactly 5 heads out of 20 coin tosses is given by = C (20,5) * (1/2)20
How do we generalize this method of computing
probabilities?
Consider an example: In an experiment of 20 “biased” coin tosses, we want to calculate the probability of heads falling exactly 5 times. How do we do this?Given probability of HEAD = p
19
Question
Use of Combinations to Calculate Probabilities
Consider an example: In an experiment of 20 “biased” coin tosses, we want to calculate the probability of heads falling exactly 5 times. How do we do this?Assume that the probability of HEAD = p
Solution:Probability of heads in 1 coin toss = pProbability of heads falling in 5 of the coin tosses =
p*p*p*p*p = (p)5
Probability of heads not falling in 1 coin toss = 1-pProbability of heads not falling in the rest (20-5=15)
coin tosses = (1-p)15
20
Use of Combinations to Calculate Probabilities
If we generalize the number of trials and the number of HEADs or successes also we obtain
Assume that in n trails of an event we want to compute the probability P of getting k successes when the probability of success in each trial is p
We denote this by the following expression P(number of heads=k) = C(n,k) * pk *
(1-p)n-k
21
Binomial Distribution
Schedule for the next 2 weeks5 Oct – Tutorial Session II
Covers Expectations, Permutations & Combinations, Basic Distributions
7 Oct – Mini Project 115 % of Final GradeCan do it as a take home if the time
provided in the class is not sufficient12 Oct – Fall Break Holiday14 Oct – Project Proposal report due and in
class discussion on the proposals
22
Project Discussion
23
ContentsPermutations and CombinationsCalculating probabilities using
combinationsDistributionBinomial DistributionNormal Distribution
24
Consider the following experiment
Event
Event 1
Event 2
..Etc.
Probabilities
p1
p2
..Etc.
Define a variable x which takes as manyvalues as the number of events
Event X
Event 1
1
Event 2
2
..Etc. ..Etc.Therefore using the probabilities of the events, we can define a function which relates the
variable x and the probabilities of the events
ipixp )(Probability (Event i)
Here ‘x’ is called a random variable.Where i={1,2,…}
What is a distribution?
Distribution
What is a distribution?A distribution is a function defined on the random variable that gives the value of the
probability of the random variable taking a particular value
i p(i)
1 1/6
2 1/6
3 1/6
4 1/6
5 1/6
6 1/6
The probability distribution describes the range of possible values that a random variable can attain and the probability that the value
of the random variable is within any (measurable) subset of that range.
Guassian DistributionUniform Distribution Binomial Distribution
Examples of a Distribution
}spaceevent{x
Example of a DistributionSuppose you flip a coin two times.
This experiment can have four possible outcomes: HH, HT, TH, and TT.
Now, let the variable random X represent the number of Heads that result from this experiment. X can take on the values 0, 1, or 2.
The table, equation and graph below, which associate each outcome with its probability, are all representations of probability distribution for above example.
27
25.0)0X(P
50.0)1X(P
25.0)2X(P
Distribution Table Distribution Equation Distribution graph
Video on Terms in Distributions
28
VarianceVariance of a random variable or probability
distribution is a measure of statistical dispersion, averaging the squared distance of its possible values from the expected value (mean).
If random variable X has expected value (mean) μ = E(X), then the variance Var(X) of X is given by:
29
Variance
Standard DeviationStandard deviation is the positive square
root of the variance. It is given by:
Low standard deviation indicates that the data points tend to be very close to the same value (the mean), while high standard deviation indicates that the data are “spread out” over a large range of values
30 A plot of a normal distribution (or bell curve). Each colored band has a width of one standard deviation.
Useful derivation for VarianceIn probability theory, the computational
formula for the variance Var(X) of a random variable X is the formula
31
Derivation
(from definition)
(expansion of expectation formula)
ContentsPermutations and CombinationsCalculating probabilities using
combinationsDistributionProof of Law of Large NumbersBinomial DistributionNormal Distribution
32
Law of Large NumbersThe law of large numbers (LLN) describes the long-
term stability of the mean of a random variable.
Given a random variable with a finite expected value, if its values are repeatedly sampled, as the number of these observations increases, their mean will tend to approach and stay close to the expected valuefor example, consider the coin toss experiment. The frequency
of heads (or tails) will increasingly approach 50% over a large number of trials.
Mathematically, it can be represented as,if Mean is , then
33
Proof of Law of Large Numbers
First, let us derive the Chebyshev Inequality which simplifies the derivation of law of large numbers
Chebyshev Inequality: Let X be a discrete random variable with expected value µ= E(X), and let > 0 be any positive real number
Let m(x) denote the distribution function of X. Then the probability that X differs from µ by at least is given by
34
Proof of Chebyshev Inequality
Proof of Law of Large Numbers
We know that,
But, V(X) is clearly at least as large as
Replacing (x- µ)2 with , to get a lower bound,
Hence, we get
35
Proof of Law of Large Numbers
Let X1, X2, . . . , Xn be an independent trials process, with finite expected value µ = E(Xj) and finite variance = V (Xj ).
Let Xn be the mean of X1,X2,… Xn. Hence,
Equivalently,
But from Chebyshev’s inequality, we have
36
Proof of Law of Large Numbers
Replacing X with Xn, we get
Hence, we get
As n approaches infinity, the expression approaches 1. Hence, we have obtained,
37
Binomial Distribution
38
Binomial distribution is the discrete probability distribution of the number of successes in a sequence of n independent yes/no experiments, each of which yields success with probability p
It can be applied in a wide variety of practical situations
for k = 0,1,2,3…. n, where
is called the ‘Binomial Coefficient’
ContentsPermutations and CombinationsCalculating probabilities using
combinationsDistributionProof of Law of Large NumbersBinomial DistributionNormal Distribution
39
Binomial Distribution
40
Binomial distribution is a very interesting distribution in the sense that it can be applied in a wide variety of practical situations.
An example, Assume 5% of a very large population to be
green-eyed. You pick 40 people randomly. The number of green-eyed people you pick is a
random variable X which follows a binomial distribution with n = 40 and p = 0.05.
Let us see how this distribution varies with different values of n and p with respect to X.
Binomial Distribution
41
Another elementary example of a binomial distribution is: Roll a standard die ten times and count the number of
sixes. Denote the number of sixes by the random variable XThe distribution of this random number X is a binomial
distribution with n = 10 and p = 1/6.
Can you plot this distribution and see how it varies with X
For the previousexample, this graph showsthe variation in probability
Notice how it peaks in the middle and dies away at the ends
X=number of green eyed peoplepro
babili
ty(p
)
In-Class ExerciseLet us try out an example of a binomial
distribution: Consider a standard die roll for 20 times
Q) Denote the number of times the outcome of the roll an even number by a random variable X. Compute the probability distribution of X = 8 for this event.
Q) Denote the number of times the outcome of the roll is ‘6’ by the random variable Y. Compute the probability distribution of Y equal to 4 for this event.
Q) Denote the number of times the outcome of the roll is ‘2’ by the random variable Z. Compute the probability distribution of Z less than or equal to 4 for this event.
42Use Binomial Distribution to solve these questions.
Attributes of Binomial Distribution
If X ~ B(n, p) (that is, X is a binomially distributed random variable with total ‘n’ events and probability of success ‘p’ in each event),Expected value or mean of X is
Variance of X is
Standard deviation of X is
43
Video on Binomial Distribution : A Summary
44
We have seen that variance is equal to
In using this formula we see that we now also need the expected value of X 2:
We can use our experience gained before in deriving the mean. We know how to process one factor of k. This gets us as far as
45
Derivation of Variance of Binomial Distribution
(again, with m = n − 1 and s = k − 1). We split the sum into two separate sums and we recognize each one
The first sum is identical in form to the one we calculated in the Mean (above). It sums to mp. The second sum is unity.
Using this result in the expression for the variance, along with the Mean (E(X) = np), we get
46
Derivation of Variance of Binomial Distribution
Deriving the Expectation of Binomial Distribution
If X ~ B(n, p) (that is, X is a binomially distributed random variable with total ‘n’ events and probability of success ‘p’ in each event), then the expected value of X is
We apply the definition of the expected value of a discrete random variable to the binomial distribution
The first term in the summation (for k=0) equals to 0 and can be removed. In the rest of the summation, we expand the C(n,k) term,
47
Since n and k are independent of the sum, we get
Assume, m = n − 1 and s = k − 1. Limits are changed accordingly
where x=1-p, y=p, m=n & s=k
Hence, as (x+y) = ((1-p)+p) = 1, we get
48
This is similar to the expansion of a binomial theorem
Deriving the Expectation of Binomial Distribution
We have seen that variance is equal to
We now compute the value of E(X2):
Use a similar approach as in the derivation of the mean to expand C(n,k)assume m = n − 1 and s = k − 1
49
Derivation of Variance of Binomial Distribution
We split the sum into two separate sums
The first sum is identical in form to the one we calculated in the Mean (above). It sums to mp. The second sum is unity (binomial theorem).
Hence, we get
50
Derivation of Variance of Binomial Distribution
In-Class ExerciseLet us continue the previous example of the
binomial distribution: Consider a standard die roll for 100 times
instead of 20 timesQ) Denote the number of times the outcome of the roll is ‘2’
by the random variable X. Compute the probability distribution of X greater than or equal to 60 for this event.
What if we consider the die roll a million times and need to compute the probability that X is greater than or equal to 100,000 for this event?
51
Difficult
Impossible!
How to Compute Distributions for Large ‘N’?
Abraham de Moivre noted that the shape of the binomial distribution approached a very smooth curve when the number of events increasedhe considered a coin toss experiment
De Moivre tried to find a mathematical expression for this curve to find the probabilities involving large number of
events more easily.led to the discovery of the Normal curve
52
Example by De Moivre
53
Can be approximated as a curve
Number of events ‘N’ increases
Coin Toss Experiment
Random variable X = Number of heads
Video on Galton Board GameDemonstrates how Binomial distribution
gives rise to a Normal/Gaussian distribution as number of trials/events tends to infinity
54
ContentsPermutations and CombinationsCalculating probabilities using
combinationsDistributionBinomial DistributionNormal Distribution
55
Video on Normal Distribution
56 First 2 mins only
Normal DistributionTo indicate that a real-valued random variable X is
normally distributed with mean μ and variance σ2 ≥ 0, we write
The normal distribution is defined by the following equation:
All normal distributions are symmetric and have bell-shaped density curves with a single peak.
57Note: Normal distribution is a continuous probability distribution while Binomial distribution is a discrete probability distribution
In-Class ExerciseLet us try out an example of a normal distribution: Consider a coin toss experiment for 1000 tosses
Q) Denote the number of times the outcome of the toss is heads by a random variable X. Compute the probability distribution of X occurring at most 600 times.
A)
A) Since, the original event is a binomial distribution and we use normal distribution to approximate it, we can use µ=np & = np(1-p). Hence,
x<=600; µ = 1000*1/2 = 500 and = 1000*1/2*(1-1/2) =250
Substituting this in the normal distribution equation, we get
Calculating, we get Probability of x<=600 = 0.65542
58
How would you use Binomial Distribution to solve this question?
600
0k
1000)2/1(*)k,1000(C Difficult
How would you use Normal Distribution to solve this question?
Source of calculation: http://stattrek.com/Tables/Normal.aspx
Examples of Few Applications of Normal Distribution
Approximately normal distributions occur in many situationsIn counting problems
Binomial random variables, associated with yes/no questions;Poisson random variables, associated with rare events;
In physiological measurements of biological specimens:logarithm of measures of size of living tissue (length, height,
weight);length of inert appendages (hair, claws, nails, teeth) of
biological specimens, in the direction of growthMeasurement errors Financial variablesLight intensity
intensity of laser light is normally distributed;
59
Normal DistributionTo indicate that a real-valued random variable X is
normally distributed with mean μ and variance σ2 ≥ 0, we write
The normal distribution is defined by the following equation:
All normal distributions are symmetric and have bell-shaped density curves with a single peak.
60Note: Normal distribution is a continuous probability distribution while Binomial distribution is a discrete probability distribution
In-Class ExerciseLet us try out the previously stated “nearly
impossible” problem using a normal distribution:
Consider a coin toss experiment for 1,000,000 tosses
Q) Denote the number of times the outcome of the toss is heads by a random variable X. Compute the probability distribution of X occurring at most 100,000 times.
A)
61
How would you use Binomial Distribution to solve this question?
000,100
0k
000,000,1)2/1(*)k,1000000(C Difficult
How would you use Normal Distribution to solve this question?
In-Class ExerciseSince, the original event is a binomial distribution and
we can use normal distribution to approximate it.
We know that µ=np & = np(1-p). Hence,x<=100000; µ = 1,000,000*1/2 = 500,000 and = 1,000,000*1/2*(1-1/2) =250,000
Substituting this in the normal distribution equation, we get
Calculating the integral with limits from 0 to 100,000;
we get Probability of x<=100,000 = 0.0548
62
Source of calculation: http://stattrek.com/Tables/Normal.aspx
Examples of Few Applications of Normal Distribution
Approximately normal distributions occur in many situationsIn counting problems
Binomial random variables, associated with yes/no questions;
Poisson random variables, associated with rare events;In sports statistical analyses:
calculating mean physical attributes like heights, weights etc and their standard deviations
estimating the probabilities of winning the gamesMeasurement errors Financial variablesLight intensity
intensity of laser light is normally distributed;
63
END
64
Example Application of Bayes Theorem
65