Prof. Dr. S. K. Bhattacharjee Department of Statistics

University of Rajshahi

Random Variable and its Properties

Basic concepts, Discrete and Continuous random variables, Density and Distribution functions, Mathematical expectations and Variance, Functions of a random variable. Marginal and Conditional distributions. Conditional expectation and conditional variance. Conditional independence, Entropy.

Random Variable

Let Ω be a sample space corresponding to some experiment ξ and let X: Ω→ R be a function from the sample space to the real line. Then X is called a random variable.

A random variable is a function from a sample space into the real numbers.

the outcome of the experiment is a random variable

Examples of Random Variables

Discrete and Continuous R.V.

Examples of Discrete R.V.

No. of students attending in a class.No. of accident in a particular day.No. of mobile calls in an hour.

Examples of Continuous R.V.

Height of a man.Amount of water drinks a person per day.Amount of rice consumption of a family per

month.

Probability DistributionSince Random variables cannot be predicted

exactly, they must be described in the language of probability.

Every outcome of the experiment will have a probability associated with it.

The pattern of probabilities that are assigned to the values of the random variable is called the probability distribution of the random variable.

Two types of Distribution: Discrete Distribution and Continuous Distribution.

Probability Distributions of Discrete Random Variables

The probability distribution of a discrete random variable Y is the set of values that this random variable can take, together with their associated probabilities. Probabilities are numbers between zero and one inclusive that always add to one when summed over all possible values of the random variable. Example: If we toss a fair coin twice, and Y is the total number of heads that turn up.

Probability Distribution of Y isPossible values of the random variable Y

0 1 2

Associated probabilities .25 .50 .25

Probability Distribution and CDFMathematical form of this distribution is Pry=x=

The CDF is defined by F(x)=PrY≤x = xxx

Cx 2

0

2 )211()21(

xxCx 22 )211()21(

Graph of CDF for Discrete Distribution

Example

Probability Distribution and CDF

Density Functions of Binomial and Poisson Distributions.

Examples of Discrete Probability Distribution

BernoulliBinomialPoissonNegative BinomialUniformHypergeometric

Density Function for Continuous Random Variables

Density Function and CDF for Continuous R.V.

Density Function for Continuous Random Variables

Examples of Continuous DistributionUniformNormalExponentialGammaBeta

Normal Probability Distribution

The pdf of this distribution is given by

Density Curve of Normal Distribution

Example of Normal Distribution

Answer

Answer

Answer

Answer

Expectations of Random Variables

Variance of R.V.

ExampleConsider as an example the New York State Daily Numbers lottery game. The simplest version of the game works as follows: each day a three digit number between 000 and 999, inclusive, is chosen. You pay $1 to bet on a particular number in a game. If your umber comes up, you get back $500 (for a net profit of $499), and if your number doesn’t come up, you get nothing (for a net loss of $1). Consider the random process that corresponds to one play of the game, and define the random variable W to be the net winnings from a play. The following table summarizes the properties of the random variable as they relate to the random process:

Outcome of process Probability Value of WYour number comes up p = 1/1000 W = 499Your number doesn’t come up p = 999/1000 W = −1

MeanIn the long run, we expect to win 1 time out of every 1000 plays, where we’d win $499, and we expect to lose 999 out of every 1000 plays, where we’d lose $1 (this is just the frequency theory definition of probabilities). That is, our rate of winnings per play, in the long run, would be $499, .001 of the time, and −$1, .999 of the time, or (499)(.001) + (−1)(.999) = −.5.In the long run, we lose 50/c each time we play. Note that on any one play, we never lose 50/c (we either win $499 or lose $1); rather, this is saying that if you play the game 10000 times, you can expect to be roughly $5000 down at the end. An even better way to look at it is that if 10 million people play the game every day, the state can expect to only have to give back about $5 million, a daily profit of a cool $5 million (this is why states run lotteries!).

Variance

Examples

Expectations of Functions of Random Variable

Properties of Expectations

Properties of Expectations

Properties of Expectations

Joint Probability Mass Function

Marginal Distribution

Conditional Distribution

Conditional Variance

Marginal and Conditional Distribution

X↓ Y→ 0 1 2 P(X) P(X|Y=1)

0 1/8 1/8 0 1/4 1/4

1 1/8 1/4 1/8 1/2 1/2

2 0 1/8 1/8 1/4 1/4

P(Y) 1/4 1/2 1/4

P(Y|X=2)

0 1/2 1/2

Conditional Mean and Variance

Marginal Distribution of X: X : 0 1 2 P(X) : ¼ ½ ¼Conditional Distribution of X given Y = 1: X : 0 1 2 P(X|Y=1) : ¼ ½ ¼ E(X|Y=1) = 0x1/4 + 1x1/2 + 2x1/4 = 1E(X2|Y=1) = 02 x1/4 +12 x1/2 +22 x1/4 = 3/2V(X|Y=1) = E(X2|Y=1) - E(X|Y=1)2 = 1/2

Conditional Density and Conditional Expectation of Continuous R.V.

Conditional Density of Continuous R.V.

Conditional Variance of Continuous R.V.

Marginal and Conditional Distribution for Continuous R.V.

Conditional Mean and Variance for Continuous R.V

Independence of R.V.

Entropy

For a random variable with outcomes x1,x2,...xn, the Shannon entropy, a measure of uncertainty is defined as

where is the probability mass function of outcome .

Conditional EntropyTo understand the meaning of Eq. (1), first consider a set of possible outcomes (events)

, with equal probability . An example would be a fair die with values, from to . The uncertainty for such a set of outcomes is defined by

the conditional entropy of two events X and Y taking values xi and yj respectively, as

Differential Entropy

Example: Uniform Distribution

Relative Entropy and Mutual InformationThe Shannon entropy is restricted to random variables taking discrete values. The formula

where f denotes a probability density function on the real line, is analogous to the Shannon entropy and could thus be viewed as an extension of the Shannon entropy to the domain of real numbers.

Another useful measure of entropy for the continuous case is the relative entropy of a distribution, defined as the Kullback-Leibler divergence from the distribution to a reference measure m(x),

The negative differential entropy of the normal distribution can be evaluated simply as

Joint and Conditional Differential Entropy