Probability Probability is used to quantify the likelihood of things that can happen, when it is not yet known whether they will happen. Sometimes it is also used to quantify the likelihood of things that could have happened in the past, when it is not yet known whether they actually happened.

Since we usually speak of the "probability of an event", the next section introduces a formal definition of the concept of event. We then discuss the properties that probability needs to satisfy. Finally, we mention some possible interpretations of the concept of probability.

Sample space, sample points and events Let be a set of things that can happen. We say that is a sample space, or space of all possible outcomes, if it satisfies the following two properties:

1. Mutually exclusive outcomes. Only one of the things in will happen. That is, when we learn that has happened, then we also know that none of the things in the set has happened.

2. Exhaustive outcomes. At least one of the things in will happen.

An element is called a sample point, or a possible outcome.

When, and if, we learn that has happened, is called the realized outcome.

A subset is called an event. In the section below, you will see that not every subset of the sample space is, strictly speaking, an event; however, on a first reading you can be happy with this definition.

Note that itself is an event, because every set is a subset of itself, and the empty set is also an event, because it can be considered a subset of .

Example Suppose that we toss a die. Six numbers, from 1 to 6, can appear face up, but we do not yet know which one of them will appear. The sample space is:Each of the six numbers is a sample point. The outcomes are mutually exclusive, because only one number at a time can appear face up. The outcomes are also exhaustive, because at least one of the six numbers will appear face up, after we toss the die. Define: is an event (a subset of ). It can be described as "an odd number appears face up". Now define:Also is an event and it can be described as "the number 6 appears face up".

Probability and its properties The probability of an event is a real number, attached to the event, that tells us how likely that event is. We denote the probability of an event by .

Probability needs to satisfy the following properties:

1. Range. For any event , .

2. Sure thing. . 3. Sigma-additivity (or countable additivity). Let be a sequence of events. Let all the

events in the sequence be mutually exclusive, i.e., if . Then:

Property (1) is self-explanatory. It just means that the probability of an event is a real number between 0 and 1.

Property (2) says that at least one of all the things that can possibly happen will happen with probability 1.

Property (3) is a bit more cumbersome. It can be proved (see below) that if sigma-additivity holds, then also the following

holds:

This property, called finite additivity, while very similar to sigma-additivity, is easier to interpret. It says that if two events are disjoint, then the probability that either one or the other happens is equal to the sum of their individual probabilities.

Example Suppose that we flip a coin. The possible outcomes are either tail () or head (), i.e.:There are a total of four subsets of (events): itself, the empty set , the event and the event . The following assignment of probabilities satisfies the properties enumerated

above: All these probabilities are between 0 and 1, so the range property is satisfied. , so the sure thing property is satisfied. Also sigma-additivity is satisfied,

because: and the four couples , , , are the only four possible couples of disjoint sets.

Before ending this section, two remarks are in order. First, we have not discussed the interpretations of probability, but below you can find a brief discussion of the interpretations of probability. Second, we have been somewhat sloppy in defining events and probability, but you can find a more rigorous definition of probability below.

Other properties of probability The following subsections discuss other properties enjoyed by probability.

The probability of the empty set is 0

Here we prove that .

Proof

Define a sequence of event as follows: The sequence is a sequence of disjoint events, because the empty set is disjoint from any other set. Then:

which impliesand .

A sigma-additive function is additive

A sigma-additive function is also

additive:

Proof

Define a sequence of events as follows: The sequence is a sequence of disjoint events, because the empty set is disjoint from any other set.

Then:

Probability of the complement

Let be an event and its complement (i.e. the set of all elements of that do not belong to ). Then

Proof

Note that:and that and are disjoint sets. Then, using the sure thing property and finite additivity, we obtain:which implies

In other words, the probability that an event does not occur is equal to one minus the probability that it occurs.

Probability of a union

We have already seen how to compute in the special case in which and are two disjoint events. In the more general case, in which they are not necessarily disjoint, the formula is:

Proof

This is proved as follows. First note that:

so that:Furthermore the event can be written as

follows: and the three events on the right hand side

are disjoint. Thus:

Monotonicity of probability

If two events and are such that , then:

Proof

This is easily proved using additivity:where the latter inequality is a consequence of the fact that (by the range property of probability).

In other words, if occurs less often than , because the latter contemplates more occurrences, then the probability of must be less than the probability of .

Interpretations of probability This subsection briefly discusses some common interpretations of probability. Although none of these interpretations is sufficient per se to clarify the meaning of probability, they all touch upon important aspects of probability.

Classical interpretation of probability

According to the classical definition of probability, when all the possible outcomes of an experiment are equally likely, the probability of an event is the ratio between the number of outcomes that are favorable to the event and the total number of possible outcomes. While intuitive, this definition has two main drawbacks:

1. it is circular, because it uses the concept of probability to define probability: it is based on the assumption of 'equally likely' outcomes, where equally likely means 'having the same probability';

2. it is limited in scope, because it does not allow to define probability when the possible outcomes are not all equally likely.

Frequentist interpretation of probability

According to the frequentist definition of probability, the probability of an event is the relative frequency of the event itself, observed over a large number of repetitions of the same experiment. In other words, it is the limit to which the

ratio: converges when the number of repetitions of the experiment tends to infinity. Despite its intuitive appeal, also this definition of probability has some important drawbacks:

1. it assumes that all probabilistic experiments can be repeated many times, which is false;

2. it is also somewhat circular, because it implicitly relies on a Law of Large Numbers, which can be derived only after having defined probability.

Subjectivist interpretation of probability

According to the subjectivist definition of probability, the probability of an event is related to the willingness of an individual to accept bets on that event. Suppose a lottery ticket pays off 1 dollar in case the event occurs and 0 in case the event does not occur. An individual is asked to set a price for this lottery ticket, at which she must be indifferent between being a buyer or a seller of the ticket. The subjective probability of the event is defined to be equal to the price thus set by the individual. Also this definition of probability has some drawbacks:

1. different individuals can set different prices, therefore preventing an objective assessment of probabilities;

2. the price an individual is willing to pay to participate in a lottery can be influenced by other factors that have nothing to do with probability; for example, an individual's betting behavior can be influenced by her preferences.

Rigorous definitions

A more rigorous definition of event

The definition of event given above is not entirely rigorous. Often, statisticians work with probability models where some subsets of the sample space are not considered events. This happens mainly for the following two reasons:

1. sometimes, the sample space is a really complicated set; to make things simpler, attention is restricted to only some subsets of the sample space;

2. sometimes, it is possible to assign probabilities only to some subsets of the sample space; in these cases, only the subsets to which probabilities can be assigned are considered events.

Denote by the space of events, i.e. the set of subsets of that are considered events. In rigorous probability theory, it is required to be a sigma-algebra.

Definition is a sigma-algebra on if it is a set of subsets of satisfying the following three properties:

1. Whole set. . 2. Closure under complementation. If then also (the complement is the set of all

elements of that do not belong to ). 3. Closure under countable unions. If is a sequence of subsets of belonging to , then:

Why is a space of events required to satisfy these properties? Besides a number of mathematical reasons, it seems pretty intuitive that they must be satisfied. Property 1) means that the space of events must include the event "something will happen", quite a trivial requirement! Property 2) means that if "one of the things in the set will happen" is considered an event, then also "none of the things in the set will happen" is considered an event. This is quite natural: if you are considering the possibility that an event will happen, then, by necessity, you must also be simultaneously considering the possibility that the same event will not happen. Property 3) is a bit more complex. However, the following property, implied by 3), is probably easier to interpret:It means that if "one of the things in will happen" and "one of the things in will happen" are considered two events, then also "either one of the things in or one of the things in will happen" must be considered an event. This simply means that if you are able to separately assess the possibility of two events happening, then, of course, you must be able to assess the possibility of either one or the other happening. Property 3) simply extends this intuitive property to countable collection of events: the extension is needed for mathematical reasons, to derive certain continuity properties of probability measures.

A more rigorous definition of probability

The definition of probability given above was not entirely rigorous. Now that we have defined sigma-algebras and spaces of events, we can make it completely rigorous.

Definition Let be a sigma-algebra on the sample space . A function is a probability measure if and only if it satisfies the following two properties:

1. Sure thing. . 2. Sigma-additivity. Let be any sequence of elements of such that implies . Then:

Nothing new has been added to the definition given above. This definition just clarifies that a probability measure is a function defined on a sigma-algebra of events. Hence, it is not possible to properly speak of probability for subsets of the sample space that do not belong to the sigma-algebra.

A triple is called a probability space.

Solved exercises Below you can find some exercises with explained solutions:

Exercise 1.1 A ball is drawn at random from an urn containing colored balls. The balls can be either red or blue (no other colors are possible). The probability of drawing a blue ball is . What is the probability of drawing a red ball?

Solution

The sample space can be represented as the union of two disjoint events and :where the event can be described as 'a red ball is drawn' and the event can be described as 'a blue ball is drawn'. Note that is the complement of :

We know , the probability of drawing a a blue ball:

We need to find , the probability of drawing a a red ball. Using the formula for the probability of a complement:

Exercise 1.2 Consider a sample space comprising three possible outcomes:

Suppose the probabilities assigned to the three possible outcomes are:

Can you find an event whose probability is ?

Solution

There are two events whose probability is .

The first one is:

Using the formula for the probability of a union of disjoint

events:

The second one is:

Using the formula for the probability of a union of disjoint

events:

Exercise 1.3

Consider a sample space comprising four possible outcomes:

Consider the three events , and defined as follows:

Suppose their probabilities are:

Now, consider a fourth event defined as follows: H={2, 4}

Find P(H).

Solution

First note that, by additivity:

Therefore, in order to compute , we need to compute and .

is found using additivity on : so

that:

is found using the fact that one minus the probability of an event is equal to the probability of

its complement and the fact that :

As a consequence: