Natural Language Processing

Introduction to Probability

Joakim Nivre

Uppsala University

Department of Linguistics and Philology

joakim.nivre@lingfil.uu.se

Natural Language Processing 1(11)

Probability and Statistics

“Once upon a time, there was a . . . ”

I Can you guess the next word?I Hard in general, because language is not deterministicI But some words are more likely than others

I We can model uncertainty using probability theoryI We can use statistics to ground our models in empirical data

Natural Language Processing 2(11)

Probability and Statistics

“Once upon a time, there was a . . . ”

I Can you guess the next word?I Hard in general, because language is not deterministicI But some words are more likely than others

I We can model uncertainty using probability theoryI We can use statistics to ground our models in empirical data

Natural Language Processing 2(11)

Probability and Statistics

“Once upon a time, there was a . . . ”

I Can you guess the next word?I Hard in general, because language is not deterministicI But some words are more likely than others

I We can model uncertainty using probability theoryI We can use statistics to ground our models in empirical data

Natural Language Processing 2(11)

The Mathematical Notion of Probability

I The probability of A, P(A), is a real number between 0 and 1:1. If P(A) = 0, then A is impossible (never happens)

2. If P(A) = 1, then A is necessary (always happens)

3. If 0 < P(A) < 1, then A is possible (may happen)

IA is an event in a sample space ⌦

ISample space = all possible outcomes of an “experiment”

IEvent = a subset of the sample space

IEvents can be described as a variable taking a certain value

1. {w 2 ⌦ |w is a noun} , PoS = noun

2. {s 2 ⌦ | s consists of 8 words} , #Words = 8

Natural Language Processing 3(11)

The Mathematical Notion of Probability

I The probability of A, P(A), is a real number between 0 and 1:1. If P(A) = 0, then A is impossible (never happens)

2. If P(A) = 1, then A is necessary (always happens)

3. If 0 < P(A) < 1, then A is possible (may happen)

IA is an event in a sample space ⌦

ISample space = all possible outcomes of an “experiment”

IEvent = a subset of the sample space

IEvents can be described as a variable taking a certain value

1. {w 2 ⌦ |w is a noun} , PoS = noun

2. {s 2 ⌦ | s consists of 8 words} , #Words = 8

Natural Language Processing 3(11)

Logical Operations on Events

I Often we are interested in combinations of two or more eventsI This can be represented using set theoretic operationsI Assume a sample space ⌦ and two events A and B :

1. Complement A (also A

0) = all elements of ⌦ that are not in A

2. Subset A ✓ B = all elements of A are also elements of B

3. Union A [ B = all elements of ⌦ that are in A or B

4. Intersection A \ B = all elements of ⌦ that are in A and B

Natural Language Processing 4(11)

Venn Diagrams

Natural Language Processing 5(11)

Axioms of Probability

IP(A) = The probability of event A

I Axioms:1. P(A) � 0

2. P(⌦) = 1

3. If A and B are disjoint, then P(A [ B) = P(A) + P(B)

Natural Language Processing 6(11)

Probability of an Event

I If A is an event and {x1, . . . , xn} its individual outcomes, then

P(A) =nX

i=1

P(xi )

I Assume all 3-letter strings are equally probableI What is the probability of a string of three vowels?

1. There are 26 letters, of which 6 are vowels

2. There are N = 26

33-letter strings

3. There are n = 6

33-vowel strings

4. Each outcome (string) is equally with P(xi ) =1N

5. So, a string of three vowels has probability

P(A) = nN = 63

263 ⇡ 0.012

Natural Language Processing 7(11)

Probability of an Event

I If A is an event and {x1, . . . , xn} its individual outcomes, then

P(A) =nX

i=1

P(xi )

I Assume all 3-letter strings are equally probableI What is the probability of a string of three vowels?

1. There are 26 letters, of which 6 are vowels

2. There are N = 26

33-letter strings

3. There are n = 6

33-vowel strings

4. Each outcome (string) is equally with P(xi ) =1N

5. So, a string of three vowels has probability

P(A) = nN = 63

263 ⇡ 0.012

Natural Language Processing 7(11)

Rules of Probability

I Theorems:1. If A and A are complementary events, then P(A) = 1 � P(A)2. P(;) = 0 for any sample space ⌦3. If A ✓ B , then P(A) P(B)4. For any event A, 0 P(A) 1

Natural Language Processing 8(11)

Addition Rule

I Axiom 3 allows us to add probabilities of disjoint eventsI What about events that are not disjoint?

I Theorem: If A and B are events in ⌦, thenP(A [ B) = P(A) + P(B)� P(A \ B)

IA = “has glasses”, B = “is blond”

IP(A) + P(B) counts blondes with glasses twice

Natural Language Processing 9(11)

Addition Rule

I Axiom 3 allows us to add probabilities of disjoint eventsI What about events that are not disjoint?

I Theorem: If A and B are events in ⌦, thenP(A [ B) = P(A) + P(B)� P(A \ B)

IA = “has glasses”, B = “is blond”

IP(A) + P(B) counts blondes with glasses twice

Natural Language Processing 9(11)

Quiz 1

I Assume that the probability of winning in a lottery is 0.01I What is the probability of not winning?

1. 0.01

2. 0.99

3. Impossible to tell

Natural Language Processing 10(11)

Quiz 2

I Assume that A and B are events in a sample space ⌦I Which of the following could possibly hold:

1. P(A [ B) < P(A \ B)2. P(A [ B) = P(A \ B)3. P(A [ B) > P(A \ B)

Natural Language Processing 11(11)

Natural Language Processing

Joint, Conditional and Marginal Probability

Joakim Nivre

Uppsala University

Department of Linguistics and Philology

joakim.nivre@lingfil.uu.se

Natural Language Processing 1(11)

Conditional Probability

I Given events A and B in ⌦, with P(B) > 0, the conditionalprobability of A given B is:

P(A|B) =DEF

P(A \ B)

P(B)

I P(A \ B) or P(A,B) is the joint probability of A and B .

IThe prob that a person is rich and famous – joint

IThe prob that a person is rich if they are famous – conditional

IThe prob that a person is famous if they are rich – conditional

Natural Language Processing 2(11)

Conditional Probability

P(A) = size of A relative to ⌦

P(A,B) = size of A \ B relative to ⌦

P(A|B) = size of A \ B relative to B

Natural Language Processing 3(11)

Example

I We sample word bigrams (pairs) from a large text TI Sample space and events:

I ⌦ = {(w1

,w2

) 2 T} = the set of bigrams in TI A = {(w

1

,w2

) 2 T |w1

= run} = bigrams starting with run

I B = {(w1

,w2

) 2 T |w2

= amok} = bigrams ending with amok

I Probabilities:I P(run

1

) = P(A) = 10

�3

I P(amok

2

) = P(B) = 10

�6

I P(run

1

, amok

2

) = (A,B) = 10

�7

I Probability of amok following run? Of run preceding amok?

I P(run before amok) = P(A|B) = 10

�7

10

�6 = 0.1

I P(amok after run) = P(B|A) = 10

�7

10

�3 = 0.0001

Natural Language Processing 4(11)

Example

I We sample word bigrams (pairs) from a large text TI Sample space and events:

I ⌦ = {(w1

,w2

) 2 T} = the set of bigrams in TI A = {(w

1

,w2

) 2 T |w1

= run} = bigrams starting with run

I B = {(w1

,w2

) 2 T |w2

= amok} = bigrams ending with amok

I Probabilities:I P(run

1

) = P(A) = 10

�3

I P(amok

2

) = P(B) = 10

�6

I P(run

1

, amok

2

) = (A,B) = 10

�7

I Probability of amok following run? Of run preceding amok?I P(run before amok) = P(A|B) = 10

�7

10

�6 = 0.1

I P(amok after run) = P(B|A) = 10

�7

10

�3 = 0.0001

Natural Language Processing 4(11)

Multiplication Rule for Joint Probability

I Given events A and B in ⌦, with P(B) > 0:

P(A,B) = P(B)P(A|B)

I Since A \ B = B \ A, we also have:

P(A,B) = P(A)P(B |A)

I The multiplication rule is also known as the chain rule

Natural Language Processing 5(11)

Quiz 1

I The probability of winning the Nobel Prize if you have a PhDin Physics is 1 in a million [P(A|B) = 0.000001]

I Only 1 in 10,000 people have a PhD in Physics [P(B) = 0.0001]

I What is the probability of a person both having a PhD inPhysics and winning the Nobel Prize? [P(A,B) = ?]

1. Smaller than 1 in a million

2. Greater than 1 in a million

3. Impossible to tell

Natural Language Processing 6(11)

Marginal Probability

I Marginalization, or the law of total probabilityI If events B1, . . . ,Bk constitute a partition of the sample space

⌦ (and P(Bi ) > 0 for all i), then for any event A in ⌦:

P(A) =kX

i=1

P(A,Bi ) =kX

i=1

P(A|Bi )P(Bi )

I Partition = pairwise disjoint and B1 [ · · · [ Bk = ⌦

Natural Language Processing 7(11)

Joint, Marginal and Conditional

I Joint probabilities for rain and wind:

no wind some wind strong wind storm

no rain 0.1 0.2 0.05 0.01

light rain 0.05 0.1 0.15 0.04

heavy rain 0.05 0.1 0.1 0.05

I Marginalize to get simple probabilities:I P(no wind) = 0.1 + 0.05 + 0.05 = 0.2I P(light rain) = 0.05 + 0.1 + 0.15 + 0.04 = 0.34

I Combine to get conditional probabilities:I P(no wind|light rain) = 0.05

0.34 = 0.147

I P(light rain|no wind) = 0.050.2 = 0.25

Natural Language Processing 8(11)

Joint, Marginal and Conditional

I Joint probabilities for rain and wind:

no wind some wind strong wind storm

no rain 0.1 0.2 0.05 0.01

light rain 0.05 0.1 0.15 0.04

heavy rain 0.05 0.1 0.1 0.05

I Marginalize to get simple probabilities:I P(no wind) = 0.1 + 0.05 + 0.05 = 0.2I P(light rain) = 0.05 + 0.1 + 0.15 + 0.04 = 0.34

I Combine to get conditional probabilities:I P(no wind|light rain) = 0.05

0.34 = 0.147

I P(light rain|no wind) = 0.050.2 = 0.25

Natural Language Processing 8(11)

Joint, Marginal and Conditional

I Joint probabilities for rain and wind:

no wind some wind strong wind storm

no rain 0.1 0.2 0.05 0.01

light rain 0.05 0.1 0.15 0.04

heavy rain 0.05 0.1 0.1 0.05

I Marginalize to get simple probabilities:I P(no wind) = 0.1 + 0.05 + 0.05 = 0.2I P(light rain) = 0.05 + 0.1 + 0.15 + 0.04 = 0.34

I Combine to get conditional probabilities:I P(no wind|light rain) = 0.05

0.34 = 0.147

I P(light rain|no wind) = 0.050.2 = 0.25

Natural Language Processing 8(11)

Bayes Law

I Given events A and B in sample space ⌦:

P(A|B) = P(A)P(B |A)P(B)

IFollows from definition using chain rule

IAllows us to “invert” conditional probabilities

I Denominator can be computed using marginalization:

P(B) =kX

i=1

P(B ,Ai ) =kX

i=1

P(B |Ai )P(Ai )

ISpecial case of partition: P(A), P(A)

Natural Language Processing 9(11)

Independence

I Two events A and B are independent if and only if:

P(A,B) = P(A)P(B)

I Equivalently:P(A) = P(A|B)P(B) = P(B |A)

I Example:I P(run

1

) = P(A) = 10

�3

I P(amok

2

) = P(B) = 10

�6

I P(run

1

, amok

2

) = (A,B) = 10

�7

I A and B are not independent

Natural Language Processing 10(11)

Independence

I Two events A and B are independent if and only if:

P(A,B) = P(A)P(B)

I Equivalently:P(A) = P(A|B)P(B) = P(B |A)

I Example:I P(run

1

) = P(A) = 10

�3

I P(amok

2

) = P(B) = 10

�6

I P(run

1

, amok

2

) = (A,B) = 10

�7

I A and B are not independent

Natural Language Processing 10(11)

Quiz 2

I Research has shown that people with disease D exhibitsymptom S with 0.9 probability

I A doctor finds that a patient has symptom SI What can we conclude about the probability that the patient

has disease D1. The probability is 0.1

2. The probability is 0.9

3. Nothing

Natural Language Processing 11(11)

Natural Language Processing

Statistical Inference

Joakim Nivre

Uppsala UniversityDepartment of Linguistics and Philology

joakim.nivre@lingfil.uu.se

Natural Language Processing 1(12)

Statistical Inference

IInference from a finite set of observations (a sample) to a

larger set of unobserved instances (a population or model)

ITwo main kinds of statistical inference:

1. Estimation2. Hypothesis testing

IIn natural language processing:

I Estimation – learn model parameters (probability distributions)I Hypothesis tests – assess statistical significance of test results

Natural Language Processing 2(12)

Random Variables

IA random variable is a function X that partitions the sample

space ⌦ by mapping outcomes to a value space ⌦X

IThe probability function can be extended to variables:

P(X = x) = P({! 2 ⌦ |X (!) = x})

IExamples:

1. The part-of-speech of a word X : ⌦ ! {noun, verb, adj, . . .}2. The number of words in a sentence Y : ⌦ ! {1, 2, 3, . . .}.

IWhen we are not interested in particular values, we write P(X )

Natural Language Processing 3(12)

Expectation

IThe expectation E [X ] of a (discrete) numerical variable X is:

E [X ] =X

x2⌦X

x · P(X = x)

IExample: The expectation of the sum Y of two dice:

E [Y ] =12X

y=2

y · P(Y = y) =252

36

= 7

Natural Language Processing 4(12)

Entropy

IThe entropy H[X ] of a discrete random variable X is:

H[X ] = E [� log2 P(X )] = �X

x2⌦X

P(X = x) log2 P(X = x)

IEntropy can be seen as the expected amount of information

(in bits), or as the difficulty of predicting the variable

I Sum of two dice: �P

12

y=2

P(Y = y) log

2

P(Y = y) ⇡ 3.27

I 11-sided die (2–12): �P

12

z=2

1

11

log2

1

11

⇡ 3.46

Natural Language Processing 5(12)

Quiz 1

ILet X be a random variable that map (English) words to the

number of characters they concern

I For example, X (run) = 3 and X (amok) = 4I

Which of the following statements do you think are true:

1. P(X = 0) = 02. P(X = 5) < P(X = 50)3. E [X ] < 50

Natural Language Processing 6(12)

Statistical Samples

IA random sample of a variable X is a vector (X1, . . . ,X

N

) of

independent variables Xi

with the same distribution as XI It is said to be i.i.d. = independent and identically distributedI In practice, it is often hard to guarantee thisI Observations may not be independent (not i.)I Distribution may be biased (not i.d.)

IWhat is the intended population?

I A Harry Potter novel is a good sample of J.K. Rowling, orfantasy fiction, but not of scientific prose

I This is relevant for domain adaptation in NLP

Natural Language Processing 7(12)

Estimation

IGiven a random sample of X , we can define sample variables,

such as the sample mean:

X =1

N

NX

i=1

Xi

ISample variables can be used to estimate model parameters

(population variables)

1. Point estimation: use variable X to estimate parameter �2. Interval estimation: use variables X

min

and Xmax

to constructan interval such that P(X

min

< � < Xmax

) = p, where p is theconfidence level adopted

Natural Language Processing 8(12)

Maximum Likelihood Estimation (MLE)

ILikelihood of parameters ✓ given sample x1, . . . , x

N

:

L(✓|x1, . . . , xN

) = P(x1, . . . , xN

|✓) =NY

i=1

P(xi

|✓)

IMaximum likelihood estimation – choose ✓ to maximize L:

max

✓L(✓|x1, . . . , x

N

)

IBasic idea:

I A good sample should have a high probability of occurringI Thus, choose the estimate that maximizes sample probability

Natural Language Processing 9(12)

Examples

ISample mean is an MLE of expectation:

E [X ] = X

I For example, estimate expected sentence length in a certaintype of text by mean sentence length in a representative sample

IRelative frequency is an MLE of probability:

P(X = x) =f (x)

NI For example, estimate the probability of a word being a noun

by the relative frequency of nouns in a suitable corpus

Natural Language Processing 10(12)

MLE for Different Distributions

IJoint distribution of X and Y :

PMLE

(X = x ,Y = y) =f (x , y)

N

IMarginal distribution of X :

PMLE

(X = x) =X

y2⌦Y

PMLE

(X = x ,Y = y)

IConditional distribution of X given Y :

PMLE

(X = x |Y = y) = P

MLE

(X=x ,Y=y)

P

MLE

(Y=y)

= P

MLE

(X=x ,Y=y)Px2⌦

X

P

MLE

(X=x ,Y=y)

Natural Language Processing 11(12)

Quiz 2

IConsider the following sample of English words:

{once, upon, a, time, there,was, a, frog}

IWhat is the MLE of word length (number of characters) based

on this sample?

1. 42. 83. 3.25

Natural Language Processing 12(12)