An Introduction to Statistical and Probabilistic …...An Introduction to Statistical and...

An Introduction to Statistical andProbabilistic Linear Models

Maximilian Mozes

Proseminar Data MiningFakultat fur Informatik

Technische Universitat Munchen

June 07, 2017

Introduction

In statistical learning theory, linear models areused for regression and classification tasks.

Introduction

I What is regression?

Introduction

I What is classification?

Introduction

I What is classification?

I How can we model such concepts in amathematical context?

Linear regression

Linear regression - basics

What is regression?

I Approximation of data using a (closed)mathematical expression

What is regression?

I Approximation of data using a (closed)mathematical expression

I Achieved by estimating the model parametersthat maximize the approximation

Linear regression - example

I A company changes the price of their products for thenth time.

I They know how the price changes affected theconsumer behavior n− 1 times before.

I Using linear regression, they can predict the consumerbehavior for the nth price change.

I Let D denote a set of n-dimensional datavectors

I Let x be an n-dimensional observation

I How can we approximate x?

Create a linear function

y(x,w) = w0 + w1x1 + · · ·+ wnxn

that approximates x with w.

Example for n = 2.

y(x,w) = w0 + w1x1 + w2x2

ProblemWeight parameters wi are simply values.

=⇒ significant limitation!

IdeaUse weighted non-linear functions φj instead!

y(x,w) = w0 +n∑

wjφj(x) = w>φ(x),

where φ = (φ0, ..., φn)>.

Polynomial regression

ExampleLet φi(x) = xi.

y(x,w) = w0 +n∑

= w0 + w1x+ w2x2 + · · ·+ wnx

Approximation with a 2nd-order polynomial.

Approximation with a 6th-order polynomial.

Approximation with an 8th-order polynomial.

Problem

=⇒ Overfitting with higher polynomial degree

Linear classification

Linear classification - basics

What is classification?

I Aims to partition the data into predefinedclasses

I A class contains observations with similarcharacteristics

Linear classification - example

I We have n different cucumbers andcourgettes.

I Each record contains the weight and thetexture (smooth, rough).

I We want to predict the correct label withoutknowing the real label.

I Assume a two-class classification problem

I How can we categorise the data intopredefined classes?

Cucumbers vs. courgettes

Discriminant functions

I Given a dataset D,

I we aim to categorise x into either class C1 orC2.

Use y(x) such that

x ∈ C1 ⇐⇒ y(x) ≥ 0 and x ∈ C2 otherwise

Use y(x) such that

x ∈ C1 ⇐⇒ y(x) ≥ 0 and x ∈ C2 otherwise

withy(x) = w>x + w0.

Decision boundary H:

H := {x ∈ D : y(x) = 0}

Decision boundary H:

H := {x ∈ D : y(x) = 0}

Non-linearity

However, it often occurs that the data are notlinearly-separable:

Non-linearity

SolutionUse non-linear functions instead!

Non-linearity

SolutionUse non-linear functions instead!

=⇒ y(x) = w>φ(x)

φ(x) = (φ0, φ1, ..., φM−1)>

Common classification algorithms

Other commonly-used classification algorithms:

I Naive Bayes classifier

I Logistic regression (Cox, 1958)

I Support vector machines (Vapnik and Lerner,1963)

Summary

I Linear modelsLinear combinations of weighted (non-)linearfunctions

Summary

I RegressionApproximation of a given amount of datausing a closed mathematical representation

Summary

I RegressionApproximation of a given amount of datausing a closed mathematical representation

I ClassificationCategorisation of data according to individualcharacteristics and common patterns

Further readings

Further readings:I J. Aldrich, 1997. R.A. Fisher and the Making of Maximum Likelihood

1912 - 1922. Statistical Science Vol. 12, No. 3, 162-176.

I D. Barber, Bayesian Reasoning and Machine Learning. CambridgeUniversity Press. 2012.

I C. M. Bishop, Pattern Recognition and Machine Learning. Springer.2006.

I T. Hastie, R. Tibshirani and J. Friedman, The Elements of StatisticalLearning - Data Mining, Inference, and Prediction. Second edition.Springer. 2009.

I K. Murphy, Machine Learning - A Probabilistic Perspective. The MITPress, Cambridge, Massachusetts, London, England. 2012.

I A. Y. Ng and M. I. Jordan, 2002. On Discriminative vs. Generativeclassifiers: A comparison of logistic regression and naive bayes.Advances in Neural Information Processing Systems.

References

References:

I J. Aldrich, 1997. R.A. Fisher and the Making of Maximum Likelihood1912 - 1922. Statistical Science Vol. 12, No. 3, 162-176.

I D. Barber, Bayesian Reasoning and Machine Learning. CambridgeUniversity Press. 2012.

I C. M. Bishop, Pattern Recognition and Machine Learning. Springer.2006.

I D. R. Cox, 1958. The regression analysis of binary sequences. Journalof the Royal Statistical Society Vol. XX, No. 2.

References

References:

I T. Hastie, R. Tibshirani and J. Friedman, The Elements of StatisticalLearning - Data Mining, Inference, and Prediction. Second edition.Springer. 2009.

I K. Murphy, Machine Learning - A Probabilistic Perspective. The MITPress, Cambridge, Massachusetts, London, England. 2012.

I F. Rosenblatt, 1958. The Perceptron: A probabilistic model forinformation storage and organization in the brain. PsychologicalReview Vol. 65, No. 6.

Thank you for yourattention.

Questions?

Backup slides

Sum of least squares

Parameter estimation

Estimating the weight parametersHow to choose the wi?

Parameter estimation

Estimating the weight parametersHow to choose the wi?

=⇒ Find the set of parameters that maximize

p(D |w)

I Method to optimize the weights w

I Aims to minimize the residual sum of squares(RSS) by optimizing weights

I Minimize

RSS(w) =N∑i=1

(zi − y(xi,w))2

=N∑i=1

zi − w0 −M−1∑j=1

I RSS can be simplified by using an N ×Mmatrix X with the xi as rows

I Then

RSS(w) = (z−Xw)>(z−Xw),

where z is a vector of target values

I Building the derivatives leads to

∂RSS

∂w= −2X>(z−Xw)

∂2RSS

∂w∂w>= 2X>X

I Building the derivatives leads to

∂RSS

∂w= −2X>(z−Xw)

∂2RSS

∂w∂w>= 2X>X

I Setting the first derivative to zero and solvingfor w results in

w = (X>X)−1X>z

I Note: the approach assumes X>X to bepositive definite

I Therefore, X is assumed to have full columnrank

I Approximation function can be described as

zi = y(xi,w) + ε,

where ε represents the data noise

I Approximation function can be described as

zi = y(xi,w) + ε,

where ε represents the data noise

I RSS provides a measurement for theprediction error ED defined as

ED(w) =1

N∑i=1

(zi −w>φ(xi))2,

where φ = (φ0, ..., φM−1)>

Maximum LikelihoodEstimation

Maximum Likelihood Estimation

Maximum Likelihood Estimation (MLE)

I Introduced by Fisher (1922)

I Commonly-used method to optimize themodel parameters

GoalFind the optimal parameters w such thatp(D |w) is maximized, i.e.

w , arg maxw

p(D |w).

GoalFind the optimal parameters w such thatp(D |w) is maximized, i.e.

w , arg maxw

p(D |w).

For target values zi, p(zi |xi,w) are assumed tobe independent such that

p(D |w) =N∏i=1

p(zi |xi,w)

holds for all zi ∈ D.38

The product makes the equation unwieldy. Let’ssimplify it using the log!

log p(D |w) =N∑i=1

log p(zi |xi,w)

log p(D |w) =N∑i=1

log p(zi |xi,w)

We can now compute ∇ log p(D |w) = 0 andsolve for w.

=⇒ w , w are the optimal weight parameters!

I Assumption: data noise ε follows Gaussiandistribution

I Then p(D |w) can be described as

p(z |x,w, β) = N (z | y(x,w), β−1),

where β is the model’s inverse variance

I For z we get

log p(z |X,w, β) =N∑i=1

logN (zi |w>φ(xi), β−1)

I It follows that

∇ log p(z |X,w, β) = β ·N∑i=1

(zi −w>φ(xi)) · φ(xi)>

I It follows that

∇ log p(z |X,w, β) = β ·N∑i=1

(zi −w>φ(xi)) · φ(xi)>

I Setting the gradient to zero and solving for wleads to

wML = (Φ>Φ)−1Φ>z

I Φ is called design matrix and is described as

φ0(x1) φ1(x1) . . . φM−1(x1)φ0(x2) φ1(x2) . . . φM−1(x2)

...... . . . ...

φ0(xN) φ1(xN) . . . φM−1(xN)

Regularization and ridgeregression

Ridge regression

I Method to prevent overfitting

Ridge regression

I Method to prevent overfitting

I Add regularization term compensating thebiased prediction

Ridge regression

I Regularization term λ||w||22

Ridge regression

I Regularization term λ||w||22I Error function E(w) becomes

E(w) =1

N∑i=1

(zi −w>φ(xi))2 + λ||w||22

Ridge regression

I Regularization term λ||w||22I Error function E(w) becomes

E(w) =1

N∑i=1

(zi −w>φ(xi))2 + λ||w||22

I Minimizing E(w) and solving for w results in

wridge = (λI + Φ>Φ)−1Φ>z

Non-linear classification

Non-linearity

In our example

Non-linearity

In our example

the appropriate decision boundary is

φ : (x1, x2)> 7→ (r · cos(x1), r · sin(x2))

>, r ∈ R.48

Multi-class classification

I The discriminant function for two-classclassification can be extended to a k-classproblem

I Use k-class discriminator

yk(x) = w>k x + wk0

I Use k-class discriminator

yk(x) = w>k x + wk0

I x is assigned to class Cj if yj(x) > yi(x) for alli 6= j

I The decision boundary is then

yj(x) = yi(x)

which can be transformed to

(wj −wi)>x + wj0 − wi0 = 0

Perceptron algorithm

Perceptron

Given an input vector x and a fixed non-linearfunction φ(x), the class of x is estimated by

y(x) = f(w>φ(x)),

where f(t) is called non-linear activationfunction.

Perceptron

Given an input vector x and a fixed non-linearfunction φ(x), the class of x is estimated by

y(x) = f(w>φ(x)),

where f(t) is called non-linear activationfunction.

f(t) =

{+1 if t ≥ 0,

−1 otherwise.

Probabilistic generativemodels

Probabilistic generative models

I Compute probability p(x, z) directly insteadof optimizing the weight parameters

I Apply Bayes’ theorem on p(z |x)

Bayes’ theorem

For a set of disjunct samples A1, ..., An and agiven sample B the probabilityp(Ai|B), i ∈ {1, ..., n} can be computed as follows:

p(Ai|B) =p(B|Ai) · p(Ai)∑nj=1 p(B|Aj) · p(Aj)

I Consider a two-class classification problem forC1 and C2

I Then

p(C1 |x) =p(x | C1)p(C1)

p(x | C1)p(C1) + p(x | C2)p(C2)

1 + e−a

= σ(a),

a = log

(p(x | C1)p(C1)p(x | C2)p(C2)

)and σ(a) =

1 + e−a

Probabilistic discriminativemodels

Probabilistic discriminative models

I Predict the correct class by directlycomputing the posterior probability

p(z |x)

I Predict the correct class by directlycomputing the posterior probability

p(z |x)

I This makes the computation of p(x | z)(Bayes’ theorem) redundant

I Computing the posterior probability

p(Ck |x, θoptCk|x)

is then achieved by using MLE

I Computing the posterior probability

p(Ck |x, θoptCk|x)

is then achieved by using MLE

I Disadvantage: only little knowledge about thegiven data is required (”black-box”)

Logistic regression

I Commonly-used classification algorithm forbinary classification problems

Logistic regression

I Commonly-used classification algorithm forbinary classification problems

I Assumption: data noise ε follows Bernoullidistribution

Ber(n) = pn(1− p)1−n, n ∈ {0, 1}

as this distribution is more appropriate for abinary classification problem

Logistic regression

I Predict the correct class label on theprobability

p(Ck |x,w) = Ber(Ck |σ(w>x)),

where σ is a squashing function (e.g. sigmoid)

An Introduction to Statistical and Probabilistic …...An Introduction to Statistical and...

Documents

Transcript of An Introduction to Statistical and Probabilistic …...An Introduction to Statistical and...

1 Statistical Modeling and Probabilistic Analysis of Cellular … · 2014. 12. 8. · arXiv:1412.2087v1 [cs.IT] 5 Dec 2014 1 Statistical Modeling and Probabilistic Analysis of Cellular

Probabilistic programming in Python using PyMC3...Keywords Bayesian statistic, Probabilistic Programming, Python, Markov chain Monte Carlo, Statistical modeling INTRODUCTION Probabilistic

Topics in Probabilistic and Statistical Databases Lecture 10 ...

Statistical methods for verification of probabilistic ...

Statistical Model Checking for Distributed Probabilistic-Control Hybrid Automata …jmartins/papers/icfem11.pdf · 2013-10-24 · Statistical Model Checking for Distributed Probabilistic-Control

A Statistical-Probabilistic Pattern for Determination of ... · A Statistical-Probabilistic Pattern for Determination of Tunnel Advance Step by Quantitative Risk Analysis Sasan Ghorbani

Tutorial for Metrologists on the probabilistic and statistical apparatus underlying ... · · 2017-04-28on the probabilistic and statistical apparatus ... [Glymour, 1980] does not

Probabilistic Foundations of Statistical Network Analysis ... · Probabilistic Foundations of Statistical Network Analysis Chapter 5: Statistical modeling paradigm Harry Crane Based

Process performance modelling statistical probabilistic

CO902 Probabilistic and statistical inference · 2013. 1. 24. · CO902 Probabilistic and statistical inference Lecture 3 Tom Nichols Department of Statistics & Warwick Manufacturing

Probabilistic Latent Factor Induction and Statistical Factor Analysis

Introduction to statistical and probabilistic method · Munich Personal RePEc Archive Introduction to statistical and probabilistic method Keita, Moussa February 2016 Online at

EFFECT OF DIFFERENT STATISTICAL MODELS IN PROBABILISTIC ...

Use of Probabilistic Statistical Techniques in AERMOD Modeling Evaluations

Probabilistic Latent Variable Models in Statistical Genomicsetheses.whiterose.ac.uk/8326/1/final_version_thesis.pdfProbabilistic Latent Variable Models in Statistical Genomics Nicol

Automatic and Probabilistic Foehn Diagnosis with a Statistical ...

Tutorial for Metrologists on the probabilistic and statistical apparatus

PROBABILISTIC PROPERTIES AND STATISTICAL INFERENCE …studentsrepo.um.edu.my/5865/1/Liew_Kian_Wah.pdf · PROBABILISTIC PROPERTIES AND STATISTICAL INFERENCE FOR A FAMILY OF GENERALISED

Probabilistic and Statistical Techniques 1 Lecture 9 Dr. Nader Okasha.

Statistical and probabilistic analyses of impact pressure ...swrh.whu.edu.cn/dianqing/images/pdf-files/2015-Statistical_and... · Statistical and probabilistic analyses of impact