PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf ·...

51
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 1: INTRODUCTION

Transcript of PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf ·...

Page 1: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

PATTERN  RECOGNITION    AND  MACHINE  LEARNING  CHAPTER  1:  INTRODUCTION  

Page 2: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Polynomial  Curve  Fi0ng    

Page 3: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Sum-­‐of-­‐Squares  Error  Func9on  

Page 4: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

0th  Order  Polynomial  

Page 5: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

1st  Order  Polynomial  

Page 6: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

3rd  Order  Polynomial  

Page 7: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

9th  Order  Polynomial  

Page 8: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Over-­‐fi0ng  

Root-­‐Mean-­‐Square  (RMS)  Error:  

Page 9: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Polynomial  Coefficients      

Page 10: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Data  Set  Size:    

9th  Order  Polynomial  

Page 11: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Data  Set  Size:    

9th  Order  Polynomial  

Page 12: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Regulariza9on  

Penalize  large  coefficient  values  

Page 13: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Regulariza9on:    

Page 14: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Regulariza9on:    

Page 15: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Regulariza9on:                      vs.    

Page 16: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Polynomial  Coefficients      

Page 17: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

The  Rules  of  Probability  

 Sum  Rule  

 Product  Rule  

Page 18: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Bayes’  Theorem  

Posterior            likelihood  ×  prior  /

Page 19: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Probability  Densi9es  

Page 20: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Transformed  Densi9es  

Page 21: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Expecta9ons  

Condi9onal  Expecta9on  (discrete)  

Approximate  Expecta9on  (discrete  and  con9nuous)  

Page 22: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Variances  and  Covariances  

Page 23: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

The  Gaussian  Distribu9on  

Page 24: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Gaussian  Mean  and  Variance  

Page 25: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

The  Mul9variate  Gaussian  

Page 26: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Gaussian  Parameter  Es9ma9on  

Likelihood  func9on  

Page 27: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Maximum  (Log)  Likelihood  

Page 28: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Curve  Fi0ng  Re-­‐visited  

Page 29: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Maximum  Likelihood  

Determine                        by  minimizing  sum-­‐of-­‐squares  error,                          .  

Page 30: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Predic9ve  Distribu9on  

Page 31: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

MAP:  A  Step  towards  Bayes  

Determine                              by  minimizing  regularized  sum-­‐of-­‐squares  error,                          .  

Page 32: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Bayesian  Curve  Fi0ng  

Page 33: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Bayesian  Predic9ve  Distribu9on  

Page 34: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Curse  of  Dimensionality  

Page 35: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Curse  of  Dimensionality  

Polynomial  curve  fi0ng,  M = 3          Gaussian  Densi9es  in    higher  dimensions

Page 36: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Decision  Theory  

Inference  step    Determine  either                        or                      .  

 Decision  step    For  given  x,  determine  op9mal  t.  

Page 37: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Minimum  Misclassifica9on  Rate  

Page 38: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Minimum  Expected  Loss  

Example:  classify  medical  images  as  ‘cancer’  or  ‘normal’    

Decision  Truth  

Page 39: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Minimum  Expected  Loss  

Regions              are  chosen  to  minimize  

Page 40: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Reject  Op9on  

Page 41: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Why  Separate  Inference  and  Decision?  

•  Minimizing  risk  (loss  matrix  may  change  over  9me)  •  Reject  op9on  •  Unbalanced  class  priors  •  Combining  models  

Page 42: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Decision  Theory  for  Regression  

Inference  step    Determine                        .  

 Decision  step    For  given  x,  make  op9mal    predic9on,  y(x),  for  t.  

 Loss  func9on:  

Page 43: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

The  Squared  Loss  Func9on  

Page 44: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Genera9ve  vs  Discrimina9ve  

Genera9ve  approach:      Model    Use  Bayes’  theorem  

 Discrimina9ve  approach:      Model                      directly  

Page 45: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Entropy  

Important  quan9ty  in  •   coding  theory  •   sta9s9cal  physics  •   machine  learning  

Page 46: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Entropy  

Coding  theory:  x  discrete  with  8  possible  states;  how  many  bits  to  transmit  the  state  of  x?  

 All  states  equally  likely  

Page 47: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Entropy  

Page 48: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Differen9al  Entropy  

Put  bins  of  width  ¢ along  the  real  line        Differen9al  entropy  maximized  (for  fixed          )  when    in  which  case  

Page 49: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Condi9onal  Entropy  

Page 50: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

The  Kullback-­‐Leibler  Divergence  

Page 51: PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch1.pdf · DecisionTheory) Inference)step))Determine)either)))))or))))).)) Decision)step))Forgiven x,)determine)op9mal)t.

Mutual  Informa9on  


PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch3.pdf · Linear’Basis’Func-on’Models’(2)’ Generally’ ’ ’ Where ’are’known’as’basis%funcons.

PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’shirani/vision/prml_ch3.pdf · Linear’Basis’Func-on’Models’(2)’ Generally’ ’ ’ Where ’are’known’as’basis%funcons.

Machine Learning – some basics - Bishop PRML Ch. 1vda.univie.ac.at/Teaching/FDA/17w/LectureNotes/01_basics.pdf · Admin MachineLearning CurveFitting DecisionTheory ProbabilityTheory

Machine Learning – some basics - Bishop PRML Ch. 1vda.univie.ac.at/Teaching/FDA/17w/LectureNotes/01_basics.pdf · Admin MachineLearning CurveFitting DecisionTheory ProbabilityTheory

Rational Decisions - Making rational decision under ...profs.sci.univr.it/~farinelli/courses/ia/slides/decisionTheory-hnd.pdf · Rational Decisions Utilities Utilities map states

Rational Decisions - Making rational decision under ...profs.sci.univr.it/~farinelli/courses/ia/slides/decisionTheory-hnd.pdf · Rational Decisions Utilities Utilities map states

PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’cbcb.umd.edu/~hcorrada/PML/pdf/lectures/prml-slides-8.pdf · prml-slides-8.ppt Author: Hector Corrada Bravo Created Date: 11/26/2012

PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’cbcb.umd.edu/~hcorrada/PML/pdf/lectures/prml-slides-8.pdf · prml-slides-8.ppt Author: Hector Corrada Bravo Created Date: 11/26/2012

Financial Economics - A Concise Introduction to Classical ... · T.Hens,M.Rieger (Zürich/Trier) FinancialEconomics July31,2010 10/57. DecisionTheory ExpectedUtilityTheory St.PetersburgLottery

Financial Economics - A Concise Introduction to Classical ... · T.Hens,M.Rieger (Zürich/Trier) FinancialEconomics July31,2010 10/57. DecisionTheory ExpectedUtilityTheory St.PetersburgLottery

PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’szhou/568/BayesianCurveFitting_1.pdfPolynomial(Curve(Fing ((• The input data set x of 10 points was generated by choosing values

PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’szhou/568/BayesianCurveFitting_1.pdfPolynomial(Curve(Fing ((• The input data set x of 10 points was generated by choosing values

I1,web.pdx.edu/~stipakb/download/PA557/DecisionTheory-Overview.pdf · I1,/k M +.rr i ~Gr,S~ Dry w r tt I DECISION THEORY: A FRAMEWORK FOR DECISION MAKING Decision theory provides

I1,web.pdx.edu/~stipakb/download/PA557/DecisionTheory-Overview.pdf · I1,/k M +.rr i ~Gr,S~ Dry w r tt I DECISION THEORY: A FRAMEWORK FOR DECISION MAKING Decision theory provides

PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’users.umiacs.umd.edu/~hcorrada/PML/pdf/lectures/prml-slides-8.pdf · The(SumPProductAlgorithm((8)(To(compute(local(marginals:(•

PATTERN’RECOGNITION’’ AND’MACHINE’LEARNING’users.umiacs.umd.edu/~hcorrada/PML/pdf/lectures/prml-slides-8.pdf · The(SumPProductAlgorithm((8)(To(compute(local(marginals:(•