Consistency of Learning Processes

Classification and Linear ClassifiersMohammad Ali KeyvanradMachine LearningFall 1392In the Name of GodThanks to: M. Soleymani (Sharif University of Technology)R. Zemel (University of Toronto)p. Smyth (University of California, Irvine)OutlineClassificationLinear classifiersPerceptronMulti-class classificationGenerative approachNave Bayes classifier

2Classification: Oranges and Lemons3

Classification: Oranges and Lemons4

Classification problem5Linear classifiers6

Linear classifiers7

7Decision boundary8Linear Decision boundary (Perceptron)9

Linear Decision boundary (Decision Tree)10t1t3t2IncomeLinear Decision boundary (K Nearest Neighbor)11OOOxxxFeature 1Feature 2Non-Linear Decision boundary12

Decision BoundaryDecision Region 1Decision Region 2Decision boundaryLinear classifier13

Non-linear decision boundaryChoose non-linear featuresClassifier still linear in parameters 14

Linear boundary: geometry15

SSE cost function for classification SSE cost function is not suitable for classificationSum of Squared Errors loss penalizes too correct predictionsSSE also lack robustness to noise16

SSE cost function for classification 17

Perceptron algorithm18

Perceptron criterion19

Batch gradient for descent PerceptronGradient Descent to solve the optimization problem

Batch Perceptron converges in finite number of steps for linearly separable data

20

Stochastic gradient descent for Perceptron21

Convergence of Perceptron22

Convergence of Perceptron23

Multi-class classification24Multi-class classificationOne-vs-all (one-vs-rest)25

Multi-class classificationOne-vs-one26

Multi-class classification: ambiguityConverting the multi-class problem to a set of two-class problems can lead to regions in which the classification is undefined27

Probabilistic approachBayes theorem

28

Bayes theorem29

Bayes decision theory30

Probabilistic classifiersProbabilistic classification approaches can be divided in two main categoriesGenerativeDiscriminative31Discriminative vs. generative approach32

Generative approach33Discriminative approach34Nave Bayes classifier35Nave Bayes classifier36

Nave Bayes: discrete example37

38