Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11....
Transcript of Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11....
![Page 1: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/1.jpg)
Support VectorMachines
Optimization objective
Machine Learning
![Page 2: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/2.jpg)
Andrew Ng
If , we want ,
Alternative view of logistic regression
If , we want ,
![Page 3: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/3.jpg)
Andrew Ng
Alternative view of logistic regression
Cost of example:
If (want ): If (want ):
![Page 4: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/4.jpg)
Andrew Ng
Support vector machineLogistic regression:
Support vector machine:
![Page 5: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/5.jpg)
Andrew Ng
SVM hypothesis
Hypothesis:
![Page 6: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/6.jpg)
Large MarginIntuition
Machine Learning
Support VectorMachines
![Page 7: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/7.jpg)
Andrew Ng
If , we want (not just )If , we want (not just )
Support Vector Machine
-1 1 -1 1
![Page 8: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/8.jpg)
Andrew Ng
SVM Decision Boundary
Whenever :
Whenever :
-1 1
-1 1
![Page 9: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/9.jpg)
Andrew Ng
SVM Decision Boundary: Linearly separable case
x1
x2
Large margin classifier
![Page 10: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/10.jpg)
Andrew Ng
Large margin classifier in presence of outliers
x1
x2
![Page 11: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/11.jpg)
The mathematics behind large margin classification (optional)
Machine Learning
Support VectorMachines
![Page 12: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/12.jpg)
Andrew Ng
Vector Inner Product
![Page 13: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/13.jpg)
Andrew Ng
SVM Decision Boundary
![Page 14: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/14.jpg)
Andrew Ng
SVM Decision Boundary
![Page 15: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/15.jpg)
커널(Kernels) I
Machine Learning
Support VectorMachines
![Page 16: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/16.jpg)
Andrew Ng
비선형결정경계
x1
x2
특징 의또다른/ 더나은선택이있는가?
![Page 17: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/17.jpg)
Andrew Ng
Kernel
x1
x2
주어진 에대하여, 랜드마크 의근접도에의존하는새로운특징을계산한다.
![Page 18: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/18.jpg)
Andrew Ng
커널과유사도(Similarity)
![Page 19: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/19.jpg)
Andrew Ng
Example:
![Page 20: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/20.jpg)
Andrew Ng
x1
x2
![Page 21: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/21.jpg)
Support VectorMachines
커널(Kernels) II
Machine Learning
![Page 22: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/22.jpg)
Andrew Ng
x1
x2
Choosing the landmarks
Given :
Predict ifWhere to get ?
![Page 23: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/23.jpg)
Andrew Ng
SVM with KernelsGivenchoose
Given example :
For training example :
![Page 24: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/24.jpg)
Andrew Ng
SVM with Kernels
Hypothesis: Given , compute featuresPredict “y=1” if
Training:
![Page 25: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/25.jpg)
Andrew Ng
Small : 특징 가좀덜부드럽게변화한다.
Lower bias, higher variance.
SVM parameters:
C ( ). Large C: Lower bias, high variance.Small C: Higher bias, low variance.
Large : 특징 가좀더부드럽게변화한다.
Higher bias, lower variance.
![Page 26: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/26.jpg)
Support Vector Machines
Using an SVM
Machine Learning
![Page 27: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/27.jpg)
Andrew Ng
Use SVM software package (e.g. liblinear, libsvm, …) to solve for parameters .
Need to specify:Choice of parameter C.Choice of kernel (similarity function):
E.g. No kernel (“linear kernel”)Predict “y = 1” if
Gaussian kernel:
, where . Need to choose .
![Page 28: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/28.jpg)
Andrew Ng
Kernel (similarity) functions:
function f = kernel(x1,x2)
return
주의 : 가우시안커널을사용하기전에특징스케일링을수행한다.
x1 x2
![Page 29: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/29.jpg)
Andrew Ng
Other choices of kernel
Note: 모든유사도함수 가 유효한커널은아니다.
( SVM 패키지의최적화가정확히실행되고발산하지않으려면, “Mercer’s Theorem”라는기술적조건을만족할필요가있다).
많은이용가능한커널들:
- Polynomial kernel:
- 좀더난해한: String kernel, chi-square kernel, histogram intersection kernel, …
![Page 30: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/30.jpg)
Andrew Ng
Multi-class classification
많은 SVM 패키지들이이미내장된다중-클래스분류기능을갖는다.
그렇지않으면, 일-대-모두방법을사용한다
(Train SVMs, one to distinguish from the rest, for ), get
Pick class with largest
![Page 31: Support Vector Machines Optimizationjun.hansung.ac.kr/ML/docs-slides-Lecture12-kr.pdf · 2016. 11. 17. · Support Vector Machine-1 1 -1 1. Andrew Ng SVM Decision Boundary Whenever](https://reader036.fdocuments.in/reader036/viewer/2022063022/5fea330fab0ba47dbb4a3ec6/html5/thumbnails/31.jpg)
Andrew Ng
Logistic regression vs. SVMs
number of features ( ), number of training examplesIf is large (relative to ):Use logistic regression, or SVM without a kernel (“linear kernel”)
If is small, is intermediate:Use SVM with Gaussian kernel
If is small, is large:Create/add more features, then use logistic regression or SVM without a kernel
Neural network likely to work well for most of these settings, but may be slower to train.