19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 1/19

SVM Tutorial

Support Vector Regression with RIn this article I will show how to use R to perform a Support Vector Regression.

We will 䁀䁀rst do a simple linear regression, then move to the Support Vector Regression so that you

can see how the two behave with the same data.

A simple data set

To begin with we will use this simple data set:

I just put some data in excel. I prefer that over using an existing well-known data-set because the

purpose of the article is not about the data, but more about the models we will use.

As you can see there seems to be some kind of relation between our two variables X and Y, and it

look like we could 䁀䁀t a line which would pass near each point.

Let's do that in R !

Step 1: Simple linear regression in R

Here is the same data in CSV format, I saved it in a 䁀䁀le regression.csv :

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 2/19

We can now use R to display the data and 䁀䁀t a line:

The code above displays the following graph:

010203040506070809101112

# Load the data from the csv filedataDirectory <‐ "D:/" # put your own folder heredata <‐ read.csv(paste(dataDirectory, 'regression.csv', sep=""), header = TRUE) # Plot the dataplot(data, pch=16) # Create a linear regression modelmodel <‐ lm(Y ~ X, data) # Add the fitted lineabline(model)

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 3/19

Step 2: How good is our regression ?

In order to be able to compare the linear regression with the support vector regression we 䁀䁀rst need

a way to measure how good it is.

To do that we will change a little bit our code to visualize each prediction made by our model

This produces the following graph:

0102030405060708091011

dataDirectory <‐ "D:/"data <‐ read.csv(paste(dataDirectory, 'regression.csv', sep=""), header = TRUE) plot(data, pch=16)model <‐ lm(Y ~ X , data) # make a prediction for each XpredictedY <‐ predict(model, data) # display the predictionspoints(data$X, predictedY, col = "blue", pch=4)

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 4/19

For each data point the model makes a prediction displayed as a blue cross on the graph.

The only di၀倀erence with the previous graph is that the dots are not connected with each other.

In order to measure how good our model is we will compute how much errors it makes.

We can compare each value with the associated predicted value and see how far away they

are with a simple di၀倀erence.

Note that the expression is the error, if we make a perfect prediction will be equal to

and the error will be zero.

If we do this for each data point and sum the error we will have the sum of the errors, and if we

takes the mean we will get the Mean Squared Error (MSE)

A common way to measure error in machine learning is to use the Root Mean Squared Error

(RMSE) so we will use it instead.

To compute the RMSE we take the square root and we get the RMSE

Xi Y i

Yi Y i

−Y i Yi Y i Yi

MSE = ( −1n

∑i=1

n

Y i Yi)2

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 5/19

Using R we can come with the following code to compute the RMSE

We know now that the RMSE of our linear regression model is 5.70. Let's try to improve it with SVR !

Step 3: Support Vector Regression

In order to create a SVR model with R you will need the package e1071. So be sure to install it and to

add the library(e1071) line at the start of your 䁀䁀le.

Below is the code to make predictions with Support Vector Regression:

As you can see it looks a lot like the linear regression code. Note that we called the svm function (not

svr !) it's because this function can also be used to make classi䁀䁀cations with Support Vector

Machine. The function will automatically choose SVM if it detects that the data is categorical (if the

variable is a factor in R).

The code draws the following graph:

RMSE = MSE− −−−−

√

1234567

rmse <‐ function(error){  sqrt(mean(error^2))} error <‐ model$residuals  # same as data$Y ‐ predictedYpredictionRMSE <‐ rmse(error)   # 5.703778

12345

model <‐ svm(Y ~ X , data) predictedY <‐ predict(model, data) points(data$X, predictedY, col = "red", pch=4)

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 6/19

This time the predictions is closer to the real values ! Let's compute the RMSE of our support vector

regression model.

As expected the RMSE is better, it is now 3.15 compared to 5.70 before.

But can we do better ?

Step 4: Tuning your support vector regression model

In order to improve the performance of the support vector regression we will need to select the best

parameters for the model.

In our previous example, we performed an epsilon-regression, we did not set any value for epsilon ( ), but it took a default value of 0.1. There is also a cost parameter which we can change to avoid

over䁀䁀tting.

The process of choosing these parameters is called hyperparameter optimization, or model

1234

# /!\ this time  svrModel$residuals  is not the same as data$Y ‐ predictedY# so we compute the error like thiserror <‐ data$Y ‐ predictedYsvrPredictionRMSE <‐ rmse(error)  # 3.157061

ϵ

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 7/19

selection.

The standard way of doing it is by doing a grid search. It means we will train a lot of models for the

di၀倀erent couples of and cost, and choose the best one.

There is two important points in the code above:

we use the tune method to train models with and cost =

which means it will train 88 models (it can take a long time)

the tuneResult returns the MSE, don't forget to convert it to RMSE before comparing the value

to our previous model.

The last line plot the result of the grid search:

ϵ

12345678

# perform a grid searchtuneResult <‐ tune(svm, Y ~ X,  data = data,              ranges = list(epsilon = seq(0,1,0.1), cost = 2^(2:9)))print(tuneResult)# best performance: MSE = 8.371412, RMSE = 2.89 epsilon 1e‐04 cost 4# Draw the tuning graphplot(tuneResult)

ϵ = 0,0.1,0.2, . . . , 1, , , . . . ,22 23 24 29

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 8/19

On this graph we can see that the darker the region is the better our model is (because the RMSE

is closer to zero in darker regions).

This means we can try another grid search in a narrower range we will try with values between 0

and 0.2. It does not look like the cost value is having an e၀倀ect for the moment so we will keep it as it

is to see if it changes.

We trained di၀倀erent 168 models with this small piece of code.

As we zoomed-in inside the dark region we can see that there is several darker patch.

From the graph you can see that models with C between 200 and 300 and between 0.8 and 0.9

have less error.

Hopefully for us, we don't have to select the best model with our eyes and R allows us to get it very

easily and use it to make predictions.

ϵ

123456

tuneResult <‐ tune(svm, Y ~ X,  data = data,                   ranges = list(epsilon = seq(0,0.2,0.01), cost = 2^(2:9)))  print(tuneResult)plot(tuneResult)

ϵ

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 9/19

We improved again the RMSE of our support vector regression model !

If we want we can visualize both our models. The 䁀䁀rst SVR model is in red, and the tuned SVR model

is in blue on the graph below :

I hope you enjoyed this introduction on Support Vector Regression with R.

You can download the source code of this tutorial. Each step has its own 䁀䁀le.

If you have any question feel free to comment.

Alexandre KOWALCZYKI am passionate about machine learning and Support Vector Machine. When I am not writing thisblog, you can �赯nd me on Kaggle participating in some competition.

12345678

tunedModel <‐ tuneResult$best.modeltunedModelY <‐ predict(tunedModel, data)  error <‐ data$Y ‐ tunedModelY   # this value can be different on your computer# because the tune method  randomly shuffles the datatunedModelRMSE <‐ rmse(error)  # 2.219642  

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 10/19

This entry was posted in R and tagged Linear Regression, R, Support Vector Regression on October

23, 2014 [http://www.svm-tutorial.com/2014/10/support-vector-regression-r/] .

36 thoughts on “Support Vector Regression with R”

Good stu၀倀. How would this behave if for example, I wanted to predict some more X variables that

are not in the training set? Is this useful in those instances? - In that case, how?

Many thanks

You just need to use the predict method with two parameters: the trained model and your new data.

This will give you the predicted values. This is useful because that is our original goal, we want to

predict unseen data.

JoseNovember 8, 2014 at 12:35 pm

Alexandre KOWALCZYKNovember 8, 2014 at 12:53 pm

Post author

Joshua DunnJanuary 7, 2015 at 5:30 pm

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 11/19

I have tried predicting unseen data but it always seems to underestimate the e၀倀ect of it. For

example, with temperature as my x-variable, if my SVR has not seen temperatures below zero

degrees C (ie minus 2 degrees C) it e၀倀ectively predicts them as it would zero. Would you be able to

tell me what this is called or point me in a direction to solve this? Regards

For me it looks like you are over䁀䁀tting your model with your training data. What you should try is to

modify increase the weight of the regularization parameter (or use regularization if you were not)

Dear

Thank you very much. Actually I want to predict the future value of univariate time series by SVM. I

have used the library e1071. I am able to predict the value over the study period but i want to

forecast the future value.

Alexandre KOWALCZYKJanuary 8, 2015 at 9:42 pm

Post author

Md. Moyazzem HossainMarch 12, 2015 at 5:58 pm

What is the software on which you do the programming??? tks

In this case this is RStudio which can be downloaded here

linaFebruary 19, 2015 at 4:13 pm

Alexandre KOWALCZYKFebruary 19, 2015 at 10:56 pm

Post author

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 12/19

"we use the tune method to train models with ϵ=0,0.1,0.2,...,1 and cost = 22,23,24,...,29 which means

it will train 88 models (it can take a long time)"

Hello. Can you explain how the number 88 is calculated? Thank you.

There is 11 values of epsilon, and 8 values for the cost. We can associate each epsilon with the 8 cost

values to create 8 couples. As there is 11 epsilons, there is couples.

LizMarch 6, 2015 at 10:22 pm

Alexandre KOWALCZYKMarch 10, 2015 at 7:24 pm

Post author

11 × 8

Hello Mr. Kowalczyk.

This tutorial is very helpful. Actually i am trying to forecast the future value of a time-series data by

using SVR method, but i am quite confused how to perform it in R. Could you explain the steps on

how to do it?

Thank you

Thanks for your comment. Unfortunately I have never used SVR to forecast timeseries. However I

found this question and one of the answer is pointing to this article. As suggested in the answer you

will need to transform the classi䁀䁀cation problem to a regression one but this might be a good

Fakhrul AgustriwanApril 2, 2015 at 7:35 am

Alexandre KOWALCZYKApril 3, 2015 at 11:49 pm

Post author

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 13/19

starting point for you.

As I understand, SVM implemented in R uses the Radial Basis Kernel by default. Therefore, there is

another parameter (called gamma). How do you deal with this one? I think you should 䁀䁀t it also.

One article mentionned to take the median of pairwise distances between the learning points. (After

the scaling process)

You just need to add the gamma parameter in the tune function. There is an example in the e1071

package documentation :

obj < ‐ tune.svm(Species~., data = iris, gamma = 2^(‐1:1), cost = 2^(2:4))

loicApril 23, 2015 at 1:31 pm

Alexandre KOWALCZYKApril 23, 2015 at 9:33 pm

Post author

Ok thanks for your reply.

Using tune.svm I noticed that this function is very very long (around 3 seconds per con䁀䁀guration of

parameters for 1000 observations of 7 variables).

Surprisingly if you use svm(..., cross = 10) you can get the cross validation error for less than 0.5

second on the same data. So, I concluded that tune.svm was very badly coded, do you have any idea

about this issue?

Therefore I coded my own parameters tuning function using svm(...,cross=10).

Also, I have found several papers that use a BFGS optimization algorithm (on a log2 scale) instead of

grid search. I tried this, it turned out to be very e䁀䁀cient.

loicApril 24, 2015 at 12:06 pm

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 14/19

When you are using svm(..., cross = 10) you are performing a 10-fold cross-classi䁀䁀cation on the

training data. This is not the same as doing a grid search. If the method tune.svm is so slow, it is not

because it is poorly coded, but because it trains one svm model per combination of hyperparameter.

So if you want to try gamma=0.1,0.01 and C=1,10,100 for instance it will train 6 di၀倀erent svm models

([0.1,1][0.1,10][0.1,100] [0.01,1][0.01,10][0.01,100]) In other word, it will try each couple in the

cartesian product of the gamma set with the C set. If you try it for 10 values of gamma and 10 values

of C, it will train 100 models. Which should indeed be much slower than training only 10 models.

Alexandre KOWALCZYKApril 25, 2015 at 12:16 pm

Post author

That's not what I meant. I am aware of that of course. But actually, I made grid search "by hand" with

a loop on 10x10 values of gamma and C using svm(...,cross = 10). Therefore I called 100 times svm

and then keep the minimum cv error. The overall time it took was something like 10 times less than

calling once tune.svm() on a 10x10 grid.

That was what made me think this function was poorly coded or it might use so䁀䁀sticated techniques

I am not aware of.

I've been trying to 䁀䁀nd the reason in vain.

Actually, I am a bit doubtful about the results of svm(..., cross = 10), it seems that it does not

compute the sv error on a stochastic way and the results are only one decimal digit precise which is

weird comparing to tune.svm()

I can't really help you more without seeing your code. Maybe you can ask on stackover䁀䁀ow or cross

validated if you want to dig deeper and understand what happens in your particular case. Feel free

to post the link here afterward and I'll take a look.

loicApril 27, 2015 at 11:38 am

Alexandre KOWALCZYKApril 29, 2015 at 11:57 pm

Post author

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 15/19

Great tutorial for svm, clearly de䁀䁀ning its function as a classi䁀䁀er or a regressor, thanks Alexandre.

SpartanApril 29, 2015 at 8:06 pm

Thank you for this valuable post. If I have more than one X variable including some dummy variables

can I 䁀䁀t the SVR for that case?

Yes you can. SVR also works when X is multidimensional.

HAPJune 3, 2015 at 7:26 am

Alexandre KOWALCZYKJune 4, 2015 at 9:54 pm

Post author

Thanks. I'l try.

HAPJune 8, 2015 at 6:16 am

Hello,

I read your blog posts. I am not very clear about how to forecast future values of time series using

SVR. I looks to me that SVR 䁀䁀ts a model using training set. But how about using predict() to predict

future values ( n.ahead values ) in R? I couldnt 䁀䁀nd this feature so far..

IlgazJuly 3, 2015 at 2:17 pm

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 16/19

Sincerely, Ilgaz

I think you should take a look at the kernlab package as suggested in this stackexchange answer.

Alexandre KOWALCZYKJuly 3, 2015 at 11:22 pm

Post author

First of all, thanks for the very helpful tutorial. I'm using R 3.2.1, but svm doesn't work correctly. On

step 3, when I'm running this: model <- svm(Y ~ X , data), the error is :

Error in predict(ret, xhold, decision.values = TRUE) :

unused argument (decision.values = TRUE)

Can you please help me?

Thanks,

Hello. I don't have a lot of idea about this one. You might want to take a look at this answer and try

the provided solution. Otherwise I would advise you to try the code on another machine to see if it

works and if it does try to replicate the environment on your machine. Best regards.

AseelJuly 6, 2015 at 8:54 pm

Alexandre KOWALCZYKJuly 6, 2015 at 11:33 pm

Post author

AseelJuly 7, 2015 at 1:25 am

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 17/19

Pingback: Support Vector Regression in R | logicalerrors

I really appreciate your replay. I think the problem is there is same function name in two packages,

for example predict() function in both ".GlobalEnv" and "package:stats" packages.

I will try to 䁀䁀gure out how to solve that.

Thanks a lot,

I've found that I have function with the same name with predict. So, simply, I've copied my function

to another name and remove predict function. That was making the confusion.

Thanks again for helping me.

AseelJuly 7, 2015 at 1:38 am

Hi Alexandre. Thank you so much for all the information, I have a few questions.

1. Can I use any kínd of variables in a SVM ? Continuous, categorical?

2. If I am using a SVM to classify two groups, is there a way to get a probability of assignment to each

group?

3. How do you validate that the SVM is a good model?

1. Yes. For continuous data it is called SVR and SVM for categorical data

2. Yes. Most framework provide a method "predict probabilities" to do so

3. You use a score to measure the quality of your model, if you want to learn more I recommend you

this book.

EspartacoJuly 17, 2015 at 3:34 am

Alexandre KOWALCZYKJuly 19, 2015 at 2:36 pm

Post author

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 18/19

Hi Alexandre,

Thanks for such a comprehensive tutorial. Much appreciated. I am trying to SVR for predicting time

series. As mentioned in your post, tune() shu䁀၀es the data. Is there any option or way to not to

shu䁀၀e the data?

Hello. You can specify a tunecontrol parameter to specify the behavior of the tune method. I think

tune.control(sampling = "䁀䁀x") might suit your need.

Yes. Worked perfectly!

DannyAugust 12, 2015 at 1:43 am

Alexandre KOWALCZYKAugust 14, 2015 at 10:31 pm

Post author

DannyAugust 15, 2015 at 1:23 am

hi Alexandre I am asking for using ls-svm in regression are there any R packages support it

tahirAugust 14, 2015 at 3:40 pm

Alexandre KOWALCZYKAugust 14, 2015 at 10:24 pm

Post author

19/08/2015 Support Vector Regression with R ­ SVM Tutorial

http://www.svm­tutorial.com/2014/10/support­vector­regression­r/ 19/19

Hello, you can use the function lssvm available in the kernlab package.

thank you very much, but when I use lssvm I get this message

(Using automatic sigma estimation (sigest) for RBF or laplace kernel

Error in if (n != dim(y)[1]) stop("Labels y and data x dont match") :

argument is of length zero)

tahirAugust 18, 2015 at 6:32 am