Machine Learning in R

Alexandros Karatzoglou1

1Telefonica ResearchBarcelona, Spain

December 15, 2010

Outline1 Introduction to R

CRANObjects and OperationsBasic Data StructuresMissing ValuesEntering DataFile Input and OutputInstalling PackagesIndexing and Subsetting

2 Basic Plots3 Lattice Plots4 Basic Statistics & Machine Learning

Tests5 Linear Models6 Naive Bayes7 Support Vector Machines8 Decision Trees9 Dimensionality Reduction10 Factor Analysis11 Cluster Analysis

Environment for statistical data analysis, inference andvisualization.Ports for Unix, Windows and MacOSXHighly extensible through user-defined functionsGeneric functions and conventions for standard operations likeplot, predict etc.∼ 1200 add-on packages contributed by developers from all overthe worlde.g. Multivariate Statistics, Machine Learning, Natural LanguageProcessing, Bioinformatics (Bioconductor), SNA, .Interfaces to C, C++, Fortran, Java

Comprehensive R Archive Network (CRAN)

CRAN includes packages which provide additional functionality tothe one existing in RCurrently over 1200 packages in areas like multivariate statistics,time series analysis, Machine Learning, Geo-statistics,environmental statistics etc.packages are written mainly by academics, PhD students, orcompany staffSome of the package have been ordered into Task Views

CRAN Task Views

Some of the CRAN packages are ordered into categories calledTask ViewsTask Views are maintained by people with experience in thecorresponding fieldthere are Task Views on Econometrics, Environmental Statistics,Finance, Genetics, Graphics Machine Learning, MultivariateStatistics, Natural Language Processing, Social Sciences, andothers.

Online documentation

1 Go to http://www.r-project.org

2 On the left side under Documentation3 Official Manuals, FAQ, Newsletter, Books4 Other and click on other publications contains a collection of

contributed guides and manuals

Mailing lists

R-announce is for announcing new major enhancements in R

R-packages announcing new version of packagesR-help is for R related user questions!R-devel is for R developers

Command Line Interface

R does not have a “real” guiAll computations and statistics are performed with commandsThese commands are called “functions” in R

click on helpsearch help functionalityFAQlink to reference web pagesapropos

Objects

Everything in R is an objectEverything in R has a class.

Data, intermediate results and even e.g. the result of a regressionare stored in R objectsThe Class of the object both describes what the object containsand what many standard functionsObjects are usually accessed by name. Syntactic names forobjects are made up from letters, the digits 0 to 9 in any non-initialposition and also the period “.”

Assignment and Expression Operations

R commands are either assignments or expressionsCommands are separated either by a semicolon ; or newlineAn expression command is evaluated and (normally) printedAn assignment command evaluates an expression and passes thevalue to a variable but the result is not printed

Expression Operations

> 1 + 1

Assigment Operations

> res <- 1 + 1

“<-” is the assignment operator in R

a series of commands in a file (script) can be executed with thecommand : source(“myfile.R”)

Sample session

[1] 1 2 3 4 5

> powers.of.2 <- 2^(1:5)> powers.of.2

[1] 2 4 8 16 32

> class(powers.of.2)

[1] "numeric"

> ls()

[1] "powers.of.2" "res"

> rm(powers.of.2)

Workspace

R stores objects in workspace that is kept in memoryWhen quiting R ask you if you want to save that workspaceThe workspace containing all objects you work on can then berestored next time you work with R along with a history of the usedcommands.

Basic Data Structures

VectorsFactorsData FrameMatricesLists

Vectors

In R the building blocks for storing data are vectors of various types.The most common classes are:

"character", a vector of character strings of varying length. Theseare normally entered and printed surrounded by double quotes."numeric", a vector of real numbers."integer", a vector of (signed) integers."logical", a vector of logical (true or false) values. The values areprinted and as TRUE and FALSE.

Numeric Vectors

> vect <- c(1, 2, 99, 6, 8, 9)> is(vect)

[1] "numeric" "vector"

> vect[2]

> vect[2:3]

[1] 2 99

> length(vect)

> sum(vect)

[1] 125

Character Vectors

> vect3 <- c("austria", "spain",+ "france", "uk", "belgium",+ "poland")> is(vect3)

[1] "character"[2] "vector"[3] "data.frameRowLabels"[4] "SuperClassMethod"

> vect3[2]

[1] "spain"

> vect3[2:3]

[1] "spain" "france"

> length(vect3)

Logical Vectors

> vect4 <- c(TRUE, TRUE, FALSE, TRUE,+ FALSE, TRUE)> is(vect4)

[1] "logical" "vector"

Factors

> citizen <- factor(c("uk", "us",+ "no", "au", "uk", "us", "us"))> citizen

[1] uk us no au uk us usLevels: au no uk us

> unclass(citizen)

[1] 3 4 2 1 3 4 4attr(,"levels")[1] "au" "no" "uk" "us"

> citizen[5:7]

[1] uk us usLevels: au no uk us

> citizen[5:7, drop = TRUE]

[1] uk us usLevels: uk us

Factors Ordered

> income <- ordered(c("Mid", "Hi",+ "Lo", "Mid", "Lo", "Hi"), levels = c("Lo",+ "Mid", "Hi"))> income

[1] Mid Hi Lo Mid Lo HiLevels: Lo < Mid < Hi

> as.numeric(income)

[1] 2 3 1 2 1 3

> class(income)

[1] "ordered" "factor"

> income[1:3]

[1] Mid Hi LoLevels: Lo < Mid < Hi

Data Frames

A data frame is the type of object normally used in R to store adata matrix.It should be thought of as a list of variables of the same length, butpossibly of different types (numeric, factor, character, logical, etc.).

Data Frames

> library(MASS)> data(painters)> painters[1:3, ]

Composition Drawing ColourDa Udine 10 8 16Da Vinci 15 16 4Del Piombo 8 13 16

Expression SchoolDa Udine 3 ADa Vinci 14 ADel Piombo 7 A

> names(painters)

[1] "Composition" "Drawing"[3] "Colour" "Expression"[5] "School"

> row.names(painters)

[1] "Da Udine" "Da Vinci"[3] "Del Piombo" "Del Sarto"[5] "Fr. Penni" "Guilio Romano"[7] "Michelangelo" "Perino del Vaga"[9] "Perugino" "Raphael"

[11] "F. Zucarro" "Fr. Salviata"[13] "Parmigiano" "Primaticcio"[15] "T. Zucarro" "Volterra"[17] "Barocci" "Cortona"[19] "Josepin" "L. Jordaens"[21] "Testa" "Vanius"[23] "Bassano" "Bellini"[25] "Giorgione" "Murillo"[27] "Palma Giovane" "Palma Vecchio"[29] "Pordenone" "Tintoretto"[31] "Titian" "Veronese"[33] "Albani" "Caravaggio"[35] "Corregio" "Domenichino"[37] "Guercino" "Lanfranco"[39] "The Carraci" "Durer"[41] "Holbein" "Pourbus"[43] "Van Leyden" "Diepenbeck"[45] "J. Jordaens" "Otho Venius"[47] "Rembrandt" "Rubens"[49] "Teniers" "Van Dyck"[51] "Bourdon" "Le Brun"[53] "Le Suer" "Poussin"

> summary(painters)

Composition DrawingMin. : 0.00 Min. : 6.001st Qu.: 8.25 1st Qu.:10.00Median :12.50 Median :13.50Mean :11.56 Mean :12.463rd Qu.:15.00 3rd Qu.:15.00Max. :18.00 Max. :18.00

Colour ExpressionMin. : 0.00 Min. : 0.0001st Qu.: 7.25 1st Qu.: 4.000Median :10.00 Median : 6.000Mean :10.94 Mean : 7.6673rd Qu.:16.00 3rd Qu.:11.500Max. :18.00 Max. :18.000

SchoolA :10D :10E : 7G : 7B : 6C : 6(Other): 8

Data Frames

Data frames are by far the commonest way to store data in R

They are normally imported by reading a file or from aspreadsheet or database.However, vectors of the same length can be collected into a dataframe by the function data.frame

Data Frame

> mydf <- data.frame(vect, vect3,+ income)> summary(mydf)

vect vect3 incomeMin. : 1.00 austria:1 Lo :21st Qu.: 3.00 belgium:1 Mid:2Median : 7.00 france :1 Hi :2Mean :20.83 poland :13rd Qu.: 8.75 spain :1Max. :99.00 uk :1

> mydf <- data.frame(vect, I(vect3),+ income)

Matrices

A data frame may be printed like a matrix, but it is not a matrix.Matrices like vectors have all their elements of the same type

> mymat <- matrix(1:10, 2, 5)> class(mymat)

[1] "matrix"

> dim(mymat)

[1] 2 5

A list is a vector of other R objects.Lists are used to collect together items of different classes.For example, an employee record might be created by :

> Empl <- list(employee = "Anna",+ spouse = "Fred", children = 3,+ child.ages = c(4, 7, 9))> Empl$employee

[1] "Anna"

> Empl$child.ages[2]

Basic Mathematical Operations

> 5 - 3

> a <- 2:4> b <- rep(1, 3)> a - b

[1] 1 2 3

> a * b

[1] 2 3 4

Matrix Multiplication

> a <- matrix(1:9, 3, 3)> b <- matrix(2, 3, 3)> c <- a %*% b

Missing Values

Missing Values in R are labeled with the logical value NA

Missing Values

> mydf[mydf == 99] <- NA> mydf

vect vect3 income1 1 austria Mid2 2 spain Hi3 NA france Lo4 6 uk Mid5 8 belgium Lo6 9 poland Hi

Missing Values

> a <- c(3, 5, 6)> a[2] <- NA> a

[1] 3 NA 6

> is.na(a)

[1] FALSE TRUE FALSE

Special Values

R has the also special values NaN , Inf and -Inf

> d <- c(-1, 0, 1)> d/0

[1] -Inf NaN Inf

Entering Data

For all but the smallest datasets the easiest way to get data into Ris to import it from a connection such as a file

Entering Data

> a <- c(5, 8, 5, 4, 9, 7)> b <- scan()

Entering Data

> mydf2 <- edit(mydf)

File Input and Output

read.table can be used to read data from text file like csvread.table creates a data frame from the values and tries to guessthe type of each variable

> mydata <- read.table("file.csv",+ sep = ",")

Packages

Packages contain additional functionality in the form of extrafunctions and dataInstalling a package can be done with the functioninstall.packages()

The default R installation contains a number of contributedpackages like MASS, foreign, utilsBefore we can access the functionality in a package we have toload the package with the function library()

Packages

Help on the contents of the packages is available

> library(help = foreign)

Help on installed packages is also available by help.start()Vignettes are also available for many packages

SPSS files

Package foreign contains functions that read data from SPSS(sav), STATA and other formats

> library(foreign)> spssdata <- read.spss("ees04.sav",+ to.data.frame = TRUE)

Excel files

To load Excel file into R an external package xlsReadWrite isneededWe will install the package directly from CRAN usinginstall.packages

> install.packages("xlsReadWrite",+ lib = "C:\temp")

Excel files

> library(xlsReadWrite)> data <- read.xls("sampledata.xls")

Data Export

You can save your data by using the functions write.table()or save()write.table is more data specific, while save can save any Robjectsave saves in a binary format while write.table() in textformat.

> write.table(mydata, file = "mydata.csv",+ quote = FALSE)> save(mydata, file = "mydata.rda")> load(file = "mydata.rda")

Indexing

R has a powerful set of Indexing capabilitiesThis facilitates Data manipulation and can speed up datapre-processingIndexing Vectors, Data Frames and matrices is done in a similarmanner

Indexing by numeric vectors

> letters[1:3]

[1] "a" "b" "c"

> letters[c(7, 9)]

[1] "g" "i"

> letters[-(1:15)]

[1] "p" "q" "r" "s" "t" "u" "v" "w" "x"[10] "y" "z"

Indexing by logical vectors

> a <- 1:10> a[c(3, 5, 7)] <- NA> is.na(a)

[1] FALSE FALSE TRUE FALSE TRUE FALSE[7] TRUE FALSE FALSE FALSE

> a[!is.na(a)]

[1] 1 2 4 6 8 9 10

Indexing Matrices and Data Frames

> mymat[1, ]

[1] 1 3 5 7 9

> mymat[, c(2, 3)]

[,1] [,2][1,] 3 5[2,] 4 6

> mymat[1, -(1:3)]

[1] 7 9

Selecting Subsets

> attach(painters)> painters[Colour >= 17, ]

Composition Drawing ColourBassano 6 8 17Giorgione 8 9 18Pordenone 8 14 17Titian 12 15 18Rembrandt 15 6 17Rubens 18 13 17Van Dyck 15 10 17

Expression SchoolBassano 0 DGiorgione 4 DPordenone 5 DTitian 6 DRembrandt 12 GRubens 17 GVan Dyck 13 G 59

Selecting Subsets

> painters[Colour >= 15 & Composition >+ 10, ]

Composition Drawing ColourTintoretto 15 14 16Titian 12 15 18Veronese 15 10 16Corregio 13 13 15Rembrandt 15 6 17Rubens 18 13 17Van Dyck 15 10 17

Expression SchoolTintoretto 4 DTitian 6 DVeronese 3 DCorregio 12 ERembrandt 12 GRubens 17 GVan Dyck 13 G 60

Graphics

R has a rich set of visualization capabilitiesscatter-plots, histograms, box-plots and more.

histograms

> data(hills)> hist(hills$time)

Scatter Plot

> plot(climb ~ time, hills)

Paired Scatterplot

> pairs(hills)

Box Plots

> boxplot(count ~ spray, data = InsectSprays,+ col = "lightgray")

Formula Interface

A formula is of the general form response ∼ expression where theleft-hand side, response, may in some uses be absent and theright-hand side, expression, is a collection of terms joined byoperators usually resembling an arithmetical expression.The meaning of the right-hand side is context dependente.g. in linear and generalized linear modelling it specifies the formof the model matrix

Scatterplot and line

> library(MASS)> data(hills)> attach(hills)> plot(climb ~ time, hills, xlab = "Time",+ ylab = "Feet")> abline(0, 40)> abline(lm(climb ~ time))

Multiple plots

> par(mfrow = c(1, 2))> plot(climb ~ time, hills, xlab = "Time",+ ylab = "Feet")> plot(dist ~ time, hills, xlab = "Time",+ ylab = "Dist")> par(mfrow = c(1, 1))

Plot to file

> pdf(file = "H:/My Documents/plot.pdf")> plot(dist ~ time, hills, xlab = "Time",+ ylab = "Dist", main = "Hills")> dev.off()

Paired Scatterplot

> splom(~hills)

Box-whisker plots

> data(michelson)> bwplot(Expt ~ Speed, data = michelson,+ ylab = "Experiment No.")> title("Speed of Light Data")

Box-whisker plots

> data(quine)> bwplot(Age ~ Days | Sex * Lrn *+ Eth, data = quine, layout = c(4,+ 2))

Discriptive Statistics

> mean(hills$time)

[1] 57.87571

> colMeans(hills)

dist climb time7.528571 1815.314286 57.875714

> median(hills$time)

[1] 39.75

> quantile(hills$time)

0% 25% 50% 75% 100%15.950 28.000 39.750 68.625 204.617

> var(hills$time)

[1] 2504.073

> sd(hills$time)

[1] 50.0407273

Discriptive Statistics

> cor(hills)

dist climb timedist 1.0000000 0.6523461 0.9195892climb 0.6523461 1.0000000 0.8052392time 0.9195892 0.8052392 1.0000000

> cov(hills)

dist climb timedist 30.51387 5834.638 254.1944climb 5834.63782 2621648.457 65243.2567time 254.19442 65243.257 2504.0733

> cor(hills$time, hills$climb)

[1] 0.8052392

Distributions

R contains a number of functions for drawing from a number ofdistribution like e.g. normal, uniform etc.

Sampling from a Normal Distribution

> nvec <- rnorm(100, 3)

Testing for the mean T-test

> t.test(nvec, mu = 1)

One Sample t-test

data: nvect = 21.187, df = 99, p-value <2.2e-16alternative hypothesis: true mean is not equal to 195 percent confidence interval:2.993364 3.405311

sample estimates:mean of x3.199337

Testing for the mean T-test

> t.test(nvec, mu = 2)

One Sample t-test

data: nvect = 11.5536, df = 99, p-value <2.2e-16alternative hypothesis: true mean is not equal to 295 percent confidence interval:2.993364 3.405311

sample estimates:mean of x3.199337

Testing for the mean Wilcox-test

> wilcox.test(nvec, mu = 3)

Wilcoxon signed rank test withcontinuity correction

data: nvecV = 3056, p-value = 0.06815alternative hypothesis: true location is not equal to 3

Two Sample Tests

> nvec2 <- rnorm(100, 2)> t.test(nvec, nvec2)

Welch Two Sample t-test

data: nvec and nvec2t = 7.4455, df = 195.173, p-value =3.025e-12alternative hypothesis: true difference in means is not equal to 095 percent confidence interval:0.8567072 1.4741003

sample estimates:mean of x mean of y3.199337 2.033933

Two Sample Tests

> wilcox.test(nvec, nvec2)

Wilcoxon rank sum test withcontinuity correction

data: nvec and nvec2W = 7729, p-value = 2.615e-11alternative hypothesis: true location shift is not equal to 0

Paired Tests

> t.test(nvec, nvec2, paired = TRUE)

Paired t-test

data: nvec and nvec2t = 7.6648, df = 99, p-value =1.245e-11alternative hypothesis: true difference in means is not equal to 095 percent confidence interval:0.863712 1.467096

sample estimates:mean of the differences

1.165404

Paired Tests

> wilcox.test(nvec, nvec2, paired = TRUE)

Wilcoxon signed rank test withcontinuity correction

data: nvec and nvec2V = 4314, p-value = 7.776e-10alternative hypothesis: true location shift is not equal to 0

Density Estimation

> library(MASS)> truehist(nvec, nbins = 20, xlim = c(0,+ 6), ymax = 0.7)> lines(density(nvec, width = "nrd"))

Density Estimation

> data(geyser)> geyser2 <- data.frame(as.data.frame(geyser)[-1,+ ], pduration = geyser$duration[-299])> attach(geyser2)> par(mfrow = c(2, 2))> plot(pduration, waiting, xlim = c(0.5,+ 6), ylim = c(40, 110), xlab = "previous duration",+ ylab = "waiting")> f1 <- kde2d(pduration, waiting,+ n = 50, lims = c(0.5, 6, 40,+ 110))> image(f1, zlim = c(0, 0.075), xlab = "previous duration",+ ylab = "waiting")> f2 <- kde2d(pduration, waiting,+ n = 50, lims = c(0.5, 6, 40,+ 110), h = c(width.SJ(duration),+ width.SJ(waiting)))> image(f2, zlim = c(0, 0.075), xlab = "previous duration",+ ylab = "waiting")> persp(f2, phi = 30, theta = 20,+ d = 5, xlab = "previous duration",+ ylab = "waiting", zlab = "")

Simple Linear Model

Fitting a simple linear model in R is done using the lm function> data(hills)> mhill <- lm(time ~ dist, data = hills)> class(mhill)

[1] "lm"

> mhill

Call:lm(formula = time ~ dist, data = hills)

Coefficients:(Intercept) dist

-4.841 8.330

Simple Linear Model

> summary(mhill)> names(mhill)> mhill$residuals

Multivariate Linear Model

> mhill2 <- lm(time ~ dist + climb,+ data = hills)> mhill2

Call:lm(formula = time ~ dist + climb, data = hills)

Coefficients:(Intercept) dist climb

-8.99204 6.21796 0.01105

Multifactor Linear Model

> summary(mhill2)> update(update(mhill2, weights = 1/hills$dist^2))> predict(mhill2, hills[2:3, ])

Bayes Rule

p(y |x) =p(x |y)p(y)

p(x)(1)

Consider the problem of email filtering: x is the email e.g. in the formof a word vector, y the label spam, ham

Prior p(y)

p(ham) ≈ mham

mtotal, p(spam) ≈

mtotal(2)

Likelihood Ratio

p(y |x) =p(x |y)p(y)

p(x)(3)

key problem: we do not know p(x |y) or p(x). We can get rid of p(x) bysettling for a likelihood ratio:

L(x) :=p(spam|x)

p(ham|x)=

p(x |spam)p(spam)

p(x |ham)p(ham)(4)

Whenever L(x) exceeds a given threshold c we decide that x is spamand consequently reject the e-mail

Computing p(x |y)

Key assumption in Naive Bayes is that the variables (or elements of x)are conditionally independent i.e. words in emails are independent ofeach other. We can then compute p(x |y) in a naive fashion byassuming that:

p(x |y) =Nwords∈x∏

p(w j |y) (5)

wj denotes the j-th word in document x . Estimates for p(w |y) can beobtained, for instance, by simply counting the frequency occurrence ofthe word within documents of a given class

Computing p(x|y)

p(w |spam) ≈∑m

i=1∑Nwords∈xi

j=1 {yi = spam&w ji = w}∑m

i=1∑Nwords∈xi

j=1 {yi = spam}(6)

{yi = spam&w ji = w} equals 1 if xi is labeled as spam and w occurs

as the j-th word in xi . The denominator counts the number of words inspam documents.

Laplacian smoothing

Issue: estimating p(w |y) for words w which we might not have seenbefore.Solution:increment all counts by 1. This method is commonly referred to asLaplace smoothing.

Naive Bayes in R

> library(kernlab)> library(e1071)> data(spam)> idx <- sample(1:dim(spam)[1], 300)> spamtrain <- spam[-idx, ]> spamtest <- spam[idx, ]> model <- naiveBayes(type ~ ., data = spamtrain)> predict(model, spamtest)> table(predict(model, spamtest),+ spamtest$type)> predict(model, spamtest, type = "raw")

k-Nearest Neighbors

An even simpler estimator than Naive Bayes is nearest neighbors. Inits most basic form it assigns the label of its nearest neighbor to an

observation xIdentify k nearest neigbhboors given distance metric d(x , x ′) (e.g.euclidian distance) and determine label by a vote.

k-Nearest Neighbors

> library(animation)> knn.ani(k = 4)

KNN in R

> model <- knn(spamtrain[, -58],+ spamtest[, -58], spamtrain[,+ 58])> predict(model, spamtest)> table(predict(model, spamtest),+ spamtest$type)

Support Vector Machines

Support Vector Machines work by maximizing a margin between ahyperplane and the data.SVM is a simple well understood linear methodThe optimization problem is convex thus only one optimal solutionexistsExcellent performance

Kernels: Data Mapping

Data are implicitly mapped into a high dimensional feature spaceΦ : X → H

Φ : R2 → R3

(x1, x2)→ (z1, z2, z3) := (x21 ,√

2x1x2, x22 )

Kernel Methods: Kernel Function

In Kernel Methods learning takes place in the feature space, dataonly appear inside inner products 〈Φ(x),Φ(x ′)〉Inner product is given by a kernel function (“kernel trick”)

k(x , x ′) = 〈Φ(x),Φ(x ′)〉

Inner product :

〈Φ(x),Φ(x ′)〉 = (x21 ,√

2x1x2, x22 )(x ′

12,√

2x ′1x ′

2, x ′2

= 〈x , x ′〉2

= k(x , x ′)

Kernel Functions

Gaussian kernel k(x , x ′) = exp(−σ‖x − x ′‖2)

Polynomial kernel k(x , x ′) = (〈x , x ′〉+ c)p

String Kernels (for text):

k(x , x ′) =∑

svx ,s′vx ′

λsδs,s′ =∑s∈A

nums(x)nums(x ′)λs

Kernels on Graphs, Mismatch kernels etc.

SVM Primal Optimization Problem

minimize t(w, ξ) =12‖w‖2 +

m∑i=1

subject to yi(〈Φ(xi),w〉+ b) ≥ 1− ξi (i = 1, . . . ,m)

ξi ≥ 0 (i = 1, . . . ,m)

where m is the number of training patterns, and yi = ±1.SVM Decision Function

f (xi) = 〈w,Φ(xi)〉+ b

fi =m∑

k(xi , xj)αj

SVM for Regression

SVM in R

> filter <- ksvm(type ~ ., data = spamtrain,+ kernel = "rbfdot", kpar = list(sigma = 0.05),+ C = 5, cross = 3)> filter> mailtype <- predict(filter, spamtest[,+ -58])> table(mailtype, spamtest[, 58])

SVM in R

> filter <- ksvm(type ~ ., data = spamtrain,+ kernel = "rbfdot", kpar = list(sigma = 0.05),+ C = 5, cross = 3, prob.model = TRUE)> filter> mailpro <- predict(filter, spamtest[,+ -58], type = "prob")> mailpro

SVM in R

> x <- rbind(matrix(rnorm(120), ,+ 2), matrix(rnorm(120, mean = 3),+ , 2))> y <- matrix(c(rep(1, 60), rep(-1,+ 60)))> svp <- ksvm(x, y, type = "C-svc",+ kernel = "vanilladot", C = 200)> svp> plot(svp, data = x)

SVM in R

> svp <- ksvm(x, y, type = "C-svc",+ kernel = "rbfdot", kpar = list(sigma = 1),+ C = 10)> plot(svp, data = x)

SVM in R

> svp <- ksvm(x, y, type = "C-svc",+ kernel = "rbfdot", kpar = list(sigma = 1),+ C = 10)> plot(svp, data = x)

SVM in R

> data(reuters)> is(reuters)> tsv <- ksvm(reuters, rlabels, kernel = "stringdot",+ kpar = list(length = 5), cross = 3,+ C = 10)> tsv

SVM in R

> x <- seq(-20, 20, 0.1)> y <- sin(x)/x + rnorm(401, sd = 0.03)> plot(x, y, type = "l")> regm <- ksvm(x, y, epsilon = 0.02,+ kpar = list(sigma = 16), cross = 3)> regm> lines(x, predict(regm, x), col = "red")

Decision Trees

|Petal.Length< 2.45

Petal.Width< 1.75setosa

versicolor virginica

Decision Trees

Tree-based methods partition the feature space into a set ofrectangles, and then fit a simple model in each one. The partitioning isdone taking one variable so that the partitioning of that variableincreases some impurity measure Q(x). In a node m of a classificationtree let

pmk =1N

∑xi∈Rm

I(yi = k) (7)

the portion of class k observation in node m. Trees classify theobservations in each node m to class

k(m) = argmaxkpmk (8)

Impurity Measures

Misclassification error:

∑i∈Rm

I(y 6= k(m)) (9)

Gini Index: ∑k 6=k ′

pmkpmk ′ =K∑

pmk (1− pmk ) (10)

Cross-entropy or deviance:

−K∑

pmk log(pmk ) (11)

Classification Trees in R

> data(iris)> tree <- rpart(Species ~ ., data = iris)> plot(tree)> text(tree, digits = 3)

Classification Trees in R

> fit <- rpart(Kyphosis ~ Age + Number ++ Start, data = kyphosis)> fit2 <- rpart(Kyphosis ~ Age ++ Number + Start, data = kyphosis,+ parms = list(prior = c(0.65,+ 0.35), split = "information"))> fit3 <- rpart(Kyphosis ~ Age ++ Number + Start, data = kyphosis,+ control = rpart.control(cp = 0.05))> par(mfrow = c(1, 3), xpd = NA)> plot(fit)> text(fit, use.n = TRUE)> plot(fit2)> text(fit2, use.n = TRUE)> plot(fit3)> text(fit3, use.n = TRUE)

Regression Trees in R

> data(cpus, package = "MASS")> cpus.ltr <- tree(log10(perf) ~+ syct + mmin + mmax + cach ++ chmin + chmax, cpus)> cpus.ltr

Random Forests

RF is an ensemble classifier that consist of many decision trees.Decisions are taken using a majority vote from the trees.

Number of training cases be N, and the number of variables in theclassifier be M.m is the number of input variables to be used to determine thedecision at a node of the tree; m should be much less than M.Sample a training set for this tree by choosing N times withreplacement from all N available training cases (i.e. take abootstrap sample).For each node of the tree, randomly choose m variables on whichto base the decision at that node.Calculate the best split based on these m variables in the trainingset.

Random Forests in R

> library(randomForest)> filter <- randomForest(type ~ .,+ data = spam)> filter

Principal Component Analysis (PCA)

A projection method finding projections of maximal variabilityi.e. it seeks linear combinations of the columns of X with maximal(or minimal) varianceThe first k principal components span a subspace containing the“best” kdimensional view of the dataProjecting the data on the first few principal components is oftenuseful to reveal structure in the dataPCA depends on the scaling of the original variables,thus it is conventional to do PCA using the correlation matrix,implicitly rescaling all the variables to unit sample variance.

Principal Component Analysis (PCA)

Principal Components Analysis (PCA)

> ir.pca <- princomp(log(iris[, -5]),+ cor = T)> summary(ir.pca)> loadings(ir.pca)> ir.pc <- predict(ir.pca)

PCA Plot

> plot(ir.pc[, 1:2], xlab = "first principal component",+ ylab = "second principal component")> text(ir.pc[, 1:2], labels = as.character(iris[,+ 5]), col = as.numeric(iris[,+ 5]))

kernel PCA

> test <- sample(1:150, 20)> kpc <- kpca(~., data = iris[-test,+ -5], kernel = "rbfdot", kpar = list(sigma = 0.2),+ features = 2)> plot(rotated(kpc), col = as.integer(iris[-test,+ 5]), xlab = "1st Principal Component",+ ylab = "2nd Principal Component")> text(rotated(kpc), labels = as.character(iris[-test,+ 5]), col = as.numeric(iris[-test,+ 5]))

kernel PCA

> kpc <- kpca(reuters, kernel = "stringdot",+ kpar = list(length = 5), features = 2)> plot(rotated(kpc), col = as.integer(rlabels),+ xlab = "1st Principal Component",+ ylab = "2nd Principal Component")

Distance methods

Distance Methods work by representing the cases in a lowdimensional Euclidian space so that their proximity reflects thesimilarity of their vairablesA distance measure needs to be defined when using DistanceMethodsThe function dist() implements four distance measuresbetween the points in the p-dimensional space of variables; thedefault is Euclidean distancecan be used with categorical variables!

Multidimensional scaling

> dm <- dist(iris[, -5])> mds <- cmdscale(dm, k = 2)> plot(mds, xlab = "1st coordinate",+ ylab = "2nd coordinate", col = as.numeric(iris[,+ 5]))

Miltidimensional scaling for categorical v.

> library(kernlab)> data(income)> inc <- income[1:300, ]> daisy(inc)> csc <- cmdscale(as.dist(daisy(inc)),+ k = 2)> plot(csc, xlab = "1st coordinate",+ ylab = "2nd coordinate")

Shannon’s non-linear mapping

> snm <- sammon(dist(ir[-143, ]))> plot(snm$points, xlab = "1st coordinate",+ ylab = "2nd coordinate", col = as.numeric(iris[,+ 5]))

Biplot

> data(state)> state <- state.x77[, 2:7]> row.names(state) <- state.abb> biplot(princomp(state, cor = T),+ pc.biplot = T, cex = 0.7, expand = 0.8)

Stars plot

stars(state.x77[, c(7, 4, 6, 2, 5, 3)], full = FALSE,key.loc = c(10, 2))

Factor Analysis

Principal component analysis looks for linear combinations of thedata matrix that are uncorrelated and of high varianceFactor analysis seeks linear combinations of the variables, calledfactors, that represent underlying fundamental quantities of whichthe observed variables are expressionsthe idea being that a small number of factors might explain a largenumber of measurements in an observational study

Factor Analysis

> data(swiss)> swiss.x <- as.matrix(swiss[, -1])> swiss.FA1 <- factanal(swiss.x,+ method = "mle")> swiss.FA1> summary(swiss.FA1)

k-means Clustering

Partition data into k sets S = {S1,S2, . . . , §k} so as to minimize thewithin-cluster sum of squares (WCSS):

argminS

k∑i=1

∑xj∈Si

|xj − µi |2 (12)

k-means Clustering

> data(iris)> clust <- kmeans(iris[, -5], centers = 3)> clust

k-means Clustering I

> data(swiss)> swiss.x <- as.matrix(swiss[, -1])> km <- kmeans(swiss.x, 3)> swiss.pca <- princomp(swiss.x)> swiss.px <- predict(swiss.pca)

k-means Clustering II

> dimnames(km$centers)[[2]] <- dimnames(swiss.x)[[2]]> swiss.centers <- predict(swiss.pca,+ km$centers)> plot(swiss.px[, 1:2], xlab = "first principal component",+ ylab = "second principal component",+ col = km$cluster)

k-means Clustering III

> points(swiss.centers[, 1:2], pch = 3,+ cex = 3)> identify(swiss.px[, 1:2], cex = 0.5)

kernel k-means

Partition data into k sets S = {S1,S2, . . . , §k} so as to minimize thewithin-cluster sum of squares (WCSS) in kernel feature space:

argminS

k∑i=1

∑xj∈Si

|Φ(xj)− Φ(µi)|2 (13)

kernel k-means

> sc <- kkmeans(as.matrix(iris[,+ -5]), kernel = "rbfdot", centers = 3)> sc> matchClasses(table(sc, iris[, 5]))

kernel k-means

> str <- stringdot(lenght = 4)> K <- kernelMatrix(str, reuters)> sc <- kkmeans(K, centers = 2)> sc> matchClasses(table(sc, rlabels))

Spectral Clustering in kernlab

Embedding data points into the subspace of eigenvectors of akernel matrixEmbedded points clustered using k -meansBetter performance (embedded points form tighter clusters)Can deal with clusters that do not form convex regions

Spectral Clustering in R

> data(spirals)> plot(spirals)> sc <- specc(spirals, centers = 2)> sc> plot(spirals, col = sc)

Spectral Clustering

> sc <- specc(reuters, kernel = "stringdot",+ kpar = list(length = 5), centers = 2)> matchClasses(table(sc, rlabels))> par(mfrow = c(1, 2))> kpc <- kpca(reuters, kernel = "stringdot",+ kpar = list(length = 5), features = 2)> plot(rotated(kpc), col = as.integer(rlabels),+ xlab = "1st Principal Component",+ ylab = "2nd Principal Component")> plot(rotated(kpc), col = as.integer(sc),+ xlab = "1st Principal Component",+ ylab = "2nd Principal Component")

Hierarchical Clustering

> data(state)> h <- hclust(dist(state.x77), method = "single")> plot(h)

Hierarchical Clustering

> pltree(diana(state.x77))

Machine Learning in R

Documents

Transcript of Machine Learning in R

Introduction to Machine Learning · Machine Learning in R. 4 There are many packages dedicated to machine learning that you can download from CRAN. Some popular ones are {glmnet},

Introduction to Machine Learning using R - Dublin R User Group - Oct 2013

Max Kuhn's talk on R machine learning

Machine Learning Algorithms - Summary + R Code

Some packages relative to machine learning in R

Machine Learning with R and H2O - University of Idahostevel/504/Machine... · Machine Learning with R and H2O by Spencer Aiello, Eric Eckstrand, Anqi Fu, Mark Landry, & Patrick Aboyoun

High Performance Machine Learning in R with H2O

Using R for Big Data Advanced Analytics and Machine ... · Using R for Big Data Advanced Analytics and Machine Learning Hands-On Lab ... Using R for Big Data AA and Machine Learning

bartMachine: Machine Learning with Bayesian Additive ...€¦ · Keywords: Bayesian, machine learning, statistical learning, non-parametric, R, Java. 1. Introduction Ensemble-of-trees

Machine Learning: Machine Learning:

Machine Learning with R Second Edition - Sample Chapter

Large scale machine learning projects with r suite

The PI System in Product Genealogy and Machine Learning€¦ · Machine Learning •Azure Machine Learning Studio •In house functions •R programming •Principal Component Analysis

Open-Source Machine Learning: R Meets Wekazeileis/papers/GfKl-2007.pdfOpen-Source Machine Learning: R Meets Weka Weka • Comprehensive collection of machine-learning algorithms for

Copyright R. Weber Machine Learning, Data Mining ISYS370 Dr. R. Weber.

R Machine Learning By Example - Sample Chapter

Machine Learning, Information R etrieval and User Modeling

Automatic Machine Learning in R - schd.wsschd.ws/hosted_files/user2017/97/business_machine_learning_555.pdf · Automatic Machine Learning in R Erin LeDell Ph.D. H2O.ai UseR! Brussels

Machine Learning example using R - French National Centre ...

Applied Machine learning using H2O, python and R Workshop