Ryan Curtin, Principal Research Scientist, Symantec at MLconf ATL 2016

1 / 47

mlpack: or, How I Learned To Stop Worrying and

Love C++

Ryan R. Curtin

September 23, 2016

Introduction

❖ Introduction

❖ Graph #1

❖ The Big Tradeoff

❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML

❖ Command-lineprograms

❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?

❖ Compile-timeexpressions

❖ Take-home

❖ kNN benchmarks

❖ vs. Spark

❖ What’s coming?

❖ Questions andcomments?

2 / 47

Why are we here?

Introduction

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



2 / 47

Why are we here?

speed

Introduction

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



2 / 47

Why are we here?

speed programming languages

Introduction

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



2 / 47

Why are we here?

speed programming languages machine learning

Introduction

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



2 / 47

Why are we here?


C++

Introduction

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



2 / 47

Why are we here?


C++ mlpack

Introduction

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



2 / 47

Why are we here?


C++ mlpack fast

Graph #1

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



3 / 47

Graph #1

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



3 / 47

low-level high-level

Graph #1

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



3 / 47


ASM

Graph #1

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



3 / 47


ASM VB

Graph #1

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



3 / 47


ASM VB

C

C++

Java

PythonScala

Ruby

INTERCAL

C#

PHP

JS

Go

Tcl

Lua

Note: this is not a scientific or particularly accurate representation.

Graph #1

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



3 / 47


ASM VB

C

C++

Java

PythonScala

Ruby

INTERCAL

C#

PHP

JS

Go

Tcl

Lua


fast easy

Graph #1

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



3 / 47


ASM VB

C

C++

Java

PythonScala

Ruby

INTERCAL

C#

PHP

JS

Go

Tcl

Lua


fast easy

specific portable

The Big Tradeoff

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



4 / 47

speed vs. portability and readability

The Big Tradeoff

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



4 / 47

speed vs. portability and readability

If we’re careful, we can get speed, portability, and

readability by using C++.

So, mlpack.

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



5 / 47

What is it?

http://www.mlpack.org/

https://github.com/mlpack/mlpack/

So, mlpack.

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



5 / 47

What is it?

● a fast general-purpose C++ machine learning library

● contains flexible implementations of common and

cutting-edge machine learning algorithms

● for fast or big runs on single workstations

● bindings are available for R, and are coming for other

languages

● commonly used inside the machine learning community

● 60+ developers from around the world

● frequent participation in the Google Summer of Code

program

R.R. Curtin, J.R. Cline, N.P. Slagle, W.B. March, P. Ram, N.A. Mehta, A.G. Gray, “mlpack: ascalable C++ machine learning library”, in The Journal of Machine Learning Research, vol. 14,p. 801–805, 2013.



Standard ML

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



6 / 47

mlpack implements a lot of standard machine learning

techniques.

● k-means and mean shift clustering

● nearest neighbor search and range search

● naive Bayes classifier

● regression: linear, logistic, LARS, softmax

● independent components analysis

● matrix factorization and collaborative filtering

● AdaBoost (plus some weak learners)

● GMMs and HMMs

● deep learning (still experimental)

● kernel PCA and regular PCA

● data preprocessing utilities

● numerous generic optimizers

Cutting-edge ML

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



7 / 47

mlpack also implements new, cutting-edge machine learning

techniques.

● several variants of the k-means algorithm

● density estimation trees

● Hoeffding (streaming) decision trees (and experimental

streaming forests)

● sparse coding and local coordinate coding

● dual-tree minimum spanning tree calculation

● dual-tree fast max-kernel search

● neighborhood components analysis

● rank-approximate nearest neighbor search

● automatically tuned locality-sensitive hashing (still

experimental)

● QUIC-SVD (fast approximate SVD)

Command-line programs

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



8 / 47

You don’t need to be a C++ expert.


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



9 / 47


# Train AdaBoost model.

$ mlpack_adaboost -t training_file.h5 -l training_labels.h5 \

> -M trained_model.bin

# Predict with AdaBoost model.

$ mlpack_adaboost -m trained_model.bin -T test_set.csv \

> -o test_set_predictions.csv


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



10 / 47








# Use PCA to reduce to 5 dimensions.

$ mlpack_pca -i dataset.txt -d 5 -o new_dataset.csv


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



11 / 47








# Use PCA to reduce to 5 dimensions.

$ mlpack_pca -i dataset.txt -d 5 -o new_dataset.csv

# Impute missing values ("NULL") in the input dataset to the

# mean in that dimension.

$ mlpack_preprocess_imputer -i dataset.h5 -s mean -o imputed.h5

Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



12 / 47

Why write an algorithm for one specific situation?

Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



13 / 47


NearestNeighborSearch n(dataset);

n.Search(query_set, 3, neighbors, distances);

What if I don’t want the Euclidean distance?

Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



14 / 47


// The numeric parameter is the value of p for the p-norm to

// use. 1 = Manhattan distance, 2 = Euclidean distance, etc.

NearestNeighborSearch n(dataset, 1);


Ok, this is a little better!

Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



15 / 47


// ManhattanDistance is a class with a method Evaluate().

NearestNeighborSearch<ManhattanDistance> n(dataset);


This is much better! The user can specify whatever

distance metric they want, including one they write

themselves.

Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



16 / 47


// This will _definitely_ get me best paper at ICML! I can

// feel it!

class MyStupidDistance

{

static double Evaluate(const arma::vec& a,

const arma::vec& b)

{

return 15.0 * std::abs(a[0] - b[0]);

}

};

// Now we can use it!

NearestNeighborSearch<MyStupidDistance> n(dataset);


Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



17 / 47


// We can also use sparse matrices instead!

NearestNeighborSearch<MyStupidDistance, arma::sp_mat>

n(sparse_dataset);

n.Search(sparse_query_set, 3, neighbors, distances);

Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



18 / 47


// Nearest neighbor search with arbitrary types of trees!

NearestNeighborSearch<EuclideanDistance, arma::mat, KDTree> kn;

NearestNeighborSearch<EuclideanDistance, arma::sp_mat, CoverTree> cn;

NearestNeighborSearch<ManhattanDistance, arma::mat, Octree> on;

NearestNeighborSearch<ChebyshevDistance, arma::sp_mat, RPlusTree> rn;

NearestNeighborSearch<MahalanobisDistance, arma::mat, RPTree> rpn;

NearestNeighborSearch<EuclideanDistance, arma::mat, XTree> xn;

Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



19 / 47









R.R. Curtin, “Improving dual-tree algorithms”. PhD thesis, Georgia Institute of Technology, At-lanta, GA, 8/2015.

Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



20 / 47










Genericity

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



21 / 47










Pros of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



22 / 47

C++ is great!

Pros of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



22 / 47

C++ is great!

● Generic programming at compile time via templates.

Pros of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



22 / 47

C++ is great!


● Low-level memory management.

Pros of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



22 / 47

C++ is great!



● Little to no runtime overhead.

Pros of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



22 / 47

C++ is great!




● Well-known!

Pros of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



22 / 47

C++ is great!




● Well-known!

● The Armadillo library gives us good linear algebra

primitives.

Pros of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



23 / 47

C++ is great!




● Well-known!

● The Armadillo library gives us good linear algebra

primitives.

using namespace arma;

extern mat x, y;

mat z = (x + y) * chol(x) + 3 * chol(y.t());

Cons of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



24 / 47

C++ is not great!

Cons of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



24 / 47

C++ is not great!

● Templates can be hard to debug because of error

messages.

Cons of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



24 / 47

C++ is not great!


messages.

● Memory bugs are easy to introduce.

Cons of C++

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



24 / 47

C++ is not great!


messages.

● Memory bugs are easy to introduce.

● The new language revisions are not making the language

any simpler...

Why templates?

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



25 / 47

What about virtual inheritance?

Why templates?

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



26 / 47


class MyStupidDistance : public Distance

{

virtual double Evaluate(const arma::vec& a,

const arma::vec& b)

{

return 15.0 * std::abs(a[0] - b[0]);

}

};

NearestNeighborSearch n(dataset, new MyStupidDistance());

n.Search(3, neighbors, distances);

Why templates?

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



27 / 47


class MyStupidDistance : public Distance

{

virtual double Evaluate(const arma::vec& a,

const arma::vec& b)

{

return 15.0 * std::abs(a[0] - b[0]);

}

};

NearestNeighborSearch n(dataset, new MyStupidDistance());

n.Search(3, neighbors, distances);

vtable lookup penalty!

Why templates?

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



28 / 47

Using inheritance to call a function costs us instructions:

Distance* d =

new MyStupidDistance();

d->Evaluate(a, b);

MyStupidDistance::Evaluate(a, b);

Why templates?

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



29 / 47


Distance* d =


d->Evaluate(a, b);


; push stack pointer

movq \%rsp, \%rdi

; get location of function

movq \$_ZTV1A+16, (\%rsp)

; call Evaluate()

call _ZN1A1aEd

; just call Evaluate()!

call _ZN1B1aEd.isra.0.constprop.1

Why templates?

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



30 / 47


Distance* d =


d->Evaluate(a, b);


; push stack pointer

movq \%rsp, \%rdi

; get location of function

movq \$_ZTV1A+16, (\%rsp)

; call Evaluate()

call _ZN1A1aEd

; just call Evaluate()!

call _ZN1B1aEd.isra.0.constprop.1

Up to 10%+ performance penalty in some situations!

Compile-time expressions

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



31 / 47

What about math? (Armadillo)


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



32 / 47


In C:

extern double** a, b, c, d, e;

extern int rows, cols;

// We want to do e = a + b + c + d.

mat_copy(e, a, rows, cols);

mat_add(e, b, rows, cols);

mat_add(e, c, rows, cols);

mat_add(e, d, rows, cols);


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



33 / 47


In C with a custom function:

extern double** a, b, c, d, e;

extern int rows, cols;

// We want to do e = a + b + c + d.

mat_add4_into(e, a, b, c, d, rows, cols);

Fastest! (one pass)


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



34 / 47


In MATLAB:

e = a + b + c + d


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



35 / 47


In MATLAB:

e = a + b + c + d

Beautiful!


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



36 / 47



❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



37 / 47



❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



38 / 47


In C++ (with Armadillo):


extern mat a, b, c, d;

mat e = a + b + c + d;

No temporaries, only one pass! Just as fast as the fastest C

implementation.


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



39 / 47





mat e = a + b + c + d;

● operator+(mat, mat) returns a placeholder type

op<mat, mat, add>


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



40 / 47





mat e = a + b + c + d;


op<mat, mat, add>

● operator+() is also overloaded for op<...> arguments


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



41 / 47





mat e = a + b + c + d;


op<mat, mat, add>


● So a + b + c + d returns

op<mat, op<mat, op<mat, mat, add>, add>, add>


❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



42 / 47





mat e = a + b + c + d;


op<mat, mat, add>


● So a + b + c + d returns

op<mat, op<mat, op<mat, mat, add>, add>, add>

● operator=() is overloaded to ‘unwrap’ op<> expressions

Take-home

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



43 / 47

● Templates give us generic code.

● Templates allow us to generate fast code.

kNN benchmarks

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



44 / 47

dataset d N mlpack mlpy matlab scikit shogun Weka

isolet 617 8k 15.65s 59.09s 50.88s 44.59s 59.56s 220.38scorel 32 68k 17.70s 95.26s fail 63.32s fail 29.38scovertype 54 581k 18.04s 27.68s >9000s 44.55s >9000s 42.34stwitter 78 583k 1573.92s >9000s >9000s 4637.81s fail >9000smnist 784 70k 3129.46s >9000s fail 8494.24s 6040.16s >9000stinyImages 384 100k 4535.38s 9000s fail >9000s fail >9000s

vs. Spark

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



45 / 47

We can use mmap() for out-of-core learning because our

algorithms are generic!

vs. Spark

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



45 / 47

We can use mmap() for out-of-core learning because our

algorithms are generic!

D. Fang, P. Chau. M3: scaling up machine learning via memory mapping, SIGMOD/PODS 2016.

What’s coming?

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



46 / 47

● Bindings to other languages! (more than just R)

● Non-experimental support for deep learning

● Automatic tuning of machine learning algorithms

● Further compile-time speedups

Questions and comments?

❖ Introduction

❖ Graph #1


❖ So, mlpack.

❖ Standard ML

❖ Cutting-edge ML


❖ Genericity

❖ Pros of C++

❖ Cons of C++

❖ Why templates?


❖ Take-home

❖ kNN benchmarks

❖ vs. Spark



47 / 47





Ryan Curtin, Principal Research Scientist, Symantec at MLconf ATL 2016

Technology

Transcript of Ryan Curtin, Principal Research Scientist, Symantec at MLconf ATL 2016