Exploring Language ClassificationWith Apache Spark and the Spark Notebook

A practical introduction to interactive Data Engineering

Gerard Maas

Gerard MaasLead Engineer @ Kensu

Computer EngineerScala ProgrammerEarly Spark AdopterSpark Notebook Dev

Cassandra MVP (2015, 2016)

Stack Overflow Top Contributor(Spark, Spark Streaming, Scala)

Wannabe IoT HackerArduino Enthusiast

@maasg

https://github.com/maasg

https://www.linkedin.com/in/gerardmaas/

https://stackoverflow.com/users/764040/maasg

DATA SCIENCE GOVERNANCE

Adalog helps enterprises to ensure that data pipelines continually deliver

their value by combining the contextual information when the pipeline was

created with the evolving environment where the pipelines execute.

CONNECT - COLLECT - LEARN

Language Classification

Language ClassificationSome inspiration...

What’s is a language? How is it composed?

Letter FrequencyCould we characterize a language by calculating the relative frequency of letters in some text ?

Spanish vs English letter frequency

n-grams

"cavnar and trenkle"

bi-grams: ca,av,vn,na,ar,r_,_a,an,nd,d_,_t,tr,re,en,nk,kl,le,e_

tri-grams: cav,avn,vna,nar,ar_,r_a,_an,and,nd_,d_t,_tr,tre,ren,enk,nkl,kle,le_

quad-grams: cavn,...

http://odur.let.rug.nl/~vannoord/TextCat/textcat.pdf

Could we characterize a language by calculating the relative frequency of sequence of letters in some text ?

Tech

Spark APIs

RDD -> Resilient Distributed Datasets

- Lazy, functional-oriented, low level API- Basis for execution of all high-level libraries

Dataframes

- Column-oriented, SQL-inspired DSL- Many optimizations under the hood (Catalyst, Tungsten)

Dataset

- Best of both worlds (except …)

Spark NotebookA dynamic and visual web-based notebook for Spark with Scala

Spark Notebook - Open Source Roadmap

2017

GIT KerberosProject Generator

Q1 Q2 Q3

Announcements: blog.kensu.io

Notebooks

Notebooks for this presentation are located at:

https://github.com/maasg/spark-notebooks

- have fun!

https://github.com/maasg/spark-notebooks/languageclassification/language-detection-letter-freq.snb

Implements the idea of using a letter frequency model to classify the language in a doc.

Uses the dataset found in https://github.com/maasg/spark-notebooks/languageclassification/data/

It produces a training set of sampled strings that will be used also for the n-gram classifier

(Note: this notebook is missing a function that’s left as an exercise to the reader. The folder /solutions contains the full working version.)

Notebook 1 : Naive Language Classification

Notebook 2 : n-gram Language Classification

https://github.com/maasg/spark-notebooks/languageclassification/n-gram-language-classification.snb

Implements the n-gram algorithm described in the paper.

Uses the dataset found in https://github.com/maasg/spark-notebooks/languageclassification/data/

Uses the resulting classifier to implement a custom Spark ML Transformer that can be easily used to classify new texts. Transformers can be combined into Spark ML Pipelines of arbitrary complexity.

(Note: this notebook is missing a function that’s left as an exercise to the reader. The folder /solutions contains the full working version.)