Distributed Computing Using...

Albert-Ludwigs-Universität Freiburg

Practical / Praktikum WS17/18

October 18th, 2017

Distributed Computing Using Spark

Prof. Dr. Georg Lausen

Anas Alzogbi

Victor Anthony Arrascue Ayala

Agenda

Introduction to Spark

Case-study: Recommender system for scientific papers

Organization

Hands-on session

18.10.2017 Distributed Computing Using Spark WS17/18 2

Agenda



Organization

Hands-on session



Distributed programming

MapReduce

Spark


Distributed programming - problem

Data grows faster than processing capabilities

- Web 2.0: users generate content

- Social networks, online communities, etc.


Source: https://www.flickr.com/photos/will-lion/2595497078

Big Data


Source: https://www.domo.com/blog/2015/08/data-never-sleeps-3-0/

Source: http://www.bigdata-startups.com/open-source-tools/

Big Data

Buzzword

Often less-structured

Requires different techniques, tools, approaches- To solve new problems or old ones in a better way


Network Programming Models

Requires a communication protocol for programming parallel computers (slow)- MPI (wiki)

Locality of the data and the code across the network have to be done manually

No failure management

Network problems not solved (e.g. stragglers)


https://en.wikipedia.org/wiki/Message_Passing_Interface

Data Flow Models

Higher-level of abstraction: algorithms are parallelized on large clusters

Fault-recovery by means of data replication

Job divided into a set of independent tasks

- Code is shipped to where the data is located

Good scalability


MapReduce – Key ideas

1. Problem is split into smaller problems (map step)

2. Smaller problems are solved in a parallel fashion

3. Finally, a set of solutions to the smaller problems get synthesized into a solution of the original problem (Reduce step)


MapReduce – Overview


split 1

split 0 Map

Map

Map

Reduce

Reduce

output 0

output 1

<k,v> Data

split 2

Input Data

…

A target problem has to be parallelizable!!!

MapReduce – Wordcount example


Google Maps charts new territory into businesses

Google selling new tools for businesses to build their own maps

Google promises consumer experience for businesses with Maps Engine Pro

Google is trying to get its Maps service used by more businesses

Google 4

Maps 4

Businesses 4

Engine 1

Charts 1

Territory 1

Tools 1

…

MapReduce – Wordcount’s map






Map

Map

Google 2

Charts 1

Maps 2

Territory 1

…

Google 2

Businesses 2

Maps 2

Service 1

…

MapReduce – Wordcount’s reduce


Reduce

Reduce

Google 2

Google 2

Maps 2

Maps 2

…

Businesses 2

Businesses 2

Charts 1

Territory 1

…

Google 4

Maps 4

…

Businesses 4

Charts 1

Territory 1

…

MapReduce

Automatic

- Partition and distribution of data

- Parallelization and assignment of tasks

- Scalability, fault-tolerance, scheduling


Apache Hadoop

Open-source implementation of MapReduce


So

urc

e: h

ttp

://w

ww

.bo

go

tob

og

o.c

om

/Ha

do

op

/Big

Da

ta_h

ad

oo

p_E

cosy

ste

m.p

hp

MapReduce – Parallelizable algorithms

Matrix-vector multiplication

Power iteration (e.g. PageRank)

Gradient descent methods

Stochastic SVD

Matrix Factorization (Tall skinny QR)

etc…


MapReduce – Limitations

Inefficient for multi-pass algorithms

No efficient primitives for data sharing

State between steps is materialized and distributed

Slow due to replication and storage


Source: http://stanford.edu/~rezab/sparkclass

Limitations – PageRank

Requires iterations of multiplications of sparse matrix and vector


Source: http://stanford.edu/~rezab/sparkclass

Limitations – PageRank

MapReduce sometime requires asymptotically more communication or I/O

Iterations are handled very poorly

Reading and writing to disk is a bottleneck

- In some cases 90% of time is spent on I/O


Spark Processing Framework

Developed in 2009 in UC Berkeley’s

In 2010 open sourced at Apache

- Most active big data community

- Industrial contributions: over 50 companies

Written in Scala

- Good at serializing closures

Clean APIs in Java, Scala, Python, R


Spark Processing Framework


Contributors (2014)

Spark – High Level Architecture


HD

FS

Source: https://mapr.com/ebooks/spark/

Spark - Running modes

Local mode: for debugging

Cluster mode

- Standalone mode

- Apache Mesos

- Hadoop Yarn


Spark – Programming model

Spark context: the entry point

Spark Session: since Spark 2.0- New unified entry point. It combines SQLContext,

HiveContext and future StreamingContex

Spark Conf: to initialize the context

Spark’s interactive shell- Scala: spark-shell

- Python: pyspark


Spark – RDDs, the game changer

Resilient distributed datasets

A typed data-structure (RDD[T]) that is not language specific

Each element of type T is stored locally on a machine

- It has to fit in memory

An RDD can be cached in memory


Resilient Distributed Datasets

Immutable collections of objects, spread across cluster

User controlled partitioning and storage

Automatically rebuilt on failure

RDDs are replaced by Dataset, which is strongly-typed like an RDD (Spark > 2.0)


Spark – Wordcount example

text_file = sc.textFile("...")

counts = text_file.flatMap(lambda line: line.split(" ")) \

.map(lambda word: (word, 1)) \

.reduceByKey(lambda a, b: a + b)

counts.saveAsTextFile("...")


http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.SparkContext

http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.SparkContext

Spark – Data manipulation

Transformations: always yield a new RDD instance (RDDs are immutable)

- filter, map, flatMap, etc.

Actions: triggers a computation on the RDD’s elements

- count, foreach, etc.

Lazy evaluation of transformations


Spark – DataFrames

DataFrame API introduced since Spark 1.3

Handles table-like representation with named columns and declared column types

Do not confuse with Python’s Pandas DataFrames

DataFrames translate SQL code into RDD low-level operations

Since Spark 2.0, DataFrame is implemented as a special case of DataSet


DataFrames – How to create DFs

1. Convert existing RDDs

2. Running SQL queries

3. Loading external data


Spark SQL

SQL context

http://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html


// Run SQL statements. Returns a DataFrame

students = sqlContext.sql( "SELECT name FROM people WHERE occupation>=‘student’)

http://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html

Spark – DataFrames


So

urc

e: S

pa

rk in

Act

ion

(b

oo

k, s

ee

lit

era

ture

)

Machine Learning (ML) with Spark

ML project steps1. Data collection

2. Data cleaning and preparation

3. Data analysis and feature extraction

4. Model training

5. Model evaluation

6. Model application


So

urc

e:

Sp

ark

in

Act

ion

(b

oo

k, s

ee

lit

era

ture

)


ML with Spark- Perfect for ML parallelizable algorithms!!

- A single platform (the same system and the same API) for performing most tasks:

• Collect, prepare, analyze the data

• Train, evaluate, use the model

- Training and applying ML algorithms on very large datasets

- Offer most of the popular ML algorithms



MLlib- Spark’s machine learning library

- Provides a generalized API for training and tuning different algorithms in the same way (influenced by scikit-learn)

- Relies on several low-level libraries for performing optimized linear algebra operations:

• Breeze, jblas for Scala and java

• NumPy for Python



MLlib two APIs- RDD-based API

• Will be removed in Spark 3.0 (spark.mllib)

- Dataframe-based API, will keep add new features (spark.ml)

• More user-friendly API than RDDs

• A uniform API across ML algorithms and across multiple languages

• Facilitate practical ML Pipelines (feature transformations)


MLlib abstractions

Transformer- Main method: transform- Examples:

• ML model• Feature transformer

Estimator- main method: fit- Example: ML algorithm

Evaluator- Example: RMSE metric


Estimator Transformer EvaluatorFit

Input

dataset

Evaluation

results

Transforme

d dataset

Estimate

Tra

nsfo

rm

Source: Spark in Action (book, see literature)

A pipeline chains multiple Transformers and Estimators together to specify an ML workflow

ExampleLearn a prediction model using features extracted from text document

Training phase

MLlib Pipelines


Source: http://spark.apache.org/docs/latest/ml-pipeline.html#properties-of-pipeline-components

Test phase

Organization

Introduction Introduction to Spark


Organization

Hands-on session



Motivation- Recommend relevant papers to users

Dataset- Set of papers (~172 K)

• Textual content: Title + abstract

• Attributes: type, journal, pages, year,…

- Set of users (~ 28 K)

- Ratings (~ 828 K ratings)


Organization

Introduction Introduction to Spark


Organization

Hands-on session


Organization

Team

Educational goals

Requirements

ILIAS

Experiments’ submissions

Assessment

Discussion with the tutors

Schedule


Team

Prof. Georg Lausen

Assistants

- Anas

- Anthony

Tutors

- Polina Koleva

- Matteo Cossu


http://dbis.informatik.uni-freiburg.de/team/lausen/georg

http://dbis.informatik.uni-freiburg.de/team/alzoghbi/anas

http://dbis.informatik.uni-freiburg.de/team/arrascue/anthony

Educational goals

Distributed programming paradigm

Recommender Systems (use case)

Theoretical and practical training

- Master project and thesis

Data Science profile for work market


Requirements

Mandatory

- Registration via HisInOne

- Attendance to Kick-off meeting

Recommended

- Attendance of DAQL, SIDS or ML lectures

- Basics In Python programming


https://campus.uni-freiburg.de/

http://dbis.informatik.uni-freiburg.de/lehre/SS17/Spezialvorlesung/Data Analysis and Query Languages/

http://websci.informatik.uni-freiburg.de/teaching/ws201617/infosys/infosys

ILIAS

Distributed Computing Using Spark -WS1718https://ilias.uni-freiburg.de/goto.php?target=crs_878841

Access with course password

Forum for clarification questions of tasks

- Do not post solutions or suggestions


https://ilias.uni-freiburg.de/goto.php?target=crs_878841&client_id=unifreiburg

Experiments’ submissions

6 experiments, 2-3 weeks of working time

Submissions in groups of 2 students (Form your group)

Submissions via ILIAS


Assessment

Each experiment: 50 points. Overall 300 points.

At least 70% of the points required to pass

Corrections done by tutors


Discussion of solutions with tutors

Mandatory attendance

Each member has to be able to explain all tasks!- 0 points for that task

Copied solutions- First time: 0 points for that experiment

- Second time: failure of the practical


Schedule


Experiment Content Release Submission Discussion

1.Familiarizing with Tools, Loading Data, and Basic Analysis of Data

18.10.2017 01.11.2017, 11h 08.11.2017

2. Experiment 2 01.11.2017 15.11.2017, 11h 22.11.2017

3. Experiment 3 15.11.2017 29.11.2017, 11h 06.12.2017

4. Experiment 4 29.11.2017 13.12.2017, 11h 20.12.2017

5. Experiment 5 13.12.2017 10.01.2018, 11h 17.01.2018

6. Experiment 6 10.01.2018 31.01.2018, 11h 07.02.2018

Literature

Spark in Action [book] by Petar Zečević Marko Bonaći

Machine Learning with Spark [book] by Nick Pentreath

Apache Spark documentation:http://spark.apache.org/docs/latest


http://spark.apache.org/docs/latest/

Organization



Organization

Hands-on session


Distributed Computing Using...

Documents

Transcript of Distributed Computing Using...