PySpark Best Practices by Juliet Hougland
-
Upload
spark-summit -
Category
Data & Analytics
-
view
2.737 -
download
1
Transcript of PySpark Best Practices by Juliet Hougland
‹#›© Cloudera, Inc. All rights reserved.
Juliet Hougland Spark Summit Europe 2015 @j_houg
PySpark Best Practices
‹#›© Cloudera, Inc. All rights reserved.
‹#›© Cloudera, Inc. All rights reserved.
RDDssc.textFile(“hdfs://…”, 4) .map(to_series) .filter(has_outlier) .count()
HDFS
Partition 1
Partition 2
Partition 3
Partition 4
Thanks: Kostas Sakellis
‹#›© Cloudera, Inc. All rights reserved.
RDDs
…RDD
HDFS
Partition 1
Partition 2
Partition 3
Partition 4
Thanks: Kostas Sakellis
sc.textFile(“hdfs://…”, 4) .map(to_series) .filter(has_outlier) .count()
‹#›© Cloudera, Inc. All rights reserved.
RDDs
…RDD …RDD
HDFS
Partition 1
Partition 2
Partition 3
Partition 4
Partition 1
Partition 2
Partition 3
Partition 4
Thanks: Kostas Sakellis
sc.textFile(“hdfs://…”, 4) .map(to_series) .filter(has_outlier) .count()
‹#›© Cloudera, Inc. All rights reserved.
RDDs
…RDD …RDD
HDFS
Partition 1
Partition 2
Partition 3
Partition 4
Partition 1
Partition 2
Partition 3
Partition 4
…RDD
Partition 1
Partition 2
Partition 3
Partition 4
Thanks: Kostas Sakellis
sc.textFile(“hdfs://…”, 4) .map(to_series) .filter(has_outlier) .count()
‹#›© Cloudera, Inc. All rights reserved.
…RDD …RDD
RDDs
HDFS
Partition 1
Partition 2
Partition 3
Partition 4
Partition 1
Partition 2
Partition 3
Partition 4
…RDD
Partition 1
Partition 2
Partition 3
Partition 4
Count
Thanks: Kostas Sakellis
sc.textFile(“hdfs://…”, 4) .map(to_series) .filter(has_outlier) .count()
‹#›© Cloudera, Inc. All rights reserved.
Spark Execution Model
‹#›© Cloudera, Inc. All rights reserved.
PySpark Execution Model
‹#›© Cloudera, Inc. All rights reserved.
PySpark Driver Program
sc.textFile(“hdfs://…”, 4) .map(to_series) .filter(has_outlier) .count()
Function closures need to be executed on worker nodes by a python process.
‹#›© Cloudera, Inc. All rights reserved.
How do we ship around Python functions?
sc.textFile(“hdfs://…”, 4) .map(to_series) .filter(has_outlier) .count()
‹#›© Cloudera, Inc. All rights reserved.
Pickle!
sc.textFile(“hdfs://…”, 4) .map(to_series) .filter(has_outlier) .count()
‹#›© Cloudera, Inc. All rights reserved.
Best Practices for Writing PySpark
‹#›© Cloudera, Inc. All rights reserved.
REPLs and Notebookshttps://flic.kr/p/5hnPZp
‹#›© Cloudera, Inc. All rights reserved.
Share your code
https://flic.kr/p/sw2cnL
‹#›© Cloudera, Inc. All rights reserved.
Standard Python Projectmy_pyspark_proj/ awesome/ __init__.py bin/ docs/ setup.py tests/ awesome_tests.py __init__.py
‹#›© Cloudera, Inc. All rights reserved.
What is the shape of a PySpark job?
https://flic.kr/p/4vWP6U
‹#›© Cloudera, Inc. All rights reserved.
• Parse CLI args & configure Spark App • Read in data • Raw data into features • Fancy Maths with Spark • Write out data
PySpark Structure?
https://flic.kr/p/ZW54
Shout out to my colleagues in the UK
‹#›© Cloudera, Inc. All rights reserved.
PySpark Structure?my_pyspark_proj/ awesome/ __init__.py DataIO.py Featurize.py Model.py bin/ docs/ setup.py tests/ __init__.py awesome_tests.py resources/ data_source_sample.csv
• Parse CLI args & configure Spark App • Read in data • Raw data into features • Fancy Maths with Spark • Write out data
‹#›© Cloudera, Inc. All rights reserved.
Simple Main Method
‹#›© Cloudera, Inc. All rights reserved.
• Write a function for anything inside an transformation • Make it static
• Separate Feature generation or data standardization from your modeling
Write Testable Code
Featurize.py …
@static_method def label(single_record): … return label_as_a_double @static_method def descriptive_name_of_feature1(): ... return a_double
@static_method def create_labeled_point(data_usage_rdd, sms_usage_rdd): ... return LabeledPoint(label, [feature1])
‹#›© Cloudera, Inc. All rights reserved.
• Functions and the contexts they need to execute (closures) must be serializable • Keep functions simple. I suggest static methods. • Some things are impossiblish • DB connections => Use mapPartitions instead
Write Serializable Code
https://flic.kr/p/za5cy
‹#›© Cloudera, Inc. All rights reserved.
• Provides a SparkContext configures Spark master • Quiets Py4J • https://github.com/holdenk/spark-testing-base
Testing with SparkTestingBase
‹#›© Cloudera, Inc. All rights reserved.
• Unit test as much as possible • Integration test the whole flow • Use sample of real data
• Test for: • Deviations of data from expected format • RDDs with an empty partitions • Correctness of results
Testing Suggestions
https://flic.kr/p/tucHHL
‹#›© Cloudera, Inc. All rights reserved.
Best Practices for Running PySpark
‹#›© Cloudera, Inc. All rights reserved.
Writing distributed code is the easy part…
Running it is hard.
‹#›© Cloudera, Inc. All rights reserved.
Get Serious About Logs
• Get the YARN app id from the WebUI or Console • yarn logs <app-id> • Quiet down Py4J • Log records that have trouble getting processed • Earlier exceptions more relevant than later ones • Look at both the Python and Java stack traces
‹#›© Cloudera, Inc. All rights reserved.
Know your environment
• You may want to use python packages on your cluster • Actively manage dependencies on your cluster
• Spark versions <1.4.0 require the same version of Python on driver and workers
‹#›© Cloudera, Inc. All rights reserved.
Complex Dependencies
‹#›© Cloudera, Inc. All rights reserved.
Many Python EnvironmentsPath to Python binary to use on the cluster can be set with PYSPARK_PYTHON
Can be set it in spark-env.sh
if [ -n “${PYSPARK_PYTHON}" ]; then export PYSPARK_PYTHON=<path> fi
http://blog.cloudera.com/blog/2015/09/how-to-prepare-your-apache-hadoop-cluster-for-pyspark-jobs/