Hadoop and Machine Learning

Machine Learning and HadoopPresent and FutureJosh Wills, Tom Pierce, and Jeff HammerbacherCloudera Data Science TeamDecember 17th, 2011

High Availability for Data Scientists

Copyright 2011 Cloudera Inc. All rights reserved

NIPS

Agenda

• Part 1: Industrial Machine Learning• Part 2: Machine Learning and Hadoop

• State of the World• Where Things Are Headed

• Part 3: Things Industry Needs From Academia


Industrial Machine Learning


Delta One: Model Evaluation

• ML Systems Are One Piece of a Complex System• Well-defined objective functions are the exception

• Multiple, often conflicting goals• Weights are fuzzy and shift with business priorities• Pareto optimization is the safest play

• Predictive Accuracy Is Only Useful Up to a Point• Examples

• Computational advertising• Friend recommendations on social networks


Delta Two: Systems Precede Algorithms

• Greenfield Projects Hardly Ever Happen• (and don’t usually launch)

• Industrial Computational Infrastructure• General-purpose• Cheap• Shared

• Constraints Drive Innovation• Vowpal Wabbit Hashing Trick• SETI @ Google


Delta Three: Workflow


Practice Over Theory Blog

http://practiceovertheory.com/blog/2010/02/15/machine-learning-who-s-the-boss/

Delta Three: Workflow

• Optimize the Overall Process• Model fitting is a small piece of the overall flow time• Parallelize everything

• Better Features > Better Models• Fast Model Deployment

• Common Feature Extraction Logic• Servable Models

• Validation as Sanity Checking• Deploy to a small subset of real data and evaluate


Agenda





Hadoop: It’s Where The Data Is


Hadoop Platform: Substrate

• Commodity servers• Open Compute

• Open source operating system• Linux

• Open source configuration management• Puppet• Chef

• Coordination service• ZooKeeper


Hadoop Platform: Storage

• Distributed schema-less storage• HDFS• Ceph

• Append-only storage formats and metadata• Avro• RCFile• HCatalog

• Mutable key-value storage and metadata• HBase


Hadoop Platform: Integration

• Tool Access• FUSE• JDBC• ODBC

• Data Ingestion• Flume• Sqoop


ML and Hadoop: The State of the World


Computation: Plain Old MapReduce

• Great for:• Data Preparation• Feature Engineering• Model Validation/Evaluation

• Works For Certain Model Fitting Problems• Recommendation Systems• Decision Trees (PLANET; Gradient Boosted Decision Trees)

• Not A Practical Option for Online Learning• Way More Detail from the KDD 2011 Talk


http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/36296.pdf

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CDkQFjAC&url=http://lccc.eecs.berkeley.edu/papers.html&ei=D8PrTqK_KonJhAfJ6OypCA&usg=AFQjCNHKovKUzaNJMDIDtinhreVtu7U82g

http://www.slideshare.net/hadoop/modeling-with-hadoop-kdd2011

Tools for Data Preparation/Feature Engineering

• Languages/Environments• PigLatin• HiveQL• Need to deal with mismatch between offline/online feature

generation

• Java/Scala APIs• Crunch (Cloudera)• Scoobi (NICTA)• Cascading (Concurrent)• Jaql (IBM)


https://github.com/cloudera/crunch

https://github.com/NICTA/scoobi

http://www.cascading.org/

Apache Mahout

• The starting place for MapReduce-based machine learning algorithms• Not machine-learning-in-a-box• Custom tweaks/modifications are the rule

• A disparate collection of algorithms for:• Recommendations• Clustering• Classification• Frequent Itemset Mining


Apache Mahout (cont.)

• Best Library: Taste Recommender• Oldest project, most widely-deployed in production• SVD implementation is particularly active

• Good Libraries: Online SGD• Does not use MapReduce• Vowpal Rabbit + AllReduce is faster, has L-BFGS option

• Roll Your Own Instead: Naïve Bayes• Challenges

• “Secret sauce” effect• Delta between Mahout + the cutting edge in ML


More Machine Learning Interfaces for Hadoop

• Based on MapReduce• SystemML (IBM)• AllReduce (Vowpal Wabbit)

• No MapReduce• Spark

• R-Based Systems (Augment MapReduce with R)• Segue• RHIPE• RHadoop• Ricardo (IBM)


https://github.com/JohnLangford/vowpal_wabbit/wiki

http://www.spark-project.org/

http://code.google.com/p/segue/

http://ml.stat.purdue.edu/rhipe/

https://github.com/RevolutionAnalytics/RHadoop

ML and Hadoop: Where Things are Headed


MRv2 and YARN

• Eliminates JobTracker bottleneck• Separate Resource Manager/Scheduler• Individual jobs have their own task masters

• Moves MapReduce into user-land• Enables Hadoop clusters to run all sorts of jobs

• MPI (Hamster; MAPREDUCE-2911)• Native BSP (Giraph)• Spark• AllReduce, GraphLab


https://issues.apache.org/jira/browse/MAPREDUCE-2911

https://issues.apache.org/jira/browse/GIRAPH-13

https://github.com/mesos/spark-yarn

Agenda





Machine Learning on Multivariate Time Series

• 1e5 writes/sec• Positive events are

relatively rare• Feature extraction

challenge• May not be clear what

the right time horizon is• Tight SLAs• Very high stakes


An Academic Language For Feature Engineering

• Feature extraction/selection is as important as model fitting• e.g., hierarchical feature representation, impact on training

time and experiment design, feature cost modeling, etc.

• Academic literature on this problem is sparse and dispersed across multiple fields• NIPS 2003• HCI, NLP, Information Retrieval, etc.

• We need a common language for talking about these problems across disciplines


http://clopinet.com/isabelle/Projects/NIPS2003/

A Broader Ontology For Model Selection

• Practical factors that enter into the “best” choice of model…• Data arrival rate• Data volume• Scoring latency• Model refresh time• Robustness/reliability

• …in addition to the standard predictive power/simplicity tradeoffs


Questions?Want A Job?

@josh_wills

Hadoop and Machine Learning

Technology

Transcript of Hadoop and Machine Learning