Building Data Science Teams: A Moneyball Approach

1© Cloudera, Inc. All rights reserved.

A Moneyball ApproachJosh Wills | Senior Director of Data Science

Building Data Science Teams


About Me


A Team Building Exercise


Data Scientist Supply vs. Data Scientist Demand


Recruiting Techniques


Moneyball and Data Science


Choosing The Right Metrics


1. Analyzing “Unstructured” Data Sources


2. Building Machine Learning Models


3. Turn Static Reports Into Analytical Applications


Answering More Questions in Less Time


How To Answer QuestionsLike A Data Scientist


1. Read and deserialize input data.

2. Project/filter input records.

3. Shuffle: serialize it, send over the network, deserialize it.

4. Apply aggregation logic.

5. Serialize output data.

The Life of a Data Processing Job


Handling the Cost of Serialization


The Traditional RDBMS Approach


The Cost of The Traditional RDBMS Approach


Query Scheduling and Exploratory Data Analysis


The Spark Approach


The Cost of the Spark Approach


The MapReduce Approach


MapReduce In The Hands of a Data Scientist


Example: Hive Multi-Insert


Our Goal: Public Transit for Questions


Data Modeling for Data Scientists


Motivating Example: Spelling Correction


Event Series Analytics


A Simple Star Schema for Spell Correction


The Combinatorial Explosion


• What parameters does this model need…• during the analysis phase?• during deployment?

• Some Candidates• Lag time between events• Similarity of queries• What else?

Designing the Spell Correction Data Product


A Supernova Schema for Search


Spell Correction in SQL


Exhibit: http://github.com/jwills/exhibit


Querying Nested Types with Impala


• Core Metric: # Outputs/ # Jobs• Measure on both an individual and

aggregate level• Drive the marginal cost of asking one

additional question towards zero• Point business analysts at output

tables for interactive analysis with Impala• Self-serve BI frees up resources

(compute + data science time)

Trading Up: From Data Analyst to Data Scientist


Thanks!@josh_wills

Building Data Science Teams: A Moneyball Approach

Data & Analytics

Transcript of Building Data Science Teams: A Moneyball Approach