Post on 23-Feb-2016
description
Ranking AlgorithmsHow to determine ranking out of complex data
types
Ranking Topics in a Presentation
Topic Supports Presentation Theme (.25)
Supports Class Understanding
(.75)
Weighted Total
Examples and Cases
.55 .9 .8125
Quarterback Rating example
.6 1 .9
Target Example .5 .8 .725Data Pre processing
.77 .7 .7175
Why .7 .7 .7How .8 .7 .725Outcomes .8 .7 .725Scaling Data For Indexing
.87 .7 .7167
Decision trees .8 .8 .8Cluster Analysis .9 .6 .675Support Vector Machines
.9 .6 .675
Collaborative Filtering
.6 .7 .675
Drawing Conclusions
.5 .7 .65
Multiple regression .5 .8 .725Anova .5 .7 .65Meta .5 .6 .575
Conclusion 1 .2 .4
Ranking QBs in the NFL: Passer Rating
• This formula is meant to measure a quarterback’s passing performance with a single numeric value.
• The passer rating scale is from 0 to 158.3
• 5 variables(completions, yards, touchdowns, interceptions and attempts)
• Each statistic is weighted, prior to the second step.• Each statistic is applied a Min/Max to make sure one outlier statistic
does not dominate the formula.
Advantages and Flaws of Passer RatingPros:
• Provides a quick and easy way of evaluating quarterback performance based on a standard formula.
• Captures key variables associated with passer performance.Cons:• Incomplete measurement: doesn’t take into consideration dropped
passes, times sacked, yards after catch, fumbles, Etc.• Can be deceptive if the amount of attempts is small.• Amount of touchdown passes scored is largely dependent on other
players.Scenario: Below are last games statistics. Which quarterback should the coach start next game?
Philip Rivers (current starter): PR = 108.7Joseph Gast (backup): PR = 158.3
Target: Using ranking algorithms to predict
pregnancyThe goal: Since birth records are public, new parents are bombarded with marketing and advertising offers. Target’s goal was to identify parents before the baby was born.More specifically, target wanted to be able to identify pregnant women in the second trimester and send them coupons for diapers, car seats, etc.The outcome: Target was successful! Women thought it was creepy. The PR following effected Target negatively.The solution: Continue to “target” (haha) pregnant women with relevant ads, however include purposefully non-relevant ads so they do not notice.
Target: How to determine what metrics to capture
• Target collected vast data on the purchase habits customers already for various other reasons (I.E: Christmas toys). • Andrew Pole (Target’s senior analyst of consumer habits)
started mining data from the baby shower registry.Pole found interesting changes in buyer behavior as their due date approaches, such as:
• Buying more unscented lotion• Buying magnesium, zinc and
calcium supplements.• Buying unscented soaps.• Buying larger quantities of cotton
balls, hand sanitizers and washcloths
Target: Putting pregnancy prediction score to use
• In the end, Pole identified 25 products that, when analyzed together, allowed him to assign each shopper a “pregnancy prediction” score.
• He also could predict shopper’s due dates well enough to send coupons timed to specific stages pregnancy.
• Target then ran studies on how to advertise to pregnant women and found out that the coupons were more likely to be used when coupled with “random” items.
“Just wait. We’ll be sending you coupons for things you want before you even know you want them.” –Andrew Polehttp://rmportal.performedia.com/node/1373
Data Pre-Processing 1: Why data must be pre-processed• Organizations often require decisions to be formed from
multi-criteria datasets; However:• Original data often suffers from:• Lacking attributes• Lacking values• Containing aggregates
• Data obtained from different sources are often inconsistent (using different attribute names, invalid codes, of different data types)
• Datasets may contain errors or outliers
Data Pre-Processing 2: How Preprocessing Works
• Fill in missing values using central tendency along with learning algorithms to predict value• Cluster values to isolate outliers• Smooth data using regression• Correct inconsistencies using decision-making
techniques
Data Pre-Processing 3: Outcomes of Preprocessing
• The purpose of data pre-processing is to produce a better data set without loss of relevant information. This:• Allows statistical analysis on incomplete datasets• Allows resulting dataset to be uploaded to data visualization
software• Can be used to test and track many relationships between
variables• Provides a method of ranking similar data from disparate
sources
Scaling Data for Indexing: Decision Trees• Most common form of index
• Uses B-Tree structure to parse values• Quick and easy given simple
metrics• Created by dividing groups of data
roughly in half and putting values into each half.• Process is repeated until each
“decision” contains exactly one value
Scaling Data for Indexing: Support-Vector Machines
• Uses machine learning to generate probability• Requires training and test data• Complexity grows exponentially with size of
training data• Used to coerce non-standard information into
standard classifications (e.g. handwriting recognition algorithms)• Sample of Support-Vector Application
Scaling Data for indexing: Cluster Analysis
• Starts with decision-tree type analysis• Data elements with arranged as objects via their
attributes• Similar objects are arranged in clusters• As clusters get too large, new clusters are formed• New data is compared against cluster ranges, indexed
accordingly• Data retrieval looks at cluster first, then objects within
appropriate clusters
Collaborative Filtering
• Crowd-sourcing rankings based on users likes/dislikes• Gives users test set of data to rank on Likert scale• Likert scale is dropped, relative ranking is retained• Users are matched with others with similar tastes,
providing ability to predict new object ranks
Drawing Conclusions: Multiple Regression
Uses multiple variables to predict a linear relationship.One dependent variable; k explanatory variables.
β = slope termsMultiple Coefficient of Determination = R2 R2 always increases the more you add explanatory variables, however this does not mean the model is better. = Adjusted R2; weighs errors more heavily by penalizing the model for adding bad explanatory variables.
Drawing Conclusions: Factorial Anova
• Used when you have 1 or more categorical independent variables. (otherwise use multiple regression)
• Tests the extent to which one variable depends on Changes in other variables.
• Great for analyzing the interaction of categorical variables. And how they relate to other variables.• The effect is measured by dividing data into categories and comparing the sum of squares mean for each category to the sum of squares total.
Drawing Conclusions: Meta Analysis• Meta-analysis contrasts and combines the results of different
studies. Usually ones with small sample sizes.• A meta analysis measures to what extent different studies on a
common topic produced the same effect.• The inverse of the variance (σ2) is often used as a weight so that larger samples have a greater effect.Pros:• Improved precision and accuracy
estimates due to more data • Results can be generalized to
larger populations.• A hypothesis test can be applied
on summary estimates.
Cons:• Publication bias: negative
results are less likely to be published.
• Agenda driven bias: cherry picked studies.
• Simpson’s paradox
Conclusion• Using rankings for analyzing multiple
variables is an extremely useful tool for expedited decision making.
• Data preprocessing allows statistical analysis for incomplete datasets (improves data hygiene).
• Indexes are scaled by common attributes: • A decision tree deals with 1 attribute • a cluster deals with multiple attributes • support vector machines rely on computer generated
algorithms for identifying similarity.• Statistical tests such as regression
analysis, factorial Anova analysis and Meta analysis can be used to draw conclusions regarding whether or not variables are related to each other.
Questions?