Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
Transcript of Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
1/24
Faraday faraday.io
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
2/24
Faraday
Tackling the customer acquisition challenge2
Introduction
TH E CUSTOMER ACQUISITION PR OB LEM
Chances are that if youre a company selling a considered consumer purchase
(>$1,000 ticket size), you have a problem: acquiring customers is too expen-
sive.
At each step of the big-ticket customer acquisition process there are costs, in-
cluding marketing to prospects, engaging with leads, and potentially, depend-
ing on your industry, on-site assessments. This problem is exacerbated by
that fact that customer engagement around considered purchases is notori-
ously difficult. For example:
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
3/24
FARADAY WHIT EPAPER SERIES 3
faraday.io
The average U.S. consumer spends just seven minutes a year interact-
ing with their electricity provider.1
41% of individuals during the open enrollment period for health care
in 2013 spent 15 minutes or less researching their options. 2
OUR SOLUTION
To begin to address these challenges, Faraday has developed
a data-driven marketing platform optimized for companies
selling considered purchases. The platform includes a machine
learning-driven prediction engine and automated marketing
instrumentation, empowering marketers to identify promising
prospects, target them, and track outcomes.
To test the platform, Faraday teamed up with one of its clients
in Massachusetts. The goal was to use the Faraday platform
to help the client reduce acquisition costs by more effectively
identifying and engaging with the individuals most likely to
accept. This paper presents our approach and the results.
1 Accenture: Actionable Insights for the New Energy Consumer
2 USA Today: Many Spend 15 Minutes or Less Picking Health Insurance
MACHINE LEARNING is a classof techniques that allow computers
to learn information that was not e
plicitly provided. Machine learning
is widely used in predictive data
analysisfamiliar examples includ
the Netflix and Amazon recommen
dation engines. Faradays platform
uses supervised classification algo
rithms to identify patterns in samp
datasets that can be generalized to
make predictions for new data.
http://www.accenture.com/us-en/Pages/insight-actionable-new-energy-consumer.aspxhttp://www.usatoday.com/story/money/personalfinance/2014/09/04/health-insurance-plans-costs/15032405/http://www.usatoday.com/story/money/personalfinance/2014/09/04/health-insurance-plans-costs/15032405/http://www.accenture.com/us-en/Pages/insight-actionable-new-energy-consumer.aspx -
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
4/24
Faraday
Tackling the customer acquisition challenge4
The project
BACKGROUND
The project was designed to test whether Faradays data-driven marketing
platform would be an asset for driving down customer acquisition costs for
our client. The project used property data, consumer data, and historical
CRM (marketing and customer data) for 15 towns in the Greater Boston met-
ropolitan area to develop a model that predicts the likelihood that a household
would invest in our clients product. We tested the model with a direct mail
campaign sent to a mix of high-scoring and randomly selected households,
none of which were existing customers. We instrumented the campaign to
allow automatic real-time response tracking, making it easy to evaluate per-
formance and incorporate the findings into an improved model for follow-on
campaigns.
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
5/24
FARADAY WHIT EPAPER SERIES 5
faraday.io
ELIGIB ILITY
To match the eligibility requirements for our clients product we excluded
non-residential buildings and multi-family buildings with more than four
units. We also excluded buildings built after 1980 because past market seg-
mentation strategies left us with insufficient data on newer homes. There
were 93,823 housing units that satisfied these criteria, representing just
under half of the 15 towns combined housing stock.
Table 1.Number of eligible homes in each project town3
TOW N H OU SIN G UN ITS E LIGIBL E U NITS PE RC EN T EL IGIBLE
Acushnet 3,985 2,312 58%
Ashland 6,578 2,324 35%
Cambridge 48,278 16,337 34%
Carver 4,506 1,685 37%
Dedham 10,098 6,895 68%
Fairhaven 7,446 4,978 67%
Framingham 27,535 14,955 54%
Holliston 5,051 3,552 70%
3 Housing units data from U.S. Census Bureau American Community Survey (2012)
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
6/24
Faraday
Tackling the customer acquisition challenge6
TOW N HO USIN G U NITS E L IG IBL E UN ITS PERC E NT E LIGIBL E
Hopkinton 5,038 1,935 38%
Maynard 4,425 2,530 57%
Needham 10,822 7,402 68%
Plymouth 25,288 10,389 41%
Sherborn 1,452 1,102 76%
Somerville 32,471 13,773 42%
Westwood 5,312 3,654 69%
DEMOGR APH ICS
The 15 towns range in size from Sherborn, with a population of about 4,000,
to Cambridge, with over 100,000 residents, and have a combined population
of approximately 472,000 living in almost 200,000 housing units.
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
7/24
FARADAY WHIT EPAPER SERIES 7
faraday.io
Figure 1.Map of project towns
At one end of the socioeconomic spectrum Sherborn has a median household
income of $150,000 and 83% of the residents over age 25 have a bachelors de-
gree. At the other end Fairhaven has a median household income of $60,000
and only 23% of its residents over age 25 have a bachelors degree.
The towns housing stock shows similar variation. In Somerville 90% of hous-
ing units were built before 1980, just 15% are single-family homes, and two
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
8/24
Faraday
Tackling the customer acquisition challenge8
thirds of residents rent. In contrast over half of Hopkintons housing stock
was built since 1980, 90% is single-family homes, and only 10% of residents
are renters. Carver is a unique case as the only town with a significant num-
ber of mobile homes, with these accounting for almost a quarter of the towns
housing units.
Table 2.Size and socioeconomic characteristics of project towns4
TOWN POPULATION MEDIAN INCOME POVERTY BACHELOR'S DEGREE
Acushnet 10,302 $65,222 5% 22%
Ashland 16,587 $95,296 4% 55%
Cambridge 105,026 $72,225 14% 74%
Carver 11,497 $67,963 7% 22%
Dedham 24,716 $82,193 5% 45%
Fairhaven 15,893 $59,933 10% 23%
Framingham 68,689 $68,906 10% 45%
Holliston 13,668 $107,192 4% 60%
Hopkinton 14,982 $127,821 2% 69%
Maynard 10,151 $79,441 4% 49%
Needham 29,005 $125,170 4% 73%
Plymouth 56,574 $77,228 6% 33%
4 ibid
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
9/24
FARADAY WHIT EPAPER SERIES 9
faraday.io
TOWN POPULATION MEDIAN INCOME POVERTY BACHELOR'S DEGREE
Sherborn 4,128 $151,944 3% 83%
Somerville 75,974 $64,603 16% 53%
Westwood 14,616 $130,824 3% 68%
Table 3.Housing unit characteristics of project towns5
TOWN MEDIAN VALUE S INGLE-FAMILY PRE-1980 RENTERS
Acushnet $299,200 81% 64% 16%
Ashland $353,900 76% 50% 19%
Cambridge $545,800 16% 80% 64%
Carver $269,100 74% 51% 8%
Dedham $374,700 72% 80% 27%
Fairhaven $272,500 76% 82% 27%
Framingham $349,800 54% 85% 44%
Holliston $386,000 85% 78% 13%
Hopkinton $510,000 93% 44% 10%
Maynard $333,000 69% 80% 35%
Needham $651,300 81% 77% 17%
Plymouth $336,200 80% 62% 20%
5 ibid
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
10/24
Faraday
Tackling the customer acquisition challenge10
TOWN MEDIAN VALUE SINGLE-FAMILY PRE-1980 RENTERS
Sherborn $721,100 94% 76% 10%
Somerville $441,300 15% 90% 67%
Westwood $620,400 84% 76% 12%
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
11/24
FARADAY WHIT EPAPER SERIES 11
faraday.io
Methodology
DATA SOURCES
MassGIS The base input to our model was publicly available
town assessor and geospatial data from MassGIS.6This mainly
provided building variables such as location, area, year built,
type, number of units, assessed value, and last sale date.
U.S. Census We also added block-level data from the U.S.
Census such as median household income and educational
achievement.
6 MassGIS
While this project made use of pub
licly-available datasets like MassG
and the U.S. Census, the Faraday
platform is built on a nationwide
database which comprises multi-
ple commercially licensed house-
hold-level datasets.
http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/ -
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
12/24
Faraday
Tackling the customer acquisition challenge12
Consumer data We enhanced the model with consumer data purchased
from a national data aggregator. This included a large number of variables
such as owner/renter, occupation, and environmental inclination.
Clients CRM data We used historical Cus-
tomer Relationship Management (CRM) data,
which included previous marketing efforts that
previously accepted or rejected an offer for our
clients product. Homes that accepted an offer
were easy to identify, but many homes simplydid not respond to prior marketing efforts. Fur-
ther investigation revealed that unresponsive
homes with three or more outreach attempts
were highly unlikely to go on to our clients offer, so we classified these homes
as did not invest. Unresponsive homes with fewer than three outreach at-
tempts were excluded from the analysis.
PR EDICTIVE MODEL
We used an iterative process to create a predictive model across all 15 pilot
towns. Working with only the homes classified as invested or did not in-
vest we created a large number of feature sets by randomly selecting subsets
of the available variables. For each of these feature sets:
INVESTMENT is a generic term for
a familys purchase or adoption of a
good or service.
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
13/24
FARADAY WHIT EPAPER SERIES 13
faraday.io
1. We randomly split the homes into a training set containing 70% of the
homes and a test set containing the remaining 30%.
2. We used the training set to train a random decision forest ensemble
to predict whether each home did or did not accept an air sealing or
insulation offer.
3. We evaluated the model by using it to predict the out-
come for the homes in the test set and comparing the
predictions to the actual outcomes.
We performed this split, train, and evaluate process 10 times
for each feature set and averaged the results. We chose the
best-performing feature set and used it to train a random forest
model on the combined training and test sets. The resulting
model takes the feature sets variables as inputs and outputs a
score from 0-100 representing the likelihood that a household
will invest in our clients product.
MAILING LIST
We used the model to score all eligible homes in the 15 pilot
towns. We then created a mailing list of 28,095 homes that had
never purchased our clients product. Half were a randomly
RANDOM DECISION FORESTSare a technique in machine learning
a branch of artificial intelligence.
They involve splitting datasets
with known outcomes at randomly
chosen points in randomly chosen
attribute dimensionsrepeatedly.
Sometimes, these splits increase
the models ability to discern good
outcomes from bad; these splits
are kept, others are discarded. The
resulting forest of decision trees
can subsequently vote on a new
candidates likely outcome based o
where its attributes fall.
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
14/24
Faraday
Tackling the customer acquisition challenge14
chosen control group and half were the highest-scoring homes not already
allocated to the control group.
INSTRUMENTATION
Faraday provided a unique phone number to be printed on each mailer. Calls
to the number are automatically linked to the mailer recipient in a databaseand are viewable in real-time through Faradays campaign tracking tool.
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
15/24
FARADAY WHIT EPAPER SERIES 15
faraday.io
Results
SUMMAR Y METR ICS
We used four metrics to evaluate feature sets for the final model:
Precision The fraction of homes the model predicted would invest that did
invest. The higher the precision, the more likely it is that homes the model
predicts will invest do invest.
Recall The fraction of investors that the model predicted would invest. High-
er recall means fewer missed opportunities; fewer investors that the model
predicted would reject.
Accuracy The fraction of the models predictions that were correct. Higher
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
16/24
Faraday
Tackling the customer acquisition challenge16
accuracy means the model made fewer mistakes in predicting both investors
and rejectors.
F1 score The harmonic mean of precision and recall. A higher F1 score
means the model is better at picking out investors. The key difference be-
tween F1 score and accuracy is that F1 score is more influenced by correct
investor predictions. Since we care more about identifying investors than
rejectors the F1 score is more relevant than accuracy.
Table 4 lists the metrics for the best-performing feature set. As describedin the methodology section these values are averaged across 10 models and
were calculated by making predictions for data that the models had never
seen. Thus they give an indication of how the model should perform in future
marketing efforts.
Table 4.Summary metrics for top feature set
PRECISION RECALL ACCURACY F1 SCORE
80% 31% 72% 45%
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
17/24
FARADAY WHIT EPAPER SERIES 17
faraday.io
TOP PREDICTIVE VARIABLES
Faradays model analyzed more than 130 variables describing housing unit
characteristics and consumer behavior. The two most predictive variables
were the household heads occupation and influence rating on social net-
works. Other top variables included the number of children, home value, and
a buildings siding type.
While these variables proved most predictive for the pilot project, Faradayhas found that predictive variables for one geographic area or product do
not necessarily perform well in other areas or with different products. This
means a universal model will be less predictive than a model tuned to the
unique demographic, financial, and property characteristics of a particular
market.
ROC CHART
A receiver operating characteristic or ROC chart is useful for deciding how to
use a models predictions. It shows how recall and false positive rate change
as the discrimination threshold varies. False positive rate is the fraction of
rejectors that the model predicted would invest. The discrimination thresh-
old is a cutoff score below which we no longer predict that a household will
invest.
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
18/24
Faraday
Tackling the customer acquisition challenge18
Figure 2 is based on the final models predictions for all investors and rejec-
tors. The black curve shows the result of using different cutoff scores while
the grey diagonal shows the result of randomly guessing whether each home
will invest. The black curve labels indicate the cutoff score. The greater the
distance between the curve and diagonal the bigger the performance increase
from using the model.
Above a cutoff score of 66 the model yields a huge improvement: over 60% of
the investors are included with less than 1% of the rejectors. Below this point
the model adds rejectors at a greater relative rate than investors.
Figure 2.ROC chart
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
19/24
FARADAY WHIT EPAPER SERIES 19
faraday.io
GAINS
A gains chart illustrates the extent to which a smaller marketing effort
informed by a predictive model could have resulted in some or all of the
investments. In Figure 3 the black curve shows the result of targeting homes
in descending order by score. The grey diagonal shows the result of random
targeting. The black curve labels indicate the cutoff score. The greater the
distance between the curve and diagonal the bigger the performance increase
from using the model.
A list of the top 25% of homes by score contains over 60% of the investors and
a list of the top 50% of homes contains over 80% of the investors. This sug-
gests that future marketing based on the model could cut costs in half if a 20%
reduction in investments is acceptable.
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
20/24
Faraday
Tackling the customer acquisition challenge20
Figure 3. Gains chart
MAILING LIST PER F OR MANCE
The ultimate test of the models performance will be the relative rates at
which the model-selected and randomly selected mailer recipients invest
in our clients product. Four months after receiving the offer, recipients
showed a statistically significant 25% higher response rate than the control
group. The percentage of customers that signed contracts to complete further
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
21/24
FARADAY WHIT EPAPER SERIES 21
faraday.io
weatherization improvements was 33% higher for the top scoring group with-
in 60 days of receiving a letter.
Figure 4.Mailing list response rate
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
22/24
Faraday
Tackling the customer acquisition challenge22
Conclusion
Both the model summary metrics and the initial results of the test mailingsuggest Faradays data-driven customer targeting increases customer acqui-
sition rate. The model ranked over 80% of past investors in the top 50% of
all households by likelihood to invest and the pilot mailing shows an initial
33% increase in conversion rate. The models performance may improve for
follow-on campaigns as it leverages data from previous campaigns to refine
its predictions.
The increase in response rate and conversion rate means that in the pilot
area our client is on track to save almost $13,000 on mailers and $311,000 on
reduced sales costs compared to a traditional campaign. If we assume a more
conservative 20% lift for a statewide campaign, our client could mail 17%
fewer homes resulting in savings of $540,000 on mailers and $13 million in
sales costs.7
7 Assuming an average of 1.5 mailers per home, $1 per mailer, and $400 in sales costs.
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
23/24
FARADAY WHIT EPAPER SERIES 23
faraday.io
-
7/23/2019 Faraday-big Ticket Blues-tackling the Customer Acquisition Challenge
24/24
Matt Pell
(802) 458-0441 x341
Scott Pellegrini
(802) 458-0441 x340
Faraday