Net Promoter Score and Semantic S imilarity of Clients
description
Transcript of Net Promoter Score and Semantic S imilarity of Clients
Hierarchical Recommender System for Improving NPS.
www.kdd.uncc.edu
CCI, UNC-Charlotte
Research sponsored by:presented by
Zbigniew W. Ras
CONSULTING COMPANY in Charlotte
Client 1 Client 2 Client 3 Client 4
Build Recommender Systemfor each client (34 clients) helping toincrease its revenue
Build Personalized Recommender Systemfor each shop helping toincrease its revenue
Services(heavy equipment repair)
Parts CUSTOMERS
shops shops shops
What we have to do
Net Promoter Score (NPS) – today’s standard for measuring customer loyalty
Promoter - {9,10}
Passive – {8,7}
Detractor – {1,2,3…,6}
NPS of a company is correlated with its revenue growth
NPS rating for all clients
NPS rating for all shops
What is our goal?
Build recommender system giving possibly the simplest and the best advice to every client/shop on the changes they need to make in order to improve their NPS ratings.
Initial Dataset – provided by Consulting CompanyAbout 100,000 records representing answers to a questionnaire collected from over 35,000 randomly chosen customers in 2010 -2013.Customers use services provided by 34 clients
A questionnaire consists of:
Information about the customerCustomer’s name, location, phone number…
Information about the serviceClient’s name, invoice amount, service type…
Information on customers’ feeling about the servicewas the job completed correctlyare you satisfied with the joblikelihood to refer to friends…
Benchmarks
Personal
BenchmarkAllOverallSatisfaction 35
BenchmarkAllLikelihoodtobeRepeatCustomer 34
BenchmarkAllDealerCommunication 33
BenchmarkServiceRepairCompletedCorrectly 32
BenchmarkReferralBehavior 31
BenchmarkServiceFinalInvoiceMatchedExpectations 31
BenchmarkEaseofContact 30
BenchmarkAllDoesCustomerhaveFutureNeeds 28
BenchmarkServiceTechPromisedinExpectedTimeframe 26
BenchmarkServiceRepairCompletedWhenPromised 26
BenchmarkServiceTimelinessofInvoice 25
BenchmarkServiceAppointmentAvailability 24
BenchmarkServiceTechEquippedtodoJob 23
BenchmarkAllContactStatusofFutureNeeds 22
BenchmarkServiceTechArrivedWhenPromised 21
BenchmarkAllHasIssueBeenResolved 19
BenchmarkAllContactStatusofIssue 17
BenchmarkServiceTechnicianCommunication 6
BenchmarkServiceContactPreference 3
BenchmarkCBCallansweredpromptly 1
BenchmarkServiceReceivedQuoteforRepair 1
BenchmarkCBAutoattendantansweredbycorrectdepartment 1
BenchmarkServiceCallAnsweredQuickly 1
BenchmarkAllMarketingPermission 1
Randomly chosen customers are asked to complete Questionnaire – It has questions concerning personal data + 30 benchmarks
To compute NPS we calculate average score of all benchmarks for all customers. Knowing thenumber of promoters and detractors we know NPS.
Data
Data Preprocessing (including reduction of benchmarks)
Knowledge Extraction (Classical Tools - WEKA + Our Software for Extracting Action Rules & Their Triggers)
Recommender Systems
Classical Approach
ClassificationSelection of Best Classification Algorithm
11
J48
RandomForest
KNN
NaiveBayes
BayesNet
RBF
PART
0 10 20 30 40 50 60 70 80
0.4 0.405 0.41 0.415 0.42 0.425 0.43 0.435
AccuracyTime Taken(secs)
PART RBF BayesNet NaiveBayes KNN RandomForest J48
Accuracy 2 3 4 1 5 7 6
Time Taken 2 4 6 7 3 1 5
Total score 4 7 10 8 8 8 11
Initial Clients Datasets Recommender Systems local approach global approach
Using Information about 34 Clients to Enlarge these Datasets
2 global approach
More Powerful Recommender Systems
1
2
Two Options
Semantic Distance between Clients
attr1
attr2
attr3 attr4
attr3
attr2
attr2
attr1
attr3 attr4
attr3
attr1
Vector describing customer 1
Vector describing customer 2
Vector describe ing customer 3
Dataset for Client-1Vector describing customer 1
Vector describing customer 2
Vector describe ing customer 3
Dataset for Client-2J48 classifier extracted from dataset D1 of Client-1
J48 classifier extracted from dataset D2 of Client-2
More similar these two classification trees, more close semantically the clients are
FIRST APPROACH for Dataset Enlargement
Semantic distance based dendrogramfor Service
n5
n6
n7n3
n2n1
Recommender System Engine based on Semantic Similarity of Clients
NPS(n1)Table1Classifier1F-score1
NPS(n2)Table2Classifier2F-score2
Start with n1.If NPS(n1) has to be improved, “move to n3”If NPS(n2) > NPS(n1), thenTable3 := Table1 Table 2 is assigned to n3.Classifier from Table3 is extracted and itsF-score is computed. If F-score3 > F-score1, then “move to n5”, otherwise STOP
\
Table 5, Classifier5
Table3Classifier3F-score3
For each client (node) we need to identify the best ancestor to be used for action rules discovery
• Extract meta-actions from relevant comments that are associated with values of attributes used in action rules;
• Find smallest sets of meta-actions triggering these action rules;• Identify all customers supporting these action rules (needed to compute the
increase in revenue);• Search for smallest groups of meta-actions that trigger maximal increase in
revenue (we present them in descending order for same-size groups of meta-actions).
• effect(m) = num(distinct records in dataset involved with m) × conf(r*)• Effect of a set of meta-action indicates the number of customers who are
expected to become promoters when these meta-actions are executed.• Organize those groups of meta-actions hierarchically.
Extracting Meta-Actions /sentiment analysis/: We use Stanford Typed Dependencies Manual & Stanford Parser to generate grammatical relationshttp://nlp.stanford.edu:8080/parser/
r1 = (A, a1 a2) (B, b1->b2) (C, c1 c2) (detractor promoter)
Sup(r1) – number of customers supporting action rule r1 in a shop which NPS has to be improved
Number of detractors for a given shop having the properties (A,a1), (B,b1), (C,c1) - we target them
Before we start searching for triggers (meta-actions) supporting r1, this number is enlarged by adding customerssemantically similar and geographically close to our shop who also support rule r1.
Op1, Op2, Op3, Op4, Op5 – comments/sentiment about the service provided by this enlarged set of customers.Comments about the service from promoters who satisfy the description (A,a2), (B,b2), (C,c2) are also listed.
Op1 - positive opinion related to AOp2 - negative opinion related to C and BOp3 – positive opinion related to BOp4 – negative opinion related to AOp5 – negative opinion related to C
Customers sentiment associated with rule r1 –{Op1, Op2, Op3, Op4, Op5}
Trigger for A -> extracted from Op1 and not(Op4)Trigger for B -> extracted from Op3 and not(Op2) Trigger for C -> extracted from not(Op2) and not(Op5)
We search for minimum set of triggers covering A, B, C
invoice+++++,outstanding bill=141invoice+++++,they billed properly=65invoice+++++,fine invoice=61invoice-----,refused pay bills=29invoice+++++,happy with bill=12
price-----,not fair pricing=108price+++++,good price=108price+++++,fair pricing=87price-----,aggressive pricing=66price-----,unreasonable charge fee=37price-----,not satisfied with price=35price+++++,better pricing=27price-----,expensive amount charged=12price+++++,the charged fairly=12
staff+++++,best manager=112staff-----,bad experience because diagnosis=108staff+++++,good technician=96staff+++++,nice guy=87staff+++++,excellent mechanic=85staff+++++,great guy=83staff+++++,excellent technician=79staff+++++,good guy=74staff+++++,wonderful dealer=71staff+++++,good team=66staff+++++,honest guy=60staff+++++,pleased with manager=40staff-----,not available technician=34staff-----,wrong diagnosis=16staff+++++,best mechanic=12staff+++++,knowledgeable mechanic=6
Examples of Comments with Sentiment Orientation
Threshold for adding triggers = 6
Seeds
How to construct optimal sets of meta-actions (triggers activating changes in customer NPS)
{TR5} {TR1,TR4}We add single triggerone by one followingtheir order with respectto support
Sup=160
{TR5,TR1} {TR5, TR2} TR5, TR3}
Sup=180 Sup=165 Sup=163
{TR5,TR1,TR2} {TR5,TR1,TR3} {TR5,TR1,TR4}Sup=184 Sup=189 Sup=185
1 2 3 4
TR5 {TR1,TR4} {TR5,TR1,TR3}
{TR5,TR1}
Triggers activating changes in customer NPS
24
• Client-2 is selected to be our target.
Client{2,4,6,1} by HAMIS
NPS 0.77 (73 detractors) 0.852
Action rules extracted 28,126
Meta-actions extracted 15
Number of meta-nodes 447(*But we only focus on the top 10 in each
size category of meta-nodes.)Maximum effect in meta-nodes 45.83 (out of 73 detractors)
Search for the best meta-actions
idx Meta-actions effect
0 • ensure service done correctly 2.00
1 • ensure service done correctly• improve price competitiveness
7.62
2 • ensure service done correctly• properly set invoice
expectations(slightly high)
5.0
3 • ensure service done correctly• sufficient staff
2.0
7 • ensure service done correctly• competitive price• improve price competitiveness
11.13
8 • ensure service done correctly• improve price competitiveness• sufficient staff
10.66
9 • ensure service done correctly• improve price competitiveness• properly set invoice
expectations(slightly high)
9.99
10 • ensure service done correctly• improve price competitiveness• keep proactive communication
8.62
Search for the best sets of meta-actionsfor Client-2
idx Meta-actions effect
22 • ensure service done correctly• competitive price• improve price competitiveness• sufficient staff
15.07
23 • ensure service done correctly• improve price competitiveness• properly set invoice
expectations(slightly high)• sufficient staff
15.00
24 • ensure service done correctly• competitive price• improve price competitiveness• reasonable invoice
13.13
25 • ensure service done correctly• improve price competitiveness• reasonable invoice• sufficient staff
11.66
46 • ensure service done correctly• competitive price• improve price competitiveness• properly set invoice
expectations(slightly high)• sufficient staff
19.41
26
• Why HAMIS helps?1. 2) Extra meta-actions extracted from datasets extended by HAMIS help improve effect further.
Set of meta-actions Expected effect
competitive priceensure service done correctlyimprove price competitiveness
sufficient staff 9.87
decrease dealer response time* 11.13competitive priceensure service done correctlyimprove price competitivenesskeep proactive communicationnice technician
reasonable invoice 7.59
knowledgeable technician* 13.3
Search for the best meta-actions
QUESTIONS?