DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date:...
Transcript of DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date:...
![Page 1: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/1.jpg)
Data-‐Driven Product Innova1on
Xin Fu LinkedIn
Hernán Asorey Salesforce
![Page 2: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/2.jpg)
WHY?
2
![Page 3: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/3.jpg)
Data Science Fuels Product Innova1on
3
![Page 4: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/4.jpg)
Example of Data Guiding Product Dev
Queries
4
Unique Searchers Unique
Searchers
Sessions Queries
Session * *
=
![Page 5: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/5.jpg)
How Did Data Science Help?
§ Metric to define product “true north”
§ Tools to help understanding
§ A/B experiments to learn and iterate
5
![Page 6: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/6.jpg)
Data Science’s Role in a Product Org
§ Model 1: Data Science as an Owner
§ Model 2: Data Science as a Service
§ Model 3: Data Science as a Partner
6
![Page 7: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/7.jpg)
Data Science as an Owner
§ Operates in a “hacky way” (early stage companies)
§ Key steps in business rely heavily on data – Recommender – Relevance – Matching – Scoring
§ Mostly back-‐end, rela1vely more stand-‐alone features
7
![Page 8: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/8.jpg)
Data Science as a Service
§ Engagement is “on-‐demand”, project-‐based § Examples:
– some of the BI roles – strategy roles (consultancy) – data API to product – modeling for specific purposes: propensity to {x}, where x: {buy, ahrite, convert, etc.}
8
![Page 9: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/9.jpg)
Data Science as a Partner
§ Plays ac1ve role in every stage of Product Life Cycle
§ Shares the ul1mate goal of product success § Oken requires an embedded engagement model
9
![Page 10: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/10.jpg)
Product Life Cycle
1. Idea1on
2. Design & Specs
3. Development
4. Test & Iterate
5. Release
10
![Page 11: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/11.jpg)
1. Idea1on
2. Design & Specs
3. Development
4. Test & Iterate
5. Release
Data-‐Driven Product Innova1on Framework
1. Hypothesis & Ques1ons
2. Holis1c Evalua1on Criteria
3. Tracking & Valida1on
4. Experimenta1on & Analysis
5. Monitoring & Learning
11
![Page 12: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/12.jpg)
5. Monitoring & Learning
§ Start from good reliable repor1ng of key product metrics
§ Par1cularly important for new partnership (“credibility projects”)
12
![Page 13: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/13.jpg)
Partner with Data Infrastructure Team to Build Great Tools
13
![Page 14: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/14.jpg)
4. Experimenta1on & Analysis
Ronny Kohavi’s key note “Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years” Ya Xu et al.’s talk “From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks”
§ A company-‐wide plaqorm for A/B tes1ng, ramping, and advanced targe1ng needs
§ Automated repor1ng and analysis capabili1es
14
![Page 15: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/15.jpg)
Streams of Innova1on around A/B Tes1ng
15
![Page 16: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/16.jpg)
3. Tracking and Valida1on
§ Joint ownership between data scien1st, engineer, QA and product manager
§ Quarterly sign-‐off process from product owners
§ Created a tool for ongoing monitoring
16
![Page 17: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/17.jpg)
2. Holis1c Evalua1on Criteria Opportunity for strong thought leadership from data scientists Past: focus on product-specific metrics
– Relevance change ⇒ CTR – Profile redesign ⇒ # Profile Views
Now: standardized, tiered metric system
– Site-wide Tier 1 metrics – Product-specific Tier 2 / Tier 3
metrics 17
Tier 1 Tier 2 Tier 3
![Page 18: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/18.jpg)
1. Hypotheses & Ques1ons
Components of a good analysis
What we would like to see
18
INSIGHTS RECOMMENDATIONS DATA
![Page 19: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/19.jpg)
Effec1ve Ways to Drive Ac1ons
Secure a sponsor and build buy in
Consider the scale of target audience and technology
Verify by tes1ng with low-‐cost prototypes 19
Outline expected benefits
![Page 20: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/20.jpg)
Data Science in Ac1on!
20
![Page 21: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/21.jpg)
OrgDNA Abstract: Develop prescrip1ve tool to help admins configure orgs. Use machine learning to iden1fy the rela1onships between perms, prefs, user behavior and adop1on metrics.
Value Crea;on: § Increased product adop1on § Increased customer engagement
Accomplishments: § Decoding of user permissions and preferences into flat binary table for analysis § Joined in ini1al test target metrics (red accounts, tenure) and behavioral variables § Data QA and sanity checks complete (explore distribu1ons and ini1al correla1ons)
Sponsor: SVP of Mobile Benefi;ng Audience: Salesforce Administrators at Customer Loca1ons 21
![Page 22: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/22.jpg)
Just the Beginning…
22
Permission Frequency By Edi1on Preference Frequency By Edi1on
![Page 23: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/23.jpg)
Just the Beginning…
23
Super High Reten;on Group (cluster 2) -‐-‐Days_S1 > 16 -‐-‐CHURN 1%
High Reten;on Group (cluster 1) -‐-‐Days_S1 between 8 and 16 -‐-‐CHURN 5%
Lower Reten;on Group (cluster 4) -‐-‐Days_S1 < 8 -‐-‐CHURN 34%
High Ac;vity, Low Reten;on Group (cluster 3) -‐-‐Days_S1 < 4 -‐-‐Ac1vi1es Per Day > 6 -‐-‐CHURN 46% Cluster Averages
NMI Score = 0.082
![Page 24: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/24.jpg)
Data Science as a Partner
Quality & Speed of Product
Decisions
24
![Page 25: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/25.jpg)
Data Science as a Partner
§ Focus on Product and balance between – Product innova1on and opera1on – Velocity and scale – Theore1cal research and prac1cal impact
§ Leverage technology for reliability and sustainability – Reliability is key – Reduce “one-‐offs” – Speed mahers
25
![Page 26: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/26.jpg)
Case Study
26
![Page 27: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/27.jpg)
Data-‐Driven Product In Ac1on
In order to increase the connec1on density of new users, we will lookup the exis1ng members whose address books contain these new users as contacts The exis1ng members will be no1fied that their contacts just joined LinkedIn and will be prompted to connect with them
27
![Page 28: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/28.jpg)
Anatomy of a Data-‐Driven Product
1. Problem Statement 2. Opportunity Analysis
– Iden1fy target audience – Verify feasibility
3. Holis1c Evalua1on Criteria – Including impact es1mate
4. Low-‐cost Tes1ng with Tracking 5. Analysis of Test Results 6. Hand off for Scalable Implementa1on
28
![Page 29: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/29.jpg)
Step 1. Problem Statement
Goal: Increase the connec1on density of new LinkedIn members. Why? § X percent of new users have Y or few
connec1ons at 30 days aker registra1on § High connec1on density => Beher engagement
29
![Page 30: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/30.jpg)
Step 2. Opportunity Analysis
30
Target Audience: 5 (50%) Of the 10 new members, 5 are eligible Feasibility Analysis: Poten1al Upside: X new connec1ons for each new member For the 5 eligible New Members, median number of poten1al ‘welcomer’ is 2 X = Median (1,2,2,1,3)
No1fica1on System Load: Of the 10 exis1ng members, 4 are poten1al ‘welcomers’ Member Overload: The volume of no1fica1ons to receive by poten1al welcomers has a distribu1on with Median=2 and Max=3
E1
N7
E10
E2
E9
E8 N8
N4
N1
N6
E5
N3
E6
E4
E3
E7
N2
N9
N5
N10
AàB : A appears in B’s address book N: New Member E: Exis1ng Member
1
2
2
3
1
![Page 31: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/31.jpg)
Step 3. Holis1c Evalua1on Criteria
§ Primary Metrics (New Members) – connec1on density – reten1on rate
§ Secondary Metrics (Exis;ng Members) – Connec1on invita1ons sent – Connec1on density – Engagement
• Impact Es;mate – Need to make assump1ons about conversion rate
31
![Page 32: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/32.jpg)
Step 4. Low-‐Cost Tes1ng
§ Email Test § Daily offline Hadoop job to generate the list of members eligible to receive the email (with eng. review)
§ Randomize for the recipients § Partner Involvement
§ Work with Engineering and QA to set up tracking § Work with Ops, SRE and Customer Service teams to set up monitoring
32
![Page 33: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/33.jpg)
Step 5. Analysis of Test Results
§ Email open rate, invite rate, invite acceptance rate
§ Es1mate impact on exis1ng members
§ Es1mate impact on new members
33
![Page 34: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/34.jpg)
Step 6. Scalable Implementa1on
Handed off to Engineering to scale up the impact:
§ Expand target audience, e.g. to dormant members
§ Offline email to online and mobile no1fica1ons
§ Introduce relevance rules
34
![Page 35: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/35.jpg)
Exercise
35
![Page 36: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/36.jpg)
Design Your Data-‐Driven Product
Please refer to the hand-‐out
§ 1. Pick an idea for a data-‐driven product either for your ins1tu1on, or for your favorite Web product
§ 2. Outline your plan to evaluate the idea – Problem statement – Opportunity analysis (target audience, feasibility) – Holis1c Evalua1on Criteria and poten1al impact – Low-‐cost tes1ng plan
§ 3. Iden1fy who you can partner with
36
![Page 37: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/37.jpg)
Data Science’s Role in a Product Org Data
Science Team
Structure Importance of Priori@za@on
Velocity Sustainability Proac@ve Approach
As an Owner Single
As a Service
Center of Excellence
As a Partner
Hub & Spoke
37 Low High
![Page 38: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/38.jpg)
Data Science as Product Partner
PARTNERSHIP
TECHNOLOGY
TALENTS
§ Start w/ credibility projects § Context and ownership § Scalable path to innova1on
§ Leverage technology to improve quality and speed
§ Reliability is key § Automate, Automate!
? 38
![Page 39: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/39.jpg)
Data Science as Product Partner
§ We put great emphasis on understanding and solving real business problems – Data sense: what is possible – Product sense: what is valuable
§ We value people who think holis1cally and in scale – Cross-‐product impact – Viral effect
39
Talents
![Page 40: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/40.jpg)
Tutorial Takeaway
PARTNERSHIP
TECHNOLOGY
TALENTS
§ Start w/ credibility projects § Context and ownership § Scalable path to innova1on
§ Leverage technology to improve quality and speed
§ Reliability is key § Automate, Automate!
§ Passion for product innova1on § Proac;ve in partnership § Impact oriented
Data Science as a Partner
40
![Page 41: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/41.jpg)
41
THANK YOU
![Page 42: DataDriven*ProductInnovaon* · 2019-09-20 · 2015KDDTutorial Author: Xin Fu Created Date: 8/11/2015 6:55:01 AM ...](https://reader034.fdocuments.in/reader034/viewer/2022042309/5ed6aba139f16f294d573ef5/html5/thumbnails/42.jpg)
Data-‐Driven Product Innova1on
Xin Fu LinkedIn
Hernán Asorey Salesforce