Getting Strategic with Digital: Intro to Analytics
-
Upload
nform-user-experience -
Category
Data & Analytics
-
view
234 -
download
0
Transcript of Getting Strategic with Digital: Intro to Analytics
Getting Strategic with Digital Intro to Analytics
Research & Product Management: How I Learned to Stop Worrying and
Love the Customer
Who am !?• ME = James Klassen• TALKING ABOUT = Research and putting the
customer 1st!• If you have a question, please just yell!
• Background in market research, online marketing, SEO, social media, analytics… blah blah blah
• Digital Product Manager @ AMA Insurance• In addition to product management, also lead a
small, elite team of marketing hackers
Questions
• Who regularly uses analytics/research to learn about their customers? Online specifically?
• Who works in digital/ directly with people in digital?
I have a case study!
First, some results
Which one converts better?
Which one converts better?
50%increase
Which one converts better?
Which one converts better?
75%decrease
Some context…
What are we trying to learn?• Big question: who are the people using our
website?– What do they want, how do they want it, what do
they already know?– How can we organize our site to help them find
what they’re looking for?– Because it’s insurance, are they looking for the
right things?• We want to help educate about insurance
policies/coverage/etc.– What functionality should we add/delete?– How can we design our site better?
Our path• Market research: understand broad strokes• Online survey 1: User intent
– We had no idea, clean slate• Traditional market research gave us direction, but we wanted to
validate online: were those people already talking to us?– Tried to keep in mind that the state of our current site was
going to influence responses• Online survey 2: Service preference
– Offline vs. online – many services not available online– How do people want to interact with us?– What type of insurance are they looking for?
• Online survey 3: Current customer?– Already insured? Already a member?
Our path
• So, we decided to change our main-page templates to reflect what we learned
• For /insurance/ and central products:1. Make it easy to accomplish high-priority
tasks2. Make it more obvious that you’re on an
insurance-related page3. (/insurance/) make it easier to access
second-level self-service
Thus…
And…
Successes are easy… so about that grey box...
• Initially:– Made the change without testing… OBVIOUSLY
THE NEW DESIGN IS BETTER– Saw big increase in bounce rate, decrease in
conversion rate– This was basically a before/after test
• Ok, we need to learn more:– Run it A/B style in a more controlled environment
(maybe external factors were impacting the results… this is what I was hoping for)
2nd test: A/B
2nd test: A/B
70%decreaseinquotes
2nd test: A/B
194%increaseinbouncerate
The horrible grey box
• Still really hate the grey box
• There must be something wrong inside of the 2-box setup– Let’s try different headings
3rd test: A/B 3 treatments
3rd test: A/B 3 treatments
58%increaseinquotes
133%increaseinquotes
The horrible, terrible grey box• Simply removing 1 of the boxes, led to 133%
increase in quotes…• Unfortunately, it looks like 2 boxes are just a
losing combination
• SIGH
• So we switched back to the original grey box…
The horrible, terrible, ugly grey box
• BUT I still think we can make it better!
• So let’s try making the single box better!– More visible text– More visible box colour– Remove overall page header
4th test: A/B 4 treatments
4th test: A/B 4 treatments
INCONCLUSIVE!!!
So what’s the outcome ofJames vs. the grey box?
So where are we now with the grey box?
• Slowly coming to terms with it
• We’re still learning and looking for ways to improve– “best practice” doesn’t work
• The fact is: it performs well!
So where are we now with the grey box?
• What we should all learn:– This stuff doesn’t always go according to plan– Users don’t always want what you want them to
want– Testing isn’t magical, it’s actually more work– Check your bias – even THIS IS OBVIOUSLY
better doesn’t always work– The same approach in 2 different contexts might
not work… and you might not realize there are different contexts!
• Can’t set it and forget it... Once you start down the dark path...
Some common mistakes• Not using the appropriate research tool for the
question• Making bias or stats-related errors in A/B tests
– Chasing 100% confidence– Not determining the parameters of your experiments– Not accounting for environmental factors like
seasonality• Putting too much weight into results• Testing too much, too big, too small• Testing things you can’t change
Research tips• If you don’t have research/ stats experience, get help
– Ask your internal business intelligence and market research team(s)
– There are loads of great resources online• Test a lot, but know you can test too much
– Easy to test ”too big” or “too small”• Get to know your analytics platform• Research isn’t just a/b testing
– Don’t forget about surveys, focus groups, analytics, internal interviews (retail, product experts)
• Think about using Agile– Puts the customer at the centre and that’s really what
we’re talking about here
Questions?
Thanks!James Klassen
Digital Product ManagerAMA Insurance
AMAInsurance.caca.linkedin.com/in/jrklassen
@forsoothed