Leading the product 2017 - Susan Teschner - Speaker Slides

Leading The Product 2017Speaker SlidesSydney, Australia

Susan Teschner99designs

For more information go towww.leadingtheproduct.com

© 99designs, Inc.© 99designs, Inc.

Experimenting with ExperimentationPro tips for doing it well

by Susan Teschner

© 99designs, Inc. 3

Who am I?some days I’m not sure

4

design

Retail catalogues & ad networks

Accounting software

VoIP

Shipping for eCommerce

Classifieds

99

iNCMYOB

Delta 3

Australia Post

FairfaxSEEK


Manage products.Manage people

managing products.

by Evilltimm by LanaD.

6

World’s largest marketplace for graphic design

We’re very busy—

450,000happy customers

We’re famous but with hidden talents—

36Design categories

by Vladson

We’re home grown and a bit grown up—

2008 Founded in Melbourne

115 Fabulous people

3Offices (BER, OAK, MEL)

$200mpaid to designers


Experimentation?


“doing a thing to discover something unknown or to test a principle”

— poorly paraphrased from dictionary.com

http://dictionary.com

9

The universe of experiments—

by Gorcha

by huhi


So this is an agile thing?


Where are they now?

by kazoe


How does it help?


8 lessons

© 99designs, Inc.

DO pay attention—1


I don’t have all the ideas—

by Thedaydreamer


How 99designs was born…—

by xeniatm


Why we created 1:1 Projects—

by ktoons

© 99designs, Inc.

DON’T underestimate the importance of experiment design—

2

© 99designs, Inc.Commercial in confidence 19

Start off on the right foot—

▪ clearly stated hypothesis

▪ definition of success

▪ specific metrics

by ktoons


Beware the dragon of statistical significance—

by ilustreishon


Here’s how—

https://unbounce.com/ab-test-duration-calculator/

Sample Size & Test Duration Calculator

Current conversion rate (%)

Desired lift (%)

Number of variations

Average daily visitors

To achieve valid test results*, you need:A total sample of (users):

Test duration of (days**):

10800

5

10

20

3

2000

© 99designs, Inc.



Desired lift (%)





10800

54

10

20

3

200



Desired lift (%)





10800

5

10

20

3

2000

22https://unbounce.com/ab-test-duration-calculator/

Here’s how—


This was a bit too long to wait—

How long will your experiment take?How many eligible visits do you have per day?

What percentage of visits will see the change?

What is your current conversion rate?

How will you change conversion?

How confident do you want to be of this?

How likely should you be to detect the change?

1,400

50%

2%

5%

95%

90%

You should run this experiment for 1 year, 7 months and 28 days.

© 99designs, Inc.

DO choose the right tools—

3

26

other data sets

PerformanceUI / UX

Resourcing

Maturity

$Things to think about—


So, who’s your favorite?—

by glasshopperart

© 99designs, Inc.

DON’T spend too much—4


Beware the gold plating—

by MattDyckStudios

https://www.quora.com/How-is-pretotyping-different-from-prototyping


Baby steps towards a Gift Shop

i1i2

© 99designs, Inc.

…but DO spend enough—5

© 99designs, Inc.© 99designs, Inc. 32

“We increased sales by 37% and all we had to do was change the color of a button!

by Pierre d’Arterie


This was never going to wow.


by Hildebrand

© 99designs, Inc.

1Testing tools—

2Variations—

3Opportunity cost—

by RaidenD

© 99designs, Inc.

DON’T ignore the results—6


If it seems too good to be true…—

by Frankie Cadillac


Share the results. Publicly.—

by Sam!


R.I.P.—

by RVST


… or put your money where your experiment is—


… or take your chances with the people who look after it—

by Thedaydreamer


A story about Social Sharing

by clouds940


by clouds940




by miljanristic by FRITO-DESIGNstudio

by Joedsign


by clouds940



WALK AWAY

extremeuncertainty.com

© 99designs, Inc.

Until one day…—

AshishYep, I’ve also flagged this feature with @Miranda Burford as

something we should critically evaluate as I’m concerned the

benefits have not outweighed the costs.

SusanGood call @Miranda Burford. And it didn’t go unnoticed that we’re

the only ones who ticked off our To-Do’s.

MirandaJust to circle back on this, we reviewed Social Sharing and made

the decision to retire it. If you’re interested in how we came to that

decision, you can check out the slide deck.


How’s it going?

by stormyfuego

© 99designs, Inc.

2017-05-17

2017-07-05

2017-05-12

2017-04-18

Start End Team

2017-04-29

Growth Activation

Growth

-2% click through-11% reduction of click through to Other design category

50%

2017-07-17 -10% clicks0% conversion impact

Growth Acquisition

/categories

Email forward in brief

Improved search on categories

Spruced up cross sell

Experiment Page Conversion goal

/work

Click to product pages

Click repeat purchase

Visual brief Increase email capture

Minimum / Aimed improvement

5%

65%

33%

Observed improvement

52

At a glance—


Facing the music?—

by jestyr37


Treat it like an experiment—

by dante banuelos


Thank you

by Thedaydreamer

Leading the product 2017 - Susan Teschner - Speaker Slides

Education

Transcript of Leading the product 2017 - Susan Teschner - Speaker Slides