5 April 20055 April 2005

Is ROI an Effective Approach for Persuading Decisionmakers on the Value of Is ROI an Effective Approach for Persuading Decisionmakers on the Value of User-Centered Design?User-Centered Design?

Organizer: Susan Dray, Organizer: Susan Dray, Dray and Associates, USADray and Associates, USAClare-Marie Karat, Clare-Marie Karat, IBM T. J. Watson Research Center, USAIBM T. J. Watson Research Center, USA, Daniel , Daniel Rosenberg, Rosenberg, Oracle Corporation, USAOracle Corporation, USA, David Siegel, , David Siegel, Dray and Associates, Dray and Associates, USAUSA, Dennis Wixon, , Dennis Wixon, Microsoft, USAMicrosoft, USATuesday, 16:30-18:00, Hall BTuesday, 16:30-18:00, Hall B

This panel examines the utility and effectiveness of various ways of making the This panel examines the utility and effectiveness of various ways of making the business case for user-centered design (UCD). Most of the discussion in our business case for user-centered design (UCD). Most of the discussion in our field has assumed that measuring and demonstrating ROI for usability is the field has assumed that measuring and demonstrating ROI for usability is the key to this effort. However, experience shows that the most brilliant ROI key to this effort. However, experience shows that the most brilliant ROI analysis may not win the day in the real world of business. Our panelists analysis may not win the day in the real world of business. Our panelists range from people who claim that ROI is an important persuasive tool as long range from people who claim that ROI is an important persuasive tool as long as the communication about ROI is happening within a healthy business as the communication about ROI is happening within a healthy business relationship, to people who claim that a focus on ROI can actually be relationship, to people who claim that a focus on ROI can actually be destructive. We also explore the idea that there are important business destructive. We also explore the idea that there are important business contexts where ROI simply does not fit. Through the presentations by the contexts where ROI simply does not fit. Through the presentations by the panelists and through discussion of a business case scenario, we explore panelists and through discussion of a business case scenario, we explore some alternatives to ROI in making the business case for user-centered some alternatives to ROI in making the business case for user-centered design.design.

Strategic ROI – Strategic ROI – A commercial A commercial

Dennis WixonDennis Wixon

Manager Games User Research MicrosoftManager Games User Research Microsoft

5 April 20055 April 2005

Building a team with Building a team with strategic ROI strategic ROI Games User Research:Games User Research:• Group started in 1997 with one contractorGroup started in 1997 with one contractor• Now 2Now 2ndnd largest user research team at Microsoft largest user research team at Microsoft • 23 employees, 3 contingent staff, and 10 moderators. 23 employees, 3 contingent staff, and 10 moderators. • Scope Scope

• Last year 10,794 playtest participants Last year 10,794 playtest participants • 56 Microsoft and non-Microsoft games.56 Microsoft and non-Microsoft games.• Over 1200 usability participants Over 1200 usability participants • Since 1998, over 1250 reports on gamers’ experiences with Since 1998, over 1250 reports on gamers’ experiences with

gamesgames• Largest consumer research effort by any game developer or Largest consumer research effort by any game developer or

publisher for the purposes of improving games. publisher for the purposes of improving games.

5 April 20055 April 2005

Tenets of strategic ROI Tenets of strategic ROI

Understand the business contextUnderstand the business context Meet management needs Meet management needs Embed methods in case studies Embed methods in case studies Partner with creative power (design) Partner with creative power (design)

5 April 20055 April 2005

Big opportunity: Big opportunity:

• Movie box office was $8.4 billionMovie box office was $8.4 billion

• $9.4 billion spent on games $9.4 billion spent on games

• Tripled in 7 yearsTripled in 7 years

• Game consoles in 30-40% of U.S. Game consoles in 30-40% of U.S. householdshouseholds

Strategic ROI: Strategic ROI: Games business contextGames business context

5 April 20055 April 2005

Strategic ROI: Strategic ROI: business context (contd) business context (contd) • The business context: The business context:

• Prolific: 600+ game products in N. AmericaProlific: 600+ game products in N. America• ““Hit” driven: 80-20 ruleHit” driven: 80-20 rule High cost of developmentHigh cost of development Success of platform depends on appeal of software Success of platform depends on appeal of software

• Halo 2: Halo 2: • Over $100 million in sales on day 1 (Shrek 2: 75 Over $100 million in sales on day 1 (Shrek 2: 75

million in sales first weekend) million in sales first weekend) • At last count 6 million sold At last count 6 million sold

5 April 20055 April 2005

Strategic ROI: Strategic ROI: management need-response management need-response Improve games quickly and effectively - Improve games quickly and effectively -

RITE methodRITE method Measure fun - playtest assessment Measure fun - playtest assessment Help sign up partners: playtest and Help sign up partners: playtest and

usability labs and case studies usability labs and case studies

5 April 20055 April 2005

Strategic ROI: RITE Strategic ROI: RITE

5 April 20055 April 2005

Strategic ROI: RITE Strategic ROI: RITE

More issues fixedMore issues fixed Fixes are verifiedFixes are verified More issues foundMore issues found Truly collaborative effort – usability & designTruly collaborative effort – usability & design

““This was a fantastic experience. Whenever we get This was a fantastic experience. Whenever we get asked "how do you like working with Microsoft," two asked "how do you like working with Microsoft," two of the first things out of our mouths are "the people of the first things out of our mouths are "the people there are AWESOME" and "usability ROCKS!" there are AWESOME" and "usability ROCKS!"

5 April 20055 April 2005

Strategic ROI: playtest Strategic ROI: playtest lab lab

5 April 20055 April 2005

Strategic ROI: playtest Strategic ROI: playtest lab lab

5 April 20055 April 2005

Strategic ROI example: Strategic ROI example: Halo 1Halo 1

• Won “Oscar” for Game of the YearWon “Oscar” for Game of the Year• 58% “attach rate” at launch58% “attach rate” at launch• Designer quote Designer quote

“Without a doubt it is one of the most valuable “Without a doubt it is one of the most valuable resources in MGS [Microsoft Games Studios] and can resources in MGS [Microsoft Games Studios] and can be a critical factor in the success of the games we be a critical factor in the success of the games we publish. publish. (I only wish UT [User-testing] had greater bandwidth so (I only wish UT [User-testing] had greater bandwidth so we could use you more frequently!)”we could use you more frequently!)”

5 April 20055 April 2005

Strategic ROI: Strategic ROI: Acknowledgements Acknowledgements Playtest: Playtest:

Bill Fulton Bill Fulton Eric Schuh Eric Schuh

RITE: RITE: Michael MedlockMichael Medlock Melissa FederoffMelissa Federoff Dan Gunn Dan Gunn

Halo 1 and 2: Halo 1 and 2: Keith SteuryKeith Steury Randy Pagulayan Randy Pagulayan Jamie Griesemer – Bungie Jamie Griesemer – Bungie

5 April 20055 April 2005

Tenets of strategic ROI Tenets of strategic ROI

Understand the business contextUnderstand the business context Meet management needs Meet management needs Embed methods in case studies Embed methods in case studies Partner with creative power (design)Partner with creative power (design)

5 April 20055 April 2005

Apply Strategic ROI: Apply Strategic ROI: you can win too you can win too

http://www.mgsuserresearch.com /http://www.mgsuserresearch.com /