Webinar:

Formative Usability Testing in Agile:Piloting New Techniques at Autodesk

Eunice Chang, AutodeskOlivia Williamson, Autodesk

#uzwebinar

Eunice ChangSenior Principal User Researcher AutodeskSpeaker

Olivia WilliamsonPrincipal User Experience DesignerAutodeskSpeaker

Alfonso de la NuezCo-Founder andCo-CEOUserZoomModerator

Speakers:

www.userzoom.com

• Chat box is available if you have any questions

• There will be time for Q&A at the end

• We will be recording the webinar for future viewing

• All attendees will receive a copy of the slides/recording

• Twitter hashtag: #uzwebinar

Quick Housekeeping

www.userzoom.com

About UserZoom

Offer online or remote user research & testing solutions, saving UXers time, money, effort, and a lot of actionable insights

UX Consultants since ’01, SaaS since ‘09

In Sunnyvale (CA) Manchester (UK), Munich (DE) and Barcelona (Spain)

90% renewal rate, 50% revenue growth rate in the last 3 years

All-in-one Enterprise software solution that helps Businesses cost-effectively test, measure and improve UX over websites & mobile apps.

Product Suite: Unmoderated Remote Usability Testing

Remote Mobile Usability Testing

Online Surveys (web & mobile)

Online Card Sorting

Tree Testing

Screenshot Click Testing

Screenshot Timeout Testing (5-sec test)

Web VOC

Mobile VOC

www.userzoom.com

• Brief introduction to Autodesk and AutoCAD

• AutoCAD’s move from waterfall to Agile development

• User feedback through Customer councils

• Testing the design

• Testing the product

• Learnings

Agenda

© 2013 Autodesk

© 2013 Autodesk

Re-structured customer feedback process to meet Agile objectives:

Become more responsive and flexible to changing customer needs & market demands

Create higher quality software with equally high levels of customer delight

Introduction of Agile into the AutoCAD development process

© 2013 Autodesk

Customer Feedback / Process Overview

8

Process overview

Minimum viable solution (MVS)Story-mapping process

Smallest usable and sellable set of functionality that provides value to the user

© 2013 Autodesk

UX team sets experience goals that define minimum viable experience (MVX)

“Mini betas” for each specific initiative (reality capture, web and mobile, etc.)

50+ customers

Committed to testing new build every 1 -2 sprints and completing key tasks and satisfaction surveys

No monetary compensation

What are customer councils?Feature-based Customer Councils

© 2013 Autodesk

What are customer councils?Building a community

© 2013 Autodesk

Testing the design:Are we building the right thing?

© 2013 Autodesk

Customer Feedback / Design validation

13

Testing the design: Customer feedback

When How Outcomes

• One or two sprints before design is scheduled to be implemented

• One-on-one conversations• Remote user testing

• Card sorting• Concept sketches• Prototypes

• Refine designs to address usability issues

• Update or add new user stories and scenarios

• Add additional detail to future feature planning

© 2013 Autodesk

Less formal approach

Ongoing customer engagement

Maintain a topic list Balance of strategic and immediate Choose items that are relevant at the moment

Use a variety of tools and techniques

Testing the design: Conversations

© 2013 Autodesk

… current environment and needs …

What we talk about

can’t be exported to Excel

© 2013 Autodesk

… concepts …

What we talk about

© 2013 Autodesk

… feature prioritization …

What we talk about

© 2013 Autodesk

… design sketches …

What we talk about

© 2013 Autodesk

… prototypes …

What we talk about

© 2013 Autodesk

… usability …

What we talk about

© 2013 Autodesk

Immediate: specific input that affects current design decisions

Ongoing: more detail on the customer’s environment and application needs

Longterm: working relationships with customers get customers invested in our success help our team gain a deeper understanding of the

customer’s world

Benefits of Design ValidationTesting the design: What we gain

© 2013 Autodesk

Testing the product: Does this work as expected?

© 2013 Autodesk23

Evaluation of MVX of scenario(s) with customer councils

When How Outcomes

• After each sprint (or most sprints, product owner to decide readiness)

• Beta-style user feedback with customer council that uses build with their environment and drawings

• Customers validate Minimum Viable Experience (MVX) by completing tasks and satisfaction surveys

• Validated scenarios deemed “Customer-Done” or not

• Prioritized usability issues for feature backlog to guide feature direction

Testing the product: customer feedback

© 2013 Autodesk

Within days of launching the council, customers are:

Immediate Enthusiasm: Point Cloud Council

Posting their own Point Cloud scans Completing tasks

Discussing use cases and improvement ideas

Immediate enthusiasm: Point Cloud Council

© 2013 Autodesk

11 teams participated in the “backyard challenge”

Case study: Connected Desktop Council

© 2013 Autodesk

Learnings

© 2013 Autodesk

Draw on existing pools of customers

Recruit broadly

Define councils by features and topics of interest

Invite targeted customers

Your response rates will vary!

Recruiting for Customer CouncilsRecruiting for customer councils

© 2013 Autodesk

Know and use your tools

Engage immediately

Make expectations clear

Planning your Customer CouncilPlanning your customer council

© 2013 Autodesk

Identify task flows

Introduce the release with care

Get people engaged with each other

Track task completion and feedback closely

Planning your MVX testing cycle

© 2013 Autodesk

Look beyond traditional usability testing

Target specific applications and workflows

Ask questions in between builds!

Be realistic about length of customer feedback cycles

Recruiting for Customer CouncilsGetting the most out of a customer council

© 2013 Autodesk

Customers see the results of their design and testing feedback Design input is gathered early and often Built features are evaluated in a realistic environment

Design and usability issues treated as seriously as code defects

MVX validated through real customer usage and feedback

Features included in the release only when they meet MVX targets

31

Customer Councils: What success looks like

Q&A