Webinar:
Formative Usability Testing in Agile:Piloting New Techniques at Autodesk
Eunice Chang, AutodeskOlivia Williamson, Autodesk
#uzwebinar
Eunice ChangSenior Principal User Researcher AutodeskSpeaker
Olivia WilliamsonPrincipal User Experience DesignerAutodeskSpeaker
Alfonso de la NuezCo-Founder andCo-CEOUserZoomModerator
Speakers:
www.userzoom.com
• Chat box is available if you have any questions
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of the slides/recording
• Twitter hashtag: #uzwebinar
Quick Housekeeping
www.userzoom.com
About UserZoom
Offer online or remote user research & testing solutions, saving UXers time, money, effort, and a lot of actionable insights
UX Consultants since ’01, SaaS since ‘09
In Sunnyvale (CA) Manchester (UK), Munich (DE) and Barcelona (Spain)
90% renewal rate, 50% revenue growth rate in the last 3 years
All-in-one Enterprise software solution that helps Businesses cost-effectively test, measure and improve UX over websites & mobile apps.
Product Suite: Unmoderated Remote Usability Testing
Remote Mobile Usability Testing
Online Surveys (web & mobile)
Online Card Sorting
Tree Testing
Screenshot Click Testing
Screenshot Timeout Testing (5-sec test)
Web VOC
Mobile VOC
www.userzoom.com
• Brief introduction to Autodesk and AutoCAD
• AutoCAD’s move from waterfall to Agile development
• User feedback through Customer councils
• Testing the design
• Testing the product
• Learnings
Agenda
© 2013 Autodesk
© 2013 Autodesk
Re-structured customer feedback process to meet Agile objectives:
Become more responsive and flexible to changing customer needs & market demands
Create higher quality software with equally high levels of customer delight
Introduction of Agile into the AutoCAD development process
© 2013 Autodesk
Customer Feedback / Process Overview
8
Process overview
Minimum viable solution (MVS)Story-mapping process
Smallest usable and sellable set of functionality that provides value to the user
© 2013 Autodesk
UX team sets experience goals that define minimum viable experience (MVX)
“Mini betas” for each specific initiative (reality capture, web and mobile, etc.)
50+ customers
Committed to testing new build every 1 -2 sprints and completing key tasks and satisfaction surveys
No monetary compensation
What are customer councils?Feature-based Customer Councils
© 2013 Autodesk
What are customer councils?Building a community
© 2013 Autodesk
Testing the design:Are we building the right thing?
© 2013 Autodesk
Customer Feedback / Design validation
13
Testing the design: Customer feedback
When How Outcomes
• One or two sprints before design is scheduled to be implemented
• One-on-one conversations• Remote user testing
• Card sorting• Concept sketches• Prototypes
• Refine designs to address usability issues
• Update or add new user stories and scenarios
• Add additional detail to future feature planning
© 2013 Autodesk
Less formal approach
Ongoing customer engagement
Maintain a topic list Balance of strategic and immediate Choose items that are relevant at the moment
Use a variety of tools and techniques
Testing the design: Conversations
© 2013 Autodesk
… current environment and needs …
What we talk about
can’t be exported to Excel
© 2013 Autodesk
… concepts …
What we talk about
© 2013 Autodesk
… feature prioritization …
What we talk about
© 2013 Autodesk
… design sketches …
What we talk about
© 2013 Autodesk
… prototypes …
What we talk about
© 2013 Autodesk
… usability …
What we talk about
© 2013 Autodesk
Immediate: specific input that affects current design decisions
Ongoing: more detail on the customer’s environment and application needs
Longterm: working relationships with customers get customers invested in our success help our team gain a deeper understanding of the
customer’s world
Benefits of Design ValidationTesting the design: What we gain
© 2013 Autodesk
Testing the product: Does this work as expected?
© 2013 Autodesk23
Evaluation of MVX of scenario(s) with customer councils
When How Outcomes
• After each sprint (or most sprints, product owner to decide readiness)
• Beta-style user feedback with customer council that uses build with their environment and drawings
• Customers validate Minimum Viable Experience (MVX) by completing tasks and satisfaction surveys
• Validated scenarios deemed “Customer-Done” or not
• Prioritized usability issues for feature backlog to guide feature direction
Testing the product: customer feedback
© 2013 Autodesk
Within days of launching the council, customers are:
Immediate Enthusiasm: Point Cloud Council
Posting their own Point Cloud scans Completing tasks
Discussing use cases and improvement ideas
Immediate enthusiasm: Point Cloud Council
© 2013 Autodesk
11 teams participated in the “backyard challenge”
Case study: Connected Desktop Council
© 2013 Autodesk
Learnings
© 2013 Autodesk
Draw on existing pools of customers
Recruit broadly
Define councils by features and topics of interest
Invite targeted customers
Your response rates will vary!
Recruiting for Customer CouncilsRecruiting for customer councils
© 2013 Autodesk
Know and use your tools
Engage immediately
Make expectations clear
Planning your Customer CouncilPlanning your customer council
© 2013 Autodesk
Identify task flows
Introduce the release with care
Get people engaged with each other
Track task completion and feedback closely
Planning your MVX testing cycle
© 2013 Autodesk
Look beyond traditional usability testing
Target specific applications and workflows
Ask questions in between builds!
Be realistic about length of customer feedback cycles
Recruiting for Customer CouncilsGetting the most out of a customer council
© 2013 Autodesk
Customers see the results of their design and testing feedback Design input is gathered early and often Built features are evaluated in a realistic environment
Design and usability issues treated as seriously as code defects
MVX validated through real customer usage and feedback
Features included in the release only when they meet MVX targets
31
Customer Councils: What success looks like
Q&A
Top Related