This slide deck is part of a presentation given by Saul Greenberg at a UIST 2007 panel on Evaluating user interface systems research (see paper by Dan Olson in ACM DL)

Its content was based on a paper written by Saul Greenberg and Bill Buxton, available at:

http://grouplab.cpsc.ucalgary.ca/papers/2007/07-UsabilityHarmful-Greenberg-Report/07-usabilityHarmful-report2007-879-31.pdf

I need to do an evaluation

What’s the problem?

It won’t get accepted if I don’t. Duh!

Dogma

Good HCI

TimeLe

arn

ing

Brian Gaines

Breakthroughmajor creative advances

Replicationideas mimicked and altered; increasing experience

Empiricismlessons formulated as empirical design rules

Theoryunderlying causal systems

Automationaccepted

Maturityused routinely

Breakthrough Replication Empiricism Theory Automation Maturity

Interface validation and refinement*

Evaluation sweet spot

*70% quantitative, 30% qualitative

Breakthrough Replication Empiricism Theory Automation Maturity

Concepts,Sketches

Memex Bush

Unimplemented and untested design. Microfilm is impractical. The work is premature.

Resubmit after you build and evaluate this design.

Sketchpad Sutherland

Complex controls,overly simplistic drawings

Recommend you evaluate a trained draftsman using Sketchpad vs. paper and pencil with real drafting tasks

Augment Engelbart

While ‘tested’ by your team, this is not an objective validation.

What to do?

Reflection

Bill Buxton, Sketching the User Experience

Breakthrough Replication Empiricism Theory Automation Maturity

Cultural Appropriation

Hypertext

What to do?

Case / field studiesCultural probesReflectionExtreme uses…

Evaluation

MoreAppropriate Evaluation

We decide what is good research

TimeLe

arni

ng

Brian Gaines

Breakthrough Replication Empiricism Theory Automation MaturityBreakthrough Replication Empiricism Theory Automation Maturity

Interface validation

Sketches Cultural appropriation

grouplab.cpsc.ucalgary.ca/~papers/