1 Usability of Programming Languages - The Computer Laboratory
Usability of Programming Languages
description
Transcript of Usability of Programming Languages
![Page 1: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/1.jpg)
Usability of Programming Languages
MPhil ACS module R201 - Alan Blackwell
![Page 2: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/2.jpg)
Overview
Practical experimental course lectures are only introductory
Lecture 1 - theoretical principles classic approaches current trends in leading research.
Lecture 2 - candidate research methods advantages and disadvantages
Lecture 3 - specific classes of user Lecture 4 - directed by your research interests
![Page 3: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/3.jpg)
Seminars
Seminar 1 – discuss background to individual usability studies focus on a particular language apply specific theories from prior research
Seminar 2 – compare notes with industry leader leading designer of end-user scripting languages
Seminar 3 – discuss your work in progress Seminar 4 – final presentations
![Page 4: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/4.jpg)
Assessment criteria
This is a specialised and focused practical research training course.
The expected outcome: You will achieve research competence in a field
such as Psychology of Programming, Program Comprehension etc
Assessment will be relative to the international standard of graduate students working in these fields. Written work will be graded relative to typical
student publications in the field Presentations will be expected to meet the
standard of first-year PhD students in the field, for example at the Doctoral Consortium of a specialised conference.
![Page 5: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/5.jpg)
Reading List
Hoc, Green, Samurçay and Gilmore (1990) Psychology of Programming.
Psychology of Programming Interest Group www.ppig.org
Cambridge guidance on human participants Cairns and Cox (2008)
Research Methods for Human-Computer Interaction
Carroll (2003) HCI Models, Theories and Frameworks: Toward a multidisciplinary science
![Page 6: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/6.jpg)
Lecture 1: Principles of human factors in
programming
![Page 7: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/7.jpg)
Cognitive models in HCI
Engineering model of human ‘I/O subsystems’ and ‘central processor’ Derived from human factors/ergonomics Speed and accuracy of movement Include working memory capacity
7 +/- 2 ‘chunks’ Single visual scene
Programming as ‘cognitive ergonomics’?
![Page 8: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/8.jpg)
UserUser
From research into visual perception
From research into physical motion
output input
input output
ComputerComputer
Classical Cognitive HCI
![Page 9: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/9.jpg)
longtermmemory
workingmemory vision
motioncontrol
problemsolving
input
output
The User (cognitive ‘boxology’)
![Page 10: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/10.jpg)
Cognitive models of programming
Deciding what to do is harder than doing it HCI models assume a ‘correct’ sequence of actions
Classic cognitive models derived from GOFAI problem solving planning knowledge representation
PoP book ch 1.4 - Human Cognition and Programming ch 3.1 - Expert Programming Knowledge:
A Schema-based Approach ch 2.3 - Language Semantics, Mental Models and
Analogy cf user interface “metaphor”
![Page 11: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/11.jpg)
Software Development Context
Cognitive science: individuals in controlled contexts carefully construct experimental tasks to explore
schemas, plans, analogy etc correspond to AI constraints of toy problems
Compare to wicked problems goals and criteria under-specified, constraints conflict etc
Commercial software development is more social understand problem domain, negotiate specification
change
PoP book ch 1.3 - The Tasks of Programming ch 3.3 - Expert Software Design Strategies ch 4.1 - The Psychology of Programming in the Large:
Team and Organizational Behaviour cf Information Systems literature
![Page 12: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/12.jpg)
Individual Variation
Cognitive theories are general theories Consistent aspects of human performance
But some programmers are far more productive Always more productive in a language they know Performance also correlated with
general intelligence self-efficacy diagnostic tests for autism
“expert” vs “novice” Study knowledge by comparing those with to those
without Study naïve users who are not ‘crippled’ or
‘mutilated’ Real expert performance may include design
research
![Page 13: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/13.jpg)
Major research centres and programmes
![Page 14: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/14.jpg)
Venues
Psychology of Programming Interest Group (PPIG) annual conference - proceedings available online “Work in Progress” meeting (PPIG-WIP)
European Association for Cognitive Ergonomics (EACE)
Empirical Studies of Programmers foundation (ESP)
IEEE Visual Languages and Human Centric Computing ESP symposia in 2002, 2003
International conference/workshop on Program Comprehension (ICPC, formerly IWPC)
ACM CHI Evaluation and Assessment in Software Engineering (EASE)
![Page 15: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/15.jpg)
NSF EUSES
End-Users Shaping Effective Software Margaret Burnett at Oregon State University Brad Myers at Carnegie Mellon University Mary Beth Rosson at Penn State University Susan Wiedenbeck at Drexel University Gregg Rothermel at University of Nebraska Alan Blackwell at Cambridge
Review publication for thorough overview Ko, A.J., Abraham, R., Beckwith, L., Blackwell, A.F.,
Burnett, M., Erwig, M., Lawrence, J., Lieberman, H., Myers, B., Rosson, M.-B., Rothermel, G., Scaffidi, C., Shaw, M., and Wiedenbeck, S. (2011). The State of the Art in End-User Software Engineering. ACM Computing Surveys 43(3), Article 21.
![Page 16: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/16.jpg)
UK/European centres
PPIG UK Salford (Maria Kutar – chair) York (Thomas Green) Sheffield Hallam (Chris Roast) Open University (Marian Petre and Judith Segal) Sussex (Judith Good) Cambridge (Alan Blackwell)
Joensuu, Finland (Sajaniemi, Tukiainen, Bednarik)
Limerick, Ireland (Buckley) INRIA Eiffel group, Paris (Détienne, Visser)
Fraunhofer (Wulf )
![Page 17: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/17.jpg)
Other US centres
University of Colorado at Boulder Gerhard Fischer & Alex Repenning
MIT Media Lab Henry Lieberman
IBM Research TJ Watson Rachel Bellamy
IBM Research Almaden Allen Cypher
Microsoft Research RedmondHuman Interactions in Programming (HIP) group Rob DeLine, Gina Venolia & Andrew Begel
![Page 18: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/18.jpg)
Current areas of theoretical attention
![Page 19: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/19.jpg)
Cognitive Dimensions of Notations
Programming as interaction with an information structure (Ch 2.2 of PoP book)
Sample dimension Viscosity: a viscous system is difficult to change
Resources: Visual language usability paper in JVLC by Green &
Petre Tutorial by Green & Blackwell Questionnaire by Blackwell & Green Chapter in Carroll book
![Page 20: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/20.jpg)
CDs Theory
Any visible notation encodes an information structure. The structure has different parts The parts have various relationships to each other
Notational Layers one structure is often derived from another with
similar parts and relationships e.g. web page, from PHP program, from UML
diagram, from whiteboard sketch, from business plan
![Page 21: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/21.jpg)
Notational Activities
Search: finding information in a familiar structure
Exploratory understanding: understand a structure you haven’t seen
Incrementation: add new items to existing structure
Modification: change an existing structure
Transcription: create a new structure derived from an existing
layer
Exploratory design: create a structure you don’t understand yet
![Page 22: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/22.jpg)
Attention Investment
Cost-benefit equation - compare mental effort: to carry out a programming task against effort saved by the program
With associated risk/uncertainty: In estimate of effort to finish the program In actual benefit if the program has a bug In chance of damage resulting from a severe bug
![Page 23: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/23.jpg)
Attention Investment Biases
Some expert programmers: under-estimate costs, and over-estimate benefits
Novices might be reluctant to engage in programming: If they over-estimate the costs If they over-estimate risk of negative return Tools can provide ‘gentle slope’ to reduce this bias E.g. surprise – explain - reward
![Page 24: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/24.jpg)
![Page 25: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/25.jpg)
Gender HCI
Attention investment + self-efficacy theory
You need confidence to start programming Attention investment means that female students
are less inclined to explore programming options.
You need to do programming to gain confidence Self-efficacy develops through time spent
experimenting
Encourage ‘tinkering’ to explore behaviour But note that the same kind of tinkering can results
in poorer learning for males, who have a tendency to be over-confident, and not to think about what they are doing
![Page 26: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/26.jpg)
Programming by Example
Based on machine learning techniques Infer programs from examples of required output
Attention Investment benefits: examples can be provided through normal direct
manipulation, so reduced perceived cost inferred program is offered to user when already
functional, so reduced perceived risk
![Page 27: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/27.jpg)
Natural Programming
Programmme of Myers’ group at Carnegie Mellon
Study natural/everyday description of algorithms
Design programming languages compatible with naïve knowledge Pane’s HANDS (for children) Miller’s LAPIS (for text manipulation) Ko’s CITRUS (constraint-based MVC platform)
![Page 28: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/28.jpg)
![Page 29: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/29.jpg)
Variable Roles
Programme of Sajaniemi’s group at Joensuu Based on analysis of source code corpuses Unlike Myer’s focus on naïve knowledge, this focuses on expert knowledge
Variables are used in only a few ways: fixed, stepper, follower, gatherer etc
Originally used for educational visualisation, instruction
May be used for intelligent compilers in future
![Page 30: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/30.jpg)
Agile/Pair Programming
Study interaction between people doing pair programming
theoretical focus on sociology rather than psychology
See Computer-Supported Collaborative Work (CSCW) rather than HCI.
![Page 31: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/31.jpg)
Programming Aptitude
How to identify good programmers? good programmers are commercially valuable Identify talented students Identify students needing additional help
Seldom any theoretical explanation, just psychometric correlations cognitive style personality measures autism spectrum diagnoses
![Page 32: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/32.jpg)
Organizational Contexts
Speciality of Microsoft HIP group Long-term studies of professional programmers in realistic teams
Maintaining code bases on an industrial scale E.g. what activities are involved when a new
programmmer joins an established team?
Hard to achieve for academics beyond the resources of academic research
budgets relies on access to commercially sensitive
information
![Page 33: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/33.jpg)
Syntax and tools
![Page 34: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/34.jpg)
Integrated Development Environments
The language is not the only usability problem Manage modules & dependencies integrated editors debugging and visualisation tools
Some research using custom plug-ins for Eclipse
Burnett’s Forms/3 research platform Complete novel IDEs for education use
BlueJ & Scratch Extensions to CMU Alice
Ko’s WhyLine Kelleher’s storytelling Alice (compare Good’s struggles with Neverwinter Nights)
![Page 35: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/35.jpg)
![Page 36: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/36.jpg)
Visual Languages
The ambition dates back to 60s and 70s Idea of measuring improvement arrived at IEEE VL
1996 IEEE VL became IEEE VL/HCC soon afterward
Pioneering commercial products National Instruments LabVIEW Prograph
Recent examples Yahoo Pipes Microsoft Kodu Google Blockly
Most could benefit from evaluation, or application of Cognitive Dimensions
![Page 37: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/37.jpg)
![Page 38: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/38.jpg)
![Page 39: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/39.jpg)
![Page 40: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/40.jpg)
![Page 41: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/41.jpg)
![Page 42: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/42.jpg)
![Page 43: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/43.jpg)
![Page 44: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/44.jpg)
Spreadsheets
Widely used, sometimes for surprising purposes A large proportion of commercial spreadsheets
contain errors (Panko) Spreadsheet research corpus (Scaffidi)
Empirical studies and extensions: Excel Burnett’s Forms/3, with free-format cells
Specific usability improvements: testing and debugging facilities such as WYSIWYT
(Burnett) type systems and generators (Erwig) Functional programming in Excel (Peyton-Jones,
Blackwell, Burnett)
![Page 45: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/45.jpg)
![Page 46: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/46.jpg)
![Page 47: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/47.jpg)
![Page 48: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/48.jpg)
![Page 49: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/49.jpg)
Scripting Languages
Allow users to customize and extend products, e.g. LISP variants in AutoCAD (2013 guest lecture) and EMACS Linden Scripting Language (LSL) in Second Life Apple Automator (and earlier Hypercard)
Key research concern in end-user programming (later) Note that many evolve into professional languages (Perl,
Flash) While others never really considered end-user needs
(TCL, JavaScript)
Can address attention investment by starting with macro recording, then exposing source code for modification Visual Basic in Microsoft Word CoScripter for Firefox (Allen Cypher)
![Page 50: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/50.jpg)
![Page 51: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/51.jpg)
![Page 52: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/52.jpg)
![Page 53: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/53.jpg)
![Page 54: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/54.jpg)
Lecture 2: Research methods in the study of
programming.
![Page 55: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/55.jpg)
Ethical Issues in Research
Review the Cambridge Technology Ethics guide What kind of study are you planning? What potential concerns might there be? What will you do to address them?
Submit a proposal to the Computer Lab Ethics committee, giving above details.
![Page 56: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/56.jpg)
Controlled Experimental Methods
Participants (subjects), potentially in groups
Experimental task Performance measures (speed & accuracy) Trials Conditions / Treatments / Manipulations
modify the programming language use different languages Use different features of the programming
environment
Effect of treatments on sample means Within-subjects (each participant uses all versions) Between-subjects (different groups use different
versions)
![Page 57: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/57.jpg)
Controlled Experiments
Based on a number of observations: How long did Fred take to fix this bug? Did he get it right?
But every observation is different. So we compare averages:
over a number of trials over a range of people (experimental subjects)
Results often have a normal distribution
![Page 58: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/58.jpg)
Sample Distribution
number ofobservations
time
1
2
3
4
5 10 15 20 25 30 35 4540
-4 -3 -2 -1 0 1 2 3 4
normalisationmean
variance
log normalisation
“outlier”
![Page 59: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/59.jpg)
Effect Size
number ofobservation
trials
task completion time(faster)
new old
![Page 60: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/60.jpg)
Significance testing
What is the likelihood that this amount of difference in means could be random variation between samples (null hypothesis)?
Hopefully very low (p < 0.01, or 1%)
onlyrandomvariationobserved
observed effectprobably does
result fromtreatment
very significanteffect of
treatment
![Page 61: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/61.jpg)
Typical experimental tasks
Production tasks write a program that is correct, and write it quickly
Comprehension tasks understanding, interpretation or recall
Search tasks find code responsible for functionality, or bug
May be possible to use standardised tasks, for comparison to previous PPIG research See Blackwell list But ‘toy problems can lack external validity
Perhaps use the six Cognitive Dimensions activities?
![Page 62: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/62.jpg)
Experimental Manipulations
Compare productivity gains (effect size) of version with new feature to one without? Will system work without the new feature? Will the experimental task be meaningful if the feature is
disabled? Must new feature be presented second in a within-subjects
comparison (order effect) Is your system sufficiently well-designed for external validity
of productivity measure?
Test a fundamental research question? e.g. imperative vs declarative paradigms, textual vs visual
syntax Are your two languages properly representative of the
paradigms, yet also equivalent in other respects? Are your experimental tasks equally suited to different
paradigms?
Is full implementation necessary? Can you simulate features with Wizard of Oz technique?
![Page 63: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/63.jpg)
Measurement
Speed (classically ‘reaction time’) E.g. time to write program
Accuracy (number of (non)errors). Is program correct?
Trade-off between speed and accuracy? Or poor performance on both? Check correlation between them
Task completion: Stop after a fixed amount of time (ideally < 1 hour) Measure proportion of the overall task completed
![Page 64: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/64.jpg)
Self-Report
Did you find this easy to use? (Likert scale) applied value: appeal to customers theoretical value: estimate ‘cognitive load’
Danger of bias Subjective impressions of performance inaccurate Suffer from experimental demand
Participants want to be nice to the experimenter Should disguise which manipulation is the novel one
May be necessary to capture affect measures: Did you enjoy it, feel creative/ enthusiastic?
Alternative is to collect ‘richer’ data …
![Page 65: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/65.jpg)
Think-aloud
“Tell me everything you are thinking” ‘concurrent verbalisation’
Problems: Hard tasks become even harder while speaking
aloud During the most intense (interesting) periods,
participants simply stop talking,
Alternative: make video recording, or eye-tracking trace playback for participant to narrate ‘retrospective verbal report’
![Page 66: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/66.jpg)
Qualitative Data
Protocol analysis methods, e.g. verbal protocol – transcript of recorded verbal data video protocol – recording of actions
Hypothesis-, or theory-driven Create ‘coding frame’ for hypothesised categories of
behaviour Segment the protocol into episodes, utterances, phrases
etc Classify these into relevant categories (with inter-rater
reliability) Compare frequency or order statistically
Grounded theory (ch 7 of HCI Research Methods) Open coding, looking for patterns in the data Stages of thematic grouping and generalization Constant comparison of emerging framework to original
data More interpretive, danger of subjective bias
![Page 67: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/67.jpg)
Experiment Design
Arrangement of participants, groups, tasks, trials, conditions, measures, and hypothesized effects of treatments
Within-subjects designs are preferred because so much variation between programmers
This leads to order effects: first condition may seem worse, because of learning effect last condition may suffer from fatigue effect task familiarity – can’t use the same task twice
Precautions: Prior training to reduce learning effects Minimise experimental session length to reduce fatigue
effects Use different tasks in each condition, but ‘balance’ with
treatment and order
These are typically combined in a ‘latin square’ where each participant gets a different combination
![Page 68: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/68.jpg)
Analysis
For an easy life, plan your analysis before collecting data!
Will quantitative data be normally distributed? t-test to compare two groups ANOVA to compare effect of multiple conditions (which
include latin square of task and order) Pearson correlation to compare relationship between
measures
Distributions of task times are often skewed: a small number of individuals complete the task quite slowly don’t exclude ‘outliers’ who have difficulty with your system log transform of time is usually found to be normally
distributed
Subjective ratings are seldom normally distributed chi-square test of categories ‘non-parametric’ comparison of means
![Page 69: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/69.jpg)
Evaluation
Rather than testing hypothesis, or comparing treatments ask ‘is my language usable’?
More typical of commercial practice, for short-term goals, rather than general understanding Formative evaluation assesses options early in design
process Summative evaluation identifies usability problems in a
system you have built Repeated for iterative refinement in user-centred design
Weaker research, because no direct contribution to theory However some mainstream applied research venues are
starting to require evidence of any claims made for new tools
e.g. ICSE, OOPSLA/SPLASH
![Page 70: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/70.jpg)
Field Study Methods
Laboratory studies are not adequate for: organizational context of software development interaction within software development teams behaviour of programmers in actual work context
Typical methods: ‘contextual inquiry’ interviews ‘focus group’ discussions ‘case studies’ of projects or organisations ‘ethnographic’ field work as participant-observer
All result in qualitative data, often transcribed, and analysed using grounded theory approaches
You won’t have time!
![Page 71: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/71.jpg)
Lecture 3: Special classes of programming language use
![Page 72: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/72.jpg)
Educational Languages
Computer Science Education vs programming for children Papert’s Logo Kay’s Smalltalk Repenning’s AgentSheets Cypher and Smith’s StageCast Kahn’s ToonTalk Kolling’s Greenfoot Carnegie Mellon’s Alice MIT’s Scratch
Many use VL techniques, to overcome syntax problems Is it ‘cheating’ to avoid teaching syntax? Or motivate children by making it easy for them to do
things that interest them (videogames or animations) ‘learning to program’ or ‘programming to learn’? ‘user-centred’ or ‘curriculum-centred’ design?
If curriculum, what theoretical principles? Logic? Functional? Objects?
![Page 73: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/73.jpg)
![Page 74: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/74.jpg)
![Page 75: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/75.jpg)
![Page 76: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/76.jpg)
![Page 77: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/77.jpg)
![Page 78: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/78.jpg)
![Page 79: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/79.jpg)
![Page 80: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/80.jpg)
![Page 81: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/81.jpg)
End-User Programming
In Information Systems ‘user’ is a (professional) organisation ‘end-user’ is a person who will actually use the
system an ‘end-user programmer’ both writes the program
and uses it. ‘end-user development’ (EUD) ‘end-user customisation’ (EUC)
Interesting research because: An externally valid source of ‘novice’ programmers Ubiquitous computing increases market for
customisation Professional programmers don’t complain enough
![Page 82: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/82.jpg)
End-User Programming
EUP is usually defined to refer to a person who has not trained as a programmer not primarily employed as a programmer does not program for its own sake, but as a means
to an end
Motivation for end-user software engineering (e.g. testing and debugging) programs may be used by other people programs may be business-critical
Domain-specific languages Programming ‘novices’ are often domain experts LabView, MATLAB are both DSLs Even some mainstream tools are increasingly
domain-specific, e.g. WPF
![Page 83: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/83.jpg)
Creative mashups and composition
Not like military, industrial, bureaucratic domains those are well structured, with ample resources. leisure, media and the arts imply ‘discretionary-use’
digital media creators collage, sample & mash-up art strategies are next generation agile methods
Current generation of artist languages Max/MSP (+ Jitter) Processing SuperCollider
Current research in Cambridge Flow in composition Live coding EUSE for improvisation processes
![Page 84: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/84.jpg)
![Page 85: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/85.jpg)
![Page 86: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/86.jpg)
![Page 87: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/87.jpg)
![Page 88: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/88.jpg)
![Page 89: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/89.jpg)
![Page 90: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/90.jpg)
![Page 91: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/91.jpg)
Domestic automation
Classic domestic HCI challenges home heating controls VCR programming privacy configuration
Home networking WiFi, Zigbee. X10 AutoHAN software plumber or software DIY?
Research opportunities Understand domestic economy of digital
technology Apply gentle slope and attention investment
![Page 92: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/92.jpg)
![Page 93: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/93.jpg)
![Page 94: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/94.jpg)
![Page 95: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/95.jpg)
![Page 96: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/96.jpg)
![Page 97: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/97.jpg)
![Page 98: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/98.jpg)
![Page 99: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/99.jpg)
![Page 100: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/100.jpg)
![Page 101: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/101.jpg)
![Page 102: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/102.jpg)
Home ServerHome Server
Media ServerMedia Server
Radio TunerRadio Tuner
![Page 103: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/103.jpg)
![Page 104: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/104.jpg)
![Page 105: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/105.jpg)
![Page 106: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/106.jpg)
![Page 107: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/107.jpg)
![Page 108: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/108.jpg)
![Page 109: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/109.jpg)
![Page 110: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/110.jpg)
![Page 111: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/111.jpg)
![Page 112: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/112.jpg)
infraredtransceivers
singlebutton
induction coilcommunication
![Page 113: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/113.jpg)
![Page 114: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/114.jpg)
![Page 115: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/115.jpg)
Lecture 4: Planning practical empirical
studies.
![Page 116: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/116.jpg)
Goal
Prepare for design of your study Previous lectures followed:
theories of programming experimental methods specific users and programming technologies
We use reverse order: specific programming technologies and users experimental methods theories of programming
![Page 117: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/117.jpg)
Candidate programming languages/tools
your own personal research e.g. MPhil dissertation
Other research other research in Cambridge recent product releases research prototypes developed elsewhere
Who is the intended user? What will they be trying to achieve?
![Page 118: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/118.jpg)
Representative tasks and measures
Identify user activities you plan to observe assigned tasks (controlled experiment) or user’s goal (observational study)
Will these explore an interesting research question?
What measures are relevant to that question?
Will qualitative data analysis be necessary?
Will there be a threat to external validity? From task, measure or analysis
![Page 119: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/119.jpg)
Review of study design options
Do you wish to carry out a comparison, an evaluation, or an open exploratory study?
If you plan to conduct a controlled experiment, will it be possible to use a within-subjects design?
What data analysis method will you use? What would you need to do in order to complete a pilot study?
What ethical issues are raised by your planned research?
![Page 120: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/120.jpg)
Theoretical goal
What do you expect to learn from conducting your study?
What contribution will it make to the research literature relevant to usability of programming languages?
Where would you publish the results?
![Page 121: Usability of Programming Languages](https://reader036.fdocuments.in/reader036/viewer/2022062322/5681437b550346895daff864/html5/thumbnails/121.jpg)
Course structure
Assignment A, presented at seminars 1 & 2 Target language, paradigm, tool or environment Review of relevant literature Study design Outline of analytic methods
Assignment B, presented at seminars 3 & 4 Full experimental report Data analysis and findings Suitable for publication at venue such as PPIG