Empirically Assessing End User Software Engineering Techniques Gregg Rothermel Department of...
-
date post
20-Dec-2015 -
Category
Documents
-
view
213 -
download
0
Transcript of Empirically Assessing End User Software Engineering Techniques Gregg Rothermel Department of...
Empirically Assessing End User Software Engineering
Techniques
Gregg Rothermel
Department of Computer Science and EngineeringUniversity of Nebraska -- Lincoln
Questions Addressed
• How can we use empirical studies to better understand issues/approaches in end user SE?
• What are some of the problems empiricists working on end-user SE face?
• What are some of the opportunities for software engineering researchers working in this area?
Outline
• Background on empirical approaches• Empiricism in the end-user SE
context• Problems for empiricism in end-user
SE• Conclusion
Outline
• Background on empirical approaches• Empiricism in the end-user SE
context• Problems for empiricism in end-user
SE• Conclusion
Empirical Approaches: Types
• Survey – interviews or questionnaires• Controlled Experiment - in the laboratory,
involves manipulation of variables• Case Study - observational, often in-situ
Empirical Approaches: Surveys
• Pose questions via interviews or questionnaires
• Process: select variables and choose sample, frame questions that relate to variables, collect data, analyze and generalize from data
• Uses: descriptive (assert characteristics), explanatory (assess why), exploratory (pre-study)
Resource: E. Babbie, Survey Research Methods, Wadsworth, 1990
Empirical Approaches:Controlled Experiments
• Manipulate independent variables and measure effects on dependent variables
• Requires randomization over subjects and objects (partial exception: quasi-experiments)
• Relies on controlled environment (fix or sample over factors not being manipulated)
• Often involves a baseline (control group)• Supports use of statistical analyses
Resource: Wohlin et al., Experimentation in Software Engineering, Kluwer, 2000
Empirical Approaches: Case Studies
• Study a phenomenon (process, technique, device) in a specific setting
• Can involve comparisons between projects• Less control, randomization, and
replicability• Easier to plan than controlled experiments• Uses include • larger investigations such as longitudinal or
industrialResource: R. K. Yin, Case Study Research Design and Methods, Sage Publications, 1994
Empirical Approaches: Comparison
Factor Survey Experiment Case Study
Execution Control
Low High Low
Measurement Control
Low High High
Investigation Cost
Low High Medium
Ease of Replication
High High Low
Outline
• Background on empirical studies• Empiricism in the end-user SE
context• Problems for empiricism• Conclusion
Three Aspects of Empiricism
1. Studies of EUSE (and SE) have two focal points
– The ability of end users to use devices/processes– The devices and processes themselves
2. Evaluation and design of devices and processes are intertwined:
– Summative evaluation helps us assess them– Formative evaluation helps us design them
3. We need families of empirical studies:– To generalize results– Studies inform and motivate further studies
DomainAnalyses
Think-Aloud, FormativeCase Studies, Surveys
ControlledExperiments
Controlled Experiments
Summative Case Studies
Exploratory,Theory Dev.
HypothesisTesting
Generalization
Building Empirical Knowledge through Families of Studies
userenvironment,device
DomainAnalyses
Think-Aloud, FormativeCase Studies, Surveys
ControlledExperiments
Controlled Experiments
Summative Case Studies
Exploratory,Theory Dev.
HypothesisTesting
Generalization
Building Empirical Knowledge through Families of Studies
userenvironment,device
Empirical Studies in WEUSE Papers
• Surveys- Scaffidi et al.: usage of abstraction, programming
practices- Miller et al.: how users generate names for form
fields- Segal: needs/characteristics of professional end
user developers- Sutcliffe: costs/benefits perceived by users of a
web-based content mgmt. system• Domain analysis
– Elbaum et al.: fault types in Matlab programs• Controlled experiments
– Fisher et al.: infrastructure support for spreadsheet studies
Cell turns more blue (more “tested”).
Testing also flows upstream, marking other affected cells too.
Example: What You See is What You Test (WYSIWYT)
At any time, user can check off correct value.
DomainAnalyses
Think-Aloud, FormativeCase Studies, Surveys
ControlledExperiments
Controlled Experiments
Summative Case Studies
Exploratory,Theory Dev.
HypothesisTesting
Generalization
Building Empirical Knowledge of End User SE through
Families of Studies
userenvironment,device
Study 1: Effectiveness of DU-adequate test suites (TOSEM
1/01)• RQ: Can DU-adequate test suites detect
faults more effectively than other types of test suites?
• Compared DU-adequate vs randomly generated suites of the same size, for ability to detect various seeded faults, across 8 spreadsheets
• Result: DU-adequate suites were significantly better than random at detecting faults
DomainAnalyses
Think-Aloud, FormativeCase Studies, Surveys
ControlledExperiments
Controlled Experiments
Summative Case Studies
Exploratory,Theory Dev.
HypothesisTesting
Generalization
Building Empirical Knowledge of End User SE through
Families of Studies
userenvironment,device
• RQs: Are WYSIWYT users more (effective, efficient) than Ad-Hoc?
• Compared two groups of users, one using WYSIWYT, one not, each on two spreadsheet validation tasks
• Participants drawn from Undergraduate Computer Science classes
• Participants using WYSIWYT were significantly better at creating DU-adequate suites, with less redundancy in testing
Study 2: Usefulness of WYSIWYT (ICSE 6/00)
DomainAnalyses
Think-Aloud, Formative Case Studies, Surveys
ControlledExperiments
Controlled Experiments
Summative Case Studies
Exploratory,Theory Dev.
HypothesisTesting
Generalization
Building Empirical Knowledge of End User SE through
Families of Studies
userenvironment,device
Study 3: Usefulness of WYSIWYT with End Users (ICSM
11/01)• RQs: Are WYSIWYT users more (accurate,
active at testing) than Ad-Hoc?• Compared two groups of users, one using
WYSIWYT, one not, each on two spreadsheet modification tasks
• Participants drawn from Undergraduate Business classes
• Participants using WYSIWYT were more accurate in making modifications, and did more testing
User can enter assertions
System can figure out more assertions
User can enter assertions
Study 4: Using Assertions (ICSE 5/03)
DomainAnalyses
Think-Aloud, FormativeCase Studies, Surveys
ControlledExperiments
Controlled Experiments
Summative Case Studies
Exploratory,Theory Dev.
HypothesisTesting
Generalization
Building Empirical Knowledge of End User SE through
Families of Studies
userenvironment,device
• RQs: will end users use assertions and do they understand the devices
• Observed persons as they worked with Forms/3 spreadsheets with assertion facilities provided
Study 4: Using Assertions (ICSE 5/03)
Outline
• Background on empirical studies• Empiricism in the end-user SE
context• Problems for empiricism in end-user
SE• Conclusion
Problems for Empiricism in EUSE
• Threats to validity – factors that limit our ability to draw valid conclusions– External: ability to generalize– Internal: ability to correctly infer
connections between dependent and independent variables
– Construct: ability of dependent variable to capture the effect being measured
– Conclusion: ability to apply statistical tests
External Validity
• Subjects (participants) aren’t representative• Programs (objects) aren’t representative• Environments aren’t representative• Problems are trivial or atypical
Internal Validity
• Learning effects, expectation bias, …• Non-homogeneity among groups (different
in experience, training, motivation)• Devices or measurement tools faulty• Timings are affected by external events• The act of observing can change behavior
(of users, certainly, but also of artifacts)
Construct Validity
• Lines of code may not adequately represent amount of work done
• Test coverage may not be a valid surrogate for fault detection ability
• Successful generation of values doesn’t guarantee successful use of values
• Self-grading may not provide an accurate measure of confidence
Conclusion Validity
• Small sample sizes• Populations don’t meet requirements for
use of statistical tests• Data distributions don’t meet
requirements for use of statistical tests
Other Problems
• Cost of experimentation• Difficulty of finding suitable subjects• Difficulty of finding suitable objects• Difficulty of getting the design right
Outline
• Background on empirical studies• Empiricism in the end-user SE
context• Problems for empiricism in end-user
SE• Conclusion
Questions Addressed• How can we use empirical studies to better
understand issues/approaches in end user SE?– Via families of appropriate studies, using
feedback and replication• What are some of the problems empiricists
working on end-user SE face?– Threats to validity, many particular to this area– Costs, and issues for experiment design/setup
• What are some of the opportunities for software engineering researchers working in this area?– Myriad, given the range of study types applicable– Better still with collaboration