Case studies in industry - fundamentals and lessons learnt
-
Upload
daniel-mendez-fernandez -
Category
Science
-
view
322 -
download
1
description
Transcript of Case studies in industry - fundamentals and lessons learnt
Fundamentals and
Technische Universität München
Case Studies in Industry
Daniel Méndez !!
Technische Universität München Germany
!IASESE 2014 Torino, Italy
@mendezfe
Based on material from a joint work with: Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering) Stefan Wagner (University of Stuttgart) !Feedback from Antonio Vetrò (Technische Universität München)
What is a case study?
Empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. [Yin 2002]
In quest for a case study definition We opt for realistic environments
Theoretical space (methods, models)
Theory
Environment made irrelevant
Artificial (controlled) environment
Realism
Reality
In quest for a case study definition We rely on a mix of qualitative & quantitative data
Quantitative Qualitative
Case studies
Surveys and interviews
Controlled experiments
Quasi experiments
Action research
Grounded Theory
Data collection procedure
In quest for a case study definition We rely on a mix of empirical methods
Case studies
Action research
Grounded Theory
Data analysis procedure
Surveys and interviews
* Example
**
!!
Most* case studies:
• are conducted in socio-economic contexts
• are of confirmatory nature (“Let’s see if my method is better than yours”)
• rely on qualitative data (e.g. as part of interview transcripts)
• rely on a mix of various research methods, often (technical) action research
In quest for a case study definition Summary of some important characteristics
Cas
e st
udy
rese
arch
Data type
Setting
Purpose
Mixed w/
•Confirmatory •Exploratory
•Qualitative •Quantitative
•Realistic environments •Realistic models
* Unsupported claim, don’t sue me.
•Action research •Interview research •…
Agenda
1. Case study research: Why and how? 2. Examples 3. Challenges in case study research 4. Time for questions
Agenda
1.Case study research: Why and how? 2. Examples 3. Challenges in case study research 4. Time for questions
Learn from realistic (socio-economic) environments with realistic conditions: • subjects with: opinions, experiences, expectations, fears, beliefs, … • cases with: realistic models, realistic complexity, realistic decisions, … … at the cost of control (and internal validity) !
» Ideally: Explore the unpredictable (e.g. when testing a SE method)
Why case studies? Learn from real life environments
Disclaimer Cases studies do not imply more “validity” or “trustworthiness” than experiments - they complement each other!
Why case studies? Case studies support scaling up to practice
Lab credibility
Street credibility
Simple model
Realistic model
Small sample Large sample
Based on: WieringaEmpirical Research Methods for Technology Validation: Scaling Up to Practice
Focus of case studies
(Focus of field studies and replications)
Scaling u
p to prac
tice
Focus of (lab) experiments
What is the lifecycle of a case study?
Case study design
Preparation for data collection
Data collection Data analysis Reporting
• Research objectives • Research goals • …
• Schedules • Protocols
• Document analyses • Interviews, Surveys • Action research • …
• Descriptive statistics • Grounded theory • …
• Packaging • (Publishing)
Further reading
Agenda
1. Case study research: Why and how?
2.Examples 3. Challenges in case study research 4. Time for questions
!
!
Example #1
Model-based process enactment
Develop and evaluate an approach to dynamically test the conformance of software project (instances) and software process models
Problem analysis Evaluation
Development (TAR*)
Case study research
InterviewsDocument
analysis
* Technical Action Research
Results
• Moving targets detected when designing the instruments for the interviews
• New goals had no scientific relevance (very low theoretical and practical impact)
Problem analysis Evaluation
Development (TAR)
Case study research
InterviewsDocument
analysis
!
!
Example #2
Develop and evaluate a model-based requirements engineering approach for embedded reactive systems in the automotive domain
Problem analysis Evaluation
Development (TAR)
Case study research
InterviewsDocument
analysis
Model-based requirements engineering
Results
No access to software project environments (“The ivory tower”)
Problem analysis Evaluation
Development (TAR)
Case study research
InterviewsDocument
analysis
!
!
Example #3
Evaluate a model-based requirements engineering approach for business information systems in comparison to previously used approach
Model-based requirements engineering (2nd try)
Case study
Assessment(Interview)
TAR workshops
Case study
Assessment(Interview)
TAR workshops
Results
The new model-based approach supported:
• more flexibility in process
• higher productivity
• higher syntactic quality in artefacts
!Conclusion: The new model-based approach better supports a flexible process with high quality artefacts.
See also: Mendez Fernandez et al. A Case Study on the Application of an Artefact-based Requirements Engineering Approach !
Conclusion: The new model-based approach better supports a flexible process with syntactically consistent results.
What problems do you see in this case study (and conclusions)?
• Action research workshops inherently threaten the internal validity
• High degree of subjectivity
• Results sensitive to their context (generalisation?)
• …
!» Each “successful” case study still offers plethora of problems and challenges!
Agenda
1. Case study research: Why and how? 2. Examples
3.Challenges in case study research 4. Time for questions
Industrial research cooperations with
• up to 3 years duration
• 3-30 participants
• multiple case studies
We distinguish three classes of challenges:
1. Approaching a case study
2. Conducting a cases study
3. Transversal challenges
(…)
Challenges in approaching a case study
» Integrate studies into ongoing / planned research projects
» Actively approach
– alumni of your university
– companies in your area
– practitioners at (local) events
Challenges in approaching a case study #1: Finding contacts
Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1 !
Challenges in approaching a case study #2: Convincing contacts to participate
» Find appropriate terminology (technology transfer)
» Find appropriate incentives for project partners
» Respond to their local needs
» Provide early feedback / results
» In funded projects, probability is higher that practitioners collaborate
Challenges in approaching a case study #3: Opportunistic case and subject selection
» Welcome to case study research!
Image source: xkcd !
You need to: • schedule meetings / workshops • allocate resources • (formally) define deliverables • clarify responsibilities • clarify NDA-related issues • …(the list can get long)
Challenges in approaching a case study #4: Planning a case study
!» Plan a case study like you’d plan a software development project:
– Time plan and resource allocation
– Quality assurance and acceptance of protocols and reports
– …
Image source: https://c1.staticflickr.com/1/186/467087455_e90b880c92.jpg !
Challenges in approaching a case study #5: Uncertainty
• What will the stakeholder characteristics be? • Availability • Skills • Motivation to participate • Commitment to goals • Representativeness • …
• What will the data look like? • Quality of the data • Quantity of the data
» Take early samples and test them!
» Test instruments and data quality via pilots
» Get to know your subjects early
» Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”
Challenges in conducting a case study
Context variables (and phenomena) • hard to determine (or unavailable) • hard to measure
» Context often hard to describe (hence, hard to reproduce by others)
Variables describedin reportChallenges in conducting a case study
#1: Properly characterising the context
» Orient yourself on
» other (similar) studies or
» on classification schemes from the area of software process models (“Tailoring”)
» Focus on whole context, not only on cases for which you want to draw conclusions
» Ask yourself: “What information would I need to understand and replicate the study?”
Variables you should measure
Variables youcan measure
Possible problems:
• No clear communication of case study in organisation
• Case study delegated to s.o. not interested
• No clear communication lines and reporting lines
» Define clear communication/reporting/escalation lines at the beginning. Take no shortcut!
» When communicating with subjects:
» Communicate purpose and goals of case study
» Find a supporter at project level (with own interests)
» Don’t call them subjects ;-)
Challenges in conducting a case study #2: Communication
Image source: https://c1.staticflickr.com/1/32/62630024_10a8fa36ce_z.jpg?zz=1 !
Challenges in conducting a case study #3: Defining the proper instruments
» Pre-tests your design, e.g.
» Pilot your surveys
» Check if planned data analysis techniques apply
» A way to calibrate the instruments is to conduct a small experiment with students first
Image source: How to lie with statistics !
Possible problems:
• Wrong assumptions about population
• (Research) questions not thought through
Challenges in conducting a case study #4: The data collection might take long…
» Be patient
» Never push!
Image source: https://c1.staticflickr.com/9/8185/8375141385_df9bd4f283_z.jpg !
Challenges in conducting a case study #5: Working with sensitive data
» Clarify the rules:
» what you need for the case study
» what can be stored and reported (internally and externally)
» Plan review and acceptance cycles for reports and publications
» The “easier” way (sometimes): Joint publication
Possible distortions in data collection: • biased researchers
(trying to sell own PhD topic) • weak moderation skills • hidden agendas of respondents
(missing trust!) • (…)
Challenges in conducting a case study #6: (Social) skills
» Don’t be biased!
» Involve other researchers in:
» Design of instruments
» Data collection (e.g. interviews)
» Continuously practice your
» Moderation skills
» Rhetoric and listening skills
» Build trust by showing loyalty and respect
Challenges in conducting a case study #7: Analysing qualitative data
Qualitative data • is inherently subjective • is difficult to interpret
(we all have different mental models!)
» When analysing qualitative data
» Work in teams (researcher triangulation)
» When making interpretations, document the rationale
» Validate the results with the subjects (if possible)
» Be careful when drawing conclusions
Image source: How to lie with statistics !
• Moving targets are something natural • The difficult part:
• Detecting moving targets • Reacting to them
Challenges in conducting a case study #8: Moving targets
» Continuously provide and get feedback
» When planning your study, agree on
» key goals and questions
» open space left for new research questions (flexible part)
» Goals should always meet the interest of both industry and academia
Challenges in conducting a case study #9: Tool support
Unfortunately, still the best tools we have…
Transversal challenges
Possible problem: Support by management…only by management.
» No backup in projects (organisational culture)
» Make clear study plan including needed subjects & cases (and formally agree on it)
» Talk early to subjects
» Find active supporter (not sponsor) with own interests in project (= their own incentives)
Transversal challenges #1: The ivory tower
» „[Even if] one swallow does not make a summer, one swallow does...
» prove the existence of swallows
» provide a great deal of reliable information about swallow anatomy“*
» Do not be that kind of reviewer that asks for both realistic models and large sample sizes
Transversal challenges #2: Sometimes, size seems to matter…
* Y. Anzai and H.A. Simon. The theory of learning by doing (1979) / H. Sharp. Keynote (EASE 2012)
Transversal challenges #3: Drawing proper conclusions
» Knowledge is more than statistical significance
» When analysing the results, be accurate and critical!
» Stay on the ground
» You may interpret, as long as the interpretation is clearly stated as such
» Rely on other case studies that might explain your observations
Image source: xkcd !
Transversal challenges #4: The quest for universal truth (“generalisation”)
1. Software engineering is about human beings (Subjectivity) 2. Reality cannot be captured in complete system of variables !» “Truth” will always be something relative to its context
» Aim for trustworthiness and transparency rather than for int./…/ext. validity
» Support scaling up to practice
» Appreciate the value of replication studies
» Align your study with similar ones
» Make clear relation to existing evidence
» Disclose your data (e.g. via promisedata)
See also: Mendez et al. In Quest for Requirements Engineering Oracles: Dependent Measurements and Variables for (good) RE
Case study data should be disclosed • reliability and trustworthiness? • reproducibility and replicability?
» The success of empirical SE relies on a collaborating community
Transversal challenges #5: Openness and transparency
» If possible, make your (anonymised) data accessible to others
Transversal challenges #6: Properly reporting the results
Case study results need to be disseminated • to practice • to academia
» How to properly report?
» Report results according to target group:
» Small presentations to management w/ focus on results
» Peer-reviewed publication w/ (selected) research scope
» (Technical) Reports with full data (and analysis)
» …
» Rely on established guidelines for reporting on case studies
!
KEY TAKE AWAYS
My personal manifesto for case study research
6. Be unbiased
7. Be accurate and critical
8. Be honest
9. Be patient
10. Be flexible (and pragmatic!)
1. Carefully plan your study like a project
2. Make clear goals, purpose, context, threats
3. Train your empirical and social skills
4. Disclose your data (reliability & replicability)
5. Collaborate
Agenda
1. Case study research: Why and how? 2. Examples 3. Challenges in case study research
4.Time for questions
Don’t ask ask us!,
!!!
Thank you!Daniel Méndez
@mendezfeFurther reading
• Slides will be made available (and probably tweeted)
• Ask me if you need material (studies, templates, …)