DIT590 Research Methods & Technical Writing

Lecture 6: Case Study, Design Research, and Action Research

Richard Berntsson Svensson http://www.rbsv.eu/courses/rmtw

Re-cap Independent&&&Dependent&Variables&

Process&Independent&variables&Dependent&variable&

What&we&want&to&study&to&see&

effect&of&changes&

What&we&manipulate&and&

control&

Produc@vity&• Development&&method&• Experience&personnel&• Tool&Support&• …..&

Factor:&use&two&treatments:&new&and&old&method&&

C. Wohlin, P. Runeson, M. Höst, C. Ohlson, B. Regnell, and A. Wessle ́n, Experimentation in Software Engineering: An Introduction. Kluwer Academic, 2000.

Case Study “Case study is a strategy for doing research which involves an empirical investigation of a particular contemporary phenomenon within its context using multiple sources of evidence”

C. Robson, Real World Research

“Case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between the phenomenon and context are not clearly evident”

R.K. Yin, Case Study Research

“A case study examines a phenomenon in its natural setting, employing multiple methods of data collection to gather information from one or a few entities (people, groups, or organizations)” Benbasat et al., The case study research strategy in studies of Information Systems

Case Studies in SE •  Software Engineering and Software Process Improvement

– Are complex activities – Success or failure depends on many interrelated factors – Cannot be fully studied in isolation – Needs empirical studies in real world setting

•  Software Engineering is different from social science and IS – Software development rather than use – Project rather than function oriented – Advanced engineers rather then routine work

Case Study

Critique •  Lack of rigor •  Generalization from a

single case? •  Take too long, results in

tedious reports •  Results in the eye of the

beholder

Countermeasures •  Rigorous data collection •  Framed with assumptions •  Understanding of many

traditions •  Procedures from many

traditions •  Idea leads to understanding •  Detailed methods for

collection, analysis and writing •  Analysis of multiple levels •  Clear writing

When to do case studies? •  Case studies are particularly appropriate when research and

theory are at their early stages, when the experience of the actors are important, and when the context is critical

•  Software engineering is carried out by individuals and organizations where social and political issues affect the process. Hence, many SE-questions are suitable for case-study research

Quality criteria for case studies •  Theoretical basis and case-study protocol •  Triangulation in methods and procedure •  Documentation of a case-study research project and case-study

report •  Designing a chain of evidence •  The logic of generalization

[Kyburz-Graber04]

Case study process

2010-10-28!

13!

Design!

Case study process

Preparation!

Collection!

Analysis!

Reporting!

•  Scope/goal •  Consent •  Instruments •  Interviews •  Recording •  Archival data •  Qualitative •  Quantitative

Runeson&Höst09

An observatory multiple case study…

•  Kitchen stories

•  Scope/goal •  Consent •  Instruments •  Interviews •  Recording •  Archival data • Qualitative • Quantitative

Case study design •  Objective—what to

achieve? •  The case—what is

studied? •  Theory—frame of

reference •  Research questions—what

to know? •  Methods—how to collect

data? •  Selection strategy—where

to seek data?

Unit(s) of analysis

Holistic vs embedded case study

Context

Case = unit of analysis

Context

Case

Unit of analysis 1

Unit of analysis 2

Single-case vs multiple-case study

Context

Case

Context

Case

Context

Case

Context

Case

Context

Case

Single-case – multiple-case E

mbe

dded

- ho

listi

c Context

Case = unit of analysis

Context

Case

Context

Case

Context

Case

Context

Case

Context

Case

Unit of analysis 1

Unit of analysis 2

Context

Case

UoA

UoA

Context

Case

UoA

UoA

Context

Case

UoA

UoA

Context

Case

UoA

UoA

Ethics •  Informed consent •  Review board approval •  Confidentiality •  Handling of sensitive results •  Inducements •  Feedback

Case study preparation •  Contact with the site •  Planning data collection •  Case study protocol •  Data sources (preparation/collection)

– First degree (direct involvement of people) – Second degree (indirect involvement of people) – Third degree (study of work artifacts only)

•  Interviews (preparation/collection) – Unstructured – Semi-structured – Fully structured

Case study analysis: Qualitative filtering/distortions

2

Data filtering

•  Quantitative •  Qualitative

Analysis!

Qualitative filtering/distorsions

Analysis!

Qualitative analysis •  Bring structure to the data

– Start by transcribing speech – Find keywords, either from the

material or from theory – Group and contrast statements – Draw conclusions

•  Coding

•  Data reduction

•  Data display

•  Conclusion drawing

[Robson 2002]

Traceability

The activity where hypotheses are identified requires some more information. This is inno way a simple step that can be carried out by following a detailed, mechanical, approach.Instead it requires ability to generalize, innovative thinking, etc. from the researcher. Thiscan be compared to quantitative analysis, where the majority of the innovative andanalytical work of the researcher is in the planning phase (i.e. deciding design, statisticaltests, etc). There is, of course, also a need for innovative work in the analysis of quantitativedata, but it is not as clear as in the planning phase. In qualitative analysis there are majorneeds for innovative and analytical work in both phases.

One example of a useful technique for analysis is tabulation, where the coded data isarranged in tables, which makes it possible to get an overview of the data. The data can, forexample be organized in a table where the rows represent codes of interest and the columnsrepresent interview subjects. However, how to do this must be decided for every case study.

There are specialized software tools available to support qualitative data analysis, e.g.NVivo and Atlas. However, in some cases standard tools such as word processors andspreadsheet tools are useful when managing the textual data.

In study XP, the transcribed interviews were initially analyzed by one of the researchers. A preliminary set ofcodes were derived from the informal notes and applied to the transcripts. The preliminary set of codes was:project model, communication, planning, follow-up, quality, technical issues and attitudes. Each statement inthe transcribed interviews was given a unique identification, and classified by two researchers. Thetranscribed data was then filled into tables, allowing for analysis of patterns in the data by sorting issuesfound by, for example, interviewee role or company. The chain of evidence is illustrated with the figure below(from Karlström and Runeson 2006)

In study RE and QA the main analysis was quantitative, although some qualitative analysis was conducted on the information that was gathered in feedback sessions. However, this analysis would probably benefit from being conducted in a more structured way, e.g. by recording, transcribing, and coding feedback data before analysis.

EMW ABB Actual events within case context

Subjects’ perceptions based on their observations and experiences

Sound recording of interview

Transcription of recording

Grouped quotes

Conclusions

152 Empir Software Eng (2009) 14:131–164

Design Science/Design Research

What is Design Science/Design Research?

Is it different from Case Study?

opportunity for IS research to make significant contributions by

engaging the complementary research cycle between design-science

and behavioral-science to address fundamental problems faced in the

productive application of information technology.

Design Science/Design Research

•  To achieve a true understanding of DS/DR: – Design is a process (set of activities) – A product (artifact)

•  Shift perspective between design process and designed artifact

[DSIS]

Design Science/Design Research

DEPARTMENT OF APPLIED INFORMATION TECHNOLOGY | www.ait.gu.se

Available'approaches'

Design'Research'

(Hevner'et'al.'2004)'''''

Ac5on'Research''

(Susman'&'Evered'1978)'''''

[DSIS]

Design Science/Design Research Hevner et al./Design Science in IS Research

MIS Quarterly Vol. 28 No. 1/March 2004 83

Table 1. Design-Science Research GuidelinesGuideline Description

Guideline 1: Design as an Artifact Design-science research must produce a viable artifact in theform of a construct, a model, a method, or an instantiation.

Guideline 2: Problem Relevance The objective of design-science research is to developtechnology-based solutions to important and relevantbusiness problems.

Guideline 3: Design Evaluation The utility, quality, and efficacy of a design artifact must berigorously demonstrated via well-executed evaluationmethods.

Guideline 4: Research Contributions Effective design-science research must provide clear andverifiable contributions in the areas of the design artifact,design foundations, and/or design methodologies.

Guideline 5: Research Rigor Design-science research relies upon the application ofrigorous methods in both the construction and evaluation ofthe design artifact.

Guideline 6: Design as a SearchProcess

The search for an effective artifact requires utilizing availablemeans to reach desired ends while satisfying laws in theproblem environment.

Guideline 7: Communication ofResearch

Design-science research must be presented effectively bothto technology-oriented as well as management-orientedaudiences.

over time. We conceive of IT artifacts not asindependent of people or the organizational andsocial contexts in which they are used but asinterdependent and coequal with them in meetingbusiness needs. We acknowledge that percep-tions and fit with an organization are crucial to thesuccessful development and implementation of aninformation system. We argue, however, that thecapabilities of the constructs, models, methods,and instantiations are equally crucial and thatdesign-science research efforts are necessary fortheir creation.

Furthermore, artifacts constructed in design-science research are rarely full-grown informationsystems that are used in practice. Instead, artif-acts are innovations that define the ideas,practices, technical capabilities, and productsthrough which the analysis, design, implemen-tation, and use of information systems can beeffectively and efficiently accomplished (Denning

1997; Tsichritzis 1998). This definition of theartifact is consistent with the concept of IS designtheory as used by Walls et al. (1992) and Markuset al. (2002) where the theory addresses both theprocess of design and the designed product.

More precisely, constructs provide the vocabularyand symbols used to define problems andsolutions. They have a significant impact on theway in which tasks and problems are conceived(Boland 2002; Schˆn 1983). They enable theconstruction of models or representations of theproblem domain. Representation has a profoundimpact on design work. The field of mathematicswas revolutionized, for example, with the con-structs defined by Arabic numbers, zero, andplace notation. The search for an effective prob-lem representation is crucial to finding an effectivedesign solution (Weber 2003). Simon (1996, p.132) states, ìsolving a problem simply meansrepresenting it so as to make the solutiontransparent.î

[DSIS]

Design Science/Design Research Hevner et al./Design Science in IS Research

86 MIS Quarterly Vol. 28 No. 1/March 2004

Table 2. Design Evaluation Methods1. Observational Case Study: Study artifact in depth in business environment

Field Study: Monitor use of artifact in multiple projects

2. Analytical Static Analysis: Examine structure of artifact for static qualities (e.g.,complexity)

Architecture Analysis: Study fit of artifact into technical IS architecture

Optimization: Demonstrate inherent optimal properties of artifact or provideoptimality bounds on artifact behavior

Dynamic Analysis: Study artifact in use for dynamic qualities (e.g.,performance)

3. Experimental Controlled Experiment: Study artifact in controlled environment for qualities(e.g., usability)

Simulation ñ Execute artifact with artificial data

4. Testing Functional (Black Box) Testing: Execute artifact interfaces to discoverfailures and identify defects

Structural (White Box) Testing: Perform coverage testing of some metric(e.g., execution paths) in the artifact implementation

5. Descriptive Informed Argument: Use information from the knowledge base (e.g.,relevant research) to build a convincing argument for the artifactís utility

Scenarios: Construct detailed scenarios around the artifact to demonstrateits utility

percent of the response time. Johansson et al.(2003) extended prior distributed database designresearch by developing a model that includesnetwork latency and the effects of parallel pro-cessing on response time.

The evaluation of designed artifacts typically usesmethodologies available in the knowledge base.These are summarized in Table 2. The selectionof evaluation methods must be matched appro-priately with the designed artifact and the selectedevaluation metrics. For example, descriptivemethods of evaluation should only be used forespecially innovative artifacts for which otherforms of evaluation may not be feasible. Thegoodness and efficacy of an artifact can berigorously demonstrated via well-selected evalua-tion methods (Basili 1996; Kleindorfer et al. 1998;Zelkowitz and Wallace 1998).

Design, in all of its realizations (e.g., architecture,landscaping, art, music), has style. Given theproblem and solution requirements, sufficientdegrees of freedom remain to express a variety offorms and functions in the artifact that areaesthetically pleasing to both the designer and theuser. Good designers bring an element of style totheir work (Norman 1988). Thus, we posit thatdesign evaluation should include an assessmentof the artifactís style.

The measurement of style lies in the realm ofhuman perception and taste. In other words, weknow good style when we see it. While difficult todefine, style in IS design is widely recognized andappreciated (Kernighan and Plauger 1978; Wino-grad 1996). Gelernter (1998) terms the essenceof style in IS design machine beauty. He de-scribes it as a marriage between simplicity and

[DSIS]

Validity •  Construct

– Are we measuring/observing the right thing?

•  Internal – Is the study conducted well?

•  External – Is the setting representative?

•  Conclusion/reliability – Are the statistics/analyses used correctly?

Reporting a case study •  An example

– Your case is to study the implementation of a sales support system called SALES in a small consultancy firmed named TriCoLor

– Title: Implementation of SALES at TriCoLor – Objective: Our objective is to study how TriCoLor

implements the SALES (version 4.07) system in their Gothenburg office.

– RQ. What challenges does TriCoLor encounter when trying to install SALES?

Don’t do this!!!

Reporting a case study •  A better example:

– Title: Implementing a sales support system in a consultancy firm

– Objective: Our objective is to understand better the process of implementing a sales support system

– RQ: What challenges does a consultancy firm encounter when installing a sales support system?

Reporting a case study •  A really good example:

– Title: Implementing an administrative system in a people-centric organization

– Objective: Our objective is to understand better the process of implementing an administrative system in an organization where people are the main assets.

– RQ: What challenges does a people-centric organization encounter when installing an administrative system

Reporting a case study •  But what about TriCoLor and SALES? •  They are replaceable parts of your context •  So you write:

To answer our research question we designed a case study at TriCoLor, a small Gothenburg-based software consultancy firm who are in the process of installing a sales support system called SALES

Reporting a case study •  Describe your case and your context carefully, both in terms of

particulars and details AND in “theoretical” terms •  i.e., describe the company as “people-centric” and show how

that is manifested by providing details from TriCoLor •  i.e., describe the system as “administrative” and show what

features and characteristics SALES has in this respect

Reporting a case study •  Describe your method •  Skip the textbook summary and write instead what you actually

did •  BUT: show that you have read the textbook by using adequate

terminology and refer whenever suitable •  Provide enough details for the reader to be able to follow your

thoughts (=transparency)

Reporting a case study •  Provide rich data from your case! •  Let the reader enter the field together with you •  You need to summaries, but let the voices of the respondents

be heard at times via quotes •  This part is very TriCoLor- and SALES-centric!!!

Reporting a case study •  In your discussion, you lift your findings and go back to the

general level •  Start in the local/specific but try to generalize by referring to

theory •  The conclusions should NOT be about TroCoLor or SALES

Action Research

DEPARTMENT OF APPLIED INFORMATION TECHNOLOGY | www.ait.gu.se

Available'approaches'

Design'Research'

(Hevner'et'al.'2004)'''''

Ac5on'Research''

(Susman'&'Evered'1978)'''''

To do … § Work on assignment 2 and 3

§ due Monday February 23, 8.00am § Attend Wednesday’s lecture about technical writing