ICT-ACTION DESIGN RESEARCH METHOD

Raimo Hälinen (2012)Lahti 14.12.2012

Agenda 12.00 – 13.30 Action Design Research 13.30 – 14.00 Cafee break 14.00 – 15.00 Using Action Design Reseach

method Discussion how to apply ADR-method

15.00 – 16.00 Comparison DSR, AR and ADR methods how to generalize results using by deduction and

induction Summary and conclusion

Information systems design science research frameworks

ISDSR Framework Activity Framework for DSR

Dominant paradigm

Design science Utility theory, Problem theories

Focus Solution focused Problem-oriented, solution focused

Perspective Researcher as experimenter (intervener)

Researcher as theory developer and experimenter

Logic Intervention-outcome Enhancement or creation of a method, product, system, practice or technique

Research question

Alternative IS interventions for classes of problems

Problem space understanding and causes and consequences

Research product

Tested and grounded technological rules (design knowledge)

Tested and evaluated product byField studies, ExperimentsAction research, Simulations

Nature of research product

Heuristic Iterative cyclical process from theoretical basis to evaluated and implemented product

Justification Saturated evidence Meta-design and meta-requirements

Type of resulting theory

Practical and abstract IS design theory and knowledge

Practical and scientific IS design theory and knowledge

Source: Walls et al (1992), van Aken (2004), Carlsson (2006), Venable (2006)

Paradigmatic dimensionParadigmatic dimension

Action research Design science research

Ontology Anti-realism Realism or anti-realism

Epistemology Mainly anti-positivism Mainly positivism but also anti-positivism especially in evaluation

Methodology Idiographic Constructive (building)Nomotethic (evaluation)Idiographic (evaluation)

Ethics Meand-endPossible interpretiveUnlikely critical

Means-endPossible interpretivePossible critical

Source: Iivari and Venable (2009)

“Considering the practical implications of this analysis, we further identify that the employment of AR in the conduct of DSR needs to be done with care, especially where there is the potential forsignificant risk to the client or other stakeholders.”

Basic assumptions

Action design research approach

Action research criterion

Design research criterion

Problem identification and definition

The principle of research-client agreement

Design as an artifactProblem relevance

Build, intervention and evaluation (BIE)

The principle of cyclical process modelThe principle of theory

Design evaluationResearch rigorDesign as a search process

Reflection and learning The principle of change through actionThe principle of learning through action

Communication of research results for practitioners and researchers.

Formalization of learning

Research contribution

Source: Cole, Purao, Rossi and Sein (2005)

Similarities of AR and DSRAction research Design science research

Action research emphasizes the utility aspect of the future system from the people’s point of view.

Design science’s products are assessed against criteria of valua or utility.

Action research produces knowledge to guide practice in modification.

Design science produces design knowledge (concepts, constructs, models, and methods.)

Action research means both action taking and evaluating.

Buildin and evaluation are the two main activities of design science.

Action researc is carried out in collaboration between action researcher and and the client system.

Design science research is initiated by the researcher(s) interest in developing technological rules for a certain type of issue. (Each individual case is primarily oriented at solvin the local problem in close collaboration wtih the local people.)

Action research modifies a given reality or develops new system.

Design science solves construction problems (producing new innovations) and improvement problems (improving the performance of existing entities).

The researcher intervenes in the problem setting.

Design science research is initiated by the researcher (s) interest in developing technological rules for certain type of issue.

Knowledge is generated, used, tested adn modified in the course of the action research project.

Knowledge is generated, used and evaluated through the building action.

Source: Järvinen P: (2005, 2007)

Action Design Reserch

1. Problem FormulationPrinciple 1. Practice-Inspired researchPrinciple 2. Theory-Ingrained ArtifactPrinciple 3. Data-inspired research

2. Building, Intervention, and EvaluationPrinciple 3. Reciprocal ShapingPrinciple 4. Mutually Influential RolesPrinciple 5. Authentic and Concurrent Evaluation

3. Relfection and learning

Principle 6. Guided Emergence

4. Formalization of LearningPrinciple 7. Generalized Outcomes

Tasks in the problem formulation

1. Identify and concepualize the research opportunity2. Formulate initial research questions3. Cast the problem as an instance of a class of

problems4. Indentify contributing theoretical bases and prior

technology advances5. Secure long-term organizational commitment6. Set up roles and responsibilities

Orlikowski and Iacono (2001) defined IT artifacts as the “bundles of material and cultural properties packaged in some socially recognizable form such as hardware and/or software”.

Problem Formulation

Principle 1. Practice-Inspired Research

The principle emphasizes to consider field problems (real-world problems)Action design research method is applied to study intersection of technological and organizational domains.

The researcher can investigate how technological solutions can be used to improve organizational processes and achieve better results.

An action design Researchers should try to generate knowledge that can be applied at the class of problems. (The cases are examples of the recognized class of problem).The result from the research activity is problem-inspired.

Problem Formulation ...

Principle 2. Theory-Ingrained ArtifactThe principle emphasizes that created artifacts are based on theories.

Gregor (2006) explored systems of statements that allow generalization and abstraction to be theories. Gregor’s theory of Type IV (explanation and prediction theories) or Type V (design theories)

The applied theories can be universal laws of natural science or ADR-researcher can utilize the specific theory (e.g. TAM, IS-succes) .

1. Identify problem, 2. Identify potential solutions, 3. design guide.

Ingrained artifact is subject to organizational practice and it provides basis for cycles of intervention, evaluation and further reshaping.

Building-intervention and evaluation

ADRteam

The Generic Schema for IT-dominant building, intervention and evaluation

Practitioners

End-users

Artifact

Design principles

Contributionto the specificensemble beingdesigned

Utility for the users

Researcher(s)

Alpha version

Beta version

Researcher(s)

Practitioners

End-users Alpha version Beta version

Implementation of Artifact

Design principles

Contributionto the specificensemble beingdesigned

Utilityfor the users

ADRteam

The Generic schema for Organization-Dominant BIE

1. Discover initial knowledge creation target2. Select or customize BIE form3. Execute BIE cycle(s)4. Assess need for additional cycles, repeat

Implementationof an artifact can lead to modification of organizational processesand work flows.

Building-intervention-evaluation

Researchers

Practitioners

End-users

ARDteam

Alpha version Beta versin

The Generic Schema for Data-Dominant BIE

Database

Utility forthe users

Contributionto the specificdatabase beingdesigned

Designprinciples

Principle 3: Reciprocal ShapingThe iterative process that are described is based on DeGrace and Stahl (1990) recursive cycles.

Scrum process

Source: scrumalliange.org, DeGrace P and Stahl L. (1990), Sutherland J. (2010)

Scrum meeting 30 min.1. What did you do yesterday?2. What will you do today?3. What obstacles got in your way?The roles of Scrum project4. The product owner5. The Team6. The Scrum MasterThe basic concecpt7. The product backlog8. The Sprint9. Sprint planning10.Daily scrum meeting11.Sprint review and retrospective

Principle 4: Mutually Influential Roles

The action design researcher’s role is to share the knowledge of theory and technological advances.The practitioner’s role is to consider practical hypotheses and knowlegde of organizational work practices.Researchers and practitioners’ role are complementary and complate each others.The clear assigment of these responsibilities is important that reflection and experience can be utilized during the research projects.

Principle 5: Authentic and Concurrent Evaluation

The evaluation is essential activities in building, intervention and evaluation process.The evaluation is not a separate stage as it is e.g. March and Smith (1995) state-gate models or compared to Peffers et al.’s (2008) model.The evaluation is ongoing and continues from start to the end of the project. The style of evaluation is formative (Remenyi and Sherwood-Smith (1999).The summative evaluation is utilized for beta version evaluation.The controlled evaluation may be difficult to achieve, so it should be carried as part of natural controls where possible.

Reflection and Learning Principle 6: Guided EmergenceThe designed artifact will reflect preliminary design and ongoing shaping by organizational use, perspective and participants.

Component of ISDT

Design product

1. Meta-requirement2. Meta-design

Describes the class of goals to which the theory applies.Describes a class of artifacts hypothesized to meet meta-requirements.

3. Kernel theories Theories from natural or social sciences governing design requirements.

4. Testable design product hypotheses

Used to test whether the meta-design satisfies the meta-requirements.

Design process

1. Design method A description of procedure(s) for artifact construction.

3. Kernel theories Theories from natural or social sciences governing design process itself.

4. Testable design process hypotheses

Used to verify whether the design method results in an artifact which is consistent with meta-design.Source: Walls et al. (1992, 2004)

Kernel theories

Meta-requiremen

s

Meta-design

Testable desing product

Kernel theories

Desing method

Testable design process

Formalization of Learning

The objective is to formalize the learning.

According to Van Aken (2004), the situated learning should develop general solution concepts for a class of field problems.

Principle 7: Generized Outcomes1. Generalization of the problem instance2. Generalization of the solution instance3. derivation of design principles from the design

research outcomes.

Tasks in the formalization of learning

1. Absract the learning into concepts for a class of real-world problems.

2. Share outcomes and assessment with practitioners3. Articulate outcomes as design principles4. Articulate learning in light of theories selected5. Formalize results for dissemination

Tsang and Williams (2012) Definition of statements:a statement is empirical when and only when it cannot be ascertained to be true or false without experience or observation. a statement is theoretical when and only when it is a generalization that purports to predict and explain the phenomena to which it refers.

Mid-range theorizing in DR

Evidence

Cause Effect

Prescribed action

Explanatory statement

Goal

Prescribtive statement

Articact evaluatio

n

Kerneltheories

Mid-rangetheories

Designtheories

Theory development

Might lead to

Is installed to lead to

Source: Kuechler and Vaishnavi (20008)

can be confirmed byleads to revision of

Kuechler and Vaishnavi (2008) pointed out to need for to create mid-range theories. The mid-range theories act as a bridge between kernel theories and design theories.

Levels of generalization from DR

Design Theory

Design Principle

s

Designproces

s

Class ofProblems

ProblemSolution as an artifact in

use

Class of Solutions

Four levels in conceptual process:1. Generalization of the problem

instance2. Generalization of the solution

instance3. Emerging design knowledge in

the form of design principles4. Feedback to design theory

Level 1 Level 2Level 3

Level 4

Rossi, Purao and Sein (2012)

Yin’s conception of generalization

Trying to generalize results of the research project can be characterized using by Yin’s process(1994, p.31).

TheoryRival

theory

Populationcharacteristi

cs

Sample of research

Case study

findings

Experimental findings

Subjects

Level-2inference

Level-1inference

A generalizability framework

Lee and Baskerville (2003) considered generalization problem and proposed the framework.

Generalizing empirical statements

Generalizing to theoretical statements

Generalizing from empirical statements

(EE) Generalizing from data to description.Measurement, observation or other description.It may be Inductive analogy.

(ET)Generalizing from desription to theory.Measurement, observation or other description to a theory

Generalizing from theoretical statements

(TE) (emprical testing)Generalizing from theory to description.Confirmed in one setting, to descriptions of other settings.(Deduction)

(TT) ?

EE = emprical to emprical, ET = empirical to theoryTE = theory to empirical, TT = theoretical to theory

Accroding to Järvinen P review this framework may be problematic?

Types of induction

Theory

Same populatio

n

Different population

Different context

Differenttime

Within-populationgeneralization

Cross-populationgeneralization

Contextualgeneralization

Temporalgeneralization

Sample

Particularinstance

Theoreticalgeneralization

Inductive analogy

Statistical syllogism

Empirical level

Source: modfied from Tsang and Williams (2012)

Definition of types of inductionTypes Definition

Theoretical Generalization from research findings to theories

Within-population

Generalizing from the characteristics of a sample to those of the corresponding population.

Cross-population

Generalizing from a sample in one population to members of another population, with both population existing in a similar context and a similar period of time.

Contextual Generalizing from a sample in one population to members of another population, with both population existing in a significantly different context but a similar period of time.

Temporal Generalizing from a sample in one population at one point in time to members of the same or a different population at another point in time, assuming that the context remains more or less the same.

Statistical syllogism

An inference of the formP1 N % of Fs are Gs.P2 X is an F.C X is a G.where “N” denotes a precise statistic or a vague range of statistics as in “Most” or“Nearly all.”

Inductive analogy

An inference of the formP1 X has properties a, b, c … and zP2 Y has properties a, b, c …C Y has property zSource: adopted from Tsang and Williams (2012

Comparison of DR, AR and ADR

Property Design research

Action research

Action design research

Artifact Central Peripheral Central

Organizational impact

Peripheral Central Central

Subject participation in research design

Possible Mandatory Mandatory

Subject Feedback Discrete Continous Continous

Transferability Explicit Implicit Explicit

Success measure Quantifiable measures of artifact behaviour

Organizational impact

Organizational learning and artifact generalizability

Source: Henfidsson (2011)

Comparison of Quantitative and Qualitative research approaches

Orientation Quantitative Qualitative

Assumption about the world

A single reality and specified real-world phenomenon.

Multiple realities and real-world phenomena.

Purpose of the research

Trying to establish relationships between measured variables.

Trying to understand social situation from participants’ perspective.

Methods and processes

Research procedures and activities are established before research begins.Hypotheses are formulated before study can begin.Deductive approach.

Research procedures and activities varies depending on data gathering and how the study is proceeding.Saturation point is essential for data collection process.Inductive approach.

Researcher’s role

Ideally an objective observator. The researcher participates in the study activities, and it can have different roles during the research project.

Participant’s role

No active role during the research project.

Participants’ role is the active participant from start to the end of the project.

Research results and generalizability

Generalizations are context-free depending on sample size and population.

Generalizations are based on context-detailed data and analysis.

Source: Modified from Thomas (2010)

Contingency theory of MIS

Contingency variables

StrategyStructureSizeEnvironmentTechnologyIndividualTask

Management

information system

DesignManagement implementation investmentUseImplementation

Management information

system Performance

FirmPerformanc

e

UseSatisfactionSuccessEffectiveness

PerceptionFinancial

Source: Weil and Olson (1987)

Case studiesIntensive cases Comparative cases

Purpose Developing theory from intensive exploration.

Developing concepts based on case comparison.

Assumption

Creativity through comparison with existing theories.

Comparison of cases leads to more useful theory.

Examples

Situation Usually evolves out of a researcher’s intensive experience with culture or organization

Usually concepts are developed from one case compared with another case.

Types Narratives, tabulation, explanatory or interpretative

Case comparison, case survey, interpretative comparison.

Source: Järvinen (2011)

Stages of case studyStage Key Charasteristics

Identify research problem and question(s).

Phenomenon is examined in a natural setting. The focus is on contemporary events.

Determine type of case study.

The complexity of the unit is studied intensively.

Select participants or groups.

One or view entities are examined (person,group, organization)

Collect data. Data are collected by multiple means.

Analyze data. Independent and dependent variables are not specified in advance.Within-case or cross-case analysis.

Compose the report. Why and How questions are useful in case studies.

Evaluate the validity and reliability.

No experimental controls or manipulation are involved.

Write proposals. The resultss derived depend heavily on the intergrative powers of investigator.

Write contributions.

The research situation

Problem

analysis

Field insights and data gathering

Resourse

development

evaluate the

validity and

reliability

Write Proposals Real-world

situation

A single case

Source: modified from Benbasat et al. (1987), Eisenhardt K.M. (1989) Järvinen P. (2012)

Case studies ...

How and Why questions are important The focus is contemporal The purpose of case studies:

Description of phenomena Grounded theory can be applied to explore

phenomena Explorative approach is used to validate,

confirm or falsify developed theory or model based on collected data.

Requirements of Case studies

Conceptual requirements: Construct validity Internal validity External validity Reliability

Structural requirements: Relevance for the audience Voyage of discovery Controversies Include all necessary data but not too much Formal structure and elaboration

User centred design activities

Plan the human centred process

Specify the context of use

Specify user requirements

Specify organizational requirements

Specify main features of an

application

Design and develop an application

Evaluate appication against user requirements

Evaluate application against

organizational requirements

Demonstrate application against speficied features(Field experiment)

Information related work

systems

Test processesmodule testIncremental

Acceptance testverification

PractitionersUsers, managers and executicesparticipates to define needed features.

Scientific rigour in action ressearch

Baskerville and Wood-Harper’s strategies: Establish an ethical client-system infrastructure Design data gathering process carefully Follow the planned iterative phases:

plan action take action evaluate action

Promote collaboration by the subject and support their subjects’ learning cycles.

Write the report that disseminate the scientific knowledge in a way that it is possible to carry out future research and by this way confirm or refute any causal suggestions or claims of generilized theory.

Action design research studies

The following slides include examples, how ADR-method has been applied by other researchers.

Saarinen L. (2012) Dissertation in Aalto University Rothengatter D. (2012) Dissertation Modified action design reserch method: Bilandzic and Venable (2011): Participatory Action Design

Research method, a new method applied to urban informatics in Australia

Wieringa and Morali (2012) Technical Action Research as a Validation Method in ISDS.

Papas, O’Keefe and Seltsikas (2012) The Action research vs design science debate: reflection from an intervention in eGovernment. They point out that ADR-method is meta-method.

Saarinen’s ADR process

Source: Saarinen (2012) Dissertation

Rothengatter’s ARD process

Source: Rothengatter (2012) Dissertation

Rothengatter’s Meta-level process in ADR

Literature review

Initial design

Problem formulation

Domain analysis

Implementation of improved design

Evaluation of improved desing

Building, intervention and evalluaution

Donain analysisImplementation of

initial design in case

EvaluationDesign updating

based on evaluation

Building, implementing and

evaluation

Analysis of initial design

Analysis of evaluation of

desingAnalysis of

improved desing

Reflection and learning

Update of underlying theories

Formalization of final design

Formalization of learning

Rothengatter’s meta-levelprocess in ADR includedtwo building, interventionand evaluation states.First iteration stage focus is to develop version 1.Second iteration stage includes an improvement of version one after evaluation is carried out, and results are available.Reflection and learning include three analyses.

Rothengatter applied contingency theory to achieve IS performance and organizational performance of Information system design (ISD).

Participatory Action Design Research

Source: Bilandzic and Venable (2011)

Diagnosis and Problem

formulation

Action planning

Action taking: design

Impact evaluation

Reflection and learning

Participative problem settingEthnographic study

Opportunity identificationParticipative planning

Participative designPrototyping and installation

Ethnographic studyParticipative evaluation

Participative client learningDesign theorizing for UI

Urban informaticsCommunity, Urban dwellersSocial goodWell-being, healt, social connectednessGovernment, public institutionOpen, mobile, diversePublic, access for allPublicCompletely discretionaryEssential

Technical action reserch processes

IT-artefact problem

investigation

IT-artefact design

It-artefact design

validation

It-artefactimplementatio

n

Implementation evaluation

Research problem

investigation

Research design

Research design

validation

Research execution

Analysis of research results

and publishing

Client-system problem

investigation

Treatment design

Design validation

Implementation

in the Client-system

Implentation evaluation and applying results

Techical Action research cycles

Engineering cycle

Engineering cycle

Idealizing assumptions Realistic assumptions

o o o o

Framework for IS design science

Environ-ment

IS design sccience

Knowledge base

Improvement problem

solving

Knowledge question

investigation

Goals

Artifact

Knowledge

The multiple realms of ADR

Real-world practices and values

LocalPractice

s

Design Research Practice

Reserch Practices

Knowledge base and values

Contribution togeneral practice

Contribution toknowledge base

Practical Realm Academic Realm

Purely practicalcontribution

Practical and scientificcontribution

Purely scientific contribution

Practicaladvancement

Scientific advancement

Nocontribution

Source: modified from Sjöström and Donellan (2012

Reflective question to select research methodResearch object

Questions AR DR ADR

The artefact What is the role of the artefact?Is the design of the artefact to improve organizational practice?Is the use cases needed to before artefact development?

? ? ?

The process and research cycles

How is the research problem going to be determined and agreed?Is a predetermined cycle of activity going to be followed?Is a software development method a necessary part of the activities?

? ? ?

The focus of evaluation

Can evaluation be a by-product of the research cycle?Is it necessary to carry out explicit evaluation activities?(e.g. ex-ante, ongoing, ex-post, verifaction and validation)How are acceptance tests included to research cycle?

? ? ?

The role of knowledge

How is the role of knowledge concerning researchers understanding of real-world phenomena? How is the role of knowledge about practitioners understanding of the research processes?

? ? ?

The role of learning

What expectations are in question of learning in organizational practices?

? ? ?

The role of practical results

How important is the artefact or improvement of the organizational process during the research process?

? ? ?

The role of scientific results

To what extent is to develop scientific knowledge during the research process?

? ? ?

Source: modified from Papas et al. (2012)

Development of design theories and knowledge

Identify real-world situations (P)

and desired outcomes (O)

Review (kernel theories)and

previous research

Propose/refinedesign theory

Test design theory

Source: Carlsson et al. (2007), Hrastinski et al. (2007)

O=f(p,i,c,m),where P = problemI = IS initiativeM = MechanismC = ContextO = Outcomes

A realist InformationSystemsintervention

Future research area in DSRResearch area

Research objects

Design science research

Design research method are used more int the future.

Design and designers are selected to the research objects.

Design science theory is developed and studied in the future.

Healt care and IT

Safety and efficiency healt-care systems.

e-recipes, electronic medical records.

clinical decision support systems.Design challengies are multiple.

Green technology and IT

Carbon footprints:1. Primary 2. Secondary

Renowable energy and energy usage in IS.

Green computing:Algritm efficiencyComputer virtualizationTelecommuting

Collaboration Two-way information channels

Collaboratice design

Education

Web 2.0 (3.0) Semantic web Social media Voice over IP

Game industry

Designer Software developers

AnimatorsSource: modified from Hevner et al. (2010)

References Bilandzic, M., & Venable, J. (2011). Towards Participatory Action Design Research: Adapting Action

Research and Design Science Research Methods for Urban Informatics. Journal of Community Informatics. Special Issue: Research in Action: Linking Communities and Universities, 7(3).

Carlsson, S.A. (2006): Towards an Information Systems Design Research Framework: A Critical Realist Perspective, Proceedings of the First International Conference on Design Science in Information Systems and Technology (DESRIST 2006), 192-212.

Henfridsson O. (2010) Action Design Research, presentation slides, University of Oslo Hevner A and Chatterjee S. (2010) Design Research in Information Systems, Theory and Practice,

Integrated Series in Information Systems 22, Springer. Järvinen P. (2012) On baundaries between field experiment, action research and design research,

University of Tamper, School of Information Sciences, Reports in Information Sciences 14, Tampere Järvinen P. (2010) IS reviews, Department of Computer Sciences, University of Tampere, DE-2010-16,

Tampere Lee, A. S., and Baskerville, R. L. 2003. “Generalizing Generalizability in Information Systems

Research,” Information Systems Research (14:3), pp. 221-243. Rothengatter D. (2012) Engineering situational methods for professional service organization, An

action design research approach, CTIT Ph D. Thesis Series No. 11-225, Enchede. Saarinen L. (2012) Enhancing ICT Supported Distributed Learning through Action Design Research,

Department of Information and Service Economy, Aalto University, Doctoral Dissertation. Sein, Henfridsson, Purao, Rossi and Lindgren (2011), Action Design research, MIS Quarterly, Vol. 35,

No. 1, pp. 37-56 Sjörström and Donellan (2012) Design research practice: A product semantics interpretation, The

International workshop on IT Artefact Design & Workpractice Intervention, 10 June 2012, Barcelona Tsang E. W. K. and Williams J.N. (2012) Generalization and induction: Clarifications, and a

classification of induction, MIS Quarterly, Vol. 36, No. 3, pp. 729-748. Yin R.K, (2003 ) Case Study Resaarch, 3rd edition, Sage Publication.