Extending the Delphi Method to Expand Its Application and ... · Delphi •Policy Delphi...

Extending the Delphi Method to Expand Its

Application and Unlock Hidden Knowledge

Donna S. Finley, MBA, PhD, CMC

Principal and Co-founder, Framework Partners Inc.

SSHRC Doctoral Fellow

2012 Symposium on Scholarship of Teaching and Learning

Mount Royal University

Banff Park Lodge

November 10, 2012

4:00 – 5:00 pm

© D.S. Finley, MBA, PhD, CMC

2012.

Discussion Outline

Overview of the Delphi Method

Methodology

Findings and Implications

Future Directions

2


2012.

The Delphi Method is a powerful tool for

co-creating solutions for complex problems.

Overview of

Delphi

• Delphi is an iterative, primary research process that

collects and distils the anonymous opinions of experts

using a series of questionnaires.

• Well suited when:

• “There is incomplete knowledge about a problem

or phenomenon . . . It investigates what does not

yet exist” [4]

• The research goal is to develop forecasts or

improve the understanding of issues,

opportunities and/or solutions.

3


2012.

Delphi is flexible and versatile.

Overview of

Delphi

• Can be reasonably situated anywhere along

the quantitative-qualitative research continuum

• Does not limit the number or types of

questions that can be asked

• Does not restrict the number of rounds for a

study (1 round [8, 9] – 11 rounds [10])

• Does not restrict the number of panelists (10

to 1685 [13])

• New ideas can emerge at any time in the

process, which influence the direction of the

results and/or impact the design of

subsequent Delphi rounds

4


2012.

The method has increased

in popularity over time.

Overview of

Delphi

Date Application

1950s Military use and forecasting (RAND

Corporation) [15]

1960s Non-military use [7]:

• Planning developing economies

• Civil service sector

• Future studies of science and

technology in Japan [16]

• Large-scale foresight studies in Europe

• Consultancy tool

1970s 670 known Delphi studies [17] of which 131

represented applications outside of

military and forecasting [7]

1980-

2004

Increasing use of the method [2, 14]

Four fundamental

characteristics of Delphi

have remained constant [5]:

• Anonymity

• Iteration

• Controlled feedback

• Statistical aggregation of

group response

5


2012.

As Delphi gains popularity across broader sectors, researchers

have evolved the traditional Delphi method to be more applicable

to their situation, sector and/or research problem.

Overview of

Delphi

60 years

• Classical Delphi

• Decision-making

Delphi

• Policy Delphi • Introduced by Helmer in 1975 [21]

• Used for research outcomes where

‘judgmental information’ is required and

the obligatory search for consensus is

eliminated [9, 1]

• Gained popularity because it evolved

traditional application beyond forecasting

into the development of public policy and

social change

6


2012.

Efforts to improve Delphi tend to focus on the

research process, rather than the research outcome.

Overview of

Delphi

Select Examples

Measuring the degree of consensus - Rayens and Hahn (2000) [18]

Use of two group process facilitators - Schnyer et al. (2005) [19]

Real-time Delphi - Gordon and Pease (2006) [9]

Opt-in/Opt-out option - Brown (2007) [20]

Expanding the definition of ‘expert’ - Gabb et al. (2006) [3]

7


2012.

Several evolving uses of Delphi provide evidence

that a focus on research outcomes is emerging.

Overview of

Delphi

Evolving Uses of Delphi:

• Learning instrument to determine whether the method could

help employees at a Dutch bank learn and understand

technical information [22]

• Construct key performance appraisal indicators for the

mobility of service industries [23, 8]

• Establish priorities in research subjects ranging from

terrorist activities to construction projects to teaching

behaviours [9, 24, 25]

• Optimize curriculum design and delivery in Turkey, Australia,

and the USA on topics ranging from sexology education to

teachers’ civic knowledge [7]

• Exploring change in the housing association sector in

England [6]

8


2012. 9

Discussion Outline


Methodology


Future Directions


2012.

Delphi was applied to conduct primary

research about knowledge translation.

10

1. How is knowledge created by Practitioners

and Academics?

2. How is knowledge transferred from

Practitioners to Academics and vice versa?

3. How can the Delphi Method be extended

beyond its original purpose to enhance the

transfer of organisational knowledge?

Primary research questions:

Secondary research question:

Overview of

Methodology

Focus of this

presentation


2012.

From a review of 151 Delphi studies (1983-2009),

eight distinct categories of Delphi application

were hypothesized.

60 years

• Classical Delphi

• Decision-making

Delphi

• Policy Delphi

Overview of

Methodology

Hypothesized Categories [7]

• Forecast (Classical Delphi) is used to articulate a future state

scenario/situation.

• Make Decisions (Decision-making Delphi) is used to promote

collaborative decision-making.

• Generate Ideas (Policy Delphi) serves to stimulate the

generation of alternatives and ideas about a specific topic.

• Articulate Core Concepts (new) develops clarity and meaning

for concepts that have not yet been defined.

• Set Priorities (new) serves to rank-order plausible alternatives.

• Measure Performance (new) articulates how organisational

effectiveness can be defined, measured and optimized.

• Build Capacity (new) enables participants to co-create new

concepts and solutions on a level, knowledge-based playing

field.

• Drive Change (new) brings new solutions to the forefront and

generates commitment amongst participants for action.

11


2012.

This Delphi study engaged Practitioners and Academics

from different sectors, disciplines and geographies.

10 Academics

“Well-known in the field of

organizational effectiveness by

virtue of their reputation.”

• 5 business faculty

• 5 non-profit faculty

10 Practitioners “Successful leaders in

organizational effectiveness by

virtue of their reputation for

sustained organisational

performance.”

• 5 business-sector leaders

• 5 non-profit sector leaders

Overview of

Methodology

12


2012.

The Delphi Feedback Questionnaire

(DFQ) had three objectives.

1. To confirm the extent of the panelists’ previous

experience with Delphi as a participant,

researcher or facilitator.

2. To better understand the potential application

and usefulness of the Delphi method in each

panelist’s occupational area.

3. To introduce eight possible classifications of the

Delphi method and receive feedback from

Practitioners and Academics about the

distinction of each classification.

Overview of

Methodology

13


2012. 14

Discussion Outline


Methodology


Future Directions


2012.

Few of the participants had

previously used Delphi.

Findings &

Implications

• 80% reported no previous experience as a Delphi

participant.

• 95% of panelists reported no previous experience

as a Delphi researcher, facilitator and/or

moderator.

• Overall, Academics were more responsive to the

DFQ than Practitioners by absolute count and in

quantity and quality of verbiage.

15


2012.

Six definitions achieved consensus

for ‘clarity’ by earning a Top 2 rating

of 75% agreement or better.

ref Options Top 2 Rank

1 Forecast 65% 7

2 Make Decisions 80% 5

3 Generate Ideas 95% 1

4 Articulate Core Concepts 85% 4

5 Set Priorities 95% 1

6 Measure Performance 90% 3

7 Build Capacity 75% 6

8 Drive Change 65% 8

Average 80%

* Measure is "Clarity" - Highest is 1 - Very Clear, lowest is 4 - Not Clear.

Findings &

Implications

16


2012.

Where overall consensus was reached, there was little

variation (i.e., 5% or less) in the clarity of five of the

definitions as perceived by Practitioners and Academics.

17

Findings &

Implications

0%

20%

40%

60%

80%

100%

0%

20%

40%

60%

80%

100%

Forecast Make Decisions Generate Ideas Articulate Core Concepts

Set Priorities Measure Performance

Build Capacity Drive Change

To

p 2

Bo

x (

Aca

de

mic

s

an

d P

ra

ctit

ion

ers)

To

p 2

Bo

x (

%)

Top 2

AcademicsPractitioners

Delphi Classifications - Definition Clarity as Reported

by Practitioners and Academics [7]


2012.

Modified Definitions Resulting from Delphi [7]

Forecast is used to articulate a future state

scenario/situation (REVISED)

Make Decisions is used to promote collaborative

decision-making (NO CHANGE)

Generate Ideas serves to stimulate the generation of

alternatives and ideas about a specific topic (NO

CHANGE)

Articulate Core Concepts develops clarity and meaning

for concepts that have not yet been defined (REVISED)

Set Priorities serves to rank-order plausible alternatives

(NO CHANGE)

Measure Performance articulates how organisational

effectiveness can be defined, measured and optimized

(REVISED)

Build Capacity enables participants to co-create new

concepts and solutions on a level, knowledge-based

playing field (REVISED)

Drive Change brings new solutions to the forefront and

generates commitment amongst participants for action

(NO CHANGE)

The wording for four Delphi classification definitions was

revised based on suggestions made by the panelists and/or

revisions in my thinking stimulated by the panelists’ input.

Findings &

Implications

18

Initial Definitions Presented in DFQ [7]

1 Forecast is used to establish facts about a situation or

topic or articulate a future forecast.

2 Make Decisions is used to promote collaborative

decision-making.

3. Generate Ideas serves to stimulate the generation of

alternatives and ideas about a specific topic.

4. Articulate Core Concepts serves to clarify

terminology and give definition to complex concepts;

providing clarity and meaning to concepts that have not

yet been articulated.

5. Set Priorities serves to rank-order plausible

alternatives.

6. Measure Performance clarifies and defines how

performance improvements can be defined, measured

and optimized.

7. Build Capacity is designed to co-create new

knowledge, concepts and solutions as participants

exchange ideas on a level knowledge playing field.

8. Drive Change brings new solutions to the forefront and

generates commitment amongst participants for action.


2012.

The reclassification to eight categories enables future Delphi studies

to benefit from specific consideration of the research design.

19

Classification

(Research Purpose) Type of Expert

Questions:

Quantitative vs.

Qualitative

Rigor in Analysis Consensus

Requirement

Reporting of

Results

Forecast Professionals Primarily

quantitative

Mean scores and

standard

deviations

Desirable

A variety of scenarios

with a possible

recommendation

Make Decisions Designated pool of

Stakeholders and/or

Professionals

Primarily

quantitative

Cost-benefit

analysis Required

Decision(s) agreed to

& supported by

rationale & action

plan(s)

Generate Ideas Pool of experts & non-

experts Primarily qualitative

Cluster like ideas

and/or generate

themes

Not required

List of all unique

ideas, regardless of

frequency of citation

Articulate Core

Concepts Pool of experts & non-


Clear, distinct,

precise &

accurate wording

Required Agreed to definitions

Set Priorities Designated pool of

Stakeholders and/or

Professionals

Primarily

quantitative

Absolute scores

or percentages

Achieved by

Default Rank-order list

Measure Performance Representative pool of

experts

Balance of

quantitative and

qualitative

Mixture of scores

& percentages

and testimony &

commentary

Desirable Report card

Build Capacity Pool of experts & non-


Scenarios

analysis Circumstantial

New skills,

knowledge and

attitude(s)

Drive Change

Representative pool of

Stakeholders and

designated

Professionals

Primarily qualitative

Pros & cons, and

implications from

change

Required Behavioral change

Findings &

Implications


2012. 20

Discussion Outline


Methodology


Future Directions


2012.

More categories increase the specificity of research

purpose that cannot be adequately captured under

the existing three classifications.

21

• Panel membership selection

• Questionnaire design

• Rigor in analysis

• Consensus requirements

• Reporting of results

Future

Directions

Elements Impacted by classification:


2012.

This work has stimulated a whole new direction

of thinking, research and application.

Publications

Finley, D.S. and M. Napier (2012). The Delphi 8: Expanded

Categorization that Increases the Delphi Method’s Application and

Effectiveness (under review by Technological Forecasting & Social

Change).

Finley, D.S., M. Napier and K. McDonnell (under development).

Knowledge Translator Roles Evolved and Defined Using Delphi.

Future

Directions

Presentations

Finley, D.S. (2011). Knowledge Mobilisation in High-performing Community

Organisations: Trends, Best Practices and Cases from the Community.

Alberta Rural Development Network.

22


2012.

Acknowledgements:

23

• Financial support received to conduct this study

from the Social Sciences and Humanities

Research Council of Canada (SSHRC),

Reference No. 752-2009-1541 05.

• The 20 Delphi panelists for their input.

• Members of my Ph.D. Supervisory Committee

(University of Calgary), specifically Dr. Margaret

(Peggy) Patterson, Dr. Gayla Rogers and Dr.

Howard Yeager for their advice and

encouragement.


2012. 24

[1] K.K. Franklin, J.K. Hart, Idea Generation and Exploration: Benefits and Limitations of the Policy Delphi Research

Method, Innovative Higher Education 31 (2007) 237-246.

[2] J. Landeta, Current validity of the Delphi method in social sciences, Technological Forecasting & Social Change 73(5)

(2006) 467-482.

[3] J. Gabb, R. Balen, G. Gibbs, C. Hall, A. Teal, Victoria Climbie Inquiry data corpus project: using the Delphi method in

multidisciplinary child protection research, The British Journal of Social Work 36(4) (2006) 577-596.

[4] G.J Skulmoski, F.T. Hartman, J. Krahn, The Delphi Method for Graduate Research, Journal of Information Technology

Education 6 (2007) 1-21.

[5] G. Rowe, G. Wright, The Delphi technique as a forecasting tool: Issues and analysis, International Journal of

Forecasting 15(4) (1999) 353-375.

[6] D. Mullins, Exploring change in the housing association sector in England using the Delphi method, Housing Studies

21(2) (2006) 227-251.

[7] D.S. Finley, Using the Delphi Method to Enhance the Transfer of Organisational Knowledge: A Bridge for Practitioners

and Academics, Thesis submitted to the Faculty of Graduate Studies in partial fulfillment of the requirements for the

degree of doctor of philosophy, University of Calgary (2012).

[8] Y. F. Kuo, P.C. Chen, Constructing performance appraisal indicators for mobility of the service industries using Fuzzy

Delphi Method, Expert Systems with Applications 35(4) (2008) 1930-1939.

[9] T. Gordon, A. Pease, RT Delphi: An efficient, "round-less" almost real time Delphi method, Technological Forecasting &

Social Change 73(4) (2006) 321-333.

[10] S. Field, D. Wall, R. Palmer, Determining generic standards for specialist registrar training, Med Teach. 25(2) (2003)

195-198.

[11] C. Bradshaw, “Research Leadership and Management: The voice of research leaders”. Thesis submitted to the Faculty

of Graduate Studies in partial fulfillment of the requirements for the degree of doctor of philosophy, University of

Calgary, (2008).


2012. 25

[12] E. Ziglio, The Delphi method and its contribution to decision making, In M. Adler and E. Ziglio (Eds.), Gazing into the

Oracle: the Delphi method and its application to social policy and public health, Jessica Kingsley Publishers, Ltd.; Bristol,

PA., 1996, 3-33.

[13] C. Powell, The Delphi technique: myths and realities, Journal of Advanced Nursing 41(4) (2003) 376-382.

[14] F. Woudenberg, An Evaluation of Delphi, Technological Forecasting and Social Change 40 (1991) 131-150.

[15] T.J. Gordon, O. Helmer, Report on a Long-range Forecasting Study, The RAND Corporation, Santa Monica, California,

1964, P-2982.

[16] K. Cuhls, Foresight with Delphi Surveys in Japan, Technology Analysis & Strategic Management 13(4) (2001) 555-569.

[17] H.A. Linstone, M. Turoff, The Delphi method: Techniques and application (Appendix: Delphi Bibliography), Addison-Wesley

Publishing, London, 1975.

[18] M.K. Rayens, E.J. Hahn, Building Consensus Using the Policy Delphi Method, Policy, Politics, & Nursing 1(4) (2000) 308-

315.

[19] N.R. Schnyer, L.A. Conboy, E. Jacobson, P. McKnight, T. Goddard, F. Moscatelli, A.T.R. Legedza, C. Kerr, T.J. Kaptchuk,

P.M. Wayne, Development of a Chinese Medicine Assessment Measure: An Interdisciplinary Approach Using the Delphi

Method, Journal of Alternative & Complementary Medicine 11(6) (2005) 1005-1013.

[20] C. A. Brown, The Opt-in/Opt-out Feature in a Multi-Stage Delphi Method Study, International Journal of Social Research

Methodology 10(2) (2007) 135-144.

[21] O. Helmer, Foreword, In H.A. Linstone, M. Turoff (Eds.), The Delphi method: Techniques and application, Addison-Wesley

Publishing, London, 1975.

[22] J.A.G.M Van Dijk, Delphi method as a learning instrument: bank employees discussing an automation project,

Technological Forecasting and Social Change 37 (1990) 399-407.

[23] I.H Saizarbitoria, How quality management models influence company results conclusions of an empirical study based on

the Delphi method, Total Quality Management & Business Excellence 17(6) (2006) 775-794.

[24] O.G. Manoliadis, J.P Pantouvakis, S.E. Christodoulou, Improving qualifications-based selection by use of the fuzzy Delphi

method, Construction Management & Economics 27(4) (2009) 373-384.

[25] F. Shands, R.R.Levary, Weighting the importance of various teacher behaviors by use of the Delphi method, Education 106

(1086) 306-326.

Extending the Delphi Method to Expand Its Application and ... · Delphi •Policy Delphi...

Documents

Transcript of Extending the Delphi Method to Expand Its Application and ... · Delphi •Policy Delphi...