Extending the Delphi Method to Expand Its Application and ... · Delphi •Policy Delphi...
Transcript of Extending the Delphi Method to Expand Its Application and ... · Delphi •Policy Delphi...
Extending the Delphi Method to Expand Its
Application and Unlock Hidden Knowledge
Donna S. Finley, MBA, PhD, CMC
Principal and Co-founder, Framework Partners Inc.
SSHRC Doctoral Fellow
2012 Symposium on Scholarship of Teaching and Learning
Mount Royal University
Banff Park Lodge
November 10, 2012
4:00 – 5:00 pm
© D.S. Finley, MBA, PhD, CMC
2012.
Discussion Outline
Overview of the Delphi Method
Methodology
Findings and Implications
Future Directions
2
© D.S. Finley, MBA, PhD, CMC
2012.
The Delphi Method is a powerful tool for
co-creating solutions for complex problems.
Overview of
Delphi
• Delphi is an iterative, primary research process that
collects and distils the anonymous opinions of experts
using a series of questionnaires.
• Well suited when:
• “There is incomplete knowledge about a problem
or phenomenon . . . It investigates what does not
yet exist” [4]
• The research goal is to develop forecasts or
improve the understanding of issues,
opportunities and/or solutions.
3
© D.S. Finley, MBA, PhD, CMC
2012.
Delphi is flexible and versatile.
Overview of
Delphi
• Can be reasonably situated anywhere along
the quantitative-qualitative research continuum
• Does not limit the number or types of
questions that can be asked
• Does not restrict the number of rounds for a
study (1 round [8, 9] – 11 rounds [10])
• Does not restrict the number of panelists (10
to 1685 [13])
• New ideas can emerge at any time in the
process, which influence the direction of the
results and/or impact the design of
subsequent Delphi rounds
4
© D.S. Finley, MBA, PhD, CMC
2012.
The method has increased
in popularity over time.
Overview of
Delphi
Date Application
1950s Military use and forecasting (RAND
Corporation) [15]
1960s Non-military use [7]:
• Planning developing economies
• Civil service sector
• Future studies of science and
technology in Japan [16]
• Large-scale foresight studies in Europe
• Consultancy tool
1970s 670 known Delphi studies [17] of which 131
represented applications outside of
military and forecasting [7]
1980-
2004
Increasing use of the method [2, 14]
Four fundamental
characteristics of Delphi
have remained constant [5]:
• Anonymity
• Iteration
• Controlled feedback
• Statistical aggregation of
group response
5
© D.S. Finley, MBA, PhD, CMC
2012.
As Delphi gains popularity across broader sectors, researchers
have evolved the traditional Delphi method to be more applicable
to their situation, sector and/or research problem.
Overview of
Delphi
60 years
• Classical Delphi
• Decision-making
Delphi
• Policy Delphi • Introduced by Helmer in 1975 [21]
• Used for research outcomes where
‘judgmental information’ is required and
the obligatory search for consensus is
eliminated [9, 1]
• Gained popularity because it evolved
traditional application beyond forecasting
into the development of public policy and
social change
6
© D.S. Finley, MBA, PhD, CMC
2012.
Efforts to improve Delphi tend to focus on the
research process, rather than the research outcome.
Overview of
Delphi
Select Examples
Measuring the degree of consensus - Rayens and Hahn (2000) [18]
Use of two group process facilitators - Schnyer et al. (2005) [19]
Real-time Delphi - Gordon and Pease (2006) [9]
Opt-in/Opt-out option - Brown (2007) [20]
Expanding the definition of ‘expert’ - Gabb et al. (2006) [3]
7
© D.S. Finley, MBA, PhD, CMC
2012.
Several evolving uses of Delphi provide evidence
that a focus on research outcomes is emerging.
Overview of
Delphi
Evolving Uses of Delphi:
• Learning instrument to determine whether the method could
help employees at a Dutch bank learn and understand
technical information [22]
• Construct key performance appraisal indicators for the
mobility of service industries [23, 8]
• Establish priorities in research subjects ranging from
terrorist activities to construction projects to teaching
behaviours [9, 24, 25]
• Optimize curriculum design and delivery in Turkey, Australia,
and the USA on topics ranging from sexology education to
teachers’ civic knowledge [7]
• Exploring change in the housing association sector in
England [6]
8
© D.S. Finley, MBA, PhD, CMC
2012. 9
Discussion Outline
Overview of the Delphi Method
Methodology
Findings and Implications
Future Directions
© D.S. Finley, MBA, PhD, CMC
2012.
Delphi was applied to conduct primary
research about knowledge translation.
10
1. How is knowledge created by Practitioners
and Academics?
2. How is knowledge transferred from
Practitioners to Academics and vice versa?
3. How can the Delphi Method be extended
beyond its original purpose to enhance the
transfer of organisational knowledge?
Primary research questions:
Secondary research question:
Overview of
Methodology
Focus of this
presentation
© D.S. Finley, MBA, PhD, CMC
2012.
From a review of 151 Delphi studies (1983-2009),
eight distinct categories of Delphi application
were hypothesized.
60 years
• Classical Delphi
• Decision-making
Delphi
• Policy Delphi
Overview of
Methodology
Hypothesized Categories [7]
• Forecast (Classical Delphi) is used to articulate a future state
scenario/situation.
• Make Decisions (Decision-making Delphi) is used to promote
collaborative decision-making.
• Generate Ideas (Policy Delphi) serves to stimulate the
generation of alternatives and ideas about a specific topic.
• Articulate Core Concepts (new) develops clarity and meaning
for concepts that have not yet been defined.
• Set Priorities (new) serves to rank-order plausible alternatives.
• Measure Performance (new) articulates how organisational
effectiveness can be defined, measured and optimized.
• Build Capacity (new) enables participants to co-create new
concepts and solutions on a level, knowledge-based playing
field.
• Drive Change (new) brings new solutions to the forefront and
generates commitment amongst participants for action.
11
© D.S. Finley, MBA, PhD, CMC
2012.
This Delphi study engaged Practitioners and Academics
from different sectors, disciplines and geographies.
10 Academics
“Well-known in the field of
organizational effectiveness by
virtue of their reputation.”
• 5 business faculty
• 5 non-profit faculty
10 Practitioners “Successful leaders in
organizational effectiveness by
virtue of their reputation for
sustained organisational
performance.”
• 5 business-sector leaders
• 5 non-profit sector leaders
Overview of
Methodology
12
© D.S. Finley, MBA, PhD, CMC
2012.
The Delphi Feedback Questionnaire
(DFQ) had three objectives.
1. To confirm the extent of the panelists’ previous
experience with Delphi as a participant,
researcher or facilitator.
2. To better understand the potential application
and usefulness of the Delphi method in each
panelist’s occupational area.
3. To introduce eight possible classifications of the
Delphi method and receive feedback from
Practitioners and Academics about the
distinction of each classification.
Overview of
Methodology
13
© D.S. Finley, MBA, PhD, CMC
2012. 14
Discussion Outline
Overview of the Delphi Method
Methodology
Findings and Implications
Future Directions
© D.S. Finley, MBA, PhD, CMC
2012.
Few of the participants had
previously used Delphi.
Findings &
Implications
• 80% reported no previous experience as a Delphi
participant.
• 95% of panelists reported no previous experience
as a Delphi researcher, facilitator and/or
moderator.
• Overall, Academics were more responsive to the
DFQ than Practitioners by absolute count and in
quantity and quality of verbiage.
15
© D.S. Finley, MBA, PhD, CMC
2012.
Six definitions achieved consensus
for ‘clarity’ by earning a Top 2 rating
of 75% agreement or better.
ref Options Top 2 Rank
1 Forecast 65% 7
2 Make Decisions 80% 5
3 Generate Ideas 95% 1
4 Articulate Core Concepts 85% 4
5 Set Priorities 95% 1
6 Measure Performance 90% 3
7 Build Capacity 75% 6
8 Drive Change 65% 8
Average 80%
* Measure is "Clarity" - Highest is 1 - Very Clear, lowest is 4 - Not Clear.
Findings &
Implications
16
© D.S. Finley, MBA, PhD, CMC
2012.
Where overall consensus was reached, there was little
variation (i.e., 5% or less) in the clarity of five of the
definitions as perceived by Practitioners and Academics.
17
Findings &
Implications
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Forecast Make Decisions Generate Ideas Articulate Core Concepts
Set Priorities Measure Performance
Build Capacity Drive Change
To
p 2
Bo
x (
Aca
de
mic
s
an
d P
ra
ctit
ion
ers)
To
p 2
Bo
x (
%)
Top 2
AcademicsPractitioners
Delphi Classifications - Definition Clarity as Reported
by Practitioners and Academics [7]
© D.S. Finley, MBA, PhD, CMC
2012.
Modified Definitions Resulting from Delphi [7]
Forecast is used to articulate a future state
scenario/situation (REVISED)
Make Decisions is used to promote collaborative
decision-making (NO CHANGE)
Generate Ideas serves to stimulate the generation of
alternatives and ideas about a specific topic (NO
CHANGE)
Articulate Core Concepts develops clarity and meaning
for concepts that have not yet been defined (REVISED)
Set Priorities serves to rank-order plausible alternatives
(NO CHANGE)
Measure Performance articulates how organisational
effectiveness can be defined, measured and optimized
(REVISED)
Build Capacity enables participants to co-create new
concepts and solutions on a level, knowledge-based
playing field (REVISED)
Drive Change brings new solutions to the forefront and
generates commitment amongst participants for action
(NO CHANGE)
The wording for four Delphi classification definitions was
revised based on suggestions made by the panelists and/or
revisions in my thinking stimulated by the panelists’ input.
Findings &
Implications
18
Initial Definitions Presented in DFQ [7]
1 Forecast is used to establish facts about a situation or
topic or articulate a future forecast.
2 Make Decisions is used to promote collaborative
decision-making.
3. Generate Ideas serves to stimulate the generation of
alternatives and ideas about a specific topic.
4. Articulate Core Concepts serves to clarify
terminology and give definition to complex concepts;
providing clarity and meaning to concepts that have not
yet been articulated.
5. Set Priorities serves to rank-order plausible
alternatives.
6. Measure Performance clarifies and defines how
performance improvements can be defined, measured
and optimized.
7. Build Capacity is designed to co-create new
knowledge, concepts and solutions as participants
exchange ideas on a level knowledge playing field.
8. Drive Change brings new solutions to the forefront and
generates commitment amongst participants for action.
© D.S. Finley, MBA, PhD, CMC
2012.
The reclassification to eight categories enables future Delphi studies
to benefit from specific consideration of the research design.
19
Classification
(Research Purpose) Type of Expert
Questions:
Quantitative vs.
Qualitative
Rigor in Analysis Consensus
Requirement
Reporting of
Results
Forecast Professionals Primarily
quantitative
Mean scores and
standard
deviations
Desirable
A variety of scenarios
with a possible
recommendation
Make Decisions Designated pool of
Stakeholders and/or
Professionals
Primarily
quantitative
Cost-benefit
analysis Required
Decision(s) agreed to
& supported by
rationale & action
plan(s)
Generate Ideas Pool of experts & non-
experts Primarily qualitative
Cluster like ideas
and/or generate
themes
Not required
List of all unique
ideas, regardless of
frequency of citation
Articulate Core
Concepts Pool of experts & non-
experts Primarily qualitative
Clear, distinct,
precise &
accurate wording
Required Agreed to definitions
Set Priorities Designated pool of
Stakeholders and/or
Professionals
Primarily
quantitative
Absolute scores
or percentages
Achieved by
Default Rank-order list
Measure Performance Representative pool of
experts
Balance of
quantitative and
qualitative
Mixture of scores
& percentages
and testimony &
commentary
Desirable Report card
Build Capacity Pool of experts & non-
experts Primarily qualitative
Scenarios
analysis Circumstantial
New skills,
knowledge and
attitude(s)
Drive Change
Representative pool of
Stakeholders and
designated
Professionals
Primarily qualitative
Pros & cons, and
implications from
change
Required Behavioral change
Findings &
Implications
© D.S. Finley, MBA, PhD, CMC
2012. 20
Discussion Outline
Overview of the Delphi Method
Methodology
Findings and Implications
Future Directions
© D.S. Finley, MBA, PhD, CMC
2012.
More categories increase the specificity of research
purpose that cannot be adequately captured under
the existing three classifications.
21
• Panel membership selection
• Questionnaire design
• Rigor in analysis
• Consensus requirements
• Reporting of results
Future
Directions
Elements Impacted by classification:
© D.S. Finley, MBA, PhD, CMC
2012.
This work has stimulated a whole new direction
of thinking, research and application.
Publications
Finley, D.S. and M. Napier (2012). The Delphi 8: Expanded
Categorization that Increases the Delphi Method’s Application and
Effectiveness (under review by Technological Forecasting & Social
Change).
Finley, D.S., M. Napier and K. McDonnell (under development).
Knowledge Translator Roles Evolved and Defined Using Delphi.
Future
Directions
Presentations
Finley, D.S. (2011). Knowledge Mobilisation in High-performing Community
Organisations: Trends, Best Practices and Cases from the Community.
Alberta Rural Development Network.
22
© D.S. Finley, MBA, PhD, CMC
2012.
Acknowledgements:
23
• Financial support received to conduct this study
from the Social Sciences and Humanities
Research Council of Canada (SSHRC),
Reference No. 752-2009-1541 05.
• The 20 Delphi panelists for their input.
• Members of my Ph.D. Supervisory Committee
(University of Calgary), specifically Dr. Margaret
(Peggy) Patterson, Dr. Gayla Rogers and Dr.
Howard Yeager for their advice and
encouragement.
© D.S. Finley, MBA, PhD, CMC
2012. 24
[1] K.K. Franklin, J.K. Hart, Idea Generation and Exploration: Benefits and Limitations of the Policy Delphi Research
Method, Innovative Higher Education 31 (2007) 237-246.
[2] J. Landeta, Current validity of the Delphi method in social sciences, Technological Forecasting & Social Change 73(5)
(2006) 467-482.
[3] J. Gabb, R. Balen, G. Gibbs, C. Hall, A. Teal, Victoria Climbie Inquiry data corpus project: using the Delphi method in
multidisciplinary child protection research, The British Journal of Social Work 36(4) (2006) 577-596.
[4] G.J Skulmoski, F.T. Hartman, J. Krahn, The Delphi Method for Graduate Research, Journal of Information Technology
Education 6 (2007) 1-21.
[5] G. Rowe, G. Wright, The Delphi technique as a forecasting tool: Issues and analysis, International Journal of
Forecasting 15(4) (1999) 353-375.
[6] D. Mullins, Exploring change in the housing association sector in England using the Delphi method, Housing Studies
21(2) (2006) 227-251.
[7] D.S. Finley, Using the Delphi Method to Enhance the Transfer of Organisational Knowledge: A Bridge for Practitioners
and Academics, Thesis submitted to the Faculty of Graduate Studies in partial fulfillment of the requirements for the
degree of doctor of philosophy, University of Calgary (2012).
[8] Y. F. Kuo, P.C. Chen, Constructing performance appraisal indicators for mobility of the service industries using Fuzzy
Delphi Method, Expert Systems with Applications 35(4) (2008) 1930-1939.
[9] T. Gordon, A. Pease, RT Delphi: An efficient, "round-less" almost real time Delphi method, Technological Forecasting &
Social Change 73(4) (2006) 321-333.
[10] S. Field, D. Wall, R. Palmer, Determining generic standards for specialist registrar training, Med Teach. 25(2) (2003)
195-198.
[11] C. Bradshaw, “Research Leadership and Management: The voice of research leaders”. Thesis submitted to the Faculty
of Graduate Studies in partial fulfillment of the requirements for the degree of doctor of philosophy, University of
Calgary, (2008).
© D.S. Finley, MBA, PhD, CMC
2012. 25
[12] E. Ziglio, The Delphi method and its contribution to decision making, In M. Adler and E. Ziglio (Eds.), Gazing into the
Oracle: the Delphi method and its application to social policy and public health, Jessica Kingsley Publishers, Ltd.; Bristol,
PA., 1996, 3-33.
[13] C. Powell, The Delphi technique: myths and realities, Journal of Advanced Nursing 41(4) (2003) 376-382.
[14] F. Woudenberg, An Evaluation of Delphi, Technological Forecasting and Social Change 40 (1991) 131-150.
[15] T.J. Gordon, O. Helmer, Report on a Long-range Forecasting Study, The RAND Corporation, Santa Monica, California,
1964, P-2982.
[16] K. Cuhls, Foresight with Delphi Surveys in Japan, Technology Analysis & Strategic Management 13(4) (2001) 555-569.
[17] H.A. Linstone, M. Turoff, The Delphi method: Techniques and application (Appendix: Delphi Bibliography), Addison-Wesley
Publishing, London, 1975.
[18] M.K. Rayens, E.J. Hahn, Building Consensus Using the Policy Delphi Method, Policy, Politics, & Nursing 1(4) (2000) 308-
315.
[19] N.R. Schnyer, L.A. Conboy, E. Jacobson, P. McKnight, T. Goddard, F. Moscatelli, A.T.R. Legedza, C. Kerr, T.J. Kaptchuk,
P.M. Wayne, Development of a Chinese Medicine Assessment Measure: An Interdisciplinary Approach Using the Delphi
Method, Journal of Alternative & Complementary Medicine 11(6) (2005) 1005-1013.
[20] C. A. Brown, The Opt-in/Opt-out Feature in a Multi-Stage Delphi Method Study, International Journal of Social Research
Methodology 10(2) (2007) 135-144.
[21] O. Helmer, Foreword, In H.A. Linstone, M. Turoff (Eds.), The Delphi method: Techniques and application, Addison-Wesley
Publishing, London, 1975.
[22] J.A.G.M Van Dijk, Delphi method as a learning instrument: bank employees discussing an automation project,
Technological Forecasting and Social Change 37 (1990) 399-407.
[23] I.H Saizarbitoria, How quality management models influence company results conclusions of an empirical study based on
the Delphi method, Total Quality Management & Business Excellence 17(6) (2006) 775-794.
[24] O.G. Manoliadis, J.P Pantouvakis, S.E. Christodoulou, Improving qualifications-based selection by use of the fuzzy Delphi
method, Construction Management & Economics 27(4) (2009) 373-384.
[25] F. Shands, R.R.Levary, Weighting the importance of various teacher behaviors by use of the Delphi method, Education 106
(1086) 306-326.