Online Scenario Planning

Large-Scale ParticipatoryFutures Systems

Harnessing Collective Intelligence forCrowdsourced Scenario Planning

Noah RafordPhD Candidate, UIS/CDD

Massachusetts Institute of Technology (MIT)Department of Urban Studies and Planning

[email protected]://news.noahraford.com/

Friday, April 9, 2010

Wednesday, 21 April 2010

mailto:[email protected]


http://news.noahraford.com


Purpose

An informal crowdsourcing charrette to harness your collective intelligence and thereby scenario plan my PhD.


Purpose


Me


Purpose


Me Your brains


Purpose


Me Your brains

Ideas

Feedback


Purpose


Me Your brains

Ideas

Feedback

PhD


Outline


Outline

1. De!nitions & overview


Outline

1. De!nitions & overview2. Research questions


Outline

1. De!nitions & overview2. Research questions3. An example, “The Future of Cities”


Outline

1. De!nitions & overview2. Research questions3. An example, “The Future of Cities”4. Theoretical foundations


Outline

1. De!nitions & overview2. Research questions3. An example, “The Future of Cities”4. Theoretical foundations5. Schema for an online approach


Outline

1. De!nitions & overview2. Research questions3. An example, “The Future of Cities”4. Theoretical foundations5. Schema for an online approach6. Case studies


Outline

1. De!nitions & overview2. Research questions3. An example, “The Future of Cities”4. Theoretical foundations5. Schema for an online approach6. Case studies7. Hypotheses


Outline

1. De!nitions & overview2. Research questions3. An example, “The Future of Cities”4. Theoretical foundations5. Schema for an online approach6. Case studies7. Hypotheses8. Evaluation criteria


Outline

1. De!nitions & overview2. Research questions3. An example, “The Future of Cities”4. Theoretical foundations5. Schema for an online approach6. Case studies7. Hypotheses8. Evaluation criteria9. Limitations & future work


De!nitions & overviewCollective Intelligence for Crowdsourced Scenario Planning



“Creation, aggregation and interpretation of strategically relevant information for decision-making [through distributed means]” (Por, 2008)



“The act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call.” (Howe, 2006)




“Tools for foresight discussions... whose purpose is not a prediction or a plan, but a change in the mindset of the people who use them.” (de Gues, 1997)





“Tools for foresight discussions... whose purpose is not a prediction or a plan, but a change in the mindset of the people who use them.” (de Gues, 1997)“Tools for foresight discussions... whose purpose is not a prediction or a plan, but a change in the mindset of the people who use them.” (de Gues, 1997)





“Tools for foresight discussions... whose purpose is not a prediction or a plan, but a change in the mindset of the people who use them.” (de Gues, 1997)“Tools for foresight discussions... whose purpose is not a prediction or a plan, but a change in the mindset of the people who use them.” (de Gues, 1997)“Tools for foresight discussions... whose purpose is not a prediction or a plan, but a change in the mindset of the people who use them.” (de Geus, 1997)




Research questions


Research questionsCan the bene!ts of face-to-face scenario planning (SP) be had using online, collective intelligence approaches?



What do online approaches add to traditional SP and what do they take away?




Which aspects of online, Web 2.0 approaches are most in"uential to the stated goals of SP, in what way and why?




Which aspects of online, Web 2.0 approaches are most in"uential to the stated goals of SP, in what way and why?

Can the design & testing of such a system provide more rigourous data for understanding the effects of SP on group process in general?


Typical scenario method*

ID Issues

Generate key themes

ID driving forces

Rank factors

Develop draft scenario logic

Create draft final scenarios

Finalise scenarios

Consider implications

Identify indicators

Meetings, conversations

Expert interviews, brainstorm with client, desktop research

Extract key themes, create trends and timelines, key events

Select key uncertainties and forces, list by uncertainty / impact, predetermined drivers

Create scenario snippets, draft systems diagrams, mix and match trends, 2x2 grids

Integrate themes from draft scenarios, create headlines and scenario narratives

Get client feedback, refine, detail, elaborate narrative to final form

Identify key strategic themes, reflect on strategic questions in the context of each scenario

ID key indicators in each scenario for strategic concerns

Client defines key questions through initial conversations & meetings

F2F & phone interviews

Group workshop

Consultant report

Group workshop

Consultant report

(Raford, 2010 after Schwartz, 1991; van der Heijden, 1997)

* i.e., The most common, Shell-style, “inductive logics” approach, distinct from a range of other rich approaches including Causal Layered Analysis (Inayatullah, 2004), La Prospective (Berger, 1964), the Manoa Method (Schultz, 1994), or the Three Horizons framework (Hodgson & Curry, 2008).


Example, “Future of Cities”


Example, “Future of Cities”

23 interviews world wide:

• Architecture

• Commercial tenants

• Entrepreneurship

• Environment

• Governance

• Infrastructure

• Non-governmental organisations

• Planning

• Real estate

• Technology


Political

• Most city governments will lack the resources to meet increasing citizens demands

• Weaken central government and open room for other players

• Civil society and community based organisations will be the !rst to !ll this role

• Bottom-up participatory approaches to development and management will become important

• Local government will need to shift from regulation to enabling and facilitating

• Boundaries of where city authorities ends will blur, administrative implications are unclear

• Grassroots' innovation could lead to transformational change

• Insecurity more important factor, with high unemployment and economic, political and environmental migration


Economic

• Increased division of wealth between rich and poor

• The world’s poor are an increasingly powerful force in urban development

• Current urban development models not !t for their emergent needs

• Global warming will disproportionately effect the poor

• International !nance will become more important, domestic capital less

• International !nance will become more selective, comparing between cities

• Taxation will continue to be a strong determinant of capital "ows

• New ecological accounting mechanisms will play an increasing role in real estate !nance and development

• Building obsolescence will become an increasingly important factor


Technological

• ICT will enable acceleration of social dynamics already in place

• Could have signi!cant destabilising effects through asymmetric warfare, etc.

• May allow for breakthroughs in decentralised infrastructure and governance

• ICT enables relocation of activities, such as public administration

• ICT will enable more social surveillance and government control


Drivers of change

Growing income inequalityLack of capital availabilityRole of centralized governanceInfrastructure decayLifestyle change & value shiftsResource shortages


Ranking key drivers


Gulliver’sWorld


Gulliver’sWorld

Massive socio-technical revolution


Gulliver’sWorld

Massive socio-technical revolution

Triumph of the Triads


Scenario 1:Gulliver’s World


Scenario 2: Massive socio-technical revolution... Or bust


Scenario 3:Triumph of the Triad’s


Components of scenarios

control

Contextual

Focused on the uncertainty & discontinuities over which we have limited or no control

Narrative, with actors, motives, chronology & events

Materially different along key strategic dimensions

Primarily qualitative but often with quantitative support

Use rich media & diverse presentation styles

(van der Heijden, 1997)

some control

no control


Theoretical foundationsStudies of group decision-making reveal important shortcomings which limit our ability to make effective decisions under conditions of dynamic uncertainty (Dorner, 1997).

These include the “availability bias”, whereby people estimate the future probability of events based on easily remembered experiences from their past (Tversky & Kahneman, 1974)

“Experimenter bias”, whereby people look for and select data that con!rms pre-existing expectations (Rosenthal, 1966)

“Ambiguity effect”, whereby subjects are ignored or discounted for which we have partial or incomplete information (Frisch & Baron, 1988)

“Groupthink biases”, whereby groups seek to minimize con"ict and reach consensus without critically testing, analyzing, or evaluating ideas (Janis, 1972).


Theoretical foundations

As a result we build operational theories of the way the world works based on past experience and data, then are reluctant to revise them in the face of change.



"We will not have any more crashes in our time."

John Maynard Keynes, 1927



"There will be no interruption of our permanent prosperity."Myron E. Forbes, President, Pierce Arrow Motor Car Co., January 12,

1928



"Stock prices have reached what looks like a permanently

high plateau.”Irving Fisher, Ph.D, Economist, Oct.

17, 1929



"This crash is not going to have much effect on

business."Arthur Reynolds, Chairman of

Continental Illinois Bank of Chicago, October 24, 1929



"... the present depression has about spent its force..."

HES, Aug 30, 1930



"Stabilization at [present] levels is clearly possible."

HES Oct 31, 1931



"All safe deposit boxes in banks or financial institutions have been sealed... and may only be opened in the presence of an agent of the

I.R.S."President F.D. Roosevelt, 1933


Scenarios focus on decision-makers’ internal landscape

“A company’s perception of its business environment is as important as its investment infrastructure because its strategy comes from this perception. I cannot over-emphasize this point: unless the corporate microcosm changes, managerial behavior will not change; the internal compass must be re-calibrated.”

(Wack, in Chermack, 2003)



Theoretical foundationsThey produce learning & strategic awareness

Scenarios are a game board of the future, designed to break down people’s perceptions of the present & their assumptions of the future, in a way that allows them to better understand changing contexts and see new insights & opportunities, today.

“Leading from the future, as it emerges in the present” Scharmer, 2009



Scenarios

Increased learning

More accuratemental models

Betterdecisions

Improved performance

Proposition 4: If changes in decision making are positively associated with !rm performance, then !rm performance will increase as a result of altered decision making strategies.

Proposition 1: If scenarios are positively associated with learning, then learning will increase as a result of participation in scenario planning.

Proposition 2: If learning is positively associated with the alteration of mental models, then mental models change as a result of learning.

Proposition 3: If a chance in mental models alters decision structure, then a change in mental models implies a change in the approach to decision making.

Proposition 5: If scenarios are positively associated with learning, learning is positively associated with altered mental models, altered mental models are positively associated with !rm performance, then scenarios can be positively associated with !rm performance.

(Chermack, 2003)


How does SP do this?

Includes diverse, contrary andnon-traditional viewpoints andevidence

Focuses explicitly on“game-changing” eventsand trends

Uses creative workshops &methods to understand organizational strategy making, “get inside”, then creatively disrupt it


(Innes & Booher, 2001)


Why does it work?

A focuses on organizantional learning, sensemaking and decision-making:

Argys & Schon (1974) Double loop organizational learning

Piaget (1977) Constructivist & social learning theory

Weick (1979) Sensemaking & organizational awareness

Klein (1999) Recognition-primed decision making

Jarzabkowski (2005), Orlikowski (1992) Activity- & practice-based strategizing

Boyd (1976) Competitive advantages of perception management



Why does it work?



Theoretical foundationsCollaborative sensemaking in the public realm

“Urban planning has lost sight of the future... creating increasingly feeble, myopic, degenerate frameworks that are more likely to react to yesterday's events than to prepare the way from here to the future.”(Isserman, 1985)

Innes & Booher (1999) - Critique of public participation

Healy (2001) - Role and need for community visioning


Theoretical foundationsCollaborative sensemaking in the public realm

“Scenarios are developed collectively to build shared images of possible futures… scenarios nurture openness to change by allowing more complexity in futures states of a system and environment to be taken into account.”(Van der Heijden, 1997)


Schema for online approach


Schema for online approachCurrent challenges (Raford, 2010; Pang, 2010)



Labor intensive & expensive




Bene!ts poorly documented (no veri!cation or reputation systems)





Limited participation (time, space & numbers)






Predominance of senior decision-making elite (participant bias)






Predominance of senior decision-making elite (participant bias)

Highly dependent on facilitation skills & consultant synthesis (facilitator &

author bias)



(Malone et al., 2010)




Who is performing the task? Why are they doing it?




Who is performing the task? Why are they doing it?

What is being accomplished? How is it being done?



Mechanisms for

collaboration

- wikis

- blogs

- discussion forums

- collaborative

workspaces, etc.

Can activities be divided into

pieces? Are necessary resources

widely distributed or in unknown

locations?

Yes No

Crowd Hiearchy

Are there adequate incentives to

participate?

What kind of activity needs to be

done?

Direct

compenstionLearning

Influence /

self-promotion

Love,

friendship

Yes

The problem

Create

Can the activity be divided into

small, independent pieces?

Yes No

Collect Collaborate

Are only a few good (best)

solutions needed?

YesNo

Finished

Decide

Does the entire group need to

abide by the same decision?

No

Individual

decisions

Yes

Group

decision

Finished

Voting Averaging ConcensusPrediction

markets

Are money or resources required to

exchange hands or motivate

decision?

Yes No

Market

exchange

Trust

networks

Finished Finished

(Raford, 2010, after Malone et al., 2010)


Mechanisms for

collaboration

- wikis

- blogs

- discussion forums

- collaborative

workspaces, etc.




locations?

Yes No

Crowd Hiearchy


participate?


done?

Direct

compenstionLearning

Influence /

self-promotion

Love,

friendship

Yes

The problem

Create



Yes No

Collect Collaborate


solutions needed?

YesNo

Finished

Decide



No

Individual

decisions

Yes

Group

decision

Finished


markets



decision?

Yes No

Market

exchange

Trust

networks

Finished Finished



Mechanisms for

collaboration

- wikis

- blogs

- discussion forums

- collaborative

workspaces, etc.




locations?

Yes No

Crowd Hiearchy


participate?


done?

Direct

compenstionLearning

Influence /

self-promotion

Love,

friendship

Yes

The problem

Create



Yes No

Collect Collaborate


solutions needed?

YesNo

Finished

Decide



No

Individual

decisions

Yes

Group

decision

Finished


markets



decision?

Yes No

Market

exchange

Trust

networks

Finished Finished


ID Issues

Generate key themes

ID driving forces

Rank factors



Finalise scenarios


Identify indicators












Group workshop

Consultant report

Group workshop

Consultant report

+


Mechanisms for

collaboration

- wikis

- blogs

- discussion forums

- collaborative

workspaces, etc.




locations?

Yes No

Crowd Hiearchy


participate?


done?

Direct

compenstionLearning

Influence /

self-promotion

Love,

friendship

Yes

The problem

Create



Yes No

Collect Collaborate


solutions needed?

YesNo

Finished

Decide



No

Individual

decisions

Yes

Group

decision

Finished


markets



decision?

Yes No

Market

exchange

Trust

networks

Finished Finished


ID Issues

Generate key themes

ID driving forces

Rank factors



Finalise scenarios


Identify indicators












Group workshop

Consultant report

Group workshop

Consultant report

+

Schema for online scenario planning


Mechanisms for

collaboration

- wikis

- blogs

- discussion forums

- collaborative

workspaces, etc.




locations?

Yes No

Crowd Hiearchy


participate?


done?

Direct

compenstionLearning

Influence /

self-promotion

Love,

friendship

Yes

The problem

Create



Yes No

Collect Collaborate


solutions needed?

YesNo

Finished

Decide



No

Individual

decisions

Yes

Group

decision

Finished


markets



decision?

Yes No

Market

exchange

Trust

networks

Finished Finished


ID Issues

Generate key themes

ID driving forces

Rank factors



Finalise scenarios


Identify indicators












Group workshop

Consultant report

Group workshop

Consultant report

+

Schema for online scenario planning

= Key decision points for system design


Case studies

Differentiating dimensions of parameter space

Type of information collected

Degree of collaboration required

Who is involved & what is their in"uence?

Level of transparency

Level of computer assistance (automation)

Presence & type of reputation accounting

User interface design

Opinion

Anecdote

Analysis

Facts

Forecasts


Case studies









Solo

Teams

Entire group

Single round

Multiple rounds


Case studies









General public

Stake holders

Experts

Moderator


Case studies









Everyone sees & can use everything

Information shared only with team

mates

Private information

Anonymous

Administrators only ones w/ access


Case studies









None

Recommendations / !ltering

Pattern matching & algorithms


Case studies









None

“Liked / Not Liked”

Scoring, levels & points


Case studies









Web-form “bare bones”

Functional, wiki-style

Designed for UX


Case studies

Clustering into representative system types

“FutureGame”

Signti!c LabsIFTF

“ExpertSystem”

FuturescaperIFF

“Human Sensor Net”

SensemakerSuite

“Futurepedia”Vegas Online

System


Future Game (Signti!c, Institute for the Future)

Case studies


Expert System (Futurescaper, International Futures Forum)

Case studies


Human Sensor Net (Sensemaker Suite)

Case studies


Futurepedia (Vegas Online Scenario System)

Case studies


Evaluation approach

Hypotheses

1. Individual learning will increase with greater participation in the system

2. Greater numbers of participants will increase the:

a. number of variables and information considered

b. amount of discussion around divergent viewpoints

c. number of scenarios produced

d. quality of !nal scenarios

3. This will produce:

a. a decrease in the probability estimation of individual future events

b. an increase in spread of probability estimates for a range of future events

c. better understanding of divergent viewpoints

d. increased consensus around key issues


Evaluation approach

Empirical measures for hypothesis testing

Number of participants

Time spent participating (when & where)

Participant demographics

Number of variables created / used

Number of scenarios created / shared / revised

Number of comments / discussion threads

Total time taken to complete exercise


Evaluation approach

Qualitative measures for hypothesis testing

Expert panel review: quality of scenarios

Self-reported review: quality of scenarios

Self-reported review: quality of process

- Appreciation of uncertainty

- Understanding of divergent viewpoints

- Consideration of alternative sources of evidence

- Level of surprise / strategic relevance

- Learning & attitude change

Interviews & observation


Evaluation approach

Evaluation example, SenseMaker Suite


Limitations & next steps

Lack of empirical evidence for “learning” (Chermack, 2005)

Hard to isolate cause / effect relationships

Dif!cult to untangle UI effects, audience effects &

“scenario effects”, i.e., Hawthorne effect?

Dif!culty attracting & curating the right audience

Dif!cult retaining interest & investment


Thank you!

Noah RafordPhD Candidate, UIS/CDD

Massachusetts Institute of Technology (MIT)Department of Urban Studies and Planning

[email protected]://news.noahraford.com/

Friday, April 9, 2010

Large-Scale ParticipatoryFutures Systems






Online Scenario Planning

Education

Transcript of Online Scenario Planning