IODA - The Promise & Perils of Narrative Research

30
The Promise & Threats of Narrative Research September 2011

description

A

Transcript of IODA - The Promise & Perils of Narrative Research

Page 1: IODA - The Promise & Perils of Narrative Research

The Promise & Threats of Narrative Research

September 2011

Page 2: IODA - The Promise & Perils of Narrative Research

This is an OD initiative because:

• The organisation makes it’s money from providing consulting services

• To ensure its staff have the skills necessary to do this, it trains them

• The suite of training programs includes four core units

• MBS was asked to validate that these units did indeed meet this goal and so, in theory, be able to help reshape the programs through feedback loops at each step from outcome, through design, implementation, workplace impacts, to the extent that they did not.

Measuring the Impact of OD Initiatives

Page 3: IODA - The Promise & Perils of Narrative Research

Basic Discovery Process Map

Discovery /Dignosis Design Delivery Evaluation

CompetenciesCapabilities

Knowledge

Do = Skills

Believe these are things possessed by individuals that enableBehaviour i.e. the use of skills and knowledge based in values that individuals might get from / learn, in a program that acts as a catalyst so they can do certain things for the workplace

Performance

Behaviour in the workplace

Performance ALL other

Things other than behaviour that contribute to performance in the workplace

Performance

Measuring behaviour in the workplace

Business GoalsWhy we bothered in the first place

Page 4: IODA - The Promise & Perils of Narrative Research

1.We Did a document review of program materials

2.Met with & explored the content and intent of the 4 programs

with client design & presenting staff

3.Did an international review of similar programs

4.Ran a workshop exploring the organisational objectives for

each program and the full suite.

5.Set up Academic Directors and undertook initial observation of

each program

6.Established a narrative capture data collection and analysis

process based on 4 captures roughly associated with the work

of Kirkpatrick and Brinkerhoff

We undertook six activities to put us in a position to evaluate this Organisation Development

program suite

We then began to collect data and run the Feedback Loops that would enable the Organisation to better achieve its goals

Page 5: IODA - The Promise & Perils of Narrative Research

Hybrid Kirkpatrick – Brickerhoff framework

Page 6: IODA - The Promise & Perils of Narrative Research

BRICKERHOFF’S Success Case Evaluation Method

Low end High end.

This method examines participants who, after being surveyed turnout to be the least impacted by the training, these he interviews to understand why.

The other end of this method examines participants who, after being surveyed turnout to be the most successfully impacted by the training, these he interviews to understand why.

The MBS Mt Eliza Impact Evaluation System simultaneously does the Survey, manages the full statistical analysis of the results, plots these in multiple formats and has the data on which these are based immediately available for analysis without any interviewing.

Page 7: IODA - The Promise & Perils of Narrative Research

This philosophy defines development as occurring in 3 main ways: through on-the-job experiences; relationships, networking & feedback; and formal training opportunities.

The “70-20-10” refers to how you should allocate professional development:

- 70% of all development should occur on-the-job,

- 20% from relationships, networking and feedback, and

- 10% from formal training opportunities. We believe that most companies stress the 10%...this creates a problem related to achieving outcomes from training. So we wanted to try to get a better balance through our learning about how the training programs worked for the organisation.

The 70-20-10 Development Philosophy

Page 8: IODA - The Promise & Perils of Narrative Research

Question to discuss….

What are your frustrations or challenges in

evaluating complex initiatives?

Page 9: IODA - The Promise & Perils of Narrative Research

A Children’s Party ……

Page 10: IODA - The Promise & Perils of Narrative Research

Key Characteristics of complex systems..(sometimes called ‘Wicked Problems”)

• context – do not happen exactly the same way twice

• emergence – not predicatable

• principles NOT prescriptive rules

• many possible outcomes- nor ONE right answer

• Too many factors to make predictable connections and predict futures

• can only understand what happened and whatconnections were operating by looking

back

Page 11: IODA - The Promise & Perils of Narrative Research

Disorder

The Cynefin Framework

Source: Cognitive Edge Pte Ltd

Page 12: IODA - The Promise & Perils of Narrative Research

Understanding and Intervening in Complex Issues - Methods

• understanding or “ making sense of ‘ the current state of a defined system in

order to take action and influence outcomes

• narrative based – narratives hold and convey complex meaning

• mass capture for patterns, insights and decision support

• minimise role of ‘experts’ filtering data capture and analysis

• evaluation focus on outcomes and impact

Page 13: IODA - The Promise & Perils of Narrative Research

Questions for Discussion:

How much of your practice is concerned with complex issues ?

What makes them complex?

Page 14: IODA - The Promise & Perils of Narrative Research

Narrative Capture Entry

Point

• Specific to each instrument• Language tailored to target group• Designed to elicit range of experiences

• Specific to each experience• Language tailored to target group• Provide basis for tracking relativenature of experiences over time• Established as double negative slider bars

Prompting Questions

Experiences from

participants

MCQ’s about experience

Filters about experience

General research questions

Demographics

Elements of a SenseMakerTM Narrative Capture

Page 15: IODA - The Promise & Perils of Narrative Research

Prompting Questions: to elicit a response

Key characteristics:

• Place people in a context they can understand

• Provoke people's memories

• Allow the question to be answered in the third person

• Do not privilege a certain type of experience over another

• They relate to the issue at hand

You have been asked to outline what has happened in applying the learning from your program in your workplace. What experience would you relate?

You have been asked by senior management to outline the benefits of the program to

•  Achieving business challenges• Improving workplace effectiveness• Improving the skills and capabilities of individuals What experiences, yours or that of another participant would you tell in answer to this?

Page 16: IODA - The Promise & Perils of Narrative Research

Narrative fragments: material captured in the field in its original form

• Narratives are the naturally occurring conversations that people have. They are not fully thought through crafted stories

• Any narrative captured is displayed in it’s original form including any spelling or grammatical mistakes

Role play

Program X offered participants the opportunity to role play difficult client scenarios with Partners - which accelerated learning and has had the impact of readying me for client situations which have since transpired. I was pushed outside my comfort zone, but at all times felt supported.

Adapting to different working preferences

Program X reminded me that not everyone has the same working style and that I need to be able to adapt to their preferred working style. Using this model, I can determine the best way to communicate with colleagues and clients. The group activity during Program X provides a good opportunity to apply these skills.

Insightful

I have used a lot of the teachings from Program X. I have recently used performed a root cause analysis (i.e. fishbone analysis) to find the root cause of an issue a client was facing. The tool helped to to find solutions which effectively addressed the root cause of the problem.

Persistence

Immediately after leaving the training course I flew to a mine site and was tasked with developing a process manual for an end to end computer system used by all Queensland X mine sites. I flew there alone and was staying in the small country town near the mine (consisting of a pub and a petrol station) and met with considerable resistance from many employees at the mine who I was relying on for the information I would need to include. I sured many of the tools I learned from Program X during this project. I broke my tasks down into manageable chunks, I was very specific when asking for information and I used many tools of interpersonal understanding and negotiation

Page 17: IODA - The Promise & Perils of Narrative Research

Signification - Triads: Assessing the relationship between three competing

Interests

Triads are one of the visual methods used to encourage participants to consider the dynamics that exist between three competing but linked aspects of an issue being explored.

To respond, the participant needs to consider each aspect, moving the ball within the triad to the point they think best reflects the degree to which that aspect is dominant

Page 18: IODA - The Promise & Perils of Narrative Research

Signification - Polarities: Opposing opposites to uncover layers of meaning

The questions are put in terms relative to two extremes of the area under consideration – an “absence of” on the left hand side and “excess of” on the right hand side. By sliding the ball to where they think their experience lies in relation to the two extremes, greater insight into the responses can be achieved when compared to a simple “good / bad” linear scale.

Absence of… Excess of…Ideal state

Page 19: IODA - The Promise & Perils of Narrative Research

Signification – Multiple Choice: The person who provides the experience decides what it

means

Multiple choice questions are also used to get the context of the experience from the participant. These questions can then be used as the lens through which the aggregate of all responses can be viewed. For example, multiple choice question could be used to view all responses from participants whose intent in telling their experience was to “inform” and about the personal outcomes associated with “flexibility and adaptability”

Page 20: IODA - The Promise & Perils of Narrative Research

SenseMakerTM

A look under the cover

Page 21: IODA - The Promise & Perils of Narrative Research

Hybrid Kirkpatrick – Brickerhoff framework

Page 22: IODA - The Promise & Perils of Narrative Research

What did we want to know more about?

Page 23: IODA - The Promise & Perils of Narrative Research

What insights did we find?

Idea

l Sta

te

Boundary management – the social contract

Limited & restricted by too many rules

A free for alldominated by a few

Level of engagement (self)

Avoiding beingnoticed

Centrestage

Group functionality - collaboration

So polite no onedared to challenge

Having no concernfor anyone but themselves

Safe learning environment

Too scared totry anything new

Over protected

Quality of Facilitator / Presenter

Inhibiting Indifferent

Self responsibility

Learnedhelplessness

Someone else’s fault

Performance as an adult learner

Accepting everythingwithout question

Just along forthe ride

Level of engagement (others)

Intense &obsessive

Detached & indifferent

Facilitator / Presenter

Great fun & entertaining,but I did not learn

Too intimidating &dismissive for me to learn

Collaborative learning

Overwhelmed withinsufficient time toshare experiences

Mostly useless & theyshould have shut up

Page 24: IODA - The Promise & Perils of Narrative Research

What insights did we find?

All cohorts

Cohort 1

Cohort 2

Cohort 3

Learnedhelplessness

Someone else’sfault

Cohort 4

Role plays

The aspect that had the biggest impact on my learning was the mock interviews with partners. At first I felt nervous and a little intimidated but as we went through the 4 different interviews I realised that it was a safe environment in which to make mistakes and developed more confidence to give different interview techniques a go. I learnt a lot about how to manage difficult clients and will be less concerned about this if the issue arises in practice. By the end, I was very happy with my presentation and felt that I had overcome my fears to an extent.

Roleplaying

I think that the roleplays had the most impact - all well and good to theoretically teach new consulting skills but experience in putting them into practice I found far more effective in making the messages stick. While it seemed a little forced initially, once into full swing I found it valuable to get to deal with different possible behaviours of clients in interview and group meeting/presentation scenarios and then receive feedback on how it all went. It was also obviously easier to try things a little different and outside my confort zone in a controlled environment.

Self responsibility

Page 25: IODA - The Promise & Perils of Narrative Research

What insights did we find?3 months back in workplace: Behaviours

Page 26: IODA - The Promise & Perils of Narrative Research

What insights did we find?3 months back in workplace: Behaviours

Page 27: IODA - The Promise & Perils of Narrative Research

Question for Group Discussion:

What challenges in using this approach do you think we faced?

Page 28: IODA - The Promise & Perils of Narrative Research

Lessons learnt – about this style of Narrative Research

• Under estimated how disruptive this would be to existing engagement processes and professional practice

• Measuring program impact over time is not appropriate for all organisations

• Upfront commitment to actively participate in collection, analysis and outcomes

• A core team that changes needs new members being effectively inducted

• Transparency and access to staff essential

• Process is not a “one shot” deal

• Need for assessment of program across all four stages to measure impact using feedback loops to improve programs

• Being prepared for unintended consequences and access to the “right” decision makers

Page 29: IODA - The Promise & Perils of Narrative Research

In addition….. normal conditions required for OD initiatives apply

• Commitment to the long term and measuring impact over time

• Senior leadership sponsorship and support

• A core team that will stay with the project

• Transparency of programs – access to all staff

• Commitment of time and resources for analysis and intervention

• Willingness to embrace continuous improvement

Page 30: IODA - The Promise & Perils of Narrative Research

Contacts

• Phill Boas [email protected]

• Viv Read [email protected]

• Chris Fletcher [email protected]

• Reference sites• www.emergingoptions.com.au• www.cognitive-edge.com