A Meta-evaluation of the “User interviews user method ...€¦ · • The ideal dialogue...

A Meta-evaluation of the“User interviews user method”

Experiences from the evaluation in Oslo

by

Geir Hyrve and Karl-Johan JohansenCES’09

2

Presentation Overview• Presentation of the method User interviews

User• The meta evaluation design• The meta-evaluation process• Findings and utilities• Paradigm shift of methods in evaluation of

services

3

Why it’s important to involve users in evaluation

• In Norway the government wish to include users in decision-making concerning services in order to improve their everyday life.

• We consider this as part of the democracy in a consumer society

• The challenge is how to get to know the consumers opinions

• The “Users interview users” method is an example of how to let the users have impact on the decision-making on different levels

4

Towards more useful evaluations?

• Is this a new approach in user involvement in evaluation?

• What can we learn from this evaluation process?

• These questions bring forward the relevance of a meta-evaluation

The origin of the method User-Interviews-User

This way of conducting evaluation has it’s origin from the service user organization in the psychiatric field in Norway. The method emphasize following elements:• Respect for human lives and lived

experiences.• Loyalty to the Service user perspective - tacit

knowledge, low voices • Empowerment for Service Users and Service

Providers.• The ideal dialogue situation with focus on

deliberative processes5

The User-interviews-user-Method

• Based on well known methods for documentation and evaluation.– Focus group interview –supplemented with

individual qualitative interviews– Qualitative Analysis.

• Dialog conferences • It is the process and the combination of

methods which are unique

6

7

The User-Interviews-User-process• Decision making• Clarifications with the principal• Recruiting informants• Data collection• Analyses• Dialog conferences • Reporting• Findings and impacts on the services• Implementation of results(The meta evaluation has been following up theevaluation process step by step)

Shared understanding

quality of services

DocumentingExperiences.

Reports

Service user Perspective

Through equal terms of communication

Ideal speech situation

Organizational development

Dialogue and learning

FeedbackMotivational

factors

The User Interviews User


factors




factors


quality of services




factors





quality of services




factors


Reports





quality of services




factors





factors







factors


Reports

Dialogue and learning Organizational development





factors

Shared understanding of

quality of servicesand the need of change


Reports





factors

8

Analyses and process documents

• Quality analyses • Process rapport with quotes from the

informants and comments from the interviewers.

• Base document for dialog conferences

9

Steps in the analyzing process

Interviews

Review of transcription of interviews

Systematic review in matrices

Models and descriptions

Process report

Dialogue conference

Final report

10

Conducting the User interviews User in Oslo

• The user evaluation process in Oslo started in the fall 2006 and was finished fall 2008.

• The evaluation was made in all 15 city districts.

• The process was followed up with a meta evaluation.

11

12

The meta-evaluation process• Interviewing staff in the Health and Welfare agency• Reading reports and documents• Observation of focus groups and dialog conferences• Interviewing service providers and managers in the

city districts (retrospective about the process and about results and implementation after the evaluation)

• Interviewing users and representatives from the user organization

• Reporting results at dialog conferences • Writing a final report

13

The meta evaluation design• An inductive approach • A follow-up research of the users interview

users method• A qualitative process oriented evaluation

project• Methods: interviews, observation, document

analysis, surveys from dialog conferences

14

Research purposesThe meta evaluation tried to reveal:

• How the method works out according to data collection, analyses; reliability, validity and relevance

• What is the method characterized by comparedwith traditionally evaluation research?

• What kind of data is this method able to captureand whether the data is relevant according to thenational goals for services?

15

Purpose…The meta evaluation project also focused on:• The evaluator role in this kind of evaluation,

where service users, with their experience and competence, carry out the data collection and analyses

• How the evaluator meet the service system in the most appropriate way?

• How the results are passed on to the principals and stakeholders?

• What happens with the results from the evaluation after the report is delivered?

16

Sample strategies

Informants have been:• The evaluators• Administrative decision makers/districts leaders• Staff personnel in the Health and Welfare Agency• Civil servants that are responsible for carrying out

the results from the evaluation in the districts of Oslo

• Representatives from the user organization

17

Findings and utilities• The method might have challenged some of

the leaders who are responsible for the services

• The method seemed to have impact on the communication in the focus groups, which appeared to be more open and thereby gave more honest answers

• The process helps the informants to be aware of their tacit knowledge and to be able to construct new realities

18

findings and utilities…• Communication at the dialog conferences

had focus on solutions and seemed also to be able to handle criticism in a constructive way

• User activity seemed to increase during and after the evaluation

• The service providers responded that they were more familiar with the results from this evaluation compared with traditional evaluation

19

Change in the paradigm?• The user’s voice is heard in a new way and have

impact on the constructions of realities• The method changes the power balance between

the users and the services in construction of realities• The users must be seen as resources in evaluation

because they have necessary experience and the possibility to develop skills in evaluation

• The method challenges traditional roles in the evaluation field

Challenges and critical factorsMeta-evaluation revealed that there are a number ofchallenges and critical steps associated with theimplementation of the method:• It is important to anchor the process with superior

authorities and persons • It is important to recruit informants who can ensure

independence and representativeness• It is easy to make mistakes in the planning process

both in terms of design and in relation to means and services

• Implementation and monitoring of results is a challenge

20

Demands and Possibilities• Using the method requires considerable expertise in

leading the process, collecting and analyzing data, implementing the results and contributing to the change of services.

• The method can help to reveal tacit knowledge, as well as the activation, participation and strengthening of users.

• The method must be assumed to have a considerable potential in the evaluation and service development also outside the field it until now has been aimed at.

21

22

Cousins & Whitmore (1998)

Summing up

• Validation in the dialog conferences are a form of negotiations between the various actors involved

• The value of truth of the statements and findings from the process reports must be discussed in a local context with involved actors (representatives of users, politicians and service providers)

23

Summing up• The conclusion from the evaluation of the

method “User interviews user” is that the evaluation method create relevant and valid data

• Both the municipality and the users communities found interesting and valuable information to further develop the services to users with mental problems

24

A Meta-evaluation of the “User interviews user method ...€¦ · • The ideal dialogue...

Documents

Transcript of A Meta-evaluation of the “User interviews user method ...€¦ · • The ideal dialogue...