THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the...
Transcript of THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the...
Ten D by Night conference Turin,
19 February 2010
THE RESULTS:
QUALITATIVE EVALUATION
Anita Villerusa, Associate Professor
Dean, Faculty of Public health
Riga Stradins University
Ten D by Night conference Turin,
19 February 2010
Ten D by Night conference Turin,
19 February 2010
What is evaluation?
• Evaluation is collecting data about an intervention and its effects, in order to judge the value of the intervention, through making a comparison, so as to make more informed decisions.
Øvretveit, 1998.
• Program evaluations aim to provide convincing evidence that a program is effective. The standards are specific criteria by which effectiveness is measured.
Fink, 1993.
Ten D by Night conference Turin,
19 February 2010
Essentials of evaluation
Evaluation is : to collect data,
to judge the value,
to make a comparison,
to make more informed decisions.
Audit evaluation – comparison what was intended (objectives) with what happened
Ten D by Night conference Turin,
19 February 2010
Project evaluation framework
Project start
Project processes
Project end
Goals/aims/objectives
Planning
Communication
Management
Training
Piloting
Field work
Outputs
Outcomes:
-New preventive approach
-Changes in alcohol/drug related accident rate
Ten D by Night conference Turin,
19 February 2010
The criteria for evaluating a health
programmeCorrespondence - if the action corresponded to objectives and
procedures?
Outcomes (Short term and long term)
Objective measures
Subjective (retrospective perception)
Efficiency - resources consumed and outputs produced
Acceptability
to assess target population and providers willingness to use and
apply the action
Sustainability
the continuation of the programme, if finance continues, or if
withdrawn
Ten D by Night conference Turin,
19 February 2010
Six standard criteria for programme
evaluations
1. What was done? (description)
2. Did all activities follow the plan and achieve the
objectives? (correspondence to plan and objectives)
3. What were the outcomes? (Did they meet needs of
target group?)
4. Was there any waste of resources, or could have used
resources more efficiently? (efficiency)
5. Did people participate in the project? (acceptability)
6. What threatens the continuation – with and without
continued funding? (sustainability)
Ten D by Night conference Turin,
19 February 2010
Methods for qualitative evaluation
• focus group discussion
• In- depth interviews
• questionnaire also with open- ended
questions for managers and for volunteers
• analyses of different documents
(Analyses of written minutes of meetings;
E- mail Correspondence; reporting tables;
project time table and financial plan, press
releseas, publications etc.
Ten D by Night conference Turin,
19 February 2010
Participants in the evaluation
Participants in the focus group discussion- 10 respondents of the
Ten D by Night Project participants from Italy took part in the focus
group; 3 were project managers and 7 were volunteers
In- depth interviews - Project technical coordinators- S&T Italy
Questionnaires – one questionnaire for managers and one for
volunteers (7 project managers and 22 volunteers completed the
questionnaires from al countries except Spain)
Ten D by Night conference Turin,
19 February 2010
Qualitative evaluation - Results
• Preparation for the work
Training of project workers/volunteers.
The equipment - alcohol tests, drug tests, console for measurement of the reaction time.
Project tools - questionnaires, leaflets, information/consent forms, posters advertising the project and action in particular places and times.
• Sampling recreation places (Selection of night clubs, their representativeness and how they correspond to the target audience)
• Field work
• Advertising the project and rising awareness among the local population.
Ten D by Night conference Turin,
19 February 2010
Involvement of volunteers in the
project
• Poland, Bulgaria, the Netherlands and
Belgium where the project partners were
NGOs they used people who they had
collaborated with in the previously
executed projects as volunteers
• Students
• the number of field workers
Ten D by Night conference Turin,
19 February 2010
Information about the project, training
and preparation for the field work
Project managers
• ~ one hour training;
• Some theoretical part
of training;
• Training on the
practical
implementation of the
tests
Volunteers
• A majority assessed
the level of
information as
sufficient
Ten D by Night conference Turin,
19 February 2010
Evaluation of information volunteers
received about the project, its mission,
objectives and tasks before field work
9
8
8
4
6
5
4
2
2
5
6
7
0% 20% 40% 60% 80% 100%
amount
type
quality
sufficient suff>insuff insuff>suff insufficient
Ten D by Night conference Turin,
19 February 2010
The level of skills regarding the
practicalities of the field work
16
8
10
11
11
1
4
3
5
6
6
2
2
2
5
4
3
4
3
0% 20% 40% 60% 80% 100%
alcohol t.
drug t.
console
communic.
questionn.
sufficient suff>insuff insuff>suff insufficient
Ten D by Night conference Turin,
19 February 2010
Selection of night clubs
• Different opinions
Some difficulties
Great number of rejections
in Italy
Less problems
Poland, Bulgaria and Belgium,
The Netherlands
Ten D by Night conference Turin,
19 February 2010
• The main problem was to choose the clubs . When they were chosen, there was no problem in gaining admission. In other words, we faced no problems”. (Poland, Bulgaria and Belgium )
• “Making appointments was difficult/the clubs and discos were poorly accessible/ it was sometimes difficult to start a discussion about drugs”
Ten D by Night conference Turin,
19 February 2010
Reasons for refusal
• luck of interest;
• unwillingness to collaborate;
• fear about control checks of people;
• possible negative effect on the number of
visitors.
Ten D by Night conference Turin,
19 February 2010
Representativeness and the
target audience
• Difficulties to reach representativeness
• the project succeeded in reaching the maximum number of participants who correspond to the target audience- young people aged from 18 to 24;
• attitudes of the customers during the field work - “Overall good even if some of them found that it took too much time to answer the questionnaire and to do the tests. Most of them liked it and found it interesting to do the tests; they cooperated well”.
Ten D by Night conference Turin,
19 February 2010
Field work
Ten D by Night conference Turin,
19 February 2010
The design and methodology of
the project
• The experts
acknowledged that
the design of the
project was
appropriate and
adequate.
• Volunteers-
“How well did you
understand your role
in the field work – the
tasks and duties to
undertake”. 73% said
very well and 27%
quite well.
Ten D by Night conference Turin,
19 February 2010
Evaluation of the suitability of
the tools
5
16
4
8
7
4
6
6
5
5
1
2
5
1
2
1
1
1
0% 20% 40% 60% 80% 100%
questionn.
alcohol t.
drug t.
console
very good good average bad very bad
Ten D by Night conference Turin,
19 February 2010
Field work
Positives• both the managers and volunteers were satisfied with
the process of the field work
• Good collaboration was established with night clubs
• the level of awareness of the necessity of preventive activities increased within the night clubs
• The informative materials provided proved to be useful and small incentives/presents helped to attract participants
• New and expensive equipment was bought
• It was possible to implement the project in several countries at the same time and compare the results
Ten D by Night conference Turin,
19 February 2010
Field work
Negatives
• In some clubs there was no space where to place questionnaires or it was not convenient to complete them
• There were queues to pass tests and some customers left because they did not want to queue.
• The drug tests took too much time
• the questionnaire was too long and some questions were misleading;
• In some cases there was not enough experience among the field workers how to deal with drunken people
Ten D by Night conference Turin,
19 February 2010
Ethical aspects
The anonymity
The system of anonymity worked very well and was
adequate for the needs of project.
no threats to volunteers
Ten D by Night conference Turin,
19 February 2010
Cultural and religious obstacles
and barriers
• No barriers were noticed.
• men showed a higher level of interest
regarding the intervention.
• there were many cases when women
refused to take part in intervention.
• although there were no restrictions on
gender, gender inequality was observed in
some countries.
Ten D by Night conference Turin,
19 February 2010
Publicity of the project
• radio interviews;
• publications in newspapers;
• the information in web-sitewww.tendbynight.com
• posters and brochures
• The experts interviewed gave 3,7 marks out of 5 for the publicity at national level and 3,5 marks at the international level.
Ten D by Night conference Turin,
19 February 2010
Evaluation of the suitability of the
project materials for field work
1
4
1
4
0
6
8
8
9
8
7
6
8
6
8
6
11
5
3
2
1
2
2
3
1
0% 20% 40% 60% 80% 100%
brochures
logo
posters
webpage
leaflets
gadgets
very good good average bad very bad
Ten D by Night conference Turin,
19 February 2010
Use of results
• for practical purposes
• for scientific and research purposes
• for constructing educational programmes
• The further development of the project
idea
Ten D by Night conference Turin,
19 February 2010
Meetings and collaboration
among project countries
• the meetings were useful and the amount
of them as optimal for the project
• Good collaboration
• Good communication via e-mail
Ten D by Night conference Turin,
19 February 2010
“This was a very good project. It
gives understanding to young people
regarding those things which are
important to their health and safety.
The project gave us the opportunity
to talk about these things … and also
opportunity to talk to each other.”
Ten D by Night conference Turin,
19 February 2010
Some final points
• Some changes in the staff of the project. .
• The decrease of the initial budget by 50
%...
• Some corrections in time table...
All the activities were implemented on time.
Ten D by Night conference Turin,
19 February 2010
Goodbye Turin!