Rapporto di valutazione POVEL def_lorena_EN

25
Code JUST/2009-2010/DAP3/AG/1349 Project POVEL Daphne programme EVALUATION REPORT Edited by This presentation has been produced with the financial support of the Daphne III Programme of the European Commission. Contents are the sole responsibility of Scuola Centrale Formazione and its partners and can in no way be taken to reflect the views of the European Commission.

description

rapporto di valutazione POVEL

Transcript of Rapporto di valutazione POVEL def_lorena_EN

Page 1: Rapporto di valutazione POVEL def_lorena_EN

Code JUST/2009-2010/DAP3/AG/1349

Project POVELDaphne programme

EVALUATION REPORTEdited by

This presentation has been produced with the financial support of the Daphne III Programme of the European Commission. Contents are the sole responsibility of Scuola Centrale Formazione and its partners and can in no way be taken to reflect the views of the European Commission.

Page 2: Rapporto di valutazione POVEL def_lorena_EN

The evaluation of the POVEL project was ensured by the survey activities established by the evaluation plan proposed by SCF to the partnership at the beginning of the project and re-adapted in compliance with the needs which emerged at that time (the survey instruments established by the evaluation plan are attached, except for the monitoring form which is attached to the monitoring report).The evaluation plan provided for a monitoring action supported by a survey card on the progress of the activities and the problems which emerged in the course of the project and by a satisfaction survey to measure the level of appreciation and effectiveness of transnational meetings. This survey activity allowed us to control the progress of the project and the quality of the products made, especially during the early project phases (research and kit planning phase), the intermediate monitoring report allows us to receive feedback on this part of the evaluation activities.The second type of evaluation activities focuses mostly on quality and refers to the kit testing phase. As mentioned, the first two phases: research and kit preparation, in consideration of their characteristics, were kept under control through monitoring. Research indeed was carried out with the aid of a qualified and recognized scientific supervision which ensured the correct preparation of the questionnaire, of data collection and processing. This questionnaire would have made an additional quality evaluation redundant, that’s why only a survey activity was put in place. As concerns the planning phase of instruments and kits, a quantitative monitoring was deemed fit for the assessment of the activities performed since the quality of the products made was evaluated over the course of testing.

For the evaluation of the testing phase three instruments were used: a structured questionnaire to measure the perception of the kits by the students involved (annex 3), a semi-structured journal edited by the coordinators and tutors of the of the activities (annex 1), a semi-structured interview made in the presence of the project partners (annex 2). Some accounts of involved students were also taken into consideration. These instruments allowed us to collect the point of view of the stakeholders involved in the activities: students, trainers and tutors who directly managed classroom testing, coordinating the partnership activities.

The testing of the four teaching kits (Media violence, Peer violence and bullying, violence related to drug assumption, violence related to alcohol abuse) is briefly described in the following chart.

Country Organization Own kit Target Other kit Target

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

2

Page 3: Rapporto di valutazione POVEL def_lorena_EN

involved in testing tested tested

Italy CEFAL, CIVIFORM, FOSF

Media Violence65 (44

questionnaires filled in and returned) + 85 (peer education

conference in CIVIFORM)

Violence related to alcohol abuse; Peer violence and bullying; Violence

related to drug assumption

Period: March – July 2012 Period: October

2012 -8 February 2013

Spain TRINIJOVE

Peer violence and bullying

40

Media Violence

40

Period: April - may 2012

Period: April - may 2012

French ID Formation

Violence related to drugs

assumption34

Violence related to alcohol abuse

41

Period: October - November

2012

Period: October - November 2012

Belgium FISSAAJ

Violence related to

alcohol abuse 52 students

Violence related to drug assumption 50 students

Period:October -

December 2012

Period: October 2012 - January

2013

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

3

Page 4: Rapporto di valutazione POVEL def_lorena_EN

The point of view of coordinators, trainers and tutors

By analysing the outcomes of the semi-structured interviews made to the coordinators of involved partners and the journals written by trainers and tutors what follows emerges.As mentioned during the monitoring phase, after each partner tested a kit it had not prepared, a phase of adaptation to the national context and the specific needs of final beneficiaries followed, which was especially necessary for the “violence related to drug assumption” kit that had been designed for an older target of beneficiaries than the others. Not all partners chose to carry out all the activities contained in the original kits, as expected, each organization followed and adapted the proposals which would better suit their local needs. In all the accounts gathered, this adaptation phase is deemed as essential for the success of the activities. An adaptation activity was required in some cases over the course of testing: for example while testing the “violence related to alcohol abuse” kit by Civiform problems were noted concerning the themes dealt with which were connected to the personal and family experiences of the students, as a consequence we chose to re-organize some of the set contents to make room for the unexpected issues which emerged. All the contents adaptation activities carried out show that the project request of preparing flexible and adaptable instruments was met.

All the kits provided for informative activities consisting of presentations and interactive activities (video-clips, games, theme discussions...) which would boost the active role of the students in facing the themes under review. The activities in almost all cases were managed by internal trainers and tutors of the organizations involved in testing. In two cases the management methods partly deviated from this model: The Belgian partner involved independent experts to manage all the set activities, because of the mistrust shown by the school trainers to face alcohol- and drug-related themes, which made it impossible to directly involve them in the activities. In this regard, please note that it would be necessary to work on raising the awareness of these figures on the importance to deal with similar themes on a regular basis in the teaching curriculum. Another organization which, for different reasons, decided to involve experts is Civiform which decided to foster a peer education-based education programme: the themes of the tested kits were proposed and managed by some peer-education experts who worked with small groups of students (12 per kit) who then had the chance, during an ad-hoc event, to carry out an information/education activity of their CFP peers (approximately one hundred). These peer education-trained students were also appointed as contact people for their classmates who would require “advice” on the

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

4

Page 5: Rapporto di valutazione POVEL def_lorena_EN

issues dealt with during the year. In the case of Civiform CFP teachers were still involved and at a later time of training they also directly participated.Most of the managers of the activities think that creating a motivated and active group is very demanding. To favour the interest of the students the strategies included:

- Favouring the more theoretical phases of presentation in the morning hours and leave the more interactive activities for the afternoon;

- Involving the students in the planning phase of the activities to be carried out;

- Involving experts from the reference sector to skilfully and punctually deal with problems concerning the personal experiences of the students;

- Referring the themes which appear to be more theoretical to the personal experiences of the students favouring the surfacing of their experiences and actual examples.

In all the cases the stress is on the fact that after an initial mistrust of the students (and in some cases of the teachers involved) on the proposed activities a clear increase in interest and participation of the group is shown that is also witnessed by the outcomes of the questionnaires to the final beneficiaries at the end of testing. From the point of view of the trainers involved some significant elements emerged, in some cases defined as unexpected, in relation to the themes dealt with, that is worth mentioning:

many of the involved students struggle to catalogue the use of alcohol as an abuse while they are more familiar talking about drug abuse;there is still much confusion about the legislation on abuse despite the fact that some themes are now usually dealt with by media, this shows the ineffectiveness of many of the conveyed messages, on the contrary after completing the proposed kits the students prove to have appreciably increased their awareness.The theme of alcohol abuse in many cases has led to the appearance of problematic personal and family experiences, an element which has us think about the need not to underestimate the importance of regularly addressing this theme in school curricula.As regards the students involved in testing the alcohol and drug abuse kits those involved in the testing of bullying and media violence kits appear to have less prior knowledge and direct experience on the issue.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

5

Page 6: Rapporto di valutazione POVEL def_lorena_EN

The point of view of students

At the end of testing the students who took part in the testing were asked to answer anonymously a questionnaire which would record their interest in the activities carried out and their perception on the effectiveness of proposed instruments.Below, there is a series of charts which outline what came out of the answers. For each question there is a corresponding chart which contains the answers expressed as a percentage of all students (notwithstanding the kit they received) and one where it is possible to distinguish the answer percentage in relation to the specific programme (Media violence, Peer violence and bullying, violence related to drug assumption, violence related to alcohol abuse). The percentage data refers to a sample of over 400 students who answered the questionnaires given by the various partners involved.

The first question asked aimed at checking how involved students perceived the usefulness of the programmes in relation to their knowledge.

1)Think about Kit topics:

13%

56%

31% I didn’t learnanything new I learned some newthings I learned many newthings

It appears that a minority of students declares that they did not learn anything new, 87% of the total on the contrary declare that they learned some or many new things. By analysing the distribution of these percentages by kit topic, it appears that the themes on which young beneficiaries declared that they learned more new things are media and peer violence and bullying. In any case it is necessary to consider that those kits were tested on the lowest number of beneficiaries each (93 for the bullying kit, 84 for the media kit).

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

6

Page 7: Rapporto di valutazione POVEL def_lorena_EN

18

66

16

22

57

20

2

45

53

5

46

49

0%

20%

40%

60%

80%

100%

Violencerelated to

alcoholabuse

Violencerelated to

drugassumption

Peerviolence

and bullying

Mediaviolence

1)

% I learned many newthings % I learned some newthings % I didn’t learnanything new

The second question aimed at checking the ability of the proposed activities to affect the opinions of involved students.

2) Your opinions about the kit topic:

20%

45%

35% They are changed

They are partiallychanged They are the same

Almost 2/3 of interviewed students declare that they partly or fully changed their opinions on the themes dealt with. 35% declare that they did not change their opinion. By analysing the comments on this question and the accounts of the students the outcome of the answers is clearer: a large part of the students declares that they already know the danger of some behaviours shown in the programmes, it then appears that most of the students who did not change their opinion are those who were already quite aware of the themes. This especially

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

7

Page 8: Rapporto di valutazione POVEL def_lorena_EN

applies to the drug abuse kit, a theme in relation to which the students already appear to be quite aware and informed.It is still worth noting that from the comments it emerges that a minority of the students who were not knowledgeable and aware of the themes, did not change their opinions (in these remaining cases the kits would appear not to affect the opinion).

9

44

47

11

35

54

30

55

15

41

47

12

0%10%20%30%40%50%60%70%80%90%

100%

violencerelated to

alcoholabuse

violencerelated to

drugsassumption

Peerviolence

and bullying

Mediaviolence

2)

They are the same

They are partiallychanged They are changed

The third question asked aimed at checking the interest of the involved students in the teaching methods used.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

8

Page 9: Rapporto di valutazione POVEL def_lorena_EN

3) During presentations and discussions:

12%

24%

64%

I wasn’t interested

The topics wereinteresting but theywere addressed in aboring wayI was interested

It appears clear that the large majority of students shows interest in the programmes proposed. However especially for the programmes concerning drug and alcohol abuse the portion of students (relatively 37% and 36%) who shows interest in the themes but does not appreciate the methods adopted, deemed boring (the comments claim that they do not appreciate that too much room is given to presentations, while active teaching methods are more appreciated) is significant. This type of remarks is much less important with reference to the two kits on bullying and media violence, programmes in relation to which the students also showed to know less of.

10

36

54

18

37

45

85

86

1310

77

0%10%20%30%40%50%60%70%80%90%

100%

violencerelated to

alcoholabuse

violencerelated to

drugsassumption

Peerviolence

and bullying

Mediaviolence

3)

I was interested

The topics wereinteresting but theywere addressed in aboring wayI wasn’t interested

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

9

Page 10: Rapporto di valutazione POVEL def_lorena_EN

This fourth question, too, aimed at checking the appreciation of the different teaching methods.

4) I was more interested in:

18%

58%

24%

PresentationsDiscussionsAnything else

As already mentioned in the analysis of the outcomes of the answers to the third question, active teaching methods are universally more appreciated for all the programmes.

22

66

12

16

60

24

13

53

35

19

49

32

0%10%20%30%40%50%60%70%80%90%

100%

violencerelated to

alcoholabuse

violencerelated to

drugassumption

Peerviolence

and bullying

Mediaviolence

4)

Anything else DiscussionsPresentations

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

10

Page 11: Rapporto di valutazione POVEL def_lorena_EN

Answers 5 and 6 were asked as a close examination of the first question (Thinking about the topics do you believe you learned some things, many things or nothing?) aiming at understanding how much the information conveyed appeared to be interesting and how much they are actually rooted in the experience of the students.

5) Did you talk about kit’s topics with friends?

49%51%

YESNO

Approximately half the students (49%) declare that they talked of their own accord with some friends about the themes. It is worth examining this question, since the results of the two pairs of kits are quite different: indeed, for the media violence kit, the percentage of young people who talked to their friends about the topics is 72%, hence broadly above half. For the bullying kit, too, the value exceeds half the youth who tested the kit (60%). This data shows an actual interest in the themes and the perception that the information received can be useful to significant people amid the group of acquaintances. Please note that the sample of youth who has tested the media kit and the bullying kit are less numerous. By analysing the reasons of those who answered “No” many state that there was no chance to talk about it, but that if the chance came up they would. A minority of statements that can be summarized in the ‘my friends would not be interested/I don’t think talking about it is useful’ is also included.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

11

Page 12: Rapporto di valutazione POVEL def_lorena_EN

34

66

42

58

60

40

72

28

0%10%20%30%40%50%60%70%80%90%

100%

v iolencerelated to

alcohol abuse

violencerelated to drug

assumption

Peer v iolenceand bully ing

Mediav iolence

5)

NOYES

6) I f your friend was in one of the situations have been described during Kit:

28%

65%

7%I would like to givehim/her some advice,but I think I ’d not beable to do it I think I would be ableto give him/her someinformation

I 'd not say anything

The effectiveness of the message conveyed by the kits is also confirmed by the answers to the following question: most students show a proactive behaviour, declaring that they feel able to give advice to a peer who would find him/herself in alcohol/drug-related situations. The picture is more or less the same in relation to all kits, a slight difference can be noticed in the drug assumption kit for which more students would not be able to give advice.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

12

Page 13: Rapporto di valutazione POVEL def_lorena_EN

17

77

6

31

62

7

22

62

4

16

79

5

0%

20%

40%

60%

80%

100%

violencerelated to

alcoholabuse

violencerelated to

drugassumption

Peerviolence

and bullying

Mediaviolence

6)

I'd not say anything

I think I would be ableto give him/her someinformation

I would like to givehim/her some advice,but I think I’d not beable to do it

This last question was asked to confirm the interest in the programmes performed.

7) Would you like to participate in other training moments like this?

68%

32%

YESNO

On average, the majority of students declare that they are inclined to repeat the experience, however 32% say the opposite. The comments to this question show that the making of the question has probably been misunderstood: many students who declare that they would not repeat the experience justify it by saying that the programme was comprehensive and it would not be useful to repeat it, however it would be interesting to propose a similar programme on different topics. These comments show that the figures are misrepresenting the situation.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

13

Page 14: Rapporto di valutazione POVEL def_lorena_EN

It is however significant that a much higher percentage of students would like to attend again the programmes on bullying and media violence, themes which the students appear to be less familiar with, also according to the other survey instruments used.

57

43

53

47

88

12

84

16

0%10%20%30%40%50%60%70%80%90%

100%

violence related toalcohol abuse

violence related todrug assumption

Peer violence andbullying

Media violence

7)

NOYES

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

14

Page 15: Rapporto di valutazione POVEL def_lorena_EN

Conclusions

To conclude, it is worth summarizing the positive and negative aspects which emerge from the point of view of all the parties involved (students, coordinators, trainers and tutors) in the project testing phase, aspects which should be kept in consideration for future activities which could be proposed again in the future.

Positive aspects:Appeal and dynamism of the instruments used with students (in particular those concerning active teaching techniques).Positive response of students: motivated and involved, many students asked to take part in the proposed activities.Greater awareness of the students on the themes.Repeatability of the kits in other contexts (flexibility of proposals).Concreteness and usability of the created and networked instruments.In some cases the fact that the kits were managed by independent experts favoured the honest expression of opinions of students.Possibility of integrating the kits in the teaching curricula of the students (which happened with the Spanish partner).Active role of students (especially clear in the use of peer education which has also guaranteed a broad impact on many students).Growing interest of teacher in the kits (in some cases during the first testing phase they were suspicious, during the second testing phase many asked to be involved).Increased knowledge and awareness of students on the themes.

Negative aspects:Logistics and time required to carry out the set activities: problems with identifying the more appropriate moments to place them in the teaching programme (ex. June is the end of courses) with a resulting reduction in the time available for testing.In some cases mistrust of trainers who do not wish to take part in the administration of the kits.Starting phase of a motivated and interested group, especially when participants have very different knowledge and experiences on the themes under examination (ex. some are usual users of drugs others are not).Problem with completing the programme in the cases in which the involved beneficiaries were not usual participants of long-term training programmes or in the cases in which they dropped out before the end of the programmes.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

15

Page 16: Rapporto di valutazione POVEL def_lorena_EN

Possibility of using the programme promoted by the kits to work on the classroom climate and the group consolidation.

From analysing these remarks it is clear that the strengths of the experience exceed the criticalities, in relation to which the partnership has also adopted effective response strategies over the course of the project.

We have already mentioned the deliberately flexible nature of the instruments offered in order to permit their subsequent re-use in other contexts. In this regard, to conclude, it is useful to mention some aspects that is worth considering to guarantee the feasibility of this transferability: especially in the cases in which independent experts have been involved in the management of the activities, the reproducibility of the experience is subject to finding new funds which permit their involvement. It also emerges that it would be useful to keep working on the project website in order to make the published instruments increasingly clear and useable.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

16

Page 17: Rapporto di valutazione POVEL def_lorena_EN

Annex 1

Project “POVEL”Daphne Programme

Trial diaryEach partner who has to do the trail will describe the mail events that will happen during experimentation specially unexpected events. The diaries:- will give information to evaluate kits transferability in other contexts- will give useful information for writing guidelines

The trial diaries will be write in english and sent to Caterina Aimé ([email protected]) at the and of experimentation.

Date and place

Who compile

Event description

Materials produced(describe and attach them)

Things unexpected?

if yes describe it

Any decision taken due to unexpected things?

Outcome of unexpected actions

- -

-

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

17

Page 18: Rapporto di valutazione POVEL def_lorena_EN

Annex 2Project “POVEL”

Daphne Programme

TRIAL INTERVIEW OUTLINE

Each partner who did the trail will describe the mail events that will happen during experimentation specially unexpected events. This interview is aimed to:- collect information to evaluate kits transferability in other contexts- collect useful information for writing guidelines

1.What Kits have you tested? 1)2)

2.How many students were involved in each kit tested? The students who are involved in first trial group were the same involved in second group?1)2)

3.How long was the trial? And how it was organized? (when it started and ended, how many hours for each day? How many students for each trial group? How many tutors for each trial group? Do you have done trial following the kit or you did some changes?)

4.During trial activities did unexpected things happen? If yes describe it:for example: discussions between students, problematic personal experiences,

5.What did you do to face up to these unexpected situations?

6.Can you tell me 2 of the two most positive aspects that you think came out of the trial?1)2)

7. Can you tell me 2 of the two most negatives aspects that you think came out of the trial?1)2)

8.Do you think that in the future your organization will continue to use the tested kits with other groups of students?

9.Are there any aspects in kits that you would change after the trial? If yes what and how?

10. Do you have any other comments?

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

18

Page 19: Rapporto di valutazione POVEL def_lorena_EN

Annex 3Project “POVEL”

Daphne Programme

POVEL-Kit Trial costumer satisfaction for students

At the end of each kit each student will complete this questionnaire. The goal is to collect student’s opinions about activities.

1. Kit Topic tested: Violence related to drug assumption Violence related to alcohol abuse Peer violence and bullying Media Violence

2. Think about Kit topics: I didn’t learn anything new I learned some new things I learned many new things

3. Your opinions about the kit topic: They are changed They are partially changed They are the same

4. During presentations and discussions: I wasn’t interested The topics were interesting but they were addressed in a boring way I was interested

5. I was more interested in: Presentations Discussions Anything else (specify):

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

19

Page 20: Rapporto di valutazione POVEL def_lorena_EN

6. Did you talk about kit’s topics with friends? Yes No

7. If your friend was in one of the situations have been described during Kit: I would like to give him/her some advice, but I think I’ll not be able to do it I think I would be able to give him/her some information I did not say anything

8. Would you like to participate in others training moments like this? Yes No

9. If you have any comments, write them here:

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

20