Blue Plasma Template

Evaluating pre-driver road safety education interventions for secondary school studentsMargaret Ryan,Dr. Michael Gormley, Trinity College Dublin

1

1 1 Evaluating

Hello, Im Margaret Ryan from the School of Psychology, Trinity College Dublin in Ireland.Today Ill be talking about my Ph.D. project where I evaluated pre-driver road safety education interventions for secondary school students. My supervisor was Dr. Michael Gormley and the Irish Road Safety Authority funded the research. OutlineBackground

Aims

Methodology

Results

Conclusions22 Outline

Ill begin by giving you some background to the study and outlining what we hoped to achieve. Then Ill describe the methods that we used and go through some of the results. Then Ill discuss some of our conclusions.

2 Young driver problemIrish Drivers - 17-24 year olds5 times greater risk of dying than other drivers (RSA, 2009)Engineering Enforcement Education

Pre-driversDevelopment of risky attitudes in adolescence (Harr et al., 2000)Cradle attitudes, grave consequences (Waylen & McKenna, 2002)Pre-driver education in secondary schools

3Young Driver ProblemWe know that young drivers are problematic because they have more crashes than other drivers. For example in Ireland, 17-24 year old drivers are 5 times more likely to die in a crash than all other drivers.We try to fix this problem by improving ENGINEERING, by better ENFOCEMENT of driving regulations and by providing more driver EDUCATION. Many of these solutions target people who drive already. Around 10 years ago researchers like Nikki Harre in New Zealand and Waylen and McKenna in Britain reported seeing risky attitudes towards driving in adolescents who were not old enough to drive legally. Apparently, these beliefs and attitudes were formed through watching people driving and noting the consequences or more typically the lack of consequences. Since then, a range of pre-driver education programmes have been developed to help to instil good thinking habits in adolescents before they actually get behind the wheel of a car. In Ireland these programmes are usually delivered in year 12, [when most schools provide a Transition Year curriculum, were students take a break from the regular activities and focus instead on developing broader life skills. This fits very well with the Goals for Driver Education framework developed Hatakka and his colleague, which suggests that in addition to basic vehicle manoeuvring skills, learner drivers should be taught to recognize personal tendencies and motivations that lead to increased risk-taking in traffic.]

3Aims of the Project Summative EvaluationImpact of 5 types of pre-driver educationKnowledge - Cognitive Skills - AttitudesSpecify the influence of proximal and distal factors

Formative Evaluation of RSA programmes

Why evaluate?Assess effectivenessSuggest possible improvementsEnhance student learningIncreased efficiencyDelivery / Content / Cost Ensure that we do no harm

44 Aims of the ProjectThe evaluation strategy that we used was largely informed by a set of guidelines produced by Kathryn Clinton and Larry Lonero for the AAA Foundation for traffic safety and which incorporate best practice in programme evaluation.

The project had 2 aimsThe main aim was to provide a SUMMATIVE EVALUATION of a range of pre-driver education programmes that are used in Ireland. We wanted to see if taking a pre-driver education programme produced any measurable change in knowledge, cognitive skills and attitudes in a representative sample of students who take these of courses. We also supposed that factors such as gender, personality and previous experiences as road users would influence these outcomes.

Our second aim was to provide a FORMATIVE EVALUATION of two specific programmes, which were developed by the Irish Road Safety Authority.

SO WHY EVALUATE?Evaluation is important because We need to know that what we are doing is effectiveWe also need to know how courses can be improved in order to enhance student learning and to maximize the benefits while limiting the costsWe also need to make sure we do no harm! Previous research suggests that some types of pre-driver education courses promote early licensure or instill overconfidence, and thus increase exposure to risk rather than alleviating it.

4ParticipantsAll second-level schools in ROI invited

5DEMOGRAPHICS Males 54%Females 46%Urban Dwellers 62% Rural Dwellers 38%Parents with 3rd level Ed. 46%Parents with 2nd Level Ed 39% ProgrammeGroups / ClustersNumber of studentsBaseline (T1)Post-intervention (T2)Follow-up(T3)Programme A5244207216Programme B8430344383Programme C6265231226Group D5269217210Group E4160126134Controls8 + 2291199243Whole School Drop-out3221--Total 411880132414125 ParticipantsThe participants in this quasi-experimental study were recruited by means of opportunity-based sampling from all of the second-level schools in the Republic of Ireland. The final sample was drawn from 33 schools, and included 27 groups who took different types of pre-driver education courses and 10 clusters that formed a control group. We conducted three tests, the Baseline tests were conducted in the Autumn of 2009, before any of the students took a pre-driver education course, the post-intervention tests were conducted with individual groups just after they finished their course and follow-up tests were conducted in the Spring of 2010. On average there was a 26 week interval between the baseline and post-intervention tests and a further 51 week interval between the post-intervention and the follow-up test. Attrition between baseline and post-intervention was 30% and between baseline and the follow up test was 25%.

5 different types of programmes were represented in the study. Programmes A,B and C are used widely with transition year classes and are delivered over several weeks or months. Programme A is delivered over 45 hours, whereas programmes B and C take just 20 hours. Students in group D who also get 20 hours tuition took courses that had been developed in their own schools. Group E represents students who took courses that were delivered in a single day.CLICKWe had 1880 students at Time 1, 1324 at Time 2 and 1412 at Time 3. The sample was reasonably well-balanced with respect to gender, location and Socio Economic Status. The modal age at Time 1 was 15 years old.Since this was a longitudinal design and the participants were nested in school groups, hierarchical multilevel modeling was used to analyze the data. This adjusts for non-normality and reduces the risk of type-1 errors.

Questionnaire MeasuresFACTORFOCUSDemographicsAge / Gender / Location / School Type / Prog. TypeDirect ExperienceRoad User / Driving / Crash ExperienceManchester D. B. Questionnaire (Reason et al.1990)Observational LearningExposure to aberrant driving stylesPersonalitySensation Seeking - AISS Scale (Arnett, 1994)Impulsiveness - BIS-Short Form (Spinella, 2007)5-Factor Model - IPIP (Goldberg, 1998)Factual KnowledgeBaseline General KnowledgePost-Intervention multiple-choice quiz (Rules of the Road)Cognitive Skill -Risk PerceptionObjective and Subjective risk estimationsSelf-efficacy beliefsVignetteProgrammeEvaluationEvaluation of programme content & deliverySuggestions for programme improvement66 Questionnaire MeasuresData were collected using questionnaires which took around 50 minutes to complete. Here is an overview of the variables that we were interested in and how they were measured.We used some well known instruments for example the Manchester Driver Behaviour Questionnaire, Barratts Impulsiveness Scale and Arnetts Inventory of Sensation Seeking. We also devised some new measures including a vignette, which Ill discuss later.

67 7 7 Knowledge Quiz ShortClassroom-based education is seen as a good way to teach factual knowledge. To measure changes in knowledge during the course of our study we devised a short knowledge quiz, using questions adapted from the driver theory test. We analysed the results using Item Response Theory because it helped us to screen for items that were poor at discriminating between students of different abilities. It also provided an index of item difficulty which allowed us to judge the appropriateness of the test by categorising test items as either easy, moderate or difficult to answer. Then we calculated proficiency scores for the students based on 11 items, including 6 difficult items, 1 moderate item and 4 easy items. After that the scores were transformed to a scale similar to the one used in IQ tests.The mean score for the entire sample at baseline was 99.96 and this increased to 105.67 in the post-intervention test and then dropped back slightly to 104.44 in the follow-up test. Both the short-term and long-term gains in proficiency that we see here were statistically significant. The most notable thing about this graph is that is shows that, the pattern of change in proficiency scores for students in the control group is very similar to those seen in the active groups, which was not what we expected to see/.

The results of hierarchical multi-level regression analyses showed that there was no significant difference between the scores for students who took pre-driver education courses and those who did not, in any of the tests. Analysis of the scores for individual groups showed that they all gained a significant amount of knowledge between the baseline and the post-intervention test. The increases ranged between 4% and 7% for the active groups and 4% for the control group. There was no significant difference in the rate of gain for the active groups in comparison with the control group when their baseline scores were controlled. The losses that we see here between the post-intervention and follow-up tests were not statistically significant. Analysis of the long-term effects of pre-driver education showed that students in groups A, B, C and the controls all gained a significant amount of knowledge during the study. However, there were no significant differences in the amount of knowledge gained by students in the active groups in comparison with those in the control group over the longer term.

This suggests that the students were probably learning about the rules of the road both inside and outside of the classroom. This might be explained by the fact that over 80% of the participants in the study intended to start driving at the earliest possible opportunity, and may have been studying the rules of the road as part of their preparations for the driver theory test in the interim between the baseline and post-intervention tests. Furthermore, since almost half of the students had obtained their learner permit by the time that the final test was conducted, we could also speculate that the decline in proficiency seen at that point may be due to decays in memory. An equally plausible explanation, however is that this pattern may have been caused by practice effects. Since there was a much shorter gap between the baseline and post-intervention tests and the gap between the post-intervention and follow-up tests one could expect that the students would be able to recall more information in the post-intervention test than in the follow-up test.

We also found that some between-student differences influenced proficiency scores in the baseline test, proficiency scores for females, urban dwellers and students with high levels of impulsiveness or high exposure to aberrant driving while travelling with their parents or friends were significant lower than those of their counterparts.

7Cognitive Skills -Risk PerceptionVignetteMark is 17 years old and has had a learner permit to drive for 6 months. One Saturday night while his parents are away he decides to use his dads car to take some of his friends to a disco in a nearby town. The disco finishes at 2am and on the way home Mark decides to see how fast the car can go. What are the possible consequences of this?

88 Cognitive Skills Risk perceptionAccurate risk perception is perhaps the most essential cognitive skill needed for safe driving. Among the measures that we used to test risk perception was this vignette. As you can see, this scenario represents the PERFECT STORM in risk taking, combining inexperience, fatigue, the presence of peers and speeding. We asked the students to list all the possible consequences that might arise in this situation, as soon as they sprang to mind. A slightly different version of this story was presented in each test. We did a thematic analysis on the results and collapsed them into 12 categories. 10 categories described negative consequences, one category contained neutral consequences and the final category contained positive consequences. We were interested in the number and types of consequences that the students listed. Then, to measure the accessibility of serious consequences, we analysed the order in which the consequences were listed.

8Vignette - ConsequencesConsequence Baseline Post-interventionFollow-upCrash85.7%88.0%85.3%Death51.3%43.1%50.7%Injury48.8%33.6%38.7%Caught by Gardai38.2%31.8%38.1%Damage to cars/property34.5%23.9%17.1%Loose Control22.9%14.9%14.9%Legal problems21.7%13.9%3.2%Trouble with parents15.6%13.0%17.5%Increased risk/danger11.5%13%9.3%Moral issues5.3%2.3%1.0%Nothing12.9%12.3%9.5%Benefits 4.9%2.6%1.5%99 Vignette ConsequencesThe results showed that on average, students listed 3.5 consequences at baseline, this decreased significantly to 2.9 in both of the remaining tests. There was no significant difference in the number of items mentioned by any of the active groups in comparison to the control group in any of the tests. Students with higher levels of trait impulsiveness mentioned significantly fewer consequences than their counterparts in the baseline test, but not in the other tests.

The percentage of respondents who listed each particular consequence remained relatively consistent across the three tests. Encouragingly, the vast majority of respondents suggested crashing as a possible consequence each time. We can also see that the frequency with which the items were mentioned is roughly in line with their seriousness, which shows that the students WERE generally aware of the physical dangers associated with high-risk driving situations. Notice that a small hard core of students suggested that there might be no consequences and that Mark and his friends would get home safely. A small minority could also see possible benefits in this situation for example, impressing their friends. Surprisingly few people mentioned moral consequences like guilt, shame or regret and this number dwindled over time.

We conducted a series of hierarchical multilevel Poisson regressions to analyze the mean order in which the consequences were listed and the results showed that the first things that came to mind for the students was either crashing or losing control of the car. This was followed in terms immediacy by comments about increased risk and then by serious consequences including death and injury.

Then we analyzed changes in immediacy for serious consequences such as crashing, death and injury over time and as a function of participating in pre-driver education. The results with respect to crashing showed that there was a significant increase in immediacy between the baseline and subsequent tests. On average, students listed crashing 7% sooner at T2 and T3 than they did in the initial test. Students who took pre-driver education mentioned crashing 15% sooner than the controls in the post-intervention test. We also found that all of the active groups mentioned crashing sooner than the controls in the post-intervention test, although this difference was only significant for groups B and C (16% - 17%) and these effects were no longer evident in the final test.

910Prototypical Images of Drivers who Speed Risky Impatient Selfish Immature Unrealistic 10 Prototypical images of drivers who speed

Speeding is a significant risk factor for all drivers. We measured the student attitudes to drivers who speed using five negative and five positive adjectives which were suggested during a pilot study.The negative adjectives were Impatient, Immature, Risky, Selfish and Unrealistic and responses were measured on a 5-point scale, where high scores indicated stronger agreement with these descriptions. As we can see from this graph, the students took a dim view of drivers who speed, because all scores were well above the mid-point on the scale. The tendency to attribute negative characteristics to speeders increased significantly between the baseline and post-intervention tests and then decreased significantly in the final test. There was no significant difference between student assessments in any of the active groups compared to the control group. However, we did find that girls took a more significantly more negative view of speeders than boys, and students with high trait impulsiveness took a less negative view than those with average to low impulsiveness.

1011Prototypical Images of Drivers who Speed Cool Popular Responsible Smart Skilful11 Prototypical Images of Drivers who speedThe images used for the positive scale were Cool, Popular, Responsible, Smart and Skilful. Here we see that all of the scores remained below the mid-point on the scale, indicating that the students did not have an overly positive opinion of drivers who speed. In fact there was very little variation in the scores for this scale either across time or between any of the experimental groups.

However we did find that girls held a significantly less positive view of speeders than boys, and students with high trait impulsiveness were more likely to see speeders in these terms than their counterparts.

11Students ViewsEnjoyable aspects of DE coursesClassroom-based groups DVDSingle session groups Practicals & LecturesOnly 3% didnt enjoy the courseBeneficial aspectsLearning the Rules of the RoadSuggested ImprovementsInclude / extend practical elementsAdd Rules of the RoadCourse is satisfactory

1212 Students ViewsAs part of the formative analysis, we asked the students to rate their courses using Likert-scale items and some open response questions. Around 35% of the students responded to the open questions, telling us what liked about their course, what they found beneficial and also suggesting ways that the courses might be improved. Students who took modular programmes mostly enjoyed watching DVDs about road safety, because they were entertaining and increased awareness of dangers of driving.Those who took single session interventions really liked the practical sessions and the lectures because they learned a lot and they were interesting. Just 3% of those who responded said they didnt enjoy their course, however it is possible that those who didnt enjoy their course didnt respond to the questions in this section at all. In general the students felt that they benefited most from learning the Rules of the Road, because this prepared them to take the drive theory test.Although we received a range of suggestions for ways to improve the courses, roughly 20% of respondents who took the modular programmes were satisfied with the courses as they stood. A further 40% wanted practical elements to be added to the course, or where this was already in place they wanted to see it extended. Students from classes that didnt study the Rules of the Road wanted this element to be added in the future.It is clear from our results that students wanted to achieve some tangible goal directed outcomes from taking a pre-driver education course over and above any gains in insight that they may have achieved. They wanted to have access to cars and to be helped to prepare for their driver theory test and given that over 80% of them intended getting a licence as soon as they possibly could, these views are quite understandable.

12Summary and conclusionsPre-driving adolescents have some knowledge, risk perception skills and attitudes towards speedingPre-driver education has a small short-term effect on knowledge, risk perception skills and attitudesConsistent effect of impulsivity on baseline scoresStudents and teachers believe that the courses are beneficialStudents want more practical contentStudy limitationsSelf-reportStudy underpowered More groups needed for HLM

13Summary and conclusionsThe results presented here represent just some of the tests that were conducted during this study and what we generally found wasthatStudents were quite knowledgeable about the rules of the road and also about factors that increase risk for drivers before they took their pre-driver education courses. This in in line with previous findings.They also displayed quite healthy attitudes towards drivers who speedAlthough we found some evidence suggesting that taking pre-driver education helped to improve knowledge, risk perception skills and attitudes in some cases, these effects were generally small in magnitude and didnt persist over the longer term.Impulsiveness was the most consistent predictor of knowledge, skills and attitudes in the baseline tests, but not in subsequent tests. We also found that both students and teachers were generally happy with the courses that were delivered in their schools although some students believed that they would benefit more if more emphasis was placed on learning the rules of the road and the practical aspects of learning to drive.The study was limited to the extent that it relied on self-report data. However, research in driver behaviour shows that self-reports provide a reasonably accurate estimates of knowledge, skills and habits.Although our large sample size was more than adequate to find significant effects using normal regression techniques, using hierarchical multilevel modelling reduced our experimental power quite considerably.

13Thanks for your attention!14

Any questions? 14ReferencesArnett, J. (1994). Sensation seeking: A new conceptualization and a new scale. Personality and Individual Differences, 16(2), 289-296. Clinton, K. M., & Lonero, L. (2006). Evaluating Driver Education Programs (pp. 335). Washington DC: AAA Foundation for Traffic Safety.Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., et al. (2006). The International Personality Item Pool and the future of public-domain personality measures. Journal of Research in Personality(40), 84-96. Reason, J. T., Manstead, A. S. R., Stradling, S. G., Baxter, J. S., & Campbell, K. (1990). Errors and violations on the road: a real distinction? Erognomics(33), 1615-1332. Spinella, M. (2007). Normative data and a short form of the Barratt Impulsiveness Scale. Intern. Journal Neuroscience(117), 359-368. 15

15