607 14th Street, NW Suite 201

Washington, DC 20005 800-993-7222

www.aaafoundation.org

October 2006

Prepared for

E v a l u a t i n g D r i v e r E d u c a t i o n P r o g r a m s

�������� ����

Prepared by

Kathryn M. Clinton Larry Lonero

Northport Associates 182 Bagot Street Cobourg, Ontario Canada K9A 3G2 905 - 377- 8883 www.northportassociates.com

C o m p r e h e n s i v e G u i d e l i n e s

2 E v a l u a t i n g D r i v e r E d u c a t i o n P r o g r a m s : C o m p r e h e n s i v e G u i d e l i n e s

This report was funded by the AAA Foundation for Traffic Safety in Washington, D.C. Founded in 1947, the AAA Foundation is a not-for-profit, publicly supported charitable research and education organization dedicated to saving lives by preventing traffic crashes and reducing injuries when crashes occur. Funding for this report was provided by voluntary contributions from AAA/CAA and their affiliated motor clubs, from individual members, from AAA-affiliated insurance companies, as well as from other organizations or sources.

This publication is distributed by the AAA Foundation for Traffic Safety at no charge, as a public service. It may not be resold or used for commercial purposes without the explicit permission of the Foundation. It may, however, be copied in whole or in part and distributed for free via any medium, provided the AAA Foundation is given appropriate credit as the source of the material.

The opinions, findings, conclusions, and recommendations expressed in this publication are those of the authors and are not necessarily those of the AAA Foundation for Traffic Safety, any individuals who peer-reviewed this report, or advisory group members. The AAA Foundation for Traffic Safety assumes no liability for the use or misuse of any information, opinions, findings, conclusions, or recommendations contained in this report.

If trade or manufacturer’s names are mentioned, it is only because they are considered essential to the object of this report and their mention should not be construed as an endorsement. The AAA Foundation for Traffic Safety does not endorse products or manufacturers.

©2006, AAA Foundation for Traffic Safety

(continued)

3E v a l u a t i n g D r i v e r E d u c a t i o n P r o g r a m s : C o m p r e h e n s i v e G u i d e l i n e s

Contents

Glossary of Acronyms ➣ 5

List of Tables and Figures ➣ 7

Acknowledgements ➣ 9

Preface ➣ 11

Executive Summary ➣ 13

1. Introduction ➣ 23 Overview of the Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 How to Read the Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Target Audiences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Driver Education Programs in the U.S. and Canada . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Approaches to Driver Education Program Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2. Evaluation Model and Framework ➣ 31 What is Program Evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Why is Evaluation Important? Lessons from Related Fields . . . . . . . . . . . . . . . . . . . . . . . 33 What is Good Evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Depicting Programs through Logic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Developing an Evaluation Model and Framework for Driver Education . . . . . . . . . . . . . . . . . 43 Driver Education Evaluation Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Integrated Driver Education Evaluation Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3. Driver Education Evaluation Guidelines ➣ 55 Applying the Framework—How to Use Evaluation Effectively . . . . . . . . . . . . . . . . . . . . . . 55 The Evaluation Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Guiding Questions for Driver Education Program Evaluation . . . . . . . . . . . . . . . . . . . . . . . 56 Steps for Evaluating Driver Education Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 STEP 1: Focus the Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 STEP 2: Select Evaluation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 STEP 3: Develop Data Collection Plan and Select Data Collection Tools . . . . . . . . . . . . . . . .101 STEP 4: Gather, Analyze, and Summarize Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 STEP 5: Interpret and Act Upon the Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . .129

4. Basic Driver Education Program Evaluation Scenario ➣ 137 Program Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .138 Evaluation Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .140 Evaluation Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145 Evaluation Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

(Table of Contents—continued)

4 E v a l u a t i n g D r i v e r E d u c a t i o n P r o g r a m s : C o m p r e h e n s i v e G u i d e l i n e s

5. Advanced Driver Education Program Evaluation Scenario ➣ 151 Program Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .153 Evaluation Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .153 Evaluation Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .157 Evaluation Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .164

6. Conclusion ➣ 165

References and Evaluation Resources ➣ 167 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Evaluation Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

APPENDIX A: Review of the Literature: Driver Education Evaluation—Past and Present . . . . . 179

APPENDIX B: Glossary of Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .213

APPENDIX C: Program Evaluation Standards Definitions . . . . . . . . . . . . . . . . . . . . . . . . . .231

APPENDIX D: Overview of Evaluation Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .235

APPENDIX E: Worksheets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .239

APPENDIX F: Data Collection Tools: Basic Evaluation Scenario . . . . . . . . . . . . . . . . . . .251

APPENDIX G: Data Collection Tools: Advanced Evaluation Scenario . . . . . . . . . . . . . . . . 281

APPENDIX H: Attitude and Risk-Taking Questionnaires . . . . . . . . . . . . . . . . . . . . . . . . 313

APPENDIX I: Benchmark Program Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317

APPENDIX J: Hiring an Evaluator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .329

5E v a l u a t i n g D r i v e r E d u c a t i o n P r o g r a m s : C o m p r e h e n s i v e G u i d e l i n e s

GLOSSARY OF ACRONYMS

AAA Formerly American Automobile Association

AAAFTS AAA Foundation for Traffic Safety

ADTSEA American Driver and Traffic Safety Education Association

CAA Canadian Automobile Association

CIPP Context, Input, Process, and Product Evaluation

CPM Critical Path Method

DSAA Driving School Association of the Americas

GDL Graduated Driver Licensing Progr