Professionalization of evaluation –recent activities within DeGEval

Dr. Jan Ulrich HenseDepartment Psychologie, Empirische Pädagogik und Pädagogische Psychologiejan.hense@psy.lmu.de

Jahreskongress SEVAL 2012, Workshop 2 Freitag, 7. September 2012Université de Fribourg

# 2Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

(1) Some comments and questions on the

professionalization of evaluation

# 3Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Two uses of „professional“

1. Referring to a job or designation (“a professional painter”)• Is evaluation developing into a full-blown profession?

2. Referring to performance on a task (“a professional repair”)• How to carry out evaluations professionally?

# 4Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

(Mieg, 2010, p. 16)

Evaluators?

# 5Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Two Classes of Professions (Mieg, 2010)

1. Care and relationship orientation Clients or patients trust professionals’ expertise and integrity

Readiness to work cost free in emergency situation

e.g. Doctors, Lawyers, Psychotherapists

2. Market orientation Price competition

Income orientation

e.g. Engineers, Consultants

# 6Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Professionalization as regulation of autonomy

Definition of education and training demands Control of market entry Determination of fees and charges Monopoly of performance evaluation

(Mieg, 2010)

# 7Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Conditions for the development of a new profession

1. Combination of a problem relevant for society and

knowledge on how to explain and deal with the problem

2. Relation to central values of society (e.g. health, justice)

3. (Mostly) academic education and training4. Professional societies

(Mieg, 2003)

# 8Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Phases of development of a profession (Wilensky, 1964)

1. Some are working in a new area (mostly) full time

2. Education and training programs are developed

3. Professional societies emerge

Shaping profile by delegating “lower” tasks

Conflict between established and graduates of new training programs

Competition with neighboring professions

4. Professional Codes und Standards

5. Move to formal closure

Rules and conditions for working in the new profession

Protection of designation

Declaring non licensed work illegal

# 9Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

(2) Recent DeGEval activities in the context of

professionalisation

1. Competencies for evaluators2. Didactics of Evaluation3. Further projects and plans

# 10Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

1. Competencies for evaluators

# 11Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Why develop "Recommendations on Education and Training in Evaluation"?

Growing demand for evaluation services

Lack of qualified personnel Lack of a professional profile

Guidelines as contribution to the professionalization of evaluation

# 12Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

A bit of history...

2004: First German edition• Authors: A. Caspari, M. Hennen,

D. Scheffler, U. Schmidt & O. Schwab

• Published by the DeGEVal

working group "Education

and Training in Evaluation“

2008: English translation

download/order on www.degeval.de

# 13Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Goals and intended uses

Standards for• Training programmes

– Direction for programme developers and teachers– Evaluation of existing programmes

• Assessing evaluators– Choice of an evaluator by clients– Self-evaluation of evaluators

# 14Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

"A competency is more than just knowledge and skills. It involves the ability to meet complex demands, by drawing on and mobilising psychosocial resources (including skills and attitudes) in a particular context."

(OECD/OCDE DeSeCo-Project)

Which knowledge, skills and attitudes are needed for conducting professional evaluations?

# 15Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

# 21Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Development of the recommendations

Theoretical approach Based on DeGEval standards of evaluation

Practical procedure Expert panel Feedback cycles from DeGEval members Officially adopted by DeGEval conference

# 22Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Who uses the guidelines?

Usage in advanced study programmes• Berne

• Saarbrücken

• Bonn

Further education courses ? Evaluation courses within

domain-specific curricula ?

# 23Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Soellner, R., Hapkemeyer, J. & Scheibner, N. (2010). Evaluationsausbildung in der Psychologie – eine Bestandsaufnahme. Zeitschrift für Evaluation, 9 (2), 297-308.

Coverage of competencies in psychology diploma programmes

# 24Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Need for revision (to be discussed)

Concept of competencies (KSA) Field-specific interpretation and differentiation Elaboration of methodological competencies Compatibility with international models of

evaluator competencies, e.g.• King et al. (2001; 2005); Mertens (1994)

• Zorzi, McGuire & Perrin (2002)

• Russ-Eft et al. (2008)

# 25Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

2. Didactics of Evaluation

# 26Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

htt

p:/

/ww

w.d

egeval.d

e/a

rbeit

skre

ise/

au

s-u

nd-w

eit

erb

ildung

-in-d

er-

evalu

ati

on/

pub

likati

onen-u

nd-p

rodukt

e

# 27Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Why didactics of Evaluation?

Competency profiles• Contents and learning goals of training programmes

• What to teach

Didactics of evaluation• Practical guidelines for teachers and trainers

• How to teach

# 28Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Goals

Provide guidance for• training and education programmes

• courses in evaluation

• self-learning materials (textbooks etc.)

Based on current state of theories on learning and instruction

Supplement existing recommendations on competency profiles

# 29Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

0 Executive Summary1. Introduction2. Learning contents and goals:

Competence profiles for evaluators

3. Programmes4. Types of programmes and target

groups5. Didactical foundation6. Learning and teaching methods7. Requirements for teachers/trainers8. Outlook

Condensed“Cookbook” guidelines

# 30Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Important Messages

Cover whole range of competencies (KSA) Apply a mix of construction and instruction

• Emphasize active, self-directed learning

• Allow for learners’ own practical experiences

• Provide instructional support

• Embed all contents in authentic evaluation contexts

Match method and learning goals Adapt to learners‘ individual needs and

preconditions

# 31Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Examples

Chocolate cookie exercise• Exercise on the logic of evaluation

Reflection exercise „An evaluator is like a ...“• Discuss different evaluator roles and responsibilities

Role play „stakeholder resistance“• Awareness for possible sources of resistance to evaluation

# 32Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

3. Further projects and plans

# 33Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan HenseFuture projects

Other topics and activities of the DeGEval working group “Education and Training in Evaluation“

Overview of training courses and seminars (online-database)

Textbook overview and review Evaluation capacity building Certification and credentialing of evaluators

# 34Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan HenseFuture projects

Overall DeGEval efforts

Work group on ethics in evaluation (“ethical codes”)

Revision of evaluation standards (until 2014)

# 35Qualitätssicherung in der Evaluation (SEVAL 2012, Universität Fribourg)

Dr. Jan Hense

Thank you for your interest!

contact:jan.hense@psy.lmu.de