1

Outcomes of Social Work Education (OSWE):

Findings from the ProjectsHilary Burgess, John Carpenter, Joanna Fox, Roxana Anghel, Sharon Vitali, Juliet Koprowska, Kish Bhatti-Sinclair, Anne Quinney, Imogen Taylor, Clare Ockwell, Suzy Braye, Michelle Lefevre

2

Outline of Session

1. Introduction to OSWE project2. Presentation of findings from six partners in

England with questions for clarification.3. Discussion of the findings in general and their

implications.4. Key issues in doing outcomes-focussed

research on Social Work Education.5. Open discussion.

3

Aims of OSWE (3 year project)

To test the feasibility of outcome measures and research designs in Social Work Education

To generate high quality evidence about the effectiveness of methods of SWE

To build capacity and capability amongst academics, including service users

To use opportunities to compare and contrast practice between programmes

4

A Collaborative Capacity & Capability building model

Peer learning through Action Learning Set Support, advice and mentoring, F2F, e-mail and by

phone (e.g. data analysis).

See Burgess, Hilary and Carpenter, John (2008) 'Building Capacity and Capability for Evaluating the Outcomes of Social Work Education (the OSWE Project): Creating a Culture Change', Social Work Education.

5

Measuring Outcomes

Measuring changes over time Before-after designs Using and adapting measures

6

Primary levels of Outcomes (after Carpenter, 2005)

Levels of Outcome 1.Learners’ reactions2.Modifications in attitudes and

perceptions Attitudes Motivational

3.Acquisition of knowledge and skills

Procedural Strategic knowledge Initial skills Compilation skills

4. Changes in Behaviour5. Benefits to users and carers

Focus

Attitudes to race and racism (Soton)Attitudes interprofessional practice

(Sussex)

Understanding partnership with SU&C (Anglia Ruskin)

Communication skills with children (Sussex)

Interviewing and communication skills (York)

Use of research skills (Bournemouth)Acquisition of Soc. Wk. competences

(Oxford Brookes)

7

The outcomes of teaching and learning about ‘race’ and racism

Kish Bhatti-SinclairDivision of Social Work StudiesSchool of Social SciencesUniversity of Southampton

8

Aims and Methods

Students’ understanding and experience of ‘race’ and racism; and

How ‘race’ and racism is addressed in the degree curriculum

Questionnaires to BSc social science students in week 2 of Semester 1 (153 students) and in Week 8 (71 students) of Semester 2.

34 respondents completed both sets. Social work students = 41.2%.

9

Main sources of learningabout race and racism (frequency)

TYPES OF LEARNING Seminars 17 Course units 16 Personal study 9 Shadowing soc wkr 4 Peer Group 3 Personal tutors 2

SPECIFIC COURSE UNITS 2 out of total of 8 (3 SW)

Social problems and social policy - 6

Sociology of everyday life - 2

10

Results

A small increase in self-rated knowledge of racism – 10%.

A increase in respondents’ confidence in challenging racism (but not statistically significant.)

Overall results suggest that formal teaching about ‘race’ and racism did influence knowledge although the impact was small.

11

Limitations

Sampling – not random. Results not generalisable – must be

interpreted with caution. Knowledge of racism based on self rating - not

possible to obtain an objective measure of knowledge.

12

Outcomes of “Using Research for Practice” using blended learning.

Bournemouth UniversityAnne Quinney

BA Social Work Year 2 students.

13

Methodology

Pre-Post testsMeasure: Research Self Efficacy (RSE) scale

(Holden et al., 1999). Subscales on research knowledge and skills. PLUS 5 items “using computer and information technology”.

Analysis: Paired t-tests for difference in mean scores.

14

Findings: Research Knowledge and Skills

“How confident are you that you can successfully analyse basic quantitative and qualitative data?”

Scale 0-10 0

1

2

3

4

5

6

Question9

T1

T2

15

Findings: using computer and information technology

“How confident are you that you can successfully access research findings from research bodies, social work organisations, government departments etc (e.g. JRF, SCIE) using the internet?”

0

1

2

3

4

5

6

7

8

Question14

T1

T2

16

Next Steps and Uses

1. Do Research Self Efficacy scores and assignment marks correlate?

2. Compare with another programme

Uses:– Adjust the curriculum for (1) present students who have to do a

dissertation in year 3 and (2) next year’s course. – Student self-assessment: “what I need to learn.”

– The RSE scale has been adopted by ESRC Researcher Development Initiative.

17

Outcomes of teaching about Partnership & Interprofessional Practice

Imogen Taylor, Clare Ockwell, Suzy Braye

University of Sussex

Note: Class-room based module does not include students from other professions.

18

Methodology

Outcomes of learning and teaching on BA & MA students’ attitudes – Stage 1 pre module teaching– Stage 2 post module teaching – [Stage 3 end of course]

Validated scales from University of West of England IPE programme

– Communication and teamwork (CT)– Interprofessional learning (ILL)– Interprofessional interaction (II)– Interprofessional relationships (IR)

19

Mean ratings at start/end of module (BA)

BA - Comparison of scores for each scale at Time 1 and Time 2

0

5

10

15

20

25

30

35

40

CT1 CT2 IL 1 IL 2 II 1 II 2 IR 1 IR 2

Scales and time point

Scor

e

20

Differences in attitudes

by years of pre-course experience

BA ratings for Communication and Teamwork Scale and number of years of experience

0

1

2

3

4

5

6

7

8

9

10

pos neutral neg pos neutral

T1 CTS rating T2 CTS rating

No. o

f res

pond

ents

MA ratings for Comunication and Teamwork Scale and years of experience

0

1

2

3

4

5

6

7

T1 CTS rating T2 CTS rating

No. o

f res

pond

ents

0-1 yr

2-5 yrs

5 yrs ormore

21

Next steps

Complete analysis of BA/MA T1 and T2 Collect and analyse end programme BA/MA

data (T3) Compare with UWE IPE findings Introduce new comparator programme (any

offers?)

22

Outcomes of Teaching and Learning communication skills

Juliet Koprowska

University of York

23

Research question & methodology

Are communication skills improved through learning within the university?

Methodology: multiple-measure case study design.

Sample: year 1 BSc Social Work

24

Research methods

Self-efficacy scale Recorded interview with service user-actor Self-evaluation of interview Service user-actor evaluation Assessment of video-recording Feedback from service users on placement

T1 = prior to teaching, T2 = 3 months later, after teaching, T3 = 9 months later, after first placement

25

Student A

0

2

4

6

8

10

1 2 3

Time

Self-efficacy Self-evaluationService user-actor evaluation Assessment

26

Student B

0

2

4

6

8

10

12

1 2 3

Time

Self-efficacy Self-evaluation

Service ucer-actor evaluation Assessment

27

Implications

Students may lose confidence as they acquire knowledge and understanding

Potential for formative assessment Effect of placement needs exploration Qualitative discussion with students and actors would

deepen understanding Should tools become regular measures to increase

participation in research, with retrospective permission?

28

Communication Skills with Children & Young People

Michelle LeFevreUniversity of Sussex

How does a Programme contribute to students’ development of confidence and competence in communicating with children and young people?

29

What and How are these learned?

‘Skilled/effective communication’ is not just techniques and micro skills. Also underpinning knowledge, values and ethical commitments, personal qualities and emotional capabilities.

Learned throughout the whole programme not just in focused skills teaching – and also through other personal and professional experiences.

30

MethodologyProspective Evaluation – 4 stages

1. Questionnaire – Students’ personal characteristics: Have particular kinds of student

learned most/least?– Measuring self-confidence in communication with children at

different stages. – Which aspects of the programme facilitated your confidence and

skills?

2. Case Vignette tool– Students demonstrate knowledge of planning, implementing,

reviewing and reflecting on communication with children.– More objective measure (though analysis is subjective?)

31

Key Findings

1. Students’ confidence has increased.2. Direct practice in placement the most important,

followed by Child Development 3. All aspects of focused skills teaching were

perceived as helpful to learning4. Knowledge of aspects of communication skills

increased (often significantly) - but not in every domain.

5. Still to analyse relation of characteristics to learning and mapping individual students through programme.

32

Learning about working in partnership with service users

Roxana Anghel & Joanna Fox

Anglia Ruskin University

Exploring the use of Concept Mapping

33

“Schematic representation of an individual’s understanding of a knowledge domain in the form of concepts meaningfully linked in propositions.”

Design – CM and questionnaire at T1 (induction week BA); T2 (end year

1); T3 (end year 2)– CM task, unstructured - ‘Working in Partnership with Service

Users’– Questionnaire – most impacting learning opportunities.

Analysis of CM – Generate scores based on validated concept-links– Raters included service user and researcher.

Method – Concept Mapping (CM)

34

Concept Map at start of programme

35

Concept map at end of Year 1

36

Concept Map at end Year 2

37

Findings – “Working in Partnership”

0

20

40

60

80

100

120

140

160

T1 T2 T3

1

2

34

5

6

Map

Sco

res

38

Sources of Learning

At T1: Ethics and Values; observational practice; Poverty, Social Exclusion and Social Work, and academic reading.

At T2: Social Work with Children and Families, Social Work with Adults, practice placement; Principles and Skills of Social Work; academic reading; and discussing with colleagues

39

CM used on two other modules

Relationship of Map Scores to Assignment Marks

30

35

40

45

50

55

60

65

70

75

30 40 50 60 70 80 90 100

Marks

Map

Sco

res

Relationship of Map Scores to Assignment Marks

40

45

50

55

60

65

70

75

40 50 60 70 80 90 100 110

Marks

Map

Sco

res

• Module B – Map scores and marks did not correlate

020406080

100120140160180

1 2 3 4 5 6 7 8 9 10 11

T1

T2

• Module A – Map scores correlated with marks

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10 11 12 13

T1

T2

• Module B (Year 1 MA, 11 pairs) – 72% T2 maps increased content and quality

• Module A (Year 3 BA, 13 pairs) – 46% T2 maps increased content and quality

Students

Ma

p S

co

res

Students

Ma

p S

co

res

40

Discussion

– Small numbers, so the value of CM as summative method in assessing outcomes needs to be explored further

– Success depends on the clarity of instructions and purpose, familiarity and time available.

– Uses: self-assessment, visual learning aid, formative assessment (identify gaps), as well as assessing outcomes.

– Being firmly based on “meaningful” learning CM can be applied to

many area of social work.

41

Measuring the Acquisition of Competency

Sharon Vitali

Oxford Brookes University

42

Methodology

Cohort Sample BSc Social Work– (n=38 at beginning n=34 at present = 89%)

Longitudinal - Baseline to Graduation Repeated Measures Fixed Outcomes (National

Occupational Standards Units) RICET Multiple Methods

43

Year One - Semester One

Baseline

December

RICET

RICETPERSONAL LEARNING PLAN

44

Consider your current state of competence (knowledge/skills) and indicate a score (from the guide below) for each of the learning outcomes listed.

0 = Can not produce any evidence of competence.

1 = Understands the learning outcome, but can produce only limited or no evidence of appropriate attempts to put it into practice. Much more knowledge/practice needed.

2 = Understands, and can offer evidence of tentative attempts to integrate into current knowledge/skill base.

3 = Demonstrates competence with some regularity.

4 = Advanced understanding and demonstrating adequate level of integration of knowledge, skills, and appropriate application.

5 = Clearly understands and demonstrates consistent and appropriate application of knowledge and skills in practice.

45

Acquisition of NOS Competencies Year One

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Unit 1

Unit 2

Unit 3

Unit 4

Unit 5

Unit 6

Unit 7

Unit 8

Unit 9

Unit 10

Unit 11

Unit 12

Unit 13

Unit 14

Unit 15

Unit 16

Unit 17

Unit 18

Unit 19

Unit 20

Unit 21

Units of Competency

Sco

re

Student Baseline Student End Sem1Tutor End Sem1 Student End Sem2

Academic Average = 61.4%

Video Scores SU=1.8 ER=2.2

46

Year Three - Semester Two

May (70 DAY PLO)

RICETPRACTICE & PORTFOLIO

TUTOR EVALUATION

PRACTICE & PORTFOLIOP/T EVALUATION

CRITICAL CAREERREVIEWTUTOR

EVALUATION RICET

47

Acquisition of NOS Competency Unit 4: Respond in a Crisis Situation

2

0333

43

43

4

1

5

1

Level of Competency (0-5)

PA End Place2

ST End Place2

PA Mid Place2

ST Mid Place 2

PA End Place1

ST End Place1

PA Mid Place1

ST Mid Place1

ST End Yr1 S2

Tutor End Yr1S1

ST End Yr1 S1

ST Baseline

48

Types of DataY1-S1

RICET

numeric

LEARNING PLAN

p/f

RICET

numeric

Grades

numeric

Y1-S2

INTERVIEW

numeric

RICET

numeric

PORTFOLIO

p/f

INTERVIEW

numericGrades

numeric

Y2-S1

RICET

numeric

RICET & PORTFOLIO

Numeric & p/f

PORTFOLIO

p/fGrades

numeric

Y2-S2

RICET

numeric

RICET & PORTFOLIO

Numeric & p/f

PORTFOLIO

numericGrades

numeric

Y3-S1

RICET

numeric

RICET & PORTFOLIO

Numeric & p/f

PORTFOLIO

p/f

CASE VIVA

numericGrades

numeric

Y3-S2

RICET

numeric

RICET & PORTFOLIO

Numeric & p/f

PORTFOLIO

numeric

CRITICAL CAREER REVIEW

numeric

Grades

numericNOW

end

49

Overview of the findings from projects increases in:

1. Attitudes to race and racism - but not to self-confidence in challenging racism.

2. Self-efficacy following learning (e.g. research skills) Do these correlate with marks?

3. Conceptual understanding (C Maps, communication vignettes).

4. Ratings of Competence (RICET)

50

But:

1. No measured increase in Communication skills

2. No overall increases in attitudes to partnership working

3. Measures do not necessarily correlate with marks.

And we need to strengthen research designs, especially comparative studies.

51

Some Questions for Discussion

1. What methods of learning and teaching should we compare? Are you interested?

2. What is their potential for use in formative evaluation and self-evaluation by students?

3. Should we use these measures routinely (like Oxford Brooks) to assess student outcomes?

4. Are you interested in replicating and developing these methods?

52

Key issues

1. Formulating an appropriate, specific research question2. Selecting/adapting appropriate methods/measures for

measuring change3. Negotiating student engagement, balancing ethical

concern to allow opt-out with maximising engagement to promote learning

4. Establishing involvement of service users/carers5. Getting support from colleagues6. Freeing space/time/resources for the research

53

Key issues (cont)

7. Balancing roles of teacher and researcher (sometimes complementary, sometimes conflicting)

8. Accessing expertise/learning to analyse and interpret data

9. Finding comparison sites

10. Feedback to programme/colleagues about knowledge/skills gained and implications for rest of programme

54

Thank you

To our sponsors: SCIE, SWAP and IRISS

Contacts: j.s.w.carpenter@bristol.ac.ukh.c.burgess@bristol.ac.uk

Resource:Evaluating Outcomes in Social Work Educationhttp://www.iriss.ac.uk/node/88