1

Using statistical process control to compare reconviction rates across local authorities

Presentation to Statistician Group

Scottish Government

Ian Morton

21 November 2011

2

Contents

Brief history of statistical process control (SPC) – (4 slides)

Summary of previous discussions on SPC (1 slide)

Background in telecommunications (6 slides) Reconviction rates (8 slides)

Our stakeholders Local authority (LA) level reconviction rates Our use of SPC in considering LA reconviction rates Causes of a difference in process Difference in interventions and/or strategies

Summary Conclusions References

3

Brief history of the inventors of SPC

SPC developed by Walter Shewhart in the 1920s and 1930s

W. Edwards Deming met Shewhart while on a scholarship at Yale University in the 1920s

Deming was a key player in taking SPC forward

Prior to 1920s, little SPC

Ford assembly line (1913).

Photo: (Author unknown) via Wikimedia Commons.

A typical assembly line of the period

4

Deming and quality improvement

Timeline 1920s – quality improvement became important 1939 – Deming teaches quality in USA 1939-1945 Second world war 1945 – USA forgets about quality 1945 – Japan suffering 1950s – Deming teaches SPC to engineers in

Japan Late 1950s – Japans quality of goods improves

(Toyota, Honda, Nissan) 1970s – low quality of USA goods (Ford,

Chrysler) 1980 – Deming teaches SPC to engineers in

USA Mid 1980s - quality of USA goods improves

(Harley Davidson, Intel, Dupont)

Final assembly (at the Lotus 60th Celebration - 14 September 2008).

Photo: Snelson on flickr, via Wikimedia Commons.

A recent assembly line

5

Demings key points on SPC

It’s a method used to improve quality Quality lowers costs and improves productivity Better quality means less defects

(fewer problems, volume goes up) Work to lessen variation

lessen variation means improve system and improve results Special cause (or assignable cause) – due to fleeting

events, and not due to chance Common causes are due to chance and are inherent

in the system

6

Recent use of SPC

Companies such as Motorola have adopted Six Sigma process quality Limit defects to a specified design tolerance Process improvement experts (black belts, green belts)

You shouldn’t rank institutions without accounting for uncertainty Royal Statistical Society (2003) Performance Indicators: Good,

Bad, and Ugly Royal Statistical Society Working Party on Performance Monitoring in the Public Services. Obtained from http://membership.rss.org.uk/pdf/PerformanceMonitoringReport.pdf

Spiegelhalter, D. J. (2005) Funnel plots for comparing institutional performance Statistics in Medicine 24 1185-1202.

Funnel plots – a snapshot in time which looks at SPC

7

Summary of previous SPC discussion Morning session – 14th July 2010

Presentations Jim Mather MSP An Introduction to SPC (Rob Wishart) Principles and methods of SPC (Chris Newson) ISD support for the Scottish Patient Safety Programme and

hospital mortality (Roger Black from ISD) Deaths on Scotland’s roads (Glen Deakin)

How have I taken SPC forward Which local authorities have an unexpected reconviction

rate ? (Reminder: lessen variation and you improve quality) Why there is variation ?

8

Telecommunications: an example from my past

9

Making telephones that work I worked in a factory that made telecommunications equipment

A specific department made telephones Consisted of a shop floor – it was a frantic environment There were n assembly lines of several workers, each with specific roles Some did electrical work Some did mechanical work Some soldered, some screwed parts together, some snapped parts on Bins of parts had to be continually replenished

There were complications Different assembly lines had different break times Some phones failed tests – there was a tester at the end of each line Staff had holidays, or were ill Friday – many of them went to the pub

Decisions were made by management They kept a record of pass/fail for each assembly line They decided on an acceptable pass rate Also decided on the number of good phones that should be made

10

Table of pass rates(fictitious example)

Some facts on each of the lines: 10 – key staff ill, temporary staff 15 – had to use old equipment 16 – rowdy and liked a drink

Foreman's impression: 11 – this is my best line! 18 – good quality but slow

But which of these assembly lines have a significantly higher/lower pass rate given the number of phones they processed ?

Assembly line

Number of phones that pass in the

month

Total number of phones in the month

Pass rate

1 1,630 2,336 70%2 2,173 3,164 69%3 1,890 2,787 68%4 778 1,040 75%5 3,829 5,534 69%6 3,827 5,818 66%7 1,252 2,156 58%8 2,981 5,117 58%9 650 1,029 63%

10 687 1,176 58%11 2,551 3,502 73%12 1,234 1,800 69%13 1,597 2,496 64%14 2,025 3,036 67%15 904 1,504 60%16 1,847 3,120 59%17 1,573 2,492 63%18 947 1,316 72%19 1,332 1,864 71%20 1,126 1,718 66%21 1,080 1,597 68%22 3,190 4,638 69%23 1,182 1,832 65%24 1,412 2,098 67%25 680 1,061 64%

Accepted pass rate

42,377 64,231 66.0%

11

Pass rate versus number of phones

This doesn’t indicate which assembly lines have an unexpected pass rate

Reminder: lessen variation tends to improve quality

11

1819

15

22 212

32124

14

620

2325 13

9 17

15

16

10 8758%

60%

62%

64%

66%

68%

70%

72%

74%

0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000

Number of phones in a month

Pas

s ra

te

Assembly Line Accepted pass rate

12

Theoretical line of variability and the 95% confidence interval (CI) 95% CI = Pass rate ± t(n;0.95)×SE, where SE= standard error

Produces two points for every value n (the number of phones) Plot on the y-axis the 95% CI for every value of n an integer

from 1 to 4000, versus n on the x-axis.

0.58

0.60

0.62

0.64

0.66

0.68

0.70

0.72

0.74

0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000

Number of phones in a month

Pas

s ra

te

Accepted pass rate -95% +95%

Upper 95% confidence interval

Lower 95% confidence interval

13

Indicating which assembly lines have a significant variability in pass rate Superimpose the previous two charts Foreman can see that assembly lines 10, 15 and 16 are “out of control” They are the result of special causes:

Require further explanation variation needs to be lessened to bring them back in control

Common cause (those within the funnel) can’t be completely eliminated They are inherent in the system

11

1819

15

22 212

32124

14

620

2325 13

9 17

15

16

10 8758%

60%

62%

64%

66%

68%

70%

72%

74%

0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000

Number of phones in a month

Pas

s ra

te

Assembly Line Accepted pass rate -95% +95%

Upper 95% confidence interval

Lower 95% confidence interval

14

Looking at this another way – other considerations

Funnel plot (snapshot in time)

Line 15 stops using an old screwdriver Quality improved Pass rate increased Volume went up

Assembly line 10 – key staff became ill

Run chart

750

760

770

780

790

800

810

820

1 2 3 4 5 6 7 8

Month

Nu

mb

er o

f p

ho

nes

0%

10%

20%

30%

40%

50%

60%

767 779 791 803 815 847

Number of phones

Rel

ativ

e fr

equ

ency

15

Using SPC on local authority reconviction rates

16

Why do our stakeholders have an interest?

Stakeholders: 32 local authorities (LAs) and 8 community justice authorities (CJAs)

For strategic planning - CJAs plan and target resources effectively.

For performance management - to demonstrate a reduction in reoffending by 2% in 2011 – it’s a National Indicator.

Twitter link http://twitter.com/#!/RRPScotland Breakdown to LA level allows more accurate

feedback on performance to partners.

17

One year reconviction rates for the 2007-08 cohort

Dundee City

Aberdeen

West Lothian Edinburgh

Falkirk

Clacks

25.0%

27.0%

29.0%

31.0%

33.0%

35.0%

37.0%

39.0%

0 2,000 4,000 6,000 8,000 10,000 12,000

Number of offenders

Rec

onvi

ctio

n ra

te

LAs National Average -95% +95%

Upper 95% confidence interval

Lower 95% confidence interval

18

Why is there a difference between organisations ?

They could be due to:

Difference due to offender mix between authorities• Age• Gender• Previous convictions

Difference in processes Difference in interventions and/or strategies

Look at these three separately

19

Difference due to offender mix

We can attempt to remove the offender mix (as much as we can)

Adjust for the effect of offender mix in order to have comparable rates across local authorities (logistic regression model)

What we then want to know is whether or not there is a real difference in practice across local authorities

20

Funnel plot of adjusted rates

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%

0 200 400 600 800 1,000 1,200

Observed-Expected offenders

Sta

ndar

dise

d re

conv

ictio

n ra

te (

O-E

)

LAs National Average -95% +95%

Adjusted for differences due to offender mix

Are there still any significant differences? Not particularly.

21

Difference in processes

Many different decision points affect conviction (and reconviction) Speed and immediacy of getting to conviction, lack of social services,

and/or system doesn’t work.

Recorded crime bulletinPolice

Procurator Fiscal

ReconvictionCriminal Proceedings bulletin

Court

Reconvictions bulletinJudiciary

Offender Re-offend

Social Work

Prison Statistics

Scottish Prison Service

Investigate

Warrants / summons

Proceed ?

Hear case

Fine

Decision(conviction ?)

Offence Community Licence

Manage offender

Manage prisoner

Breach

Lose offences - they are not brought to a conviction

22

Difference in interventions and/or strategies Deterrence based programmes

E.g. boot camps Incapacitation

E.g. prison or community Community supervision Rehabilitation

E.g. aftercare therapy Reparation

Authorities may use different strategies Moving on Renfrewshire is a ‘throughcare in custody’

programme for young offenders in Renfrewshire. Run by The Robertson Trust Working closely with offenders aims to reduce reoffending

23

In summary

Brief history of SPC SPC applied to a fictitious telecommunications example SPC applied to reconviction rates

Adjust for differences due to offender mix Are there still any significant differences? Not particularly.

Even if it is difficult to accurately remove the offender mix. It helps to understand if there are: Differences in process Differences in interventions and/or strategies

The funnel plot does indicate which authorities have a reconviction rate that needs further investigation

24

Conclusions

No evidence of a significant difference in reconviction rates across LAs after adjustment for offender mix.

Evidence that differences in reconviction rate are largely due to variability in offender characteristics.

It is complicated and technical

25

References

Deming Collaboration (2011) Deming Bio: W. Edwards Deming Obtained from http://demingcollaboration.com/?page_id=266 on the 12th April 2011

Edwards Deming, W (1986) Out of the Crisis MIT Center for Advanced Engineering Study

Leadership Institute Inc. (2011) Who is Dr. W. Edwards Deming ? Obtained from http://www.lii.net/deming.html on the 12th April 2011

Oakland, J. S. (2007) Statistical Process Control Butterworth-Heinemann

Royal Statistical Society (2003) Performance Indicators: Good, Bad, and Ugly Royal Statistical Society Working Party on Performance Monitoring in the Public Services. Obtained from http://www.rss.org.uk/pdf/PerformanceMonitoringReport.pdf

Spiegelhalter, D. J. (2005) Funnel plots for comparing institutional performance Statistics in Medicine 24 1185-1202.