Improving Improving Justice Sector Justice Sector AssessmentsAssessments

Insanity: doing the same thing over Insanity: doing the same thing over and over, while expecting the and over, while expecting the outcome to be different.outcome to be different.

Project preparationProject preparation-- Diagnostic/assessmentDiagnostic/assessment-- Formulation of priorities for Formulation of priorities for reformreform-- Design of the project – solutions Design of the project – solutions crafted crafted to meet the priority to meet the priority problems identified.problems identified.

Project implementationProject implementation-- Support/ oversightSupport/ oversight-- Monitoring and evaluationMonitoring and evaluation-- Feedback loop, learning, Feedback loop, learning, adjustments to adjustments to projectproject-- Measuring outcomes and impactMeasuring outcomes and impact-- Recording lessons learnedRecording lessons learned

What do we need to change?What do we need to change?

TC: TC: -- Develop new methodologiesDevelop new methodologies-- Define a research agendaDefine a research agenda-- Create centers for research, Create centers for research,

dissemination.dissemination.LH:LH:

-- Consolidate informationConsolidate information-- Build knowledge baseBuild knowledge base-- Disseminate better, and require Disseminate better, and require

that it that it be used.be used.

Why focus on diagnostics?Why focus on diagnostics?

-- Measuring impact requires a Measuring impact requires a baselinebaseline

-- Changing to a problem focus Changing to a problem focus (rather (rather than capacity-building) than capacity-building) requires a requires a definition of the problemdefinition of the problem

-- Understanding the political context Understanding the political context of a of a reform requires analysis of who reform requires analysis of who stands stands to benefit (and lose)to benefit (and lose)

-- It’s hard to know where to start It’s hard to know where to start without without an understanding of priorities.an understanding of priorities.

Doing diagnostics differently can be part Doing diagnostics differently can be part of building the knowledge base to change of building the knowledge base to change what we know about and what we learn what we know about and what we learn from justice reforms, and perhaps even from justice reforms, and perhaps even change their impact. change their impact.

The goal: reforms that are fact-based The goal: reforms that are fact-based rather than purely assertion-based:rather than purely assertion-based:

- an empirically-based process to - an empirically-based process to determine determine what is working and what is what is working and what is wrong and wrong and what might work as a what might work as a solutionsolution

-- based on sound social-science based on sound social-science methodologymethodology

-- to test and/or complement qualitative to test and/or complement qualitative assertionsassertions

The Bank has been a leader in good The Bank has been a leader in good diagnostic practice – in general and in diagnostic practice – in general and in the justice area.the justice area.

Do we need to change our diagnostic Do we need to change our diagnostic process? How solid is Bank practice process? How solid is Bank practice with respect to assessments and with respect to assessments and diagnostics in the justice sector?diagnostics in the justice sector?

To get a sense of that, we commissioned To get a sense of that, we commissioned an analysis of Bank diagnostics, held a an analysis of Bank diagnostics, held a workshop with Bank staff who have workshop with Bank staff who have worked on diagnostics.worked on diagnostics.

Key findings: Key findings:

-- Tap into the wealth of research Tap into the wealth of research capacity within the Bank capacity within the Bank

(from social (from social sciences, DEC, sciences, DEC, etc.) regarding the etc.) regarding the diagnostic/research diagnostic/research processprocess

-- More emphasis on collaborative More emphasis on collaborative engagement with local engagement with local

actorsactors-- Improve the relevance, quality Improve the relevance, quality

and and sophistication of sophistication of questions (and thus questions (and thus answers)answers)

(continued)(continued)-- Be explicit about choice of Be explicit about choice of

focus/scope, focus/scope, the underlying the underlying theory of what theory of what constitutes justice – i.e., what we’re constitutes justice – i.e., what we’re

measuring the system measuring the system against, the against, the tools used.tools used.

-- Add some tools to the toolkit: Add some tools to the toolkit: surveys, surveys, observation of court observation of court process, reviewing process, reviewing court court decisions, etc.decisions, etc.

-- Increase empirical accuracy of Increase empirical accuracy of descriptions, and analyze descriptions, and analyze

problems problems found in terms of found in terms of the factors that the factors that contribute contribute to them.to them.

Would it be possible to prepare a manual Would it be possible to prepare a manual on assessments, to build on the Bank’s on assessments, to build on the Bank’s existing guidelines and experience?existing guidelines and experience?

Project concept note reviewed at the end Project concept note reviewed at the end of 2005. Methodology:of 2005. Methodology:

-- analyze existing assessments for analyze existing assessments for good good practicespractices

-- workshop with Bank staffworkshop with Bank staff-- look for other manuals for diagnostics look for other manuals for diagnostics

of of systems that might be usefulsystems that might be useful-- in relatively short timeframe try in relatively short timeframe try

writing an writing an initial draft, to get a sense of initial draft, to get a sense of what what we we know and don’t know and know and don’t know and what team skills what team skills are needed to do thisare needed to do this

(continued)(continued)-- put together a team pulling in put together a team pulling in

Bank Bank staff from various departments staff from various departments with with expertise in aspects of expertise in aspects of diagnosticsdiagnostics

-- get regular input/review from get regular input/review from Bank Bank staff staff

-- produce a manual, for hard copy produce a manual, for hard copy and and to to put on-lineput on-line

-- make it interactive at least in the make it interactive at least in the sense sense that we can continually take that we can continually take feedback feedback and comments into and comments into account to improve account to improve the manual.the manual.

Some of the challenges to attempting a Some of the challenges to attempting a manual:manual:

Different types of studies are Different types of studies are appropriate for different purposes:appropriate for different purposes:

-- Desk review, pulling together what Desk review, pulling together what is is known, understanding known, understanding what what knowledge is knowledge is lackinglacking

-- Comprehensive look at the justice Comprehensive look at the justice sector sector where are major weaknesses where are major weaknesses and and strengths, priorities for reformstrengths, priorities for reform

-- Examination of one area of concern Examination of one area of concern or or interestinterest

(continued)(continued)-- Support stakeholders in coming to a Support stakeholders in coming to a consensus regarding what is wrong and consensus regarding what is wrong and priorities for change, and where it may priorities for change, and where it may

be be possible to bring about desired possible to bring about desired change.change.

Can we write a manual appropriate to all Can we write a manual appropriate to all the different sorts of diagnostic one might the different sorts of diagnostic one might engage in?engage in?

Can we write a manual appropriate to the Can we write a manual appropriate to the different levels of expertise/experience different levels of expertise/experience that our target audience (i.e., people that our target audience (i.e., people starting out to do assessments) will have?starting out to do assessments) will have?

If we review the list of types of If we review the list of types of diagnostics, each one, to be done well, diagnostics, each one, to be done well, would require an understanding of what would require an understanding of what aspects of the justice sector need to be aspects of the justice sector need to be understood in order to have a grasp of understood in order to have a grasp of how the sector’s functioning.how the sector’s functioning.

Each one requires the assessor to know Each one requires the assessor to know what sort of questions, measurement what sort of questions, measurement methods, and likely sources of data exist methods, and likely sources of data exist or can be created to understand the or can be created to understand the aspects of the sector under review.aspects of the sector under review.

For example, the assessor may hear For example, the assessor may hear from all sides that the system is slow, from all sides that the system is slow, that delay is an obstacle to access and that delay is an obstacle to access and a hindrance to the system’s credibility. a hindrance to the system’s credibility.

-- How slow is too slow? What’s the How slow is too slow? What’s the standard?standard?-- Which courts are slow? Which courts are slow? -- What kind of cases? What kind of cases? - Which proceedings? - Which proceedings?

(continued)(continued)-- In what part of the process does In what part of the process does

delay delay occur: preparation or pre-trial? occur: preparation or pre-trial? The The hearing itself? Interlocutory hearing itself? Interlocutory appeals, appeals, continuances, or other continuances, or other procedural procedural maneuvers? Time maneuvers? Time between hearing between hearing and judgment? and judgment? Enforcement of Enforcement of judgment?judgment?

-- Who is reporting delay? Judges, Who is reporting delay? Judges, parties, lawyers?parties, lawyers?

-- What incentives reinforce delay?What incentives reinforce delay?

- - What’s causing delay?What’s causing delay?

Are these the relevant questions? Under Are these the relevant questions? Under what circumstances? What methods what circumstances? What methods exist to get at answers to those exist to get at answers to those questions, and how best to employ questions, and how best to employ them?them?

This is a cursory review of some of the This is a cursory review of some of the issues raised in analyzing delay - the issues raised in analyzing delay - the manual’s goal would be to identify key manual’s goal would be to identify key questions and methods for probing those questions and methods for probing those questions, and provide references to questions, and provide references to more detailed information about both.more detailed information about both.

If the manual achieves that, it can If the manual achieves that, it can help produce diagnostics that help produce diagnostics that provide: provide:

-- a first step toward forming a a first step toward forming a solid empirical basis for reform solid empirical basis for reform efforts, efforts,

-- a first step toward establishing a a first step toward establishing a baseline from which to measure baseline from which to measure change, change,

-- and thus an important element in and thus an important element in making justice reform a fact-based making justice reform a fact-based

rather than a purely assertion-rather than a purely assertion- based based enterprise.enterprise.

Topic/Diagnostic Stage

Substance Stakeholders

1. Getting started Desk review, logistics, choosing between general or narrow approach.

Initial outreach to fill gaps in desk review and as first step for field assessment.

2. Identifying and selecting problems

Most common problems to analyze, guidance on choosing, what factors to look at, and risks in analyzing a particular theme.

Who to include as subjects of diagnostic, additional persons to be targeted by the diagnostic.

3. Tools/methods Checklists, survey techniques (pros & cons), short-cuts and best practices.

Who to interview, who to use in diagnostic team etc.

4. Report writing and Analysis

How to organize the report, different levels of analysis & description, techniques for presenting data.

Tailoring report content to have strongest impact on the report’s audience(s).

5. Dissemination How to disseminate and best reach audience (political considerations.)

Assessment Manual “Iceberg”1. Getting Started2. Identifying/

Selecting Problems

3. Tools/methods4. Report

writing/Analysis5. Dissemination

1. 2. 3. 4. 5.

Is this process of improving Is this process of improving diagnostics one that sounds diagnostics one that sounds useful/makes sense to you?useful/makes sense to you?

Is it one that would likely have a Is it one that would likely have a positive impact on our project work positive impact on our project work and in filling the knowledge gap?and in filling the knowledge gap?

Would you be willing to review and Would you be willing to review and comment on the draft manual?comment on the draft manual?