Seminar on Mid Term Evaluation in Objective 1 and 2

Regions

Lessons from the Mid Term Evaluation of

Merseyside Objective One

Overall Evaluation Approach

Review andUpdate ofEconomicBaseline

Review of SPDStrategy

Analysis ofProgramme

Monitoring Data

PartnerConsultations

Project Reviews

BeneficiarySurveys - SMEs

& individuals

Top-DownAnalysis

Bottom-upAnalysis

Analysis andIntegration of

Emerging Findings forProgramme and

Priorities

Conclusions on: -Programme and Priority

Progress- Emerging Impacts &

added value- Effectiveness of

management

Comprehensive approach, responding directly to the evaluation questions

Mix of top down and bottom up analysis

Evaluation highlighted value, but also limitations, of original research

Bottom up Analysis

Need to understand nature and performance of projects, in terms of: Benefiting target groups Programme performance Overall socio-economic performance

Analysis therefore included: Detailed project reviews Surveys of business and individual beneficiaries

Project Reviews

Purpose: Reality check for progress ‘on

ground’ Check validity of monitoring data Explore emerging impacts

140 reviews in total: Covering 50% of committed spend Mix by priority, measure, fund,

delivery agency, location and size Use of a structured questionnaire,

gathering both quantitative and qualitative data

Performance data collected before hand

Review coverage: Rationale and

purpose Value added by SF Project progress Financial

performance Output/Impact

performance Cross cutting issues Programme

management

Project Reviews (cont)

Reviewers trained to undertake a rigorous assessment, including judgements on: Adequacy of project rationale and approach Articulation of project with the Programme Objectives Appropriateness and realism of targets and outputs

achieved Value added by Structural Fund support

Surveys of Beneficiaries

Two separate surveys: Individual beneficiaries Business beneficiaries

Purpose: Explore the effectiveness of the assistance for the

individual or business (gross impact) Establish the extent of deadweight and displacement

(net impact upon the economy) Investigate participants views of the support and

opportunities to improve provision But there are limitations

The Surveys

Individual Beneficiary Survey Postal survey with 100%

coverage 1,500 completed

questionnaires – 18% response rate

Potentially powerful tool, but limited by: Ability to capture detailed

information on impacts Ability to secure a

representative sample

Business Survey Telephone survey of 200

SMEs (c20% of participants)

Richer source of data due to use of telephone survey

Survey issues: Exclusion of SMEs with

minor interventions Ability to locate most

appropriate contact Too early to draw firm

conclusions on impacts

Survey Issues

Suitability of survey methods Representativeness of sample Timing of survey Design and testing of questions Training of interviewers

Conclusions

Value of original research in answering evaluation questions

Success dependent upon: Appropriate techniques Well designed and tested research instruments Accurate beneficiary data Timing of research

But need to be aware of limitations of approach