Measuring Impact - c1593.r93.cf3.rackcdn.comc1593.r93.cf3.rackcdn.com/Measuring_Impact.pdf ·...
Transcript of Measuring Impact - c1593.r93.cf3.rackcdn.comc1593.r93.cf3.rackcdn.com/Measuring_Impact.pdf ·...
Measuring Impact
What is Impact?
Dictionary definition
1. The striking of one body against another; collision.
2. The force or impetus transmitted by a collision
3. The effect or impression of one thing on another: still
gauging the impact of automation on the lives of factory
workers
4. The power of making a strong, immediate impression: a
speech that lacked impactwww.thefreedictionary.com
Where do I start?
Work backwards
Does your coaching programme / project have an impact on coaches?
Define the impact. What impact do you hope to see?
What are the programme aims and objectives?
The importance of planning!
The Planning Stage...Project Logic Models
Rationale/ Aims & ObjectivesAll coaches are sufficiently skilled and qualified to provide
the best experience to participants. In turn increasing recruitment and retention
Challenges
Any barriers?What might you
encounter?
Opportunities
Where can you help?
What can be done to overcome challenges?
Inputs
£Resource
AdminEtc.
Activities
CPD courses / Qualifications / Educational
awareness raising etc.
Outputs
More coaches completing
Level 2 qualification
Outcomes (short and
medium term)
Coaches feel better skilled,
knowledgeable and more confident
Impact (long term)
Coaches provide better
experiences for
participants = increased
recruitment and retention
Output Measures
Increase in % of those completing Level 2
(database)
Outcome Measures
Increase in % of coaches feeling x,y,z (Coach
questionnaire)
Impact MeasuresObservational
assessment, participant feedback, tracking
survey
The Planning Stage...Project Logic Models
Rationale/ Aims & ObjectivesAll coaches are sufficiently skilled and qualified to provide
the best experience to participants. In turn increasing recruitment and retention
Challenges
Any barriers?What might you
encounter?
Opportunities
Where can you help?
What can be done to overcome challenges?
Inputs
£Resource
AdminEtc.
Activities
CPD courses / Qualifications / Educational
awareness raising etc.
Outputs
More coaches completing
Level 2 qualification
Outcomes (short and
medium term)
Coaches feel better skilled,
knowledgeable and more confident
Impact (long term)
Coaches provide better
experiences for
participants = increased
recruitment and retention
Output Measures
Increase in % of those completing Level 2
(database)
Outcome Measures
Increase in % of coaches feeling x,y,z (Coach
questionnaire)
Impact MeasuresObservational
assessment, participant feedback, tracking
survey
Measures
The Planning Stage...methodology & data collection
Costs / Resources
Aims & Objectives
Project Outline
Project Sample
Data sources and Collection
Stakeholders / data holders
Timings and Deadlines
Title
Reporting
Example; Evaluation Surveys
Project Start
Aims & Objectives
Project End
Evaluation & Conclusions
Evaluation survey
Do coaches report feeling more satisfied / confident / knowledgeable
etc. as a result of the project?
Ok, but nothing to compare the data to. Relying upon people to remember how they felt at the
start
?
Example; Tracking Studies
Has the project resulted in the desired changes in attitudes, opinions, reported behaviours etc.?
Project Start
Aims & Objectives
Baseline survey
Project End
Mid-point survey
Final survey
Look for any changes in responses Look for any changes in responses
Look for any changes in responses
Evaluation & Conclusions
Example; Interviews / Focus groups
95% of coaches feel more confident
But in what ways? Give me examples?
Quantitative is the ‘what’
Qualitative is the ‘why’
Understanding the meaning behind the numbers
Example; Case studies
It’s about telling a...
1. Beginning
Set the scene; what was the problem
2. Middle
What happened; what was the solution / project / activities
3. End
What was the result; was the solution successful?
Case study examples
Example; Video Case Studies
Content and
Format
Script and
Set-up
Capturing on Camera
Review and Edit
Presentation and
Promotion
Excellent Coaching Case Study
Take One
29th January 2013
Researching
The FA Tesco Skills Programme
Presented by Lucy Roberts and David Hollinshead
The FA Tesco Skills programme...
•Launched in 2007 through a unique partnership with Tesco and Sport England in response to the
‘skills gap’ identified between English youngsters and their European counterparts.
•Based on a unique FA coaching model that encompasses four key attributes vital for any developing
young player: Physical, Technical, Psychological and Social elements.
•Delivers high quality football coaching for 5-11s by creating learning environments that are safe and
inclusive and coaching sessions that are simple, game related & enjoyable.
•Provides after-school football courses, holiday
coaching and specialist coaching and support in
primary schools.
•An inclusive coaching programme for 5-11s -
coaching is free of charge.
Programme objectives
•To develop better and more technically gifted players and to give every child the opportunity to be
the best that they can be.
•To provide all 5-11s with a skill set to develop their sporting ability with the long term objective of
improving the technical quality at all levels of the English game.
•Tesco objectives – to give every child in England the opportunity to get active and lead healthier
happier lives.
Achievements so far•3.98 million child places filled (42.5% girls and 57.5% boys)
•3% of places taken up by disabled children and 16.2% by ethnic minority children
•7,400 Primary schools, 569,800 school children, 22,200 teachers supported
•2,700 Charter Standard football clubs and 4,300 Charter Standard teams supported
•4,300 team managers/coaches and 10,800 coaches supported
•300 weekly Skills Centre sessions involving 4,500 children
•To demonstrate impact, research is key to everything we do
What research have we done so far?
20
0 P
aren
ts
20
Tea
cher
s
10
Co
ach
es
July 2011
30
0 P
aren
ts
15
0
Teac
her
s
10
0
Co
ach
es
November 2011
330respondents
550respondents
August 2012
40
0 P
aren
ts
in s
chem
e/
left
sch
eme
40
0 P
aren
ts
no
t in
sc
hem
e
800respondents
How long does a project like this take?
Timing Task Outcome
Week 1 Internal meeting Define scope of project: objectives and outcomes and outputs needed
Week 2 Research brief written and signed off Objectives, target audience, approach and budget
Week 3 Brief distributed 2-3 agencies
Week 4 Proposal submitted Agency selected
Week 5 Agency commissioned and set up meeting Define roles, timings and logistics
Week 6 Planning Questionnaire written, sample sourced
Week 7
Fieldwork Interviews take placeWeek 8
Week 9
Week 10 Presentation of results
Questions to include when measuring the impact of an intervention
59%
64%
67%
79%
82%
32%
27%
27%
18%
16%
High quality of coaching and supervision
I can see myself attending for many …
I will attend on a regular basis
The kickabouts are really good fun
I would recommend to friends and relatives
Strongly agree Slightly agree
98%
97%
94%
91%
91%
Encouraging children to play sport now...
...and in the future
Research shows that the FA Tesco Skills programme is
successful in...
Key Findings
Encouraging confidence in playing sport
Encouraging enjoyment in playing sport
Do you think your child would have taken part in sporting activity if the programme hadn’t been available?
41% of parents stated their kids would have doneless or no sport without the programme
Parents
20%
21%
59%
Yes would have found something else
Yes but not as much
No
Parents: Q5e (100) Would your child have taken part in activity if programme not available
“it gives children the chance to join a club and there aren’t many available in this area. If it wasn’t for this, there would be like 60 children out of it doing nothing
each week”
41%
The FA Tesco Skills is successful at
encouraging children to play sport more
frequently – especially once they have
left the programme
5-11 year olds 12-16 year olds
In FA Skills
mean 5.0 times per wk
Not in FA Skills –Control areas
mean 4.7 times per wk
In FA Skills
mean 5.2 times per wk
Not in FA Skills –Control areas
mean 3.9 times per wk
46 4352
26
4746
43
56
67
3
10
6 211
Less often
Once week
Several times a week
Daily
Q: Frequency of participating in sport
... and for longer
5-11 year olds 12-16 year olds
In FA Skills
mean 7.8 hours
per wk
Not in FA Skills –Control areas
mean 7.2 hours
per wk
In FA Skills
mean 10.2 hours
per wk
Not in FA Skills -Control areas
mean 7.0 hours
per wk
67
8
12 1
2
3
45
9
1011
67
8
12 1
2
3
45
9
1011
67
8
12 1
2
3
45
9
1011
67
8
12 1
2
3
45
9
1011
Kids currently or
previously in FA Skills
are significantly more
confident playing sport,
like sport more and are
more likely to play
sport in the future
Unweighted base: Children in FA Skills Scheme areas - 5-11 year olds in Scheme (200); not in Scheme and in control areas (200); 12-16 year olds in Scheme (107); not in Scheme and in control areas (200)
In FA Skills
Scheme
Not in FA
Skills –Control areas
In FA Skills
Scheme
Not in FA
Skills –Control areas
% agreeing5-11 yrs
5-11yrs
12-16 yrs
12-16 yrs
“Likes sport a lot”
96% 83% 94% 75%
“Confident playing sports”
71% 55% 77% 56%
“Extremely likely to take part in sport
in future”
80% 56% 79% 45%
Impact on children
•78% of parents noticed their child being more active.
•97% of parents, teachers and coaches felt that the sessions were enjoyed a lot.
•93% of parents and 88% of teachers noticed improved confidence in the children.
•77% of coaches and 51% of parents noticed improved concentration in the children.
•43% of teachers felt the children were performing better in other lessons (attributed to the social
interaction in the sessions and improved concentration and focus).
•.
Impact on teachers
•99% of teachers are more confident in teaching sport.
•97% of teachers and coaches have or plan to use the techniques taught by Skills coaches, 97% of
whom, said it was easy to use/carry them on.
•43% of teachers noticed children were behaving better in the classroom.
•92% of teachers feel that it is very important that the programme is continued to be available.•..
•/,
“Because we are a primary school, the skills of
the teachers varies considerably so the input
from professionals has increased my
knowledge. I therefore have been able to teach
differently. I also think it makes valuable links
with the people outside of school and provides
good role models.”
Research benefits
•Evidence to demonstrate that the programme is meeting its objectives to develop children, coaches
and teachers.
•Provides confidence that the programme is doing the right things to meet its objectives.
•Helps us to shape the future direction of the programme.
•Provides our sponsors with a valuable asset by making a positive impact on communities.
•Potentially will help to secure sponsor and partner investment (Tesco & Sport England).
... but don’t take our word for it...
Communicating impact - The FA Tesco Skills video
The FA Group
TheFA.com