Introduction

• About me• About Involve• About evaluation...we will look at;

• Why we might evaluate• How we might evaluate• What we might measure

"Not everything that can be counted counts, and not everything that counts can be counted."

Albert Einstein’s deskat Princeton University

“Evaluation of online participation ... is still just an evaluation of

participation!”*Alice Casey’s desk at the Involve offices, London

* Less catchy, but

also true

Back to basics...why evaluate?

• Clarifying your objectives and purpose

•Ensuring goals are achievable and measurable

• Improving ongoing management

• Improved public accountability and value data

• Improving future work – learning!

A choice of purpose• Evidence generatorCollecting the evidence of success

• Learning toolPlanning our future work more effectively

Very important for emerging online methods

Its never too soon...

There’s a role for the evaluator at all stages;

• Scoping• Planning• Delivery• Reporting back

... so, begin with the end in mind.

External or internal evaluation?

Internal • Develops in-house learning• Self-assessment• Less defensive reaction• Reduces risks of

innovation• Lower Costs

External• An outside perspective• Perceived independence• Credibility• Therapeutic• Lets others focus on doing

So, if ;

"Not everything that can be counted counts, and not everything that counts can be counted."

…then how on earth do we know what to count and how to count it?

Evaluation of online participation

We’re still learning...

but here are some ideas...

How do we measure?

Use online and offline methods•Online surveys, timing is important•Pop-ups vs. email•Consider incentivising•Ongoing panel of participants•Telephone.... Skype?•Targeted publications•Peer interviews (include ‘people missed’)•Use focus groups (include ‘people missed’)

Involve your stakeholders...

Target participant groups provide insight

• Ensure they understand why you are evaluating

• Consult them about methodology• Consult them about indicators • Include them on the evaluation team• Invite them to be peer researchers

A step further... Participatory Evaluation

• What role should members of the public and/or service users play in evaluating projects?

• What are the benefits and downsides to having them involved?

• If you’re interested, read more here: Putting People Into Public Services; NationalConsumer Council (2008)

What do we measure?

Some useful definitionsTerm Definition

Outputs Tangible products (reports, meetings etc.)

Outcomes Overall direct results (increased skills, reduced crime etc.)

Impact Longer-term results (may include unintended effects)

Quantitative Objective, measurable, predetermined response options

Qualitative Subjective, interpretative, open ended response options

Web stats:(Tip: Check out GoogleAnalytics)

1.Unique visitors to the site

2. Length of time spent on a page

3. Downloads of certain resources or materials

‘Thin’ engagement

Deeper engagement

Interaction quality: eg. online forums, blogs, wikis.

1.Number of comments2.Number of user-instigated threads or topics3.Length of comments4.Number and length of chains of comments ‘discussions’5. Variety of participants, including representation of decision makers

Motivations and perceptions:Understanding the experience

•Did participants feel they had influence?

•Did participants feel that they understood what they were being asked to contribute to and why?

• How did the moderator’s role seem to affect the engagement exercise?

Outcomes and impact:What actually changed?

•Did participants have any demonstrable influence on the process?

•Did policy or decision making change as a result in the medium - long term?

• How did the working practices of decision makers change if at all, how did the participants attitudes or views change?

www.involve.org.uk22-25 Finsbury Square, EC2A 1DX

tel: +44 (0) 207 9206470 email: alice@involve.org.uk twitter: @cased

Creative commons thanks for photos go to: plindberg, memotions