A centre of excellence supported by the Australian Government © Copyright 2011

The evaluation of an open access self-help web-site Delyth Lloyd1 & Chris Clarke2

1Australian Centre for Posttraumatic Mental Health 2Defence Links, Department of Veterans’ Affairs

Acknowledgements

Andrea Phelps

John O’Connor

Funded by Department of Veterans’ Affairs

SMS Management Technology

Outline

• Background• The web-site & audience• Steps in the development of the evaluation• Factors shaping the evaluation

• Data sources and IT capacity• Policy cycle

• Challenges, lessons learned• Your thoughts?

Context

Department of Veterans’ Affairs ‘lifecycle’ program

• Aim: Provide an on-line Wellbeing resource tailored for veterans, former serving members and their families.

• Target hard-to-reach / hard-to-engage• Adding to the range of mental health and support

services available www.wellbeingtoolbox.net.au

Wellbeing Toolbox

Skills for Psychosocial recovery• Generic distress reduction skills• Principles based on risk & resilience research• Broad, accessible, tried and tested

Target population• Sensitive to information request• “Younger” (<50y)• Difficulties with government involvement in health

Open Access• Anyone can visit• Personalised journey – increase relevance• Ease of access – find what you want quickly

• Log in (optional)• Questionnaire (optional)• Modular (with guide me function)• Self management plan (optional)

accessibility priority low impact non intrusive evaluation

1. Literature review – what can we learn from others?

• Need is present in target group• On-line self-help “courses” can work • Disorder specific evidence• Organisational evidence• Youth orientated products successful

Genuine open access self-help evaluations with accessibility focus

• Not done?• Not reported?

Planning

Planning (cont’d)Utilisation discussions • Who is going to use the evaluation, why, how?• What do they need to know?• What would they like to know?• When?

Program Logic • Not a clinical TOC from use to better wellbeing • Path of use • Identify and agree on evaluation questions

Web-site created and

promoted

Users from target group

become aware of web-site

Via KIT Via direct marketing of self-care site navigate there independently from web-search from other sites e.g. At ease

Users visit SC web-site

Users perceive as useful

Sign in

Use

Re-use

Use of other

services & resources

Positive

feedback

Re-visit

Use web-site components to greater or lesser extent depending

on individual factors

Users self-refer to other services and resources as appropriate

How do users reach

the site?

Who are the “users”?

Who/How many users re-visit the

site?

Who/How many users choose sign

in?

Have site users seen anyone about their

problems since using the site or do they intend to

do so?

Use self –Management

Plan

Who/How many users re- visit the Self Management

plan?

What sections are visited by the

most people? (indicates interest)

What do first time users

think? (many will be

`1 time’ users)

What sections do people re-visit most

often? (indicates which are

most helpful)

Note: Module selections may be prompted to some extent if the ‘guide me’ function is used

General Feedback:

What do users like best about the site &

what could be

improved?

Actions Processes

Must be targeted at

veterans and also relevant

for family members

Must be clinically sound

Must contain Key features

named in contract

Outcomes

Usability testing &

stakeholder consultation

ensures appealing

and usable to target group

Have users followed links

to other recommended

resources?

How effective

were promotion marketing activities?

One time visitors

Factors shaping the evaluation 1: Data Available

• What can / could the site do? • e.g. reminders, return visits

• What evaluation tools can be built in?

• What can Google Analytics do?

• Ethics? What is it okay to know? • What we don’t/won’t/can’t know?

• e.g. questionnaire and use, but not both• Logged in versus not logged in

Web-site created and

promoted

Users from target group

become aware of web-site

Via KIT Via direct marketing of self-care site navigate there independently from web-search from other sites e.g. At ease

Users visit SC web-site

Users perceive as useful

Sign in

Use

Re-use

Use of other

services & resources

Positive

feedback

Re-visit

Use web-site components to greater or lesser extent depending

on individual factors

Users self-refer to other services and resources as appropriate

How do users reach

the site?

Who are the “users”?

Who/How many users re-visit the

site?

Who/How many users choose sign

in?

Have site users seen anyone about their

problems since using the site or do they intend to

do so?

Use self –Management

Plan

Who/How many users re- visit the Self Management

plan?

What sections are visited by the

most people? (indicates interest)

What do first time users

think? (many will be

`1 time’ users)

What sections do people re-visit most

often? (indicates which are

most helpful)

Note: Module selections may be prompted to some extent if the ‘guide me’ function is used

General Feedback:

What do users like best about the site &

what could be

improved?

Actions Processes

Must be targeted at

veterans and also relevant

for family members

Must be clinically sound

Must contain Key features

named in contract

Outcomes

Usability testing &

stakeholder consultation

ensures appealing

and usable to target group

Have users followed links

to other recommended

resources?

How effective were

promotion marketing activities?

One time visitors

How do users reach

the site?

Data – sources and strategies

Google AnalyticsVisitors/ unique visitors/ page views/ time on site

Rating scales within the topicsHow helpful was this module to you? (optional)

User survey “pop-up”Demographics, brief ratings, comment

Feedback (ad hoc) users and non-usersEvaluation Register (structured interviews)

Users and stakeholders

Benchmarking• What will it all mean? • “2685 people visited module X”…. So what?

• user numbers (proportion of sister site)

• time on site (one open access depression – 20 mins)

• extent of use (proportion start, mid, end)

• log-in rates (?1.6%/ 35%/25-90%)

• Revise and refine benchmarks over time• Link findings to recommendations

> Early indicators / likely trajectories > if this then what?

What info at what time point: • Reassurance – now > mid trial report• Decision making - soon• Accountability & bigger picture- later

Challenges of providing this as well as: • Rigorous evaluation of new concept• Useful • Ethical• Practically possible & not overly demanding on users

Factors shaping the evaluation 2: Policy Processes & Budget Cycle

Progress

Key questions: Reach, Acceptability, BenefitWhat next ?

Monitoring component (usage patterns over time)Benchmarks (meaning)Feedback (ad hoc complaints and compliments)Interviews (user stories)

> Decision making & meaning oriented report

Conclusions & our throughs so far.. 1. Evaluation needs to be incorporated in design of

web products. Privacy and Ethics are grey areas

2. Lessons/ tactics for a meaningful but low impact evaluation design. Trade-off between purpose of site and ease of evaluation.

3. Orienting reporting to stakeholder needs, timing, broader implications; what is reasonable to conclude mid way?

Questions, comments?