Developing National On-Line HIV Prevent ion Program Evaluat ion Training

David Davis, Ph.D. ,and the NHM&E Training TeamProgram Evaluation Branch, Division of HIV/AIDS Prevention

National HIV Prevention ConferenceAugust 15, 2011

National Center for HIV/AIDS, Viral Hepatitis, STD, and TB PreventionDivision of HIV/AIDS Prevention

Outline

NHM&E Training Project History Contents Data Lessons Learned

National HIV Prevent ion Program Monitoring and Evaluat ion (NHM&E) Training Project

Consists of On-line, on-demand training modules Webinars (live and recorded)

Covers Basics of monitoring and evaluation NHM&E standardized variables Data reporting system (PEMS) Local use of data

The Old Way

Two people per grantee agency (59 HDs, 130 CBOs) In person, in At lanta Week long training sessions About 30 people at a t ime Done after major changes to data report ing system

The New Way

On-line, web-based, mult i-media, interact ive training modules

Available on demand to all grantee and subcontractor staff

Webinars for most t raining modules as they are released

Webinars recorded for playback on demand Staff can take only those modules they need

Contents

Sixteen “courses” or “modules” with one or more lessons

Fifty-nine planned lessons, 43 current ly available Sixteen current lessons specific to PEMS software

Topics

M&E Guidance and Tools Data Quality Assurance HIV Prevent ion Program Indicators Community Planning Partner Services Referrals

More Topics

Client-Level Services Aggregate-Level Services HIV Test ing Agency Information Program Planning Data Security

Even More Topics

Introduct ion to NHM&E Training Data Submission Data Transfer

Usage (5/2010-5/2011)

1,794 lessons completed 6,162 visits by 899 visitors Average of 120 visits per week Average of 17 first-t ime users per week Average of about 15 minutes per visit About 1.7 hours per visitor average total t ime on site

Most Completed Lessons

Training Navigat ion Tutorial Monitoring and Evaluat ion Overview Data Submission PEMS Admin Module Data Quality Assurance

On-Demand Training FeedbackParticipants’ Satisfaction With On-Demand Training Delivery

Statement n Not at All Somewhat Neutral For the Most Part

Very Much So

N/A

The assistance I have received related to the on-demand training met my needs.

42 0 (0%) 7 (17%) 4 (10%) 16 (38%) 10 (24%) 5 (12%)

Do the elements (e.g., the navigation text, buttons on the play-bar) on the screen enable you to move through the on-demand training with ease?

42 0 (0%) 6 (14%) 4 (10%) 19 (45%) 13 (31%) N/A

The visual aids (e.g., graphics and animation) have kept me engaged in the on-demand training.

42 2 (5%) 6 (14%) 6 (14%) 15 (36%) 13 (31%) N/A

The Quick Checks have contributed to my knowledge of the subject matter.

42 3 (7%) 5 (12%) 6 (14%) 18 (43%) 10 (24%) N/A

The on-demand training was well organized.

42 0 (0%) 5 (12%) 7 (17%) 18 (43%) 12 (29%) N/A

More On-Demand FeedbackParticipants’ Satisfaction With Relevance to Job

Statement n Not at All Somewhat Neutral For the Most Part Very Much So

The training has provided me with practical insights into the content area (i.e., best practices and/or lessons learned).

41 1 (2%) 10 (24%) 3 (7%) 23 (56%) 4 (10%)

The on-demand training has provided useful tools (e.g., worksheets).

41 3 (7%) 6 (15%) 13 (32%) 9 (22%) 10 (24%)

I would recommend this training to my colleagues.

41 4 (10%) 6 (15%) 4 (10%) 12 (29%) 15 (37%)

Comments on On-Demand Training

2% said training was too basic and 8% said too advanced.

Lack of t ime was most common barrier to using the training.

Site navigat ion was a problem for a few people.

Webinars

Program Planning – June 24, 2010 Agency Information – July 28, 2010 M&E Guidance and Tools – October 14, 2010 Data Quality Assurance and Security – November 18,

2010 Client-Level Services Data – January 20, 2011 Aggregate-Level Services Data – March 31, 2011 Average 65 website logins per webinar (mult iple

part icipants on some logins)

Webinar Delivery Dates and Web Traffic

*Number of individual sessions initiated by all visitors to the site.

0

50

100

150

200

250

300

Num

ber o

f Vis

its

Visits

Program PlanningWebinar Held

Agency InformationWebinar Held

Security Measures and DataQuality Assurance Webinar HeldM&E Guidance and

Tools Webinar Held

Client-Level ServicesWebinar Held

Aggregate-LevelServices Course

Post Webinar FeedbackParticipants’ Perceptions of Overall Webinar Quality

Program Planning

Agency Information

M&E Guidance and Tools

Data Quality Assurance and

Security Measures

Client-Level Services

Aggregate-Level Services

(n = 49) (n = 25) (n = 21] (n = 50] (n = 24) (n = 12)

The Webinar was well organized.

mean = 4.31 mean = 4.48

mean = 4.38 mean = 4.18

mean = 4.04 mean = 4.83

The Webinar met my learning needs.

mean = 4.78 mean = 3.72

mean = 3.81 mean = 3.71

mean = 3.83 mean = 4.58

More Post Webinar FeedbackParticipants’ Perceptions of How Well the Webinar Has Prepared Them to Better Perform Their Jobs

Program Planning

Agency Information

M&E Guidance and Tools

Data Quality Assurance and

Security Measures

Client-Level Services

Aggregate-Level Services

(n = 49) (n = 25) (n = 21] (n = 51] (n = 24) (n = 12)

Poor 4 (8%) 1 (4%) 1 (5%) 1 (2%) 0 (0%) 0 (0%)

Marginal 2 (4%) 2 (8%) 0 (0%) 5 (10%) 1 (4%) 0 (0%)

Fair 7 (14%) 4 (16%) 4 (19%) 8 (16%) 5 (21%) 2 (17%)

Good 22 (45%) 10 (40%) 7 (33%) 20 (39%) 8 (33%) 2 (17%)

Excellent 12 (24%) 7 (28%) 9 (43%) 14 (27%) 9 (38%) 7 (58%)

N/A 2 (4%) 1 (4%) 0 (0%) 3 (6%) 1 (4%) 1 (8%)

Grantee suggest ions for webinars

Have grantees present in webinars on best pract ices from the field.

Don’t repeat on-demand training, supplement it . Separate webinars for different skill and experience

levels.

Lessons Learned

Webinars prompt use of the on-demand training. Grantees like to hear from other grantees. Not being away from the office creates t ime conflicts to

taking on-line training. On-line training appears to deal with staff and grantee

turnover. As HIV prevent ion changes, the training will need to

change as well.

Acknowledgements

CDC Staff Antonya Rakestraw

ICF Macro Staff Sonya Shropshire Jessica Wals Kira Sloop Brieon Arthur Betty Treschitta Katherine Akopov Elizabeth Jacobs John Afon Nkateko Okwera

Emily Baughman Micheal Raines Altus Waters

Thank you

For more information please contact David Davis at the Centers for Disease Control and PreventionMail Stop E-591600 Clifton Road NE, Atlanta, GA 30333Telephone: 1-404-639-0938E-mail: dad5@cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

National Center for HIV/AIDS, Viral Hepatitis, STD , and TB PreventionDivision of HIV/AIDS Prevention