Post on 24-Dec-2015
Rating Health Information Websites
Peter G. Goldschmidt, PresidentHealth Improvement Institute
Bethesda, Maryland
Consumer Reports WebWatch
Trust or Consequence Berkeley, California
June 9, 2005
2
Presentation
1. Introduction – HII-CRW partnership
– Independent ratings
– Project/ratings philosophy
– Types of health websites
2. Methods
3. Results
4. Conclusions
3
1A HII-CRW Partnership
Health Improvement Institute 1990, founded; non-profit 501(c)3 education/research charitable
organization 1991, established Aesculapius Awards for excellence in health
communication 1997, held workshop on quality of health information on the Internet 2003, entered into partnership with CU/CRW
Purpose/objectives regarding health websites To enable consumers to be more effective users To provide consumer with independent ratings
Medical Library Association is actively supporting project
4
1B Independent Ratings
Many past/present activities Criteria
– American Medical Association
– Mitretek Systems Codes of conduct
– eHealth Code of Ethics
– HI-ETHICS Trustmarks/seals
– Health on the Net Foundation (HONcode)
– URAC accreditation
Growing literature on criteria, quality of health websites
5
1C Project/Ratings Philosophy
Transparent regarding– Identity, contact information
– Ownership
– Privacy (or its invasion)
Easy to use regarding– Finding/navigating contents on
website
– Comprehending contents (clarity of writing, reading level, etc)
– Printing contents of interest
– Serving consumers with special needs
Meticulous in distinguishing advertising/selling from contents
Editorially adequate, for example, disclose & be appropriate regarding – Providing authors' credentials
– Selecting/grading & editing contents
– Resulting content
° Accurate, complete, objective, balanced
° Well-organized
° coherent/clear; not muddled
° Referenced
Current Useful to consumers
Health information websites must be
6
1D Types of Health Websites
Health communication– Health information – Decision support tool– Health ratings– Health information resource– Health website search
engine– Health advice on-line
Behavior modification – Behavior self-help – Disease management
On-line product sales/marketing– On-line pharmacy – On-line store – Health product marketing
Health care organization – Health plan
enrollment/transaction – Health care provider– Producer of health resources– Public health program– Other health care organization
To start, focus is "health information websites"
7
2 Health Website Rating Methods
Enter into partnership
HII-CRW
Enter into partnership
HII-CRW
Develop concepts/ approach
Develop concepts/ approach
Select websites to
be rated
Select websites to
be rated
Select additional
websites to be rated
Select additional
websites to be rated
Develop health
website rating instrument
Develop health
website rating instrument
Credential/ select raters
Credential/ select raters
Design ratings website
Design ratings website
Rate health websites
Rate health websites
Display ratings
Display ratings
Rate additional websites
Rate additional websites
Rerate rated websites
Rerate rated websites
Add/update ratings website
displays
Add/update ratings website
displays
Refine methods
Refine methods
Analyze feedback
Analyze feedback
8
2A Ratings Concepts/Strategy
Transparency/accountability
Editorial adequacy
Information reliability — completeness/accuracy of what is stated in any medium for any audience — excluded from present project because– Website may contain information on very many subjects
– To rate validity of health information for given subject requires panel of qualified medical/research experts
– Ultimately, assessments of reliability/validity of health information reflect state of medical science
3 levels for health information websites
9
2B Select Websites
To start Define health information websites Identify 100 most-visited "health" websites (A. C. Nielson) Select top-20 "health information websites"
To continue Rate additional top-100 websites Rate health websites suggested by consumers
Periodically, rerate rated websites
10
2C Develop Ratings Instrument
Developed general instrument applicable to rating all types of websites (CRW principles)
Adapted CRW instrument to meet project purposes
Created "Part I" after evaluation by HII volunteers of adapted CRW instrument (& compiled generic/specific criteria)
Pretested/revised Part I
Analyzed/assessed criteria sets intended to evaluate health websites
Compiled list of (generic & specific) criteria
Asked HII volunteers to evaluate criteria
Created "Part II" to focus on specific criteria
Piloted/refined Part II
Consumer Reports WebWatch Health Improvement Institute
11
2C Contents of Ratings Instrument
I Website transparency/accountability
II Health information editorial policies
III HONcode compliance
IV Raters' feedback
12
2C HWRI, Part I, CRW Principles
Developed/applied by CRW staff Identity
Advertising & sponsorships
Ease of use
Corrections & currency
Privacy
13
2C HWRI, Part II, HII principles
A. Criteria/descriptors – Characteristics of website
contents– Accessibility of contents to
consumers– Editorial policies/procedures– Authors of articles/contents– Articles– Summary scores
Optional – Raters could also complete Part-
I (to facilitate construction of evaluative narratives)
B. Evaluative narratives – Purpose/scope & intended
audience– Characteristics of website– Accessibility– Editorial policies/procedures– Contents– Website's greatest strengths– Website's greatest
weaknesses – Utility of website to
consumers
Developed by HII; applied by volunteers
14
2D Credential/Select Raters
Process Call for raters Request applicants present their
credentials Submit applicants' credentials to raters
credentialing committee Credential health website raters Select credentialed raters for panel
– Type of organization in which employed currently
– Role– Professional background– Region of country
Credentialing criteria Health professional
At least 5 years continuous relevant experience
Currently active in health field
Sufficient qualifications/experience to evaluate health websites
No apparent disqualifying event
After initial credentialing, satisfactory performance as rater
15
2E Rate Websites
CRW staff applied HWRI, Part I
Panel of HII volunteers; each applied HWRI– Raters signed HII "Policy on Conflict of Interest“
– If conflict of interest, website reassigned
HII summarized individual panel members' scores/assessments to produce coordinated ratings
HII/CRW integrated ratings for website display
CRW created ratings website displays
16
2F Create Ratings Website
Scope —website contains – Introductory/explanatory
material– Description of methods– Disclosures/disclaimers– Ratings page for each
rated website
Ratings include – Website's stated purpose– HII-CRW description of
website– Global/attribute scores– Greatest
strengths/weaknesses– Noteworthy items
Purpose/objectives– To display independent health website ratings in consumer-
friendly way
– To solicit feedback
17
3 Results
Rating health websites
Rated websites
Median attribute ratings
Distribution of ratings
Excellent top-20 health websites
Example of ratings webpage
Ratings are accessible on www.healthratings.org
18
3A Rating Health Websites
Process 20 most-visited health websites Rated for
– Transparency, 2 CRW raters
– Editorial policies, HII raters
14 HII raters (3-member panels)– 2 health practitioners
– 5 health information experts
– 2 health education specialists
– 5 media, production & related
Resultant ratings– Excellent– Very good– Good– Fair– Poor
Attributes Identity
Advertising & sponsorship
Ease of use
Corrections & currency
Privacy
Design
Coverage
Accessibility (navigation/reading level)
Contents
Overall rating
19
3B Rated Websites (20 most-visited)
webmd.com nih.gov health.yahoo.com about.com/health mayoclinic.com medicinenet.com emedicine.com drugs.com intelihealth.com pfizer.com
realage.com kidshealth.org rxlist.com qualityhealth.com healthology.com health.ivillage.com medscape.com heartcenteronline.com healthboards.com healthsquare.com
20
3C Median Attribute Ratings
Identity - Excellent Advertising & sponsorship - Excellent Ease of use - Good Corrections & currency - Fair Privacy – Excellent Design – Good Coverage – Very good Accessibility – Very good Contents – Very good
21
3D Distribution of Ratings
0
1
8
5
6 (30%)
(25%)
(40%)
(6%)
(0%)
Excellent
Very Good
Good
Fair
Poor
Top 20 Health Websites
22
3E Excellent Top-20 Health Websites
emedicine.net
kidshealth.org
mayoclinic.com
medscape.com
nih.gov
webmd.com
Ratings are accessible on www.healthratings.org
24
4 Conclusions
Designing, producing & updating excellent health information websites is costly, complex, challenging
Rating websites is equally challenging, but desirable & feasible
6 of 20 most visited health websites, were rated “excellent” overall
Quality of information is limited by state of medical science
25
4A Conclusions: Needed improvements
Generally, rated websites need to improve– Contents - descriptions of editorial policies; also policies
& procedures• Describe how select topics, search/grade information,
develop contents, assure quality of articles/contents• Name authors/reviewers• Provide authors/reviewers’ credentials; must be appropriate
to contents; disclose financial/other interests• State date last reviewed/updated• Refer to sources of facts/citations• Indicate criteria for linking to other websites
– Design & Ease of use
– Accessibility - especially for consumers with special needs
– Currency & corrections