Designing Health-Related Quality of Life Measures

Ron D. Hays, Ph.D.

February 2, 2015 (9:00-11:50 am)

HPM 214, Los Angeles, CA

Comprehensive Geriatric Assessment (n = 363 community-dwelling older persons) lead to improvements in SF-36 energy, social functioning, and

Physical functioning (4.69 points) in 64 weeks Cost of $746 over 5 years beyond control group

Keeler, E. B., et al. Cost-effectiveness of outpatient geriatric assessment with an intervention to increase adherence. Med Care, 1999, 37 (12), 1199-1206.

Ad Hoc Preference Score Estimates

Is CGA worth paying for?

Change in QALYs associated with 4.69 change in SF-36 physical functioning scale

r = 0.69 -> b = .003 QWB = 4.69 x .003 = .014 .014 x 5 yrs. = 0.07 QALYs Cost/QALY: $10,600+

<$20,000 per QALY worthwhile

[0.69x (0.103)/(23.5) = .003]

Listed below are a few statements about your relationships with others. How much is each statement TRUE or FALSE for you?

1. I am always courteous even to people who are disagreeable.2. There have been occasions when I took advantage of someone.3. I sometimes try to get even rather than forgive and forget.4. I sometimes feel resentful when I don’t get my way.5. No matter who I’m talking to, I’m always a good listener.

Definitely true; Mostly true; Don’t know; Mostly false; Definitely false

Give yourself 1 point for each item that you answered as shown below

(Possible score range is 0-5)

- I am always courteous even to people who are disagreeable. DEFINITELY TRUE

- There have been occasions when I took advantage of someone. DEFINITELY FALSE

- I sometimes try to get even rather than forgive and forget. DEFINITELY FALSE

- I sometimes feel resentful when I don’t get my - way. DEFINITELY FALSE- No matter who I’m talking to, I’m always a good

listener. DEFINITELY TRUE

How many people scored?

• 0 points? _______

• 1 point? _______

• 2 points? _______

• 3 points? _______

• 4 points? _______

• 5 points? _______

• Mean was 1-2 points in MOS

7

End goal is measure that is “Psychometrically Sound”

• Same people get same scores

• Different people get different scores and differ in the way you expect

• Measure works the same way for different groups (age, gender, race/ethnicity)

• Measure is practical

8

First law of survey development:

Only do it when necessary

9

http://www.ispor.org/TaskForces/PROInstrumentsUse.asp Developing a new PRO instrument is a labor and time intensive task.

Use of existing instruments is generally preferable to developing a new instrument or modifying an instrument.

If a PRO instrument is modified, additional validation studies may be needed to confirm the adequacy of the modified instrument’s measurement properties. The extent of additional validation recommended depends on the type of modification made.

The FDA intends to consider a modified instrument as a different instrument from the original and will consider measurement properties to be version-specific.

http://www.ispor.org/workpaper/FDA%20PRO%20Guidance.pdf

PRO Iterative Development Process

10

11

Documentation

+ Chronology of all item development activities+ Protocols for qualitative interviews, focus groups, cognitive interviewsand other research used to identify concepts, generate items, or revisean existing instrument, including training of interviewers+ Development of response options, modes of administration and scoring+ Size, characteristics, location, and (if requested) transcripts of eachqualitative interview and focus group+ Documentation on how saturation was achieved (i.e. no new informationwas obtained from additional qualitative interviews or focus groups)+ Description of any pilot test, including cognitive interviewing, cognitiveinterview transcripts (if requested)+ Versions of the instrument at various milestones of development+ Item tracking table that list the source of each item in the final instrument, and how it changed during development+ A summary statement of qualitative research in support of contentvalidity of the PRO instrument

D. Patrick et al, Value in Health 2007, 10, S125-37

12

Second law: Know thy respondent

Chapter 1.2 (Elaine McColl)

“seeking lay input on the meaning of health and the impact on quality of life of living with an index condition is now becoming the norm. A further advantage of lay involvement is that the developer can gain a greater insight into the language and terminology used by the target respondents, and can thus ensure that the vocabulary used in the instrument is culturally appropriate.”

13

14

Measurement Steps

• Review literature

• Focus groups

• Define constructs

• Draft items

• Pretest (cognitive interviews)

• Revise and finalize items

15

Literature ReviewEdgell, E. T., et al. (1996). A review of health-related quality-of-life

measures used in end-stage renal disease. Clinical Therapeutics, 18, 887-938.

Coons, S. J., et al. (2000). A comparative review of generic quality of life instruments. PharmacoEconomics, 17, 13-35.

Golomb, B., Vickrey, G., & Hays, R. D. (2001). A review of health-related quality-of-life measures in stroke. PharmacoEconomics, 19, 155-185.

Castle, N.G., Brown, J., Hepner, K. A., & Hays, R. D. (2005). Review of the literature on survey instruments used to collect data on hospital patients’ perceptions of care. Health Services Research, 40 (6 Pt. 2), 1996-2017 Shen, J., Johnston, M., & Hays, R. D. (2011). Asthma outcome measures. Expert Reviews of Pharmacoeconomic and Outcomes Research, 11, 447-453

16

Search Internet

http://www.library.ucla.edu/libraries/biomed

http://www.ingentaconnect.com/

http://www.google.com/

http://www.rand.org/health/surveys_tools.html

http://www.sf36.com/

http://www.proqolid.org/

http://healthmeasurement.org/

http://chime.med.ucla.edu/

http://www.ncbi.nlm.nih.gov/pubmed/

17

Focus Groups

• Discuss feelings, attitudes, perceptions• Learn

– Vocabulary and thinking patterns

• Conversational meeting– Moderator and 6-12 people– Questions posed– Group synergy– Economical

18

Focus Group Logistics

• HSPC/IRB• Recruitment and payment• Moderator and note-taker• Visitors• Audio-record (backup)• Transcript• Background questionnaire

Focus group excerpts follow

19

Conservation of Energy

• Because of Hepatitis C, Jenny has had to set “boundaries” with people. For example, she has had to close her office door and say “no” at times.

• She has had to figure out how much energy she has to get to the finishing line. “If I don’t have enough energy to get to the finishing line, it means I won’t be available for everyone else.”

rhays:

Example of unique issue that would be missed if survey only asked whether Jenny cut down on work instead of asking whether she adjusts her work behavior to ensure she doesn’t deplete her energy reserve.

Question 14a

rhays:

Example of unique issue that would be missed if survey only asked whether Jenny cut down on work instead of asking whether she adjusts her work behavior to ensure she doesn’t deplete her energy reserve.

Question 14a

20

Longer Recovery Period

• Candy has a full-time job fixing earthquake damage and “needs the rest of the day to be able to do that.”

• She engages in heavy work, but it takes her several hours to recover from it.

Rhays: Question 15d, 16e, 25gRhays: Question 15d, 16e, 25g

21

Work life accommodation

and physical limitations • Manny is a self-employed builder who has owned his

business for 25 years. He goes to bed at about 9:00 and gets up about 5:30 am. He wakes up several times during the night.

• Manny has changed from participating actively on the job site with his employees to only setting up the job and supervision.

Rhays: Quest. 4a-d may not capture this very well.Rhays: Quest. 4a-d may not capture this very well.

22

Symptoms of disease

• Since he went off interferon, Manny has had a lot of muscle fatigue. His legs cramp when he surfs.

• He can’t arm wrestle his 12-year old son, etc. Forcing himself to do even 10 pushups now is difficult.

• Manny has also been getting little pimples on his legs (“cryo globulins”) due to inadequate blood circulation.

Rhays: Questions 12a, b, nRhays: Questions 12a, b, n

23

Side effect of medicines

• When taking interferon Manny said that he tended to get angry and yell and scream at his wife.

• Manny also indicated that he gets a stomach ache daily somewhere between noon and 2 pm whether or not he has eaten. “All of the sudden I feel nauseous.”

Rhays: Questions 14c, 14f, 27aRhays: Questions 14c, 14f, 27a

24

Daily unpredictability; change in relationships

• Tammy lost her best friend as a result of hepatitis C, because the friend didn’t like her last minute cancellations caused by the disease.

• However, it has brought her very close to her spouse—he is very supportive and even “babies” her now.

Rhays: Question 14e, 16d, 26eRhays: Question 14e, 16d, 26e

25

Keeping disease secret from others

• Fred only recently told other people that he had hepatitis C (other than his wife). He told a few people and recommended that they get checked for it themselves.

• He is more comfortable telling people now because he has more information about the disease and can explain what it is.

Rhays: Questions 26e, i, mRhays: Questions 26e, i, m

26

Third law: Practice before you play

“Cut and try, see how it looks and sounds, see how people react to it, and then cut again, and try again” Converse & Presser (1986, p. 78)

Identify problems with

– Comprehension of items (stem/response options)– Retrieval of information– Skip patterns– Response burden

27

Draft Items

28

Target Respondent--Educational Level

• Literacy expert• Readability indices Morales et al. (2001).

Readability of CAHPS 2.0 child and adult core surveys. In M. L. Cynamon & R. A. Kulka (eds), Seventh Conference on Health Survey Research Methods (pp. 83-90). DHSS Publication No. (PHS) 01-1013, Hyattsville, Maryland.

Flesch-Kincaid Grade Level

FK GL = 0.39 * (n of words/n of sentences) + 11.8 * (n of syllables/n of words) – 15.59

• Driven by sentence length and syllables per word

• U.S. school grade level (e.g., 8.0 implies that 8th grader can understand the document).

• Possible minimum = -3.4– Green eggs and ham averages 5.7 words per sentence and 1 syllable per word – (FK GL = -1.3)

Challenges

• Items have fragments such as preamble phrases

• Response choices

In general, would you say your health is:

• FK GL = ?

• Fragment not scored because not recognized by computer as a complete statement, declaration, or question that ends with a period, exclamation mark, or question mark.

In general, would you say your health is excellent, very good, good, fair, or

poor?

• FK GL = 5.9

Items in Grid Format The following questions are about activities you

might do during a typical day. Does your health now limit you in these activities? If so, how much?

1) Vigorous activities, such as running, lifting heavy objects, participating in strenuous sports

Yes, limited a lot; Yes, limited a little; No, not limited at all.

Instructions separately

The following questions are about activities you might do during a typical day. Does your health now limit you in these activities? If so, how much?

• FK GL = 5.9

The following questions are about activities

1 3 2 1 2 4

you might do during a typical day. Does

1 1 1 2 1 3 1 1

your health now limit you in these activities?

1 1 1 2 1 1 1 4

If so, how much?

1 1 1 1

26 words, 3 sentences, 8.6 words/sentence, 40 syllables (0.39 * 8.6) + (11.8 * 40/26) – 15.59 = 5.9

Slightly revised instructions

The following questions are about activities you might do during a typical day. Does your health now limit you in these activities?

• FK GL = 8.0

The following questions are about activities

1 3 2 1 2 4

you might do during a typical day. Does

1 1 1 2 1 3 1 1

your health now limit you in these activities?

1 1 1 2 1 1 1 4

22 words, 2 sentences, 11 words/sentence, 36

syllables (0.39 * 11) + (11.8 * 36/22) – 15.59 = 8.0

ItemVigorous activities, such as running, lifting heavy objects,

participating in strenuous sports

===

Does your health now limit you in vigorous activities, such as running, lifting heavy objects, participating in strenuous sports?

• FK GL = 13.5

EQ-5D Instructions (9.3)

By filling in one circle in each group below, please indicate which statements best describe your own health state today.

Flesch-Kincaid for Items Paz, S. H., Liu, H., Fongwa, M. N., Morales, L. S., & Hays, R. D. (2009).

Readability estimates for commonly used health-related quality of life surveys. Quality of Life Research,18, 889-900. .

Median Range

HUI 9.4 6.1—12.0

QWB 8.5 1.2—12.0

SF-36 5.1 0.6—12.0

EQ-5D 4.4 0.5—10.3

41

Fourth law: Keep it simple and short

42

Use only as many words, items and response options as needed

43

Survey InstructionsThank you for taking the time to fill out this survey. The purpose of this survey

is to learn about your experiences as a cancer patient. The information you provide is very important. It will help to improve cancer services for other patients.

Many of the questions ask about your experiences at your “Cancer Center.” A Cancer Center refers to the hospital, center, or institute where you receive most of your cancer care. A Cancer Center also refers to the doctors, nurses, and other health care professionals who work with the hospital, center, or institute. In some places, the Cancer Center is all in one building. In other places the doctors, nurses, and other health care professionals who work with the Cancer Center are in different locations.

Many of the questions ask about your “Cancer Care Team.” A cancer care team refers to the doctors, nurses, and other health care professionals who provide your cancer care. Your cancer care team might also include social workers, counselors, patient navigators and others who help with your cancer care.

You may feel that some questions are easier to answer and some are harder to answer. Please remember that there are no right or wrong answers! We want to know about your experiences, both good and bad. If you truly do not know the answer to a question, then it is okay to check the box that says “Don’t know.”

44

Modified The purpose of this survey is to learn about your

experiences as a cancer patient, both good and bad.

Some of the questions ask about your “Cancer Center” and others ask about your “Cancer Care Team.”

A Cancer Center is the hospital, center, or institute where you receive most of your cancer care and includes all the health care professionals who work there.

A Cancer Care Team refers to the doctors, nurses, and

other health care professionals (social workers, counselors, patient navigators, etc.) who help provide your cancer care.

45

Use a recall period that is as short as possible, but long

enough • Now

• Last 24 hours

• Last 4 weeks (last month)

• Last 6 months

• Last 12 months

46

3-5 response options is enough

Strongly correlated with items administered using more responses options

Miller, D. G. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychology Review, 2, 81-96.

47

Response options

• None of the time

• A little of the time

• Some of the time

• Most of the time

• All of the time

A good bit of the time

48

Choice of Response Alternatives

• Respondents assume:– Average behavior (attitude) is located in

middle of scale– Extreme behavior located in extremes of

scale.

49

How Tall Is Bond?

• Very tall (+3)• Moderately tall (+2)• Tall (+1)• Short (-1)• Moderately short (-2)• Very short (-3)

(r = 0.94)

50

How Many Hours a Day Do You Watch TV? (A)

• Up to ½ hour

• ½ hour to 1 hour

• 1 hour to 1.5 hours

• 1.5 to 2 hours

• 2 hours to 2.5 hours

• More than 2.5 hours

51

How Many Hours a Day Do You Watch TV? (B)

• Up to 2.5 hours

• 2.5 hours to 3 hours

• 3 hours to 3.5 hours

• 3.5 hours to 4 hours

• 4 hours to 4.5 hours

• More than 4.5 hours

52

Percentage of Sample Who Reported Watching TV > 2.5

Hours Per Day• A: 16%

• B: 38%

PRO Iterative Development Process

53

54

Goals of Pretesting

• Identify potential problems– Item stem or response options– Comprehension– Retrieval of information– Skip patterns– Response burden

55

Cognitive Interviews

• “Think aloud”

• Intermittent probes

• Retrospective recall

56

Think Aloud

• Participant verbalizes what they are thinking when they complete a survey or are interviewed.

57

Intermittent Probe (A)

• Do you have one person you consider to be your personal doctor or nurse? Yes/No

• Is this person a medical doctor, a nurse, or what?

58

Intermittent Probe (B)

• Other than a medical doctor, nurse practitioner, physician’s assistant, nurse, or nurse-midwife, is there anyone else you consider to be a health care professional?

59

Intermittent Probe (C)

Is “health care professional” the way you would refer to this collection of people or would you use different words to describe a medical doctor, nurse practitioner, physician’s assistant, nurse, or nurse-midwife?

60

Intermittent Probe (D)

• Questions 20 and 21 were similar (read both questions again). Which answer choices were easiest for you? Never to always or Poor to Excellent?

• Why?

• Do you think these two questions measure the same thing?

61

Intermittent Probe (E)

• In the last 6 months, how many visits did you make for yourself to a doctor’s office or clinic? (Do not include staying overnight in a hospital, or visits to a dentist.)

• Tell me about these visits. Who did you visit each time? Did you include visits to chiropractors?

62

Intermittent Probe (F)

• In the last 6 months, how often do you think the chiropractors you saw listened carefully to you?

• What do you think of when I said “listened carefully”? Would you have answered differently if I had asked you “how often do you think the chiropractors you saw listened to you?”

63

Retrospective Recall

• Ask probes after the interview or survey is completed.

2nd Class Assignment

• Conduct and summarize 5 cognitive interviews with a self-administered HRQOL survey instrument. Your written summary should be no more than 3 pages in length.

http://chime.med.ucla.edu/qualitative/

http://hpm214.med.ucla.edu/

64

65

After all this upfront work, then you can …

• Collect data

• Analyze data

• Write-up results

PRO Iterative Development Process

PRO Iterative Development Process

68

Fifth law: Believe the survey respondent, but only so much

Listed below are a few statements about your relationships with others. How much is each

statement TRUE or FALSE for you?

- I am always courteous even to people who are disagreeable.

- There have been occasions when I took advantage of someone.

- I sometimes try to get even rather than forgive and forget.

- I sometimes feel resentful when I don’t get my way.

- No matter who I’m talking to, I’m always a good listener.