The Use of Non-Probability Samples to Characterize Rare Conditions

John M. Boyle, Ph.D. ICF International

Federal Committee on Statistical Methodology

November 2013

icfi.com | 1

Rare Diseases

Rare Diseases are generally defined by population prevalence, which differ somewhat across countries

United States: a rare disease is one that affects less than 200,000 persons (1 in 1,500 )

There are an estimated 5,000 to 8,000 rare diseases

The majority of these conditions are serious and potentially life-threatening

An estimated 10% of the U.S. population is affected by one or more rare diseases

In summary, rare diseases are important to public health due to the number of persons affected in the aggregate and the impact of the conditions on the affected individuals and the health care system

icfi.com | 2

Evidenced Based Medicine

Evidence based medicine (EBM) is currently treated as the best tool or gold standard to validate clinical decisions about the care of patients, individually or as a whole

Evidence based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care off individual patients

The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.

Evidence based medicine is “derived from high quality research on population samples to inform clinical decision-making”

Evidence based medicine evolved from clinical epidemiology --- using population health sciences to inform clinical practice

icfi.com | 3

Evaluating Evidence in EBM

Evidence quality can be assessed based on the source type from meta-analyses and systematic reviews of triple-blind randomized clinical trials down to conventional wisdom at the bottom

Good quality, reliable evidence does not always have to come from clinical trials

High quality evidence has good internal and external validity

Critical appraisal of evidence for internal validity that can be broken down into aspects regarding: Systematic errors as a result of selection bias, information bias and confounding

Quantitative aspects of diagnosis and treatment

The effect size and aspects regarding its precision

Clinical importance of results

External validity or generalizability

icfi.com | 4

Primary Immune Deficiency Diseases

Primary immune deficiency diseases (PIDD) is a class of approximately 150 specific disorders

Immune deficiency diseases both individually and as a class of disorders are recognized as rare diseases (NIH Office of Rare Diseases)

Library of Medicine abstract: “X-Linked Agamma-globulinemia Marks the Spot: Rare Diseases in Evidenced-Based Practice”

Uniquely, the Immune Deficiency Foundation commissioned a national probability sample of 10,000 households sampled by random digit dialing and screened by telephone in 2005 Identified 18 households and 23 persons with a PIDD diagnosis Estimated prevalence for the class would be 1/1,200 persons

icfi.com | 5

Evidence-Based Medicine and Treatment

EMB is usually used in the context of treatment decisions (treat or not)

Random control trials are gold standard for assessment of efficacy and safety

Rare diseases are a challenge for RCT in approval of new therapies

BUT what about assessing treatment and treatment outcomes after FDA approval?

The majority of patients with PIDD have antibody disorders for which immunoglobulin (IgG) therapy is the standard of care

icfi.com | 6

icfi.com |

Current IgG Use: IDF Patient Surveys and National Prevalence Survey

70% 67%74%

22%

0%

20%

40%

60%

80%

100%

1996 IDF Patient 2002 IDF Patient 2008 IDF Patient 2005 Prevalence

Sources: IDF Primary Immune Deficiency Diseases in America: 1996, 2002 & 2008, 2005 IDF National Prevalence Survey

icfi.com | 7

2010 National PIDD Treatment Survey

Characterize treatment with immunoglobulin replacement therapy among a national community-based sample of PIDD patients Obtain a sample of at least 100 diagnosed PID

patients to provide more stable estimates of immunoglobulin use Cost of probability survey would be prohibitive Member based survey may be biased toward treatment

Large web panel of the general public to screen for a nationally representative sample of persons diagnosed with primary immune deficiency diseases may offer a less biased, non-probability source of information on treatment

icfi.com | 8

Web Survey Study Procedures

Internet panel company sends invitations to samples of web panel members to participate in a survey

Interested panel members go to a “requirements page” where they self-identify their eligibility for the survey

“Eligible” respondents are given the link to the IDF survey website to conduct the treatment interview

These respondents must answer a more detailed set of questions to determine whether they are eligible for the survey

Respondents must have a member of their household or an immediate family member living outside of their household to be eligible for the survey

The household or immediate family member must have a specific appropriate diagnoses of primary immune deficiency diseases

Those respondents with eligible household or immediate family members are interviewed on-line concerning the PID patient’s disease and its treatment

icfi.com | 9

icfi.com | icfi.com | 10

E-mail Invitation to participate in new survey opportunity N=859,379

Went to web screener = 13% N=114,934 Did not respond= 87% N=744,445

No One Qualified in Household or

Family = 97%

N=111,447

Anyone in household or immediate family diagnosed

with XLA, CVID, IgA def., IgG def., SCID or other

immune deficiency? Yes=3% N=3,487

Did not go to Main Questionnaire

No=51% N=1785

Went to main questionnaire Yes=49% N=1,702

Has anyone in your household/immediate family ever been

diagnosed with a primary immune deficiency disease? What

specific types of primary immune deficiency has the (AGE)

been diagnosed with?

Good PIDD Diagnosis=9% N=144

No answer to screen 10%

No PIDD in HH/Family 47%

No Diagnosis given 6%

Bad Diagnosis given 28%

PIDD Web Survey

Goal: Obtain a sample of at least 100 diagnosed PID patients to provide more stable estimates of immunoglobulin use

Result: Web panel yielded completed interviews with 147 eligible respondents reporting on 160 PIDD

Result: Web panel yielded interviews with119 patients with antibody disorders for which IgG treatment is recommended

Result: Web panel found relatively low levels of IgG treatment in a national community based sample

Problem: How do we evaluate the quality of the evidence from a non-probability sample such as a web panel?

icfi.com | 11

External Validity

Geographic distribution --- compared to Census population distribution

Prevalence --- compared to 2005 National Prevalence Survey (RDD)

Treatment --- compared to 2005 National Prevalence Survey (RDD)

Health --- compared to large, IDF member surveys (non-probability, but not

convenience sample)

icfi.com | 12

External Validity: Distribution of Population and Patients by Region

5%

15%17%

7%

18%

6%

11%

6%

16%

6%

16%

8%

17%

6%6%

17%16%

6%

0%2%4%

6%8%

10%12%14%

16%18%20%

NewEngland

MidAtlantic E NorthCentral

W NorthCentral

S Atlantic E SouthCentral

W SouthCentral

Mountain Pacific

US Population Patients web

Patients in 2007 based on state of birth. Patients in web survey based on state of residence from sample file. Base: One or more eligible PIDD and completed interview N=144 (3 missing data) icfi.com | 13

External Validity: Household Prevalence of PIDD

144 respondents have a qualified PIDD diagnosis in their household in web survey

114,934 panelists saw the screening questionnaire so 1 out of 798 households prevalence in web survey If the distribution of eligible diagnoses was the same for those

who qualified on the refinement question and did not go to the web survey, we would expect 295 eligible respondents for a household prevalence rate of

1 in 390 households estimated prevalence in web survey Estimated household rate for diagnosed PIDD was

1 in 555 households in RDD probability survey

icfi.com | 14

Compare with a Probability Survey: Past and Current IG Treatment

44%

36%

29%

22%

0%5%

10%15%20%25%30%35%40%45%50%

IVIG 2005 RDD N=23 IVIG 2010 Web N=160

Ever Current

Q8a. Has the (AGE) ever been treated for a month or longer with the following? Q8c. Is the (AGE) currently being treated with any of the following? All living eligible PIDD N=160

icfi.com | 15

Compare with Other Non-Probability Survey: Current Health Status

7% 8%

33%

37%

12%

4%8%

21%

30%28%

10%

2%

0%5%

10%15%

20%25%

30%35%

40%

Excellent Very good Good Fair Poor Very poor

2010 Web 2002 Member

Q23. Would you describe the (AGE)’s current health status as …? Base: Selected PIDD patient N=147

icfi.com | 16

Compare to Other Non-Probability Survey: Limitations due to Health

17%

33% 34%

16%

30% 29%

25%

14%

00.050.1

0.150.2

0.250.3

0.350.4

No limitation Slightlimitation

Moderatelimitation

Severelimitation

2010 Web 2002 Member

Q24. How much, if any, is the (AGE) limited in work, play or normal physical activity as a result of his/her health? Base: Selected PIDD patient N=147

icfi.com | 17

Compare with Other Non-Probability Survey: Acute Conditions in Previous 12 Months

45%

17%

34%

25%

16%

3%

8%

3%

17%

2%

17%

36%

14%

37%

24%

11%

5%

6%

3%

20%

5%

25%

0% 10% 20% 30% 40% 50%

Bronchitis

Candida

Repeated Diarrhea

Repated Ear Infections

Eye Infections

Lymphopenia

Malabsorbtion

Neutropenia

Pneumonia

Sepsis

Urinary Infection

2010 Web Survey2002 Member Survey

Q25. Which of the following conditions, if any, has the (AGE) had during the past 12 months? Base: Selected PIDD patient N=147

icfi.com | 18

Internal Validity

Disease specific eligibility

Correct diagnoses by clinical expectations

Incorrect diagnoses by type

Location of treatment by treatment level

Contact with patient organization by treatment level

icfi.com | 19

Disease Specific Eligibility in Web: Minimizing Fraud and Error

Must report household or immediate family member with primary immune deficiency disease

Must select one or more legitimate diagnosis of primary immune deficiency disease from pre-coded list of 18 legitimate and 13 non-legitimate conditions plus one “other” category on two screens

Must report one legitimate condition, and

Not more than 5 persons in household (or immediate family) with PIDD conditions

Not three or more PIDD conditions for any person

Not two rare, non-PIDD conditions placed at beginning of list

Not combinations of PIDD conditions that were improbable

icfi.com | 20

Internal Validity: Specific PIDD Diagnoses

5%1%

3%13%

27%3%

1%1%

2%12%

14%6%

3%5%

1%1%

2%

0% 5% 10% 15% 20% 25% 30%

AgammaAtaxia Tel

CGD

CombinedCVID

Complement

DiGeorgeAngiodema

Hyper IgM

IgG subclassSelective IgA

SCID

SCNSAD

W-A

XLPMixed

Q5a/b. What specific types of primary immune deficiency has the (AGE) been diagnosed with? Base: Selected PIDD patient N=147 (Conditions using IgG in blue)

icfi.com | 21

-I

I

I

I

I . I

I

I

I

I

I

I I I I I

Internal Validity: Non PIDD Diagnoses Reported as PIDD

21%18%18%

11%9%

9%5%

5%5%

5%4%

3%2%

2%2%2%

23%2%

0% 5% 10% 15% 20% 25%

Fibromyalgia

Lupus

Rheumatoid Arthritis

Crohn or IBS

MS

Diabetes

AIDS/HIV

Sjogrens

Hashimoto's

Cancer/Leukemia

Chronic Fatigue Syndrome

ITP

Sarcodosis

CIDP

Hepatitis C

Autoimmune Hemolytic Anemia

Other

Not sure

Q5a/b. What specific types of primary immune deficiency has the (AGE) been diagnosed with? Base: PIDD Reported but Only Non-PIDD Diagnosis Given N=244 Auto-immune in Green

icfi.com | 22

http:icfi.com

Ever and Current Treatment with Immunoglobulin by Immunologist Location

67% 69%61%

25%

0%

60%

44%

20%

0%

61%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Universityhospital

Non-universityhospital

Privatepractice/HMO

None Not sure

Ever Current

Q8a. Has the (AGE) ever been treated for a month or longer with the following? Q8c. Is the (AGE) currently being treated with any of the following? Base: Selected PIDD patient N=147

icfi.com | 23

Internal Validity: Contact with IDF

5%

9%

22%

14%

8%

2%

2%

6%

1%

59%

0% 10% 20% 30% 40% 50% 60% 70%

Called them

Get newsletter

Visited website

Seen patient information

Told about by others

Other

Only this survey

No contact

Not sure

Never/not sure heard of them

Q41b. What kinds of contact, if any, have you had with the Immune Deficiency Foundation? Base: Selected PIDD patient N=147

icfi.com | 24

http:icfi.com

Internal Validity: Treatment with Immunoglobulin by IDF Contact

100% 92% 92% Ever Current 90%

80%

70% 62%60%

49% 51%50% 41%40%

30%

20%

10%

0%Receive Newsletter Heard of IDF/No Never head of IDF

newsletterQ8a. Has the (AGE) ever been treated for a month or longer with the following? Q8c. Is the (AGE) currently being treated with any of the following? Base: Selected PIDD patient N=147

icfi.com | 25

Evaluating a Non-Probability Sample for Data Quality

Patients distributed nationally similar to Census Household prevalence of diagnosed PIDD in web panel

equivalent to RDD telephone survey Treatment rates similar in web and RDD surveys Correct diagnoses distributed to clinical expectations Wrong diagnoses primarily auto-immune diseases and similar

to probability survey (not presented here) Disease characteristics of web panel patients similar to

previous member surveys Treatment rates vary by immunologist location Only a minority have heard of IDF and less than one in ten

would be in the IDF database Treatment rates of patients in web survey who receive IDF

newsletter similar to IDF member surveys

icfi.com | 26

Conclusions

Incidence of rare diseases make population estimates of treatment and treatment outcomes infeasible using probability samples

National web population panel was able to generate a national sample of a rare disease of usable size

Web panel sample was compared to Census, probability surveys, other non-probability surveys for external validity

Web panel sample was analyzed for internal validity for indications of fraud, error and internal consistency

Findings from this web survey of a rare disease appear to be internally and externally consistent

These estimates confirm a potential problem in treatment in the general community that was suggested by probability survey

Carefully constructed and conducted surveys using an appropriate web panel can provide credible measures of the treatment of a rare disease in the community

Can we develop a set of procedures for evaluating the quality of non-probability samples for rare diseases that would permit their use in evidence-based medicine?

icfi.com | 27

Thank You

John M. Boyle, Ph.D.

Lead, Survey Research Line of Business

ICF International

john.boyle@icfi.com

icfi.com | 28

mailto:john.boyle@icfi.com

The Use of Non-Probability Samples to Characterize Rare ConditionsRare DiseasesEvidenced Based MedicineEvaluating Evidence in EBMPrimary Immune Deficiency DiseasesEvidence-Based Medicine and TreatmentCurrent IgG Use: IDF Patient Surveys and National Prevalence Survey2010 National PIDD Treatment SurveyWeb Survey Study ProceduresPIDD Web SurveyExternal ValidityExternal Validity: Distribution of Population and Patients by RegionExternal Validity: Household Prevalence of PIDDCompare with a Probability Survey: Past and Current IG TreatmentCompare with Other Non-Probability Survey: Current Health StatusCompare to Other Non-Probability Survey: Limitations due to HealthCompare with Other Non-Probability Survey: Acute Conditions in Previous 12 MonthsInternal ValidityDisease Specific Eligibility in Web: Minimizing Fraud and ErrorInternal Validity: Specific PIDD DiagnosesInternal Validity: Non PIDD Diagnoses Reported as PIDDEver and Current Treatment with Immunoglobulin by Immunologist LocationInternal Validity: Contact with IDFInternal Validity: Treatment with Immunoglobulin by IDF ContactEvaluating a Non-Probability Sample for Data QualityConclusionsThank You