LQAS: Pitfalls, Controversies, and Addressing Concerns

CORE Spring Meeting

Jennifer Luna; Todd Nitkin; Bill Yaggy

May 10, 2011

LQAS is useful

However, common mistakes can lead to problems including:

Not collecting adequate information

Incorrectly interpreting findings

Improperly describing actions taken as a result of findings

LQAS session

Purpose: Discuss common mistakes and practical ways to avoid them

Agenda:

Overview of LQAS, resources and discussion of current controversies

Experience, advice from AMREF

Practical advice from MTI

• Small group work

• Brief plenary on results of small group work

LQAS

Origin in quality control for industry light bulbs

Promoted for use in small population based health surveys

Useful for monitoring because it provides information used to identify sub-divisions (supervision areas) of project area that are sub-standard More attention can be given to these areas.

Does not provide coverage levels for supervision areas

Steps

Divide project area into supervision areas (SAs)

Decide on level below which a supervision area is identified as priority for special attention; e.g. <85%

Randomly select sample of 19 per SA

Use a decision rule table to determine what number of correct responses is the cut-off for identifying SAs that need priority attention

<20%, <1 correct response out of 19

<50%, <7 correct responses out of 19

85%, <14 correct responses out of 19

Steps

Indicators must be dichotomous (yes/no)

Was child exclusively breastfed?

Not for how many months was the child exclusively breastfed?

Supervision Area (SA)

“Lots” that you divide project area into

Should have meaning:

• i.e. health facility catchment areas, sub-districts or other MOH administrative areas

A B

C D

E

Optimal LQAS Decision Rules for Sample Sizes of 12-40 for Targets of 10-95%

Targets (for Monitoring and Evaluation)

Sample Size 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%

12 na na 1 1 2 2 3 4 5 5 6 7 7 8 8 9 10 11

13 na na 1 1 2 3 3 4 5 6 6 7 8 8 9 10 11 11

14 na na 1 1 2 3 4 4 5 6 7 8 8 9 10 11 11 12

15 na na 1 2 2 3 4 5 6 6 7 8 9 10 10 11 12 13

16 na na 1 2 2 3 4 5 6 7 8 9 9 10 11 12 13 14

17 na na 1 2 2 3 4 5 6 7 8 9 10 11 12 13 14 15

18 na na 1 2 2 3 5 6 7 8 9 10 11 11 12 13 14 16

19 na na 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

20 na na 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17

21 na na 1 2 3 4 5 6 8 9 10 11 12 13 14 16 17 18

22 na na 1 2 3 4 5 7 8 9 10 12 13 14 15 16 18 19

23 na na 1 2 3 4 6 7 8 10 11 12 13 14 16 17 18 20

24 na na 1 2 3 4 6 7 9 10 11 13 14 15 16 18 19 21

25 na 1 2 2 4 5 6 8 9 10 12 13 14 16 17 18 20 21

26 na 1 2 3 4 5 6 8 9 11 12 14 15 16 18 19 21 22

27 na 1 2 3 4 5 7 8 10 11 13 14 15 17 18 20 21 23

28 na 1 2 3 4 5 7 8 10 12 13 15 16 18 19 21 22 24

29 na 1 2 3 4 5 7 9 10 12 13 15 17 18 20 21 23 25

30 na 1 2 3 4 5 7 9 11 12 14 16 17 19 20 22 24 26

31 na 1 2 3 4 6 8 9 11 13 14 16 18 19 21 23 24 26

32 na 1 2 3 5 6 8 10 11 13 15 17 18 20 22 23 25 27

33 na 1 2 3 5 6 8 10 12 13 15 17 19 21 22 24 26 28

34 na 1 2 4 5 6 8 10 12 14 16 18 19 21 23 25 27 29

35 na 1 3 4 5 6 8 10 12 14 16 18 20 22 24 26 28 30

36 na 1 3 4 5 6 9 11 13 15 17 19 20 22 24 26 28 31

37 1 2 3 4 6 7 9 11 13 15 17 19 21 23 25 27 29 31

38 1 2 3 4 6 7 9 11 13 15 17 20 22 24 26 28 30 32

39 1 2 3 5 6 7 9 12 14 16 18 20 22 24 26 28 31 33

40 1 2 3 5 6 7 10 12 14 16 18 21 23 25 27 29 32 34

na: not applicable, meaning LQAS cannot be used in this assessment because the target is either too low or too high to assess a lot.

Notes: α and β errors < 10% for all decision rules except where noted. Lightly shaded cells indicate where α or β errors are ≥10%.

Darker cells indicate where α or β errors are ≥15%.

Parallel Sampling

Conducting multiple surveys at the same time using the same logistic system

Necessary for surveys that collect information on indicators with different age groups as denominators: i.e. IYCF (denominator children 6-23 months); use of ORS for diarrhea treatment (denominator

children 0-23 months with diarrhea in the last two weeks)

Parallel sampling ensures that: Each indicator will have 19 responses/supervision

area

LQAS – Coverage estimates

Information from supervision areas combined to obtain coverage for entire project area

LQAS with information combined from multiple SAs is a form of stratified sampling where supervision areas are strata

Requires a weighted average of all supervision areas; confidence intervals

Total of 95 responses (5x19 = 95)

Remember parallel sampling

LQAS

Useful for monitoring of implementation

Can be controversial

Best not to use for evaluation

Controversy - Example

Concern raised that use of LQAS to identify SAs as having reached the target, might result in mistakenly identifying an SA as one in for which resources can be reduced.

If this is a mistaken classification, then the population will not receive needed resources

American Statistical Association (ASA) Recommendation

Report to the American Statistical Association Board of Directors on Lot Quality Assurance Sampling, August 19, 2010

“We believe that LQAS conclusions should be carefully stated to ensure that the user will never mistakenly draw a conclusion about the performance of the SA. We propose the following as a better interpretation of the conclusions that can be drawn from LQAS:

Based on considerations related to the power of the test employed, given the sample size used:

ASA recommendation continued

If 13 or more of the 19 samples of men in a supervision area (SA) can correctly name the ways to prevent sexual transmission of HIV, then we classify the SA as not requiring priority intervention at this time or

If 12 or fewer of the 19 sampled men in the SA can correctly name the ways to prevent sexual transmission of HIV, then we classify the SA as substandard requiring immediate intervention.

Articles

Rhoda, Dale A., Soledad A. Fernandez, David J. Fitch, and Stanley Lemeshow (2010), LQAS: User Beware. International Journal of Epidemiology, 39:60-68

Olives, C., M. Pagano, and J.J. Valadez (2010), Commentary: Understanding practical lot quality assurance sampling. International Journal of Epidemiology, 39: 69-71.

Resources:

MCHIP NGO/PVO support website: www.mchipngo.net

Rapid Health Surveys Handbook (Public Health Institute)

CORE website: (www.coregroup.org)

KPC Trainer of Survey Trainers (TOAST)

LQAS trainers guides

LQAS protocol for parallel sampling

LQAS FAQs