1

Quality and purpose: quality in quantitative methods

Angela Dale

University of Manchester

2

Quality in survey methods

• A survey is of adequate quality if it can answer specified question with an adequate degree of accuracy– it is fit for purpose

• If it can provide an answer at a 1% level of confidence when only 5% is required, then it may not be cost effective

3

Standards for surveys

• Random sample – best bet of avoiding bias in sample; but need adequate sampling fame

• Adequate numbers for required subgroups – depends on level of accuracy needed

• Good response rate – but extent of bias is crucial

• Questionnaire must be well developed• well designed questions; good flow; salient

4

Can a scoring system work?

• A scoring system can indicate overall strength on all these dimensions, but: – Higher quality costs more– Increased spend may have diminishing returns

• But there will be a level below which a survey would have very low value/low credibility

• Need to find a way of assessing quality against requirement

5

Analysis issues

• Results based on analysis need to reflect the quality of the survey– Importance of making clear the level of accuracy

of the results

• The quality of the analysis is also of great importance– An excellent survey may be badly analysed ,

• methods may be used poorly; interpretations incorrect• inappropriate assumptions about causality may be

made• Claims may go beyond what the data can support

6

Can a poor survey have value?

– A very poor survey may still have value:• if analysed with care and weaknesses

recognised• If only very limited conclusions, that can be

justified, are drawn

– Where there is no better alternative

7

Conclusions

• It is of value to have a reference framework for assessing quality

• But equally important to use it critically and with care

• The key to quality lies not just in good data but in ensuring that claims made can be supported by the data