IAOS 2014 Da Nang: An agile approach to question testing and satisfying a new data requirement

Pete Brodie

ONS, UK

Overview

• Background• Drivers• Questionnaire requirements• Delivery dilemma• Solution and outcome• New output requirement• “Agile” methodology• Outcome• Results• Next steps• Conclusion

Background

• ONS delivers a very broad spectrum of statistics

• Financial pressures on all NSIs• Difficult to react quickly to new

requirements• Scarcity of specialist skills

Drivers

• Changes due to ESA 10 and BPM6 - updated regulations for producing National Accounts

• Changes in the labour market – the way people are employed

Questionnaire requirements

• Early in 2014 legislative changes required substantial changes to most ONS business statistics

• First tranch of 28 Finance Surveys• Total redesign of questionnaires• Additional questions• Covering new sectors of the economy

Delivery dilemma

• Standard approach for DCM uses multiple waves of cognitive interviews with responders

• Transcription of interviews• Analysis using thematic framework • Detailed report of issues and

recommendations• Time and available resource – impossible!

Solution and outcome I

• Brand new approach needed trialled for two surveys

• Condensed usual waves of testing into one week “stints”

• Three stages to each stint

Monday Tuesday Wednesday Thursday Friday

Stage 1 Stage 2 Stage 3

Fieldwork Collaborative meetings – DCM, Clients, Stakeholders

Changes agreed

Solution and outcome II

• Traffic light document

Solution and outcome III

• Extensive changes implemented for two surveys – agreed between all stakeholders

• Real improvements made• All questions rated “green” at end• Client feedback very positive• Re-use methodology with some tweaks

New output requirements

• Autumn 2013 demand for measurement of the number of employees on “Zero Hours” Contracts

• Definition unclear• No information from businesses• Promise to produce estimates early in 2014• Conventional approach of qualitative and

quantitative research likely to take around three years

“Agile” methodology I

• Staged approach – pilot followed by live survey

• Combined qualitative and quantitative work

• Novel (for ONS) approach of telephone survey of businesses

• Used flexible “Statistical Computing” for all quantitative production

“Agile” methodology II

• Pilot survey of around 1000 businesses (30 strata)• Supplemented frame information with contact

numbers• Collaborative working between different parts of

the office• In field for four weeks achieved 60% response• Qualitative interviews with selected responders• Bespoke code for editing, outlier detection and

estimation

“Agile” methodology III

• Quantitative estimates from pilot used to optimise survey design for main survey – aim for 10% cv at top level

• Questionnaires updated using qualitative research

• Main survey of 5000 businesses included all pilot sample and 4000 new members

• In field for six weeks achieved over 60% response

Outcome

• Published results on 30th April – first business survey estimate of the number of live contracts of 1.4 million (c.v. 8.5%)

• Greater understanding of term -“contracts with no guaranteed minimum hours”

• Comparison with estimates from Labour Force Survey

• Published some more detailed estimates• Some data available for longitudinal analysis

Results I

Proportion of business using ZHC by size

Results II

Proportion by broad industry

Next Steps

• The LFS questionnaire was updated in the light of findings and to answer detailed demographic questions

• Follow up business survey of 5000 now in the field

• Considering longitudinal design• Further embedded qualitative work• Considering longer term strategy

Conclusions

• These two examples illustrate what is possible when using an agile approach.

• Too often constrained by artificial boundaries.

• Opportunities for development of staff and methods through collaborative working.

• Necessity is the mother of invention!

Thank you!

Any questions ?