Experimenting with industrial policy: The UK’s experience of industrial policy making using randomised control trials (RCTs)

Stephen Roper

Enterprise Research Centre and Warwick Business School,Coventry, CV4 7AL, UK

Stephen.Roper@wbs.ac.uk

UK policy context

• 'Dear chief secretary, I'm afraid to tell you there's no money left’. Note left for David Laws new Chief Secretary to the UK Treasury elected in 2010. Started period of ‘austerity’ government

• In policy terms:– Real terms cuts in public spending – Removal of some organisations – notably Regional Development Agencies – And, new emphasis on value for money in policy making

• This latter point has stimulated:– Focus on ‘impact’ in public support for science and social science– Creation of ‘What Works’ Centres…. – More interest in rigorous policy evaluation and RCTs

• (Also potentially important has been the increasing prominence of behavioural economics and the role of the ‘Behavioural Insights team)

More broadly …

• ‘Experimental methods’ of policy evaluation are well-established in education and social policy and development economics.

• Typically such evaluations involve individual human subjects facing some common socio-economic problem, and the random allocation of subjects to a treatment and control group. Differences in outcomes between the treatment and control groups are then attributed to the effect of the policy intervention

• In terms of industrial policy, however, experimental policy evaluation approaches remain marginal, with non-experimental, ex post policy evaluations remaining the norm.

• Potter and Storey (2007), for example, provide an extensive review of OECD best practice without any mention of either the application or potential for experimental methods.

• Similarly, experimental approaches are largely ignored in UK government guidance (BIS, 2009) and evaluating innovation policy (Laredo, 1997)

Aims of paper

• Explore the rationale for RCTs in industrial policy design and implementation

• Consider the recent UK history and the four RCTs implemented to date

• Identify some of the implementation and practical issues involved

• Consider questions of interpretation and the nature of ‘evidence’ for policy evaluation

On experiments …• Experimental methods – based on randomised allocation – offer a way of avoiding

issues of selection rather than correcting ex post (with its attendant difficulties)

• But experiments themselves are subject to series of potential implementation issues which can reduce their ‘validity’ (or ability to provide true representation of a treatment effect

• Threats to internal validity – i.e. treatment v control– Small sample issues (Bruhn and McKenzie, 2009) – Substitution bias (Heckman and Smith , 1995)– Signalling bias (e.g. Meuleman and Maeseneire, 2012)

• Threats to external validity – i.e. in-scheme v not in-scheme– Macro biases (information flows, social interaction, norm formation) (Garfinkel et al.,

1990)– Participation bias (differing characteristics) (Burtless, 1995)– Randomisation bias (Heckman and Smith, 1995).

… and the importance of replication

• Issues related to the validity (either external or internal) suggest the potential value of triangulation or meta-analytical reviews across a number of RCTs

• In medicine the Cochrane Library have established routines for assessing the quality of individual experiments and for synthesis

• In the context of medicine, however, interventions are primarily mechanical: a treatment which works with one human being is very likely to work with another

• In industrial support, however, the situation is more complex given the heterogeneity of firms, the social and interactive nature of business and the importance (and diversity) of the contexts in which firms operate.

• Internal and external validity may therefore be more difficult to maintain in industrial policy RCTs than in medicine.

• Arguably this means that standards of evidence should also be higher, requiring consistent evidence from multiple RCTs from different contexts, before robust conclusions should be drawn

Entrepreneurship and innovation policy shifts in the UK since 2010

• Move towards ‘place-based’ policy in entrepreneurship policy; innovation policy become more national. De-coupling?

• Perhaps key change has been closure of Regional Development Agencies (which used to do both)

• Entrepreneurship policy now Local Economic Partnerships (39 in England); Innovation policy national agency (InnovateUK or TSB)

• Also development of ‘Growth Hubs’ intended to provide/co-ordinate local services for growth companies

• Remember too UK part of EU and so EU initiatives such as Horizon 2020 also potentially important

RCT1: Creative Credits -2010-12• Creative Credits: a B2B voucher scheme

intended to establish innovation partnerships between SMEs and creative services suppliers, e.g. designers. – Manchester City Region – Sept 2009 to

October 2010– 150 Creative Credits worth £4,000 with

£1,000 firm contribution to stimulate collaborative project

– 672 applications, 22 per cent funded, rest to control

• Outcome measures (measured after 6 and 12 months):– Short-term – project additionality – more

projects?– Longer term

• output additionality (sales, innovation), • behavioural additionality (innovation

intention), • network additionality (partnering intention)

Application

Randomised

Creative Credit

150 companies

No Creative Credit

422 companies

Follow-up 12 months

Follow-up 12 months

RCT1: Collecting evaluation data – in practice

• Four surveys over two years so attrition was a problem – used some small financial incentives

• By survey 4. Treatment -78 per cent response (n=117), control group 52.2 per cent (n= 157)

• Clearly bias between groups but comparison of baseline characteristics suggests NO systematic bias within groups

• Longitudinal sample therefore considered ‘representative’ and preserved internal validity

RCT1: On additionality…

• Comparison of treatment and control groups suggest:

• Project additionality - firms receiving Creative Credit were 84 per cent more likely to go ahead with their project – a strong positive effect

• Output additionality – (product/service innovation, process innovation, sales growth) - strong positive effect after 6 months, no effect after 12 months – a transitory effect (and misleading Interim Report)

• Behavioural and network additionality – no significant effects after either 6 or 12 months – a negative result

• Results confirmed by 2-stage Heckman models

RCT2: Growth Vouchers• Launched in January 2014 and ends in March 2015.

Provides subsidised strategic business advice to around 20,000 small businesses through randomly allocated vouchers. Budget £30m.

• Each voucher is worth up to £2,000 excluding VAT. It can be used to cover half the cost of advice in one of five areas: – Raising finance and managing cash flow;– Recruiting and developing staff;– Improving leadership and management skills;– Marketing, attracting and keeping customers; and– Making the most of digital technology.

• Firms not getting a voucher will be given some more general information on other sources of potential support as well as getting access to the website with private suppliers of advice.

• Growth Vouchers may also help to tell us which types of advice are most valuable to firms.

• Plan is to follow up for 2-3 years after the end of the period of support. Programme is targeted at firms which do not usually use business advice.

RCT3: Growth Impact Pilot

• March 2014 March 2015 - Aim to examine the impact of business coaching on growth. Trial linked to major government programme called ‘Growth Accelerator’.

• SMEs provided with subsidised leadership and management training and some then given access to additional business coaching

• Coaching linked to access to finance, business development or innovation

Enrolment

Leadership and management

training

Randomised

Business coaching

C. 300 firms

No business coaching

c. 300 firms

Follow-up 2 years

Follow-up 2 years

RCT4: University Growth Vouchers

• Launching October 2014 – March 2015 with target of 1250 SMEs.

• Aims to test whether firms who receive subsidised leadership and management education training (10 workshops, networking events and mentoring) from six UK Business Schools perform better than those who have received no business advice.

• Growth Vouchers the scheme operates on a co-funding basis with £2000 Growth Voucher per business and the business matching this funding. (Business schools are receiving some additional funding for participation).

• Analysis planned over two years. Careful analysis of attrition to maintain internal validity

Enrolment

Diagnostic Workshop

Randomised

Receive Growth VoucherC. 500

No Growth VoucherC. 750

Follow-up 2 years

Follow-up 2 years

Assessing validity …Internal validity External validity

RCT1: Creative Credits Proven robust Problematic – small regional experiment

RCT2: Growth Vouchers Large numbers should be okay but not proven yet

Larger numbers so should be okay – not proven

RCT3: Growth Impact Pilot Okay. Careful design and allocation

Small sample so may be problematic as RCT1

RCT4: University Growth Vouchers

Okay. Careful design and allocation

Medium sample so may be okay – not proven

Implications for the future• By 2016 early results from all four RCTs. What will this mean?

• Actually hard to know!

• Key questions about whether can preserve internal validity of experiments e.g. sample biases and attrition may be a problem in smaller trials (RCT3, RCT4). External validity potential issue in all trials (again particularly smaller ones)

• English saying: ‘One swallow doesn’t make a summer’. Similarly might also argue that ‘One RCT doesn’t constitute robust evidence’. Do we need replication as the Cochrane Review approach suggests?

• Another key issue is unwarranted generalisation. RCTs can provide good evidence on very specific interventions and delivery mode and that is all. Not generalizable to all business advice, all mentoring etc. Slight changes may improve/upgrade effect. So need to be cautious in interpretation

Conclusions • Creative Credits project (RCT1) did suggest feasibility of RCT approach to

industrial policy:– Randomisation - > internal validity – Longitudinal monitoring -> captured varying output effects – Qualitative work -> generated causal insights

• UK rapidly developing some experience in this evaluation approach both design and operational aspects (and further RCTs planned)

• Studies have yet to contribute to evidence base but potentially powerful addition but ….

• Need to be somewhat cautious about generalising from single trials and also generalising beyond the real ‘test’ provided by the trials.

Thanks

Contact:

Stephen.roper@wbs.ac.uk