www.uis.unesco.org

Survey methodology and procedures: General advice from Frascati Manual

Regional Workshop on Science, Technology and Innovation (STI) Indicators for Gulf countries

Doha, Qatar15 to 17 October 2012

www.uis.unesco.org

Sources of this presentation

Chapter 7 of the Frascati Manual - R&D Survey Methodology

Section 8 of the Annex to the Frascati Manual (Measuring R&D in developing countries) - Strengthening R&D Statistical Systems

Chapter 7 of UIS Technical Paper no. 5 (Measuring R&D in developing countries) - Strengthening R&D Statistical Systems

www.uis.unesco.org

Institutionalization of STI statistics

Political support

Infrastructure and sustained staff training/capacity building

Involvement of NSOs: “Official statistics” status for R&D surveys.

Adequate legal framework

www.uis.unesco.org

User-producer networks

Recommendations:

• User-producer networks and other forms of stakeholder consultation should be instituted.

• Establishing national S&T statistics groups.

• Involve multiple actors.

• Coordinating/networking among institutions/databases.

• Partnering with business associations.

• Conducting face-to-face visits by statisticians and project leaders.

• Exploit pre-existing personnel ties.

• Get NSO involved; to deal with privacy of information.

• Training of interviewers/primary data producers.

www.uis.unesco.org

General issues

Statistics on R&D require regular, systematic and harmonised special surveys

Other sources provide information, but:

• concepts of R&D used often different from FM concepts

• concepts may change over time

• very difficult to obtain all data for the same period

• difficult to avoid double counting when tracking flows from financial statements and other sources

Estimates are a necessary supplement to surveys

• especially in higher education sector

www.uis.unesco.org

Scope of R&D surveys

R&D surveys should identify and measure all financial and personnel resources devoted to all R&D activities in all R&D units

R&D surveys are mainly addressed to R&D-performing units

Chapter 7 of the FM only addresses performer-based surveys

Statistical methodologies and other procedures have to be established to capture all R&D, especially for units in the business enterprise sector with little R&D

www.uis.unesco.org

Identifying target population and survey respondents – General issues

Exhaustive survey not possible in most countries

Constraints include:

• number of respondents may have to be restricted to keep costs down

• R&D survey may have to be taken in conjunction with another survey

• surveys of some groups may require the participation of other agencies with different data needs and hence different questions for respondents

One size does not fit all: every country has different constraints – advice is therefore of general nature

www.uis.unesco.org

Establishing registers

R&D in developing countries tends to be very much the purview of public bodies

Recommendations:

Establishing a database of public sector R&D projects

• include human and financial resources; align with national policies.

• design could reflect the R&D statistical reporting/definitions.

• source for evaluation of such projects.

Establishing Science and Technology Management Information System (STMIS)

• provide overview of research system.

• framework for establishing complete registers as sample frames for R&D surveys.

www.uis.unesco.org

Science and Technology Management Information System (STMIS) and other secondary sources

STMIS (e.g. database of scientists, research grants, CV databases, etc): frequent source for the production of R&D statistics.

Recommendations:

• need close integration between the statistical system and the STMIS.

• need adjustments to produce comparable statistics, taking into account issues of definitions and coverage.

• need a balanced approach using both STMIS and surveys.

• need different approach to Private sector organizations as they are frequently not covered by these systems.

www.uis.unesco.org

Establishing registers

Other sources

• Associations (trade, academic).

• Learned societies.

• Registers or databases of scientists and engineers.

• Database of research grants.

• Databases of scientific publications.

• Patents and other IP documents.

• Business registers.

www.uis.unesco.org

Survey procedure for each sector

Each sector has different management styles, approaches and institutional culture.

Consider the existing norms in relation to data exchange.

First R&D Survey: through interviews rather than relying on telephonic, e-mail or postal survey.

Higher cost and labour intensive.

Who is the target of the survey?

• Need to consider the sector and the size and complexity of the organizations.

www.uis.unesco.org

Government sector: Identifying target population and survey respondents

Units to include in surveys are:

• R&D institutes: Public research institutes (PRIs); Department-based research institutions (DBRIs)

• R&D activities of general administrations of central or state government.

• Public institutions dealing with STS: statistical, meteorological, geological and other public services, museums, hospitals.

• R&D activities at the municipality level.

Recommendation: the best way to survey is to send questionnaires to all units known or assumed to perform R&D.

www.uis.unesco.org

Government sector cont..

Department-based research institutions (DBRIs)

• Director-General or Permanent Secretary

• Issues: non-availability of information in compiled form

Public research institutes (PRIs)

• Chief Executive Officer or executive responsible for research management

www.uis.unesco.org

Higher education sector: Identifying target population and survey respondents

Recommendation: The surveys and estimation procedures should cover all universities and corresponding institutions, especially those awarding degrees at the doctorate level. Other institutions in the sector known or assumed to perform R&D should also be included.

• Identification generally easy.

• preferable to use smaller units, such as departments or institutes of the university, as statistical units.

www.uis.unesco.org

Higher Education sector cont..

Higher Education institutes (HEIs) are the main seat of R&D activity

Different degree of autonomy

• Staff employed as civil servant – list of employees is available

• Academics directly employed by HEI – staff details are protected

Maturity of HEIs and historic relationships with Government

Researcher CV database

Publications databases (Web of Science™ or Scopus™)

If there is no central registry – approach through Vice Chancellor or Dean of Faculties, Dean of Research or, Head of Departments.

www.uis.unesco.org

Business enterprise sector: Identifying target population and survey respondents

The enterprise is recommended as the main statistical unit in the business enterprise sector

• Some enterprises perform R&D on a regular basis from year to year, and may have one or several R&D units

• Other enterprises perform R&D only occasionally

It is recommended that all enterprises performing R&D, either continuously or occasionally, should be included in R&D surveys.

www.uis.unesco.org

Business enterprise sector: Survey population – first possible approach

1. A census-based survey of large enterprises and a sample of smaller ones in order to identify R&D performers and request the information from them

• R&D performed in the past in the enterprise is not considered

• this is the approach followed in innovation surveys

• very small enterprises and enterprises in certain less R&D-intensive industries often excluded for cost reasons

• when the sample size is very small, estimates may be less reliable, owing to raising factors

• Method not strictly followed in any country

www.uis.unesco.org

Business enterprise sector: Survey population – second possible approach

2. Try to survey all enterprises known or assumed to perform R&D, based on a register of R&D-performing enterprises• lists of enterprises receiving government grants and

contracts for R&D• lists of enterprises reporting R&D activities in previous

R&D surveys, in innovation surveys or other enterprise surveys

• directories of R&D laboratories• members of industrial research associations• employers of very highly qualified personnel • lists of enterprises claiming tax deductions for R&D.

www.uis.unesco.org

Business enterprise sector: Survey population – joint approach

Recommendation

To include in R&D surveys of the business enterprise sector all firms known or supposed to perform R&D.

To identify R&D performers not known or supposed to perform R&D by a census/sample of all other firms:

• In the industries on the next slide.

• In principle, enterprises in all size classes should be included, but if a cut-off point is necessary, it should be at ten employees.

www.uis.unesco.org

Business enterprise sector: Industries to be included

Industry ISIC Rev. 3/NACE Rev. 1

Mining 14

Manufacturing 15-37

Utilities, construction 40,41,45

Wholesale 50

Transport, storage and communication 60-64

Financial intermediation 65-67

Computer and related activities 72

R&D services 73

Architectural, engineering and other technical activities

742

Plus any other industry relevant for the country

www.uis.unesco.org

Business sector cont…

R&D performed in business sector remains low in many developing and emerging economies.

How to detect R&D activity in Business?

• trade associations, or chambers of commerce.

• businesses listed on the main stock exchange.

• large firms/MNC - discussion with the Chief Financial Officer or Chief Technology Officer.

• missing a large firm might result in significant error.

• exclude holding companies, construction, retail, and utilities as sub-sectors likely to perform little or no R&D.

• list of business beneficiaries of research or innovation grants by NRC.

• cooperation with the departments responsible for R&D tax incentives.

www.uis.unesco.org

Business enterprise sector: Structural issues

Publicly-owned businesses play a major role in R&D in some developing countries

Recommendations:

• should consider issuing data for ‘publicly-owned businesses’ separately from the ‘fully private enterprise sector’.

• private enterprises could also be disaggregated by ownership, in particular the various degrees of foreign ownership.

www.uis.unesco.org

Business enterprise sector: Structural issues cont..

Business enterprise R&D is presumed to be generally weak in developing countries when compared to industrial countries.

Recommendations:

• take into account when conducting sample surveys, perhaps by over-sampling, especially amongst larger companies.

• big companies should not be missed out as it might imply significant error.

• invest time in interviewing key firms to understand their R&D function and obtain a clear picture of their activity.

www.uis.unesco.org

Private non-profit sector: Identifying target population and survey

Private non-profit (PNP) sector: make a significant contribution to R&D in developing countries, but the sector tends to be very volatile

Same challenges as in business – difficulty in identify PNPs engaged in R&D

Not clear about, status; ownership.

Engaged in wide range of activities.

Perform in-house R&D as well as contract R&D.

www.uis.unesco.org

Private non-profit sector cont..

The sources for identifying possible survey respondents are mainly the same as for the government sector.

Register information may be less comprehensive and could be completed by information from researchers or research administrations.

This sector may be more relevant for surveys on R&D funding.

www.uis.unesco.org

Who is the right respondent?

R&D Manager

Better understanding of R&D and FM norms

But may not be able to supply exact figures

Accountant or personnel manager

May not refer exactly to R&D as defined in FM

But able to supply exact figures

Cooperation of all three may be needed

Useful to identify in advance the person responsible for providing information and for co-ordinating information from smaller sub-units

www.uis.unesco.org

Working with respondents

Questionnaire: simple and short, logical and with clear definitions and instructions

• Optional: simpler survey for smaller units

Test questionnaires on a sample of respondents

www.uis.unesco.org

Survey procedure and estimation

Recommendations:

• Attention needs to be paid to questionnaire design.

• Frequency of survey.

• Prioritize area of work; accompanied by step-by-step approach.

• Use of survey questionnaires of other countries for inspiration: need adaptations to local situation.

• Get expertise from the NSO, in conducting survey, in sampling..

• Different questionnaires might be designed for different sectors based on stakeholder consultations. “One size does not fit all”.

• Procedures need to be developed for estimating missing data.

www.uis.unesco.org

Encouraging co-operation

Secure co-operation of respondent

• Make them appreciate the potential uses of the data

• Respect confidential data

• Minimise the response burden

• Share the results (option: customised information)

• Provide technical assistance and contact details

www.uis.unesco.org

Estimations

R&D measurement could be done in three stages:

Identification of all specialised R&D units and measurement of their total activity.

Estimates of the non-R&D portions of their activity and subtraction of these estimates from the total.

Estimates of the inputs used for R&D in other units and addition of these estimates to the total.

www.uis.unesco.org

Operational criteria

Tools for “translating” theoretical FM concepts into practical questionnaire:

Explanatory notes

Hypothetical examples

Guidance to individual respondents

Documentation on treatment of different cases

Covered in FM

To be covered by data collection agency keeping good documentation is essential

www.uis.unesco.org

Estimation procedures

Imputation methods for item non-response

• Use previous answer

• Hot decking (use info from same survey)

• Cold decking (use info from previous survey)

Imputation methods for unit non-response

• Use past R&D data (adjusted for sales or employment growth)

• Impute as a function of the relation to personnel or sales (test with non-response analysis)

www.uis.unesco.org

Thank you!

http://www.uis.unesco.org

r.pathirage@unesco.org