www.uis.unesco.org

Ernesto Fernandez PolcuchUNESCO Institute for Statistics

International Conference on S&T Policy Research and Statistical Indicators

08-10 November 2006 - Colombo, Sri Lanka

Three approaches to STI measurement in developing

countries

www.uis.unesco.org

Structure of the presentation

• UIS strategy in the field of S&T stats.

• Three approaches:

• Careers of Doctorate Holders (CDH)

• Innovation

• R&D

• Developing countries’ specificities and way forward

www.uis.unesco.org

UIS S&T Strategy

www.uis.unesco.org

UNESCO International Review of S&T Statistics and Indicators

Carried out jointly with UNESCO Division of Science Analysis and Policy (SC/AP) and

Regional Office for S&T in Latin America and the Caribbean (ROSTLAC) in 2002/2003.

Objectives:• To review priority science policy information

needs.• To examine existing S&T statistical and indicator

systems.• To identify key areas for future development of

S&T statistics.• To define the future role and strategy of the UIS.

www.uis.unesco.org

General S&T Policy Issues

Extremely Important

Very Important

Not/Somewhat Important ISSUES

% % %

Resources 76 17 7 Impact on Society 53 29 18 Dissemination of Knowledge & Technology transfer

53 33 14

S&T output 45 38 17 Institutions & Mechanisms 41 41 18

0% 20% 40% 60% 80% 100%

Institutions & Mechanisms

S&T output

Dissemination of knowledge & tech. transfer

Impact on Society

ResourcesExtremely Important

VeryImportant

No

t or S

om

ewh

at Imp

ortan

t

www.uis.unesco.org

Immediate term priorities (Human & Financial Resources)

R&D personnel

Expenditure in R&D

Human resources devoted to S&T

Science education

Higher education

International mobility

Gender

www.uis.unesco.org

Medium term priorities (Innovation)

Measurement of innovation in agriculture and other non-manufacturing sectors

Promoting the use of indicators reflecting sub-national (regional) innovation systems

Measuring minor or incremental innovation

Measuring innovative applications of existing products or processes (surveys of use of technologies)

Output

www.uis.unesco.org

Longer term priorities (Output & Impact)

• Output:

• Bibliometric tools adapted for the analysis of scientific output in developing countries

• Technology output indicators

• Impact indicators:

• Measuring social impact of S&T,

• Impact of S&T on agriculture,

• Public perception of S&T

www.uis.unesco.org

Ongoing activities

• S&T Survey operation and data guardianship

• Training in S&T statistics: Workshops

• Standard setting/Methodological developments:

• Analysis / Publications

• Collaborations / Partnerships

www.uis.unesco.org

S&T Survey operation and data guardianship

Conducting global Survey on Statistics of Science & Technology: Biennially – currently second round

Maintaining database on S&T statistics

Data publishing on the UIS website

Contributions to external agencies

www.uis.unesco.org

Survey on Statistics of Science & Technology

• The 2004 S&T statistics survey was launched in May, 2004. This was the first UNESCO statistical questionnaire to be available for completion on-line via the internet. The latest resulting data were released on the UIS website May 2006.

The 2006 S&T statistics survey was launched in June/July 2006.

OECD and EUROSTAT provide data for their Member States. RICYT provides data for Latin America.

www.uis.unesco.org

Data collected: R&D personnel by occupation & gender R&D personnel by sector of employment & occupation R&D personnel by sector of employment & gender Researchers by formal qualification & sector of employment Researchers by formal qualification & gender Researchers by fields of science & sector of employment Researchers by fields of science & gender Total expenditure in R&D by sector of performance Total expenditure in R&D by source of funds

Survey on Statistics of Science & Technology (continued)

Questionnaire on Statistics of Science and Technology; Instruction Manual for completing questionnaire.

Data are collected from each country from either the institution responsible for S&T policy or statistics (e.g. Ministry of Science and Technology, Ministry of Research and Higher Education, National S&T Council or similar organization) or the National Statistical Office; through a single questionnaire which cover data on all institutions carrying out R&D activities in the particular country.

www.uis.unesco.org

2004 Survey on Statistics of Science & Technology

UIS 2004 survey 49

UIS surveys 1998 or 2001 7

OECD (MSTI) 35

Eurostat (NewCronos) 5

RICYT 18

Total 114

Results of the 2004 survey are now published on the UIS website:

Data are available for 114 countries and territories. Here is an overview of the sources:

http://www.uis.unesco.org

www.uis.unesco.org

Evolution of 2006 survey

2004 and 2006 Surveys Received questionnaires comparison

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9

10 (

2006, 6N

ov)

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

(…)

87 (

end o

f S

2004)

Weeks since deadline

Nu

mb

er

of

receiv

ed

qu

esti

on

nair

es (

tota

l)

2004 2006

www.uis.unesco.org

http://www.uis.unesco.org

Data and Indicators published:

Number of Total R&D personnel (Headcount-HC and Full-time equivalent-FTE) by sex Total R&D personnel (FTE) by sector of employment Researchers (HC and FTE) by sex Researchers (FTE) by sector of employment Technicians and equivalent staff (HC and FTE) by sex Other supporting staff (HC and FTE) by sex Researchers per million inhabitants (HC and FTE) Technicians per million inhabitants (HC and FTE) Women as a percentage of R&D personnel (HC and FTE) Women as a percentage of Researchers (HC and FTE) Total Gross Domestic Expenditure on R&D (GERD) in local currency and international

Purchasing Power Parity (PPP) dollars GERD by sector of performance (%) GERD by source of funds (%) GERD as a percentage of Gross Domestic Product (GDP) GERD per capita

www.uis.unesco.org

UIS is UN lead agency for S&T Statistics

Contributes data for: UNESCO HQs: UNESCO Science Report

2005, UNESCO World Report, International Report on Science Technology and Gender 2006.

UN Statistical Division: UN Statistical Year Book

UNDP: Human Development Report

World Bank: World Development Indicators

Specific Requests

www.uis.unesco.org

S&T Statistics WorkshopsDiagnosis: Production of S&T statistics in many developing regions (particularly in Africa

and Asia) remains low. Capacity building is needed.

UIS Response: Regional workshops To increase the number of countries regularly producing quality S&T indicators. To create local capacities for the production of such indicators, with the final

aim of establishing sustainable local S&T statistics systems To promote the use of S&T indicators, seeking comprehension for evidence-

based S&T policy making. To share experiences with other developing countries in the field of S&T

indicators, and to address the problems that countries may have encountered in collecting S&T statistics.

To gain knowledge about the particular characteristics of S&T statistics data collection and use in the context of countries in the same region.

To generate initiatives that could be used as a demonstration for good practices in other countries of the region.

www.uis.unesco.org

S&T Statistics workshops: 2005/06

Southern and Eastern Africa Regional Workshop – held in Uganda, Sept. 2005

South Asian Regional Workshop – held in India, Nov. 2005

South-East Asian Workshop – held in Indonesia, March 2006

Francophone Sub-Saharan Africa: held in Senegal, Oct. 2006

Upcoming Workshops:

• Workshop in Central Asia: will be held in Kazakhstan, Nov. 2006

• Workshop in Arab States, will be held in Jan. 2006

• Workshop in South-East Europe: will be held in Croatia, Mar. 2006

www.uis.unesco.org

Standard setting/Methodological developments:

Measuring Innovation in Developing countries: Annex to the Oslo Manual (2005):

Oslo Manual (Guidelines for collecting and interpreting innovation data):

central reference document for the statistical definition of innovation and forms the basis for surveys of innovation throughout the world.

www.uis.unesco.org

Careers of Doctoral Holders – CDH

Objectives: to track the careers of doctoral holders, with the

aim of obtaining information on the loss of highly qualified specialists, the so-called ‘brain drain’.

to design an international survey tool which would help track the careers of doctorate holders which would better inform policy makers worldwide, both at global and local levels.

www.uis.unesco.org

Analysis / Publications

Data on the web site.

UIS Publications (S&T Bulletin): 1st – Investment in R&D; 2nd – Bibliometric Indicators; 3 rd – Women in Science (can be downloaded from the UIS website)

UNESCO Science Report 2005

International Report on Science, Technology and Gender 2006

UNESCO World Report

History of Science Statistics at UNESCO

www.uis.unesco.org

Women in Science

International Report on Science, Technology and Gender 2006

to be published by

UNESCO Natural Science Sector

UIS Contribution: the chapter on ‘Statistics on Science, Technology and Gender (STG)’ with an Annex on statistical overview.

Collaboration with European Commission (Research Directorate-General) and a group of worldwide experts.

UIS Bulletin on Women in Science: available on-line in Sept 2006.

www.uis.unesco.org

Collaborations / Partnerships

UNESCO HQs

OECD

Eurostat

RICYT (Latin America)

NEPAD / AU / ATPS (Africa)

ASEAN

IDRC

INRS (Quebec)

ISESCO

Inter-Academy Council

ISDB

www.uis.unesco.org

Quality of data

Efficient use of resources

Consistency over time and space

Accessibility and affordability

Validityand reliability

Comparability through

standards

Relevance to policy

Potential for disaggregation

Currency and punctuality

Coherence across sources

Clarity and transparency

www.uis.unesco.org

Careers of Doctorate Holders (CDH)

www.uis.unesco.org

UIS approach to Human Resources statistics: CDH project

• Methodology developed “from the scratch” together with OECD & Eurostat.

• Aimed both at developed and developing countries

• With participation from experts from both developed and developing countries

• Now being piloted

• Promoting the methodology by encouraging developing countries to conduct such surveys and produce cross-nationally comparable statistics on careers of doctorate holders

• What about the Sri Lankan Tracer Study of Graduates and Postgraduates? Does it fit into this project?

www.uis.unesco.org

Relevance of the CDH project

• World economy is increasingly based on knowledge and information.

• Knowledge is now recognised as the driver of productivity and economic growth.

• As a result, there is a new focus on the crucial role of highly qualified individuals who represent a key to the production, application and transmission of knowledge.

• Statistics on the global trends in human resources for Science and Technology (HRST) are very week.

• The quality and comparability of international data on migration is particularly weak.

• Diversity of data collection methods hinders international comparability, and does not provide information on career paths and mobility patterns.

www.uis.unesco.org

Background and Process

Background: ‘Brain drain’ or loss of skilled personnel identified as priority by UIS

and UNESCO Paris OECD and EU are interested in

skills loss from developing countries Moves from university to industry

US NSF also interested in changing US patterns of academic recruitment

Process: To meet the users demands UIS, OECD & Eurostat launched a project

on “Statistics on the Careers of Doctorate Holders”. Each organization took responsibility for issues appropriate to their

mandate. Expertise and resources are shared across the three partners.

Expert group: Argentina, Belgium, Canada, China, Denmark, France, Germany, India, Italy, Japan, Malaysia, Portugal, Russian Federation, Spain, Switzerland, Ukraine, Uganda, the United States.

www.uis.unesco.org

Objectives and Task forces

Objectives: To design an internationally comparable tool for tracking the

careers of doctorates holders and highly qualified people in different countries.

To pilot tests of this instrument in volunteer countries. to collect and exchange information on the career paths of

holders of doctorates from existing data sources and the new survey tool.

Task forces: Production of output tabulations, and development of common

definitions (led by OECD and Canada) Drafting of methodological guidelines (led by Eurostat and

Portugal). Development of a model survey with a supporting methodology

(led by UIS).

www.uis.unesco.org

Output tabulation program

• Conceived to be used by countries when delivering the requested data to the international organizations;

• Consists of several tables organized in seven broad groups: personal characteristics, educational history, work history, scientific output, perception of current work situation, international mobility retrospective and future plans.

www.uis.unesco.org

• Doctorate;

• Citizenship and residential status;

• Recent doctorate recipients;

• Employed persons;

• Unemployed and inactive persons;

• Researcher;

• Temporary/permanent employment, full-time and part-time employment, etc.

• Postdoc

Definitions

www.uis.unesco.org

Methodological guidelines

• Define and structure the target population

• Describe the sources for building sampling frames

• Present some aspects of data collection, data processing and results estimation.

• Should be seen as an orientation for countries that plan to launch a CDH survey as well as a tool to improve and align their existing national surveys to an international standard.

www.uis.unesco.org

Target population

Consists of individuals fulfilling the following criteria:

• having an education at ISCED 6 level (doctorates) obtained anywhere in the world, and

• being resident (permanent or non-permanent) within the national borders of the surveying country.

www.uis.unesco.org

Target population structure • Target population, residents in country X (the surveying country)

that are:

A. Citizens of country X with doctorate awarded within country X

B. Citizens of foreign countries with doctorate awarded in country X

C. Citizens of country X with doctorate awarded abroad

D. Citizens of foreign countries with doctorate awarded abroad

• Non-target population, residents outside country X that are:

E. Citizens of country X, with doctorate awarded within country X that have left country X

F. Citizens of foreign countries, with doctorate awarded in country X that have left country X

• What to do with doctorates abroad

- Groups E and F are usually taken into consideration in the reporting countries where these doctorate holders are resident.

- If countries are able to construct a sample frame, and reach group E, the survey could also be extended to this group, even if countries are not requested to report information to the international organizations on them.

www.uis.unesco.org

Sources for constructing sampling frames

• National education registers

• Universities

• International and national foundations

• National libraries

• Population censuses

• Research institutes

• Professional organizations

• Central registers of foreigners

• Alumni organizations

• Previously conducted surveys on doctorate holders

www.uis.unesco.org

Survey types

• Cohorts of recent graduates:• administered on graduation and repeated every few years• simple and straightforward, but• limited coverage of the target population

• Snapshot surveys (cross-sectional retrospective sample of all doctorates):

• one time, administered on an occasional basis • more appropriate tool for collection of information related to the

entire target population • covers all cohorts at the same time• quickly provides information on the mobility and career

development of doctorate holders

• Should not be seen as mutually excluding but rather complementary

www.uis.unesco.org

Instruments (developed by UIS)

• Core model questionnaire containing “core questions” which would provide data to fill in the output tabulations. The questionnaire will be accompanied with a manual comprising the definitions and other guidelines.

• CDH questions data base comprises the set of questions among which the suitable questions can be chosen and applied by countries when designing CDH surveys. CDH data base will be web-based and openly accessible.

• The core model questionnaire is mainly foreseen for countries which do not have CDH type surveys yet.

www.uis.unesco.org

The following existing CDH questionnaires were used:

• USA (SED and SDR),

• Canada (SED),

• Switzerland (Survey of Tertiary Graduates plus Doctorates surveyed the year after they received their degree),

• Portugal (Survey of former PhD Scholarships receivers concerning their professional situation),

• India (Pilot study on the Career Profile and Professional Achievements of Doctorates in science from the Selected Central Universities/Institutions),

• Ukraine (Structure of database of Doctorate Holders in Ukraine)

Construction of Core model Questionnaire

www.uis.unesco.org

Modules

• Doctoral Education (EDU)

• Recent graduates (REC)

• POSTDOCS (POS)

• Employment situation (EMP)

• Career-related experience and scientific productivity (CAR)

• International mobility (MOB)

• Personal characteristics (PER)

www.uis.unesco.org

Information on educational history of doctorates holders, such as:

• institution

• field of science

• duration

• sources of financial support

• country of previous degrees

Doctoral Education

www.uis.unesco.org

• Data on the holders of doctorate who received their degree in the last 2 years.

• Provides a complete educational history, including names and locations of secondary and each postsecondary institution, dates of attendance, field of study, and date of degrees.

• Enquires about the time elapsed before obtaining first CAREER PATH JOB .

Recent graduates

www.uis.unesco.org

• Would your principal job be considered a POSTDOC position in your country of residence? (PROBLEM WITH DEFINITION)

• Title, Field of S&T, total length of POSTDOC

• Reasons for taking this POSTDOC

• Activities involved in POSTDOC (Research, teaching, other)

• Main source of financial support

POSTDOCS

www.uis.unesco.org

• Detailed information on employment status, working hours, principal employer, sector and type of employment (postdoc, temporary or permanent employment, part-time or full-time job ), annual earnings, job-satisfaction.

• Relationship of job & doctorate degree.

• Retrospective career history within ten past years (including occupation, dates, years of experience, and reasons for leaving position).

• “multi-job” pattern is foreseen.

• Section for unemployed and inactive.

Employment situation

www.uis.unesco.org

• Information on mobility patterns, including inflows (group D) and outflows (group C).

• Enables to distinguish temporary mobility from permanent mobility as well as to identify the reasons for departure and return.

• Provides data on intentions to move out of the country within the next year including the destination planned.

• Enquires about linkages with country of origin.

International mobility

www.uis.unesco.org

• Data on scientific output and experience, as well as teaching.

• Section for researchers, including reasons for taking up research careers.

• Enquires about international cooperation activities.

Career-related experience and scientific productivity

www.uis.unesco.org

• Marital status

• Number of dependents

• Place of birth, date of birth, citizenship status (in order to better deal with dual nationality as well) and residential status.

• Contact information

Personal characteristics

www.uis.unesco.org

Way forward

Instruments are being tested.

Joint publication with OECD & Eurostat

Involve more countries to create an international database

www.uis.unesco.org

Innovation

www.uis.unesco.org

UIS approach to innovation statistics

Analyse experiences in innovation measurement in developing countries.

Propose amendments to Oslo Manual.

Development of an Annex to the Oslo Manual “Measuring Innovation in Developing Countries”, with the participation of experts across the developing world.

Dissemination of Oslo Manual & Annex. Discussion of its applicability in other developing countries.

www.uis.unesco.org

Measuring Innovation in developing countries

After the publication of the 2nd Oslo Manual, also developing countries started conducting innovation surveys.

The design of the surveys was intended to comply with Oslo Manual standards, with adaptations for capturing the particular characteristics of innovation processes. Adaptations were prepared by each country separately and with different approaches.

Bogotá Manual published by RICYT (Ibero American Network on S&T Indicators) first effort to compile particularities and guide the design of cross-nationally comparable innovation surveys.

www.uis.unesco.org

Annex to OM (continued)

UIS circulated a base document prepared by RICYT to a vast network of experts in the developing world covering China, Thailand, Singapore, Malaysia, Hungary, India, Lebanon, South Africa, Tanzania, and Thailand.

UIS drafted the final annex based on this input.

www.uis.unesco.org

Characteristics of innovation in developing countries

Size and structure of markets and firms

Instability

Informality

Particular economic and innovation environments

Reduced innovation decision-making powers

Weak innovation systems

Characteristics of innovation

www.uis.unesco.org

Innovation measurement in developing countries

The definition of innovation needs to remain unchanged, as well as those concerning its subtypes.

The concept of potentially innovative firm is incorporated.

Measurement priorities:• Innovation capabilities (Human resources, Linkages,

Quality assurance systems, ICTs)• Expenditure on innovation activities• Organizational innovation

www.uis.unesco.org

Principal adaptations

ICTs in innovation surveys• strategic use of new technologies (“Front office” vs “Back office”)

Linkages • linkage agents and types of linkage • geographical location of linkages

Innovation Activities • “Hardware purchase”, and “Software purchase”• “Industrial design”, and “Engineering activities”• “Lease or rental of machinery, equipment and other capital goods”• “In-house software system development”• “Reverse engineering”

Human resources + training Quality and environmental management

www.uis.unesco.org

Methodological issues for developing country contexts

• Information systems specificities – weakness of statistical systems

• Application of the survey

• Questionnaire design

• Frequency

• Publication

• Difficulties

www.uis.unesco.org

Thinking ahead

• The role of entrepreneurs and their attitudes towards innovation.

• The intention to capture innovations driven by factors other than market forces, and in particular innovations conducted by the public sector.

• The adaptation of methodology to measure innovation in the primary sector (particularly in agriculture).

• The need for better measuring minor or incremental changes, including innovative applications of existing products or processes, and the so-called 'backwards integration' of technological capability.

• The development of indicators reflecting sub-national (regional) innovation systems.

www.uis.unesco.org

Issues arising in the follow-up to the Annex

• Innovation in informal sector?

• Innovation from traditional knowledge?

• Surveying innovation, rather than R&D, in business (and informal) sector?

www.uis.unesco.org

Innovation data collection

• In 2007, innovation data and metadata will be collected through a pilot project.

• Only countries with existing innovation surveys will be approached.

www.uis.unesco.org

Research and Experimental Development (R&D)

www.uis.unesco.org

UIS approach to R&D statistics (currently)

• Applying Frascati Manual (FM) as much as possible.

• Disseminate the FM methodology through workshops

• Listen to countries (in workshops) and be attentive to issues where the FM is not suitable for developing countries and look for solutions together with the country professionals.

www.uis.unesco.org

Breakdowns required by UIS

R&D personnel by gender R&D personnel by occupation and gender R&D personnel by sector of employment and occupation R&D personnel by sector of employment and gender Researchers by formal qualification and sector of

employment Researchers by formal qualification and gender Researchers by fields of science and sector of employment Researchers by fields of science and gender Total expenditure in R&D by sector of performance Total expenditure in R&D by source of funds

www.uis.unesco.org

HOW do we collect data?

R&D Surveys. Innovation surveys. Combined R&D-innovation surveys. -> Good quality questionnaires are needed!

Administrative data (budget, personnel list)

S&T management information systems

Time-use surveys

Estimations

Different strategies for different sectors: one size does not fit all!

www.uis.unesco.org

Conclusions

www.uis.unesco.org

Conclusions: How do we ORGANIZE a sustainable S&T statistics system?

Convince policy-makers

Involve multiple actors

User/producer consultation

Create national S&T statistics groups

Shared ownership of data

Quality is key – capacity building of producers

Step-by-step approach

Combine statistics with S&T information systems

Network with colleagues from other (similar) countries

Prioritize areas of work: R&D – innovation – CDH?

Develop sector specific approaches and tools: GOV-HE-BUS-PNP-FOR

www.uis.unesco.org

Conclusions

• Three different approaches to development of methodologies for developing countries:

• R&D, focus on comparability.

• Innovation, focus on adaptation.

• CDH, joint development from the scratch.

• The Annex to the Oslo Manual as an inspiration for further work on other areas, particularly R&D.

• Developing countries are not homogeneous, UIS is trying to get closer, but new developments will need significant participation of national experts.

www.uis.unesco.org

Thank you!

http://www.uis.unesco.org

UNESCO Institute for StatisticsC.P. 6128 Succursale Centre-ville, Montreal, Quebec, H3C 3J7,Canada.

TP: (1 514) 343-6880Fax: (1 514) 343-6872