Contextual Response Analysis of Publications of PBL ... · PBL publications. A co-word analysis...

Contextual Response Analysis of Publications of

PBL Reports 2013-2017

Ad Prins,

With Nicolas Robinson Carcia (Ingenio, Valencia/Granada SP.)

Contextual Response Analysis of PBL Reports. July 17, 2017

P a g e | 1 Ad Prins Support in ResearchManagement

Content

Content ........................................................................................................................................................ 1

Conclusions.................................................................................................................................................. 2

Summary ..................................................................................................................................................... 3

1. Introduction ........................................................................................................................................ 6

1.1. Mission and context of PBL ........................................................................................................ 6

1.1. Contextual Response Analysis of PBL ......................................................................................... 6

1.2. Materials ..................................................................................................................................... 7

1.3. Structure of the report ............................................................................................................... 7

2. Parliamentary use ............................................................................................................................... 8

3. General Societal Use ......................................................................................................................... 14

3.1. Characteristics of Users ............................................................................................................ 15

3.2. Response of Frequent Users to specific PBL publications ........................................................ 20

3.3. Findings for societal use ........................................................................................................... 28

4. PBL reports in printed news media ................................................................................................... 30

5. Scholarly response to PBL reports .................................................................................................... 34

6. Comparison with other government agencies ................................................................................. 35

7. Public engagement of PBL through social media.............................................................................. 38

7.1. Findings and recommendations from the analysis of response in social media ...................... 38

7.2. Objective, data and methods ................................................................................................... 39

7.3. Audiences following PBL in Twitter .......................................................................................... 40

7.4. Outreach of PBL output in social media ................................................................................... 43

7.5. Outreach of selected publications ............................................................................................ 48

Addendum I. Country list of tweets and Twitter accounts mentioning publications from PBL ................ 50

Addendum II: lists of tables and figures .................................................................................................... 51

Addendum III: list of investigated reports ................................................................................................. 52

Addendum IV. Analyzing PBL reports ........................................................................................................ 55

Oosterkade 8b, 9718 RS Groningen Nicolas Robinson-Garcia



Conclusions

• In parliament, PBL is seen as authoritative even though the debates about topics related to

PBL reports are dynamic, with considerable involvement of opposition parties. PBL is

appreciated not only for its research reports and recurrent structural reports but also for its

broader publications such as the signal reports. PBL has a high profile, mentioned very often

by the various sides in parliament, including the government. Mentions in parliament include

also reports critical of standing policies which adds to its high profile. (Section 2)

• Wide appreciation for the signals reports and structural reports is also found among societal

users, as these are frequently used among a broad range of users. The response to

publications on Urban Planning and Housing is largely limited to users working in these areas.

(Section 3)

• The number of societal users is increasing, particularly in the sector Urban Planning &

Housing. Among users in the broad sector of Environment, Sustainability, Climate and Energy,

the response for individual reports is similar to the previous analysis of 2012, with more

response from media specialized in the various sectors of users. The increase in numbers of

users is not translated in higher numbers of users per individual report, but in a larger and

more diverse audience responding to more publications. (Section 3)

• In terms of response in parliament, press and from societal users PBL is smaller than other

bureaus also due to differences in subject matter and policy areas. Both in parliament and

among societal users, its profile is high, with many interest groups particularly in the sectors of

environment and sustainability and related to energy and climate change. (Section 2, 3, 4)

• Visibility in social media for PBL is clearly present but there is room for growth by offering

more opportunity for search engines to index reports and by more attention to exposure in

social media. (Section 7)



Summary As a governmental research and advisory agency, the role of PBL is to investigate current environmental,

ecological and spatial quality and to explore and identify related social issues and future social trends, and

to identify possible strategic options for achieving government objectives. To this end, PBL conducts

solicited and unsolicited research that is always independent and scientifically sound. The prime concern

of the agency is the policy relevance of its studies which consist of contributions to the improvement of

the quality of all political and administrative decision-making related to these fields.

This report studies the response to a selection of PBL publications from various audiences in different

domains over the period 2013-2016 plus some earlier publications appearing in 2011 and 2012. To this

end, 66 of the PBL publications are investigated for traces of use in databases for parliamentary debates,

newspapers and scientific literature. The societal response is investigated by searching the internet with

Google and BING. The societal response is further investigated for diversity of groups of users.

Particularly, frequent users, those responding more than once to the selected publications are each

identified with respect to relevant social economic sectors in particular 1. Environment and Sustainable

Development, 2. Urban Planning and Housing, 3. Climate, and 4. Mobility and Infrastructure. Comparisons

are made with the outcomes of similar contextual response analysis for two other policy research

bureaus, SCP and CPB. The report does not trace the impact of PBL in the policy process via direct

interaction with policy makers, as this type of information is beyond the scope of this study.

PBL informs parliament and policy makers with policy relevant insights and perspectives as well as factual

information in a context in which many of the topics, such as climate change, sustainability and energy

and also agriculture are intensely debated. The topics are often politically sensitive and open to

confronting political interpretations. Even though it is seen as a bureau cooperating often with other

policy research institutes, PBL has a high profile also in relation to these topics. In the parliamentary

debates, PBL has a relative high profile in comparison with SCP and CPB. More than the other two policy

research bureaus, PBL is explicitly referred to as the source of information or source of insight and

perspectives during the debates. PBL is mostly seen as authoritative and independent both with regards

to facts and policy relevant insights in these areas, both by MPs and members of the government. In this

context, the challenges for PBL are to maintain its position as an independent policy research

organization, its high profile and to communicate possibly critical policy views also while collaborating

with other institutes. (Section 2)

In the societal domain, there is a strong increase in numbers of users from all sectors in comparison with a

similar analysis in 2012. Users are found in sectors directly relevant to the topics of the publications and

among other relevant users. There are variegated user groups of professionals, bloggers, public and

governmental organizations and firms. Publications regarding politically sensitive issues, such as on

climate change, agriculture and sustainability are used by various sides in the public debate. The increase

in numbers of users is not translated in higher numbers of users per individual report, but in a larger and

more diverse audience responding to more publications.

Among the most frequently used publications in the societal domain are the three Signals reports: De

Energieke Samenleving, Wissels Omzetten and also Vergroenen en Verdienen (if taken together with its

translation Green Gains). Various Monitoring reports too are frequently used among diverse groups of

users, including the Nationale Energieverkenning 2016 and its 2015 edition, and Balans van de

Leefomgeving 2016 and its 2014 edition. De macht van het menu, a topical report on sustainable supply of



healthy food, draws considerable attention in printed media and is especially popular among non-

frequent users, i.e. those that use less than two publications.

In comparison with a similar analysis in 2012, many more users are found for reports regarding urban

planning, housing and mobility. Frequently used publications are Vergrijzing en Ruimte, De Veerkrachtige

Binnenstad, the IABR book Slimme Steden, De Stad: magneet, roltrap, spons, and Belastingkortingen voor

zuinige auto’s.

Both the structural and signal reports show a large ambit of users from a very broad and diverse set of

sectors. To some lesser degree, this is also the case for other publications on issues of environment,

sustainability, climate and energy, with response from users in directly related sectors and mostly also to

a wider set of user interested in the specific topic. An exception is the set of publications on urban

planning & housing, which draws attention mainly from the related sector of users but only a fair number

of users from other directions. Changes in the characteristics of users can be noted, in particular

regarding those from sectors of Environment, Sustainability, Climate and Energy. Specialized media have

become more important as a category of responders. Also, but in modest numbers, responses from

governments are coming more from network organizations in which governments work together with

other public organizations and firms. (Section 3)

In printed media, PBL is particularly mentioned in relation to issues of urban planning & housing,

environment and sustainability, agriculture and climate. Regional newspapers more often refer to issues

of urban planning & housing, as the topics involve regional effects of demographic change and the

opportunities for smaller regional cities. Issues regarding climate are more likely to be addressed by

national media, and issues regarding agriculture appear in specialized media on agriculture. A content

analysis on a sample of larger newspaper articles shows that the attention for PBL is often focused on

findings and data rather than publications, but opinion pieces also appear, often citing PBL in a positive

way. The sample revealed few critical remarks, which mainly regard to issues of agriculture and urban

planning. (section 4)

Analyzed with Google Scholar, the visibility of the 66 publications in scientific literature is modest but the

numbers increase slightly in comparison to those of the previous analysis. The analysis of visibility of

scientific publications is beyond the remit of this study. (section 5)

By way of an experiment, the response on social media is also investigated, in particular regarding Twitter,

for national attention and using Altmetric.com to investigate international attention on social media.

Twitter reactions to selected publications were limited, particulary those in Dutch and included

researchers from PBL or related institutes as well as by interest groups and politicians. Frequently

retweeted reports are all about issues on environment, energy and climate: Nationale Energieverkenning

2016, the reaction to the Paris Climate Accords, Aanpassen aan Klimaatverandering, Dalende Bodems,

Stijgende Kosten, and Grondstof voor een Circulaire Economie. However, the response to PBL in general is

notable, both via the Dutch and English PBL accounts, among much more followers clearly interested in

PBL publications. A co-word analysis shows diversity among PBL followers in interests and background,

while their self-described interests show entwinement and overlap with respect to issues regarding

energy, development, sustainability and innovation. A further analysis of 282 PBL journal papers in

English, retrieved via Web of Science, shows the possibility of higher degrees of visibility for PBL

publications. This implies also the possibility of higher visibility of PBL publications in Dutch if made also



retrievable in open access via DOIs. The investigation of social media is reported in section 7 of this

report. (Section 7)

In comparison with SCP and CPB, PBL the response for PBL is smaller than for SCP and CPB, in parliament,

press and among societal users. This most probably is related to differences in scope of subject matter

and policy areas. Perhaps also because PBL is also not as vested as the other two -older- institutes. The

response to PBL increases, however, in its visibility in newspapers and in scientific literature, while its

societal audience also increases and diversifies. (section 6)



1. Introduction

1.1. Mission and context of PBL The Netherlands Environmental Assessment Agency PBL is the national institute for strategic policy

analysis in the fields of environment, nature and spatial planning. In order to certify and improve the

scientific quality and social relevance of the work of the agency, the research program and the individual

studies is to be assessed by an external committee. A self evaluation is one of the key documents for this

assessment. An analysis of its social and scientific impact is therefore to be part of this evaluation,

including also a comparison with the similar previous analysis of 2012, and with comparisons of studies

for SCP and CPB.

The tasks of PBL are to investigate and document current environmental, ecological and spatial quality, to

explore and identify related social issues and future social trends, and to identify possible strategic

options for achieving government objectives. To this end, PBL conducts solicited and unsolicited research

that is always independent and scientifically sound.

The research program is decided by the director of PBL and the relevant ministries have their say during

the preparation of the work program. The agency is organized in seven departments and is assisted by an

independent advisory board. Some of these studies are commanded by the relevant ministries as part of

the structural task of the advisory role of PBL. These consist of recurrent overviews and compendia.

The agency produces a considerable number of studies, partly in cooperation with other institutes such

as CBS (Statistics Netherlands), the Netherlands Bureau for Economic Policy Analysis CPB and the

Netherlands Institute for Social Research SCP, The Dutch National Institute for Public Health and the

Environment RIVM, and Energy research Centre of the Netherlands (ECN). There is cooperation with

advisory bodies such as The Social and Economic Council of the Netherlands (SER), The Council for the

Environment and Infrastructure (Rli), The Netherlands Scientific Council for Government Policy (WRR) and

with universities (Wageningen University, WUR) or university related institutes (LEI).

The prime concern of the agency is the policy relevance of its studies which consist of contributions to the

improvement of the quality of all political and administrative decision-making related to these fields. PBL

does so with many types of publications, with structural or recurrent reports, with specialized studies and

also with publications offering insights and perspectives about current issues placed in broader

developments. The latter is particularly done in Signals reports. All of these publications are intended to

be relevant for national policy makers and politicians and for a wide audience of stakeholders and

interested parties.

1.1. Contextual Response Analysis of PBL PBL asks to support the self evaluation by providing an analysis of its social and scientific impact related to

the mission of the agency. The focus of the analysis is on the most important publications and on the

structural products, signals reports and compendia. The Contextual Response Analysis investigates the

social impact or use in several domains, including parliament, news media, scientific literature and among

societal users. A further explanation of the method in Contextual Response Analysis is in Addendum IV.

Where appropriate, comparisons will be made with the outcomes of similar recent Contextual Response

Analyses for the Netherlands Bureau for Economic Policy Analysis (CPB) (in 2015) and for the Netherlands

Institute for Social Research SCP (in 2016). Also, comparisons are made with previous analyses for PBL

(2012) and SCP (2008).



As an experiment, a quantitative analysis is made on data of social media, in particular related to Twitter.

1.2. Materials PBL has provided 66 titles of its most important reports to be investigated with the method of Contextual

Response Analysis. This is a selection of the reports and other documents published by the agency in the

period 2013-2016/2017, with the addition of three reports appearing earlier (De Energieke Samenleving,

(2011), Natuurverkenning 2010-2040 (2012) and Nederland Verbeeld (2012, appearing in 2013). (see

Addendum III for a full list of investigated reports.)

The investigation of response in parliament and printed media extends from beginning of May 2012 to

February 2017, in order to closely follow up the previous response analysis of 2012, which ended its data

collection end of April 2012.

An extra analysis is made of social media, in particular Twitter, and with the Altmetric technique, by

Nicolas Robinson Garcia (INGENIO) (Section 7)

1.3. Structure of the report The structure of the report is in part based on the mission of the agency in relation to its audiences. The

first task of the agency is to inform and interact with policy makers and politicians. Secondly, the agency

focuses also on key policy areas in which various organizations and interested others work, denoted here

as society in general, including the various printed news media. Consequently, the analysis starts with

parliamentary use of PBL reports and of the references in the parliamentary minutes that can be linked to

the agency (section 2). Furthermore, the broader circle of audiences is studied on the basis of internet

data (section 3), which in part are classified to analyze in depth the linkage between individual reports

and groups of users. Additionally, use is searched for in two specific social domains, printed news media

(section 4) and science (section 5) that can be studied with specialized databases. In section 6, a

comparison is made with the outcomes of contextual response analyses of two other government

agencies. Section 7 reports findings in social media.



2. Parliamentary use The prime aim of PBL is its contribution to the improvement of the quality of political and administrative

decision-making. The framework for policy relevance is largely set in parliament where government

ministers and secretaries of state discuss with members of parliament of both governmental and

opposition parties. Debates in parliament are therefore considered as a suitable starting point for the

analysis of policy relevance of PBL. The analysis extends a period from the beginning of May 2012 to

February 2017, closely following the period of the previous evaluation for 2008 until end of April 2012.

Parliamentary debates are analyzed on the basis of minutes of the meetings of its two Houses (Eerste

Kamer – which functions as Upper House or Senate, and Tweede Kamer – Lower House). Between the

beginning of May 2012 and February 2017, PBL has been mentioned in 116 plenary meetings which on

average per year is similar to the period of the previous evaluation (102 meetings). In the current period

PBL is also mentioned in the meetings of parliamentary standing committees, 31 in total. 1

More than one third of the plenary meetings in which PBL was mentioned (48 out of 116), were about

general issues, such as (40) debates on budgets for Ministries, which are particularly numerous at the

start of the new government in 2014, or the 8 general initial debates at the start of the parliamentary

season (Algemene Politieke Beschouwingen and Algemene Financiele Beschouwingen). This indicates that

PBL is not only mentioned in relation to specific reports or outcomes, but for other reasons as well, such

as the request for new studies by MPs.

To investigate in more detail how PBL is referred to, by which MP or cabinet member, and for what topic,

a more detailed text analysis is made of a selection of the meetings. For the plenary meetings fifty of the

116 meetings have been selected for further text analysis. Of the 31 meetings of the standing

committees, 16 were selected. In total sixty-six meetings were selected, based on randomly attributed

numbers.

In many of the sixty-six selected meetings, PBL has been mentioned multiple times, both by MPs and

members of the cabinet alike. In the selected plenary meetings, PBL is mentioned 195 times, including

multiple mentions by the same speaker. 2 Correcting for these multiple instances in the same debate by

the same speaker,3 118 instances in the plenary sessions can be found of speakers referring one or more

times to PBL. Taking the plenary sessions and the standing committees together, there are 188 instances

of speakers mentioning PBL one or more times in a meeting.

The parliamentary minutes show recognition of the collaboration of PBL with a wide range of other policy

research bureaus, such as ECN, SCP, CPB and CBS. The appreciation or esteem of MPs and members of the

government for PBL relates therefore also to the collaborative efforts and results with many other

institutes. This is particularly the case with issues regarding climate (with ECN, in more than 50% of all

1 Substantial meetings are defined here as meetings in Handelingen excluding Ingekomen Stukken (list of incoming documents). The meetings for the Standing Committees can be found under Kamerstukken. 2 The 195 mentions are divided among the various sides in parliament: 106 mentions by the opposition, 69 times by Ministers and Secretaries of State, and 21 times by members of the government parties (including the sustaining Freedom Party -constituting a so called “gedoogpartij”- for the cabinet period until April 2012. In one other occasion the chair of parliament (Presidium) proposes to ask the government for an assignment for PBL for an evaluation of the policy initiative Green Deal. 3 Numbers of speakers are counting each different speaker per meeting. The counts exclude motions.



cases) and to somewhat less degree also on topics of energy (with ECN) and agriculture (WUR, LEI). In

other cases, esp. regarding environmental issues, mobility and housing, the debates seem more related to

the specific expertise of PBL, as much less often other institutes are mentioned.

Notwithstanding this recognition of entwinement, PBL does have a rather high profile in parliamentary

debates. This also was the case in the previous analysis of 2012, where PBL also was discussed for its role,

and in this present analysis. In this period, rather, its high profile is related to PBL as a source of research

outcomes, policy insights and perspectives on much debated topics.

The attention given to PBL as source of findings and perspectives is to be noted in a comparison with

similar samples for mentions of two other policy research institutes, SCP and CPB. Although with 116

meetings in which it is involved PBL seems to achieve the least attention, the number of 118 speakers in

the selected samples is very high. This suggests that, once meetings with PBL topics take place PBL

insights and outcomes are more explicitly debated among MPs and members of the government than is

the case for the other two institutes. (table 1)

Table 1 Comparison of numbers of speakers in selected meetings for PBL, CPB and SCP

# selected plenary

meetings

# speakers in selected

meetings mentioning the

institute

Total # plenary meetings

(Handelingen)

PBL 50 118 116

CPB 50 72 496

SCP 50 57 185

The debates are mostly disputes between members of the opposition and cabinet members, both in the

plenary debates in parliament and the discussions in the standing committees mostly take. In the 50

selected plenary sessions there were 118 speakers mentioning PBL one or more times, with 30 speakers

of the cabinet, 71 for the opposition, 16 for government parties and 1 other instance. In the 16 selected

meetings of the standing committees, there are 31 speakers mentioning PBL one or more times, with 10

from the cabinet, 15 from the opposition, six from government parties. 4 Particularly parties such as Partij

voor de Dieren, GroenLinks, D66 and SP (respectively: Animal Party, GreenLeft, Liberal Democrats and

Socialist Party) are very active on environmental issues and agriculture. PvdA (Labour) is relatively active

on climate. (Table 2)

4 In the selection for Standing Committees, three meetings had no speaker mentioning PBL, while a report or a responding letter from the government were on the agenda.



Table 2 Speakers mentioning PBL in parliament per party and subject 2012-2017

Topic

Cab

inet

Pvd

D

GL

SP

Pvd

A

D6

6

CU

VV

D

PV

V

CD

A

Oth

er

Total

Energy, Renewable Energy 5 4 1 1 3 5 1 1 1 0 22

Environment, sustainable development 8 8 4 5 1 3 1 1 0 31

Planning, City development, Housing and Mobility 6 2 1 2 1 2 2 1 2 19

Climate 2 1 3 1 4 1 0 12

Agriculture, Food & Biodiversity 7 9 3 3 2 1 2 27

Role 2 1 2 1 1 0 7

Water 0 0

other 1 1

Grand Total 30 21 15 12 9 9 7 7 2 2 4 118 Legend: Opposition: PvdD: Partij voor de Dieren, Animal Party; GL: Groen Links, Green Left; D66: Lib. Dem.; SP:

Socialistische Partij, Socialist Party; CU: Christen Unie, Christian Party;

Government parties: all years: VVD: Volkspartij voor Vrijheid en Democratie, Conservative Party; (from nov 2012 to present:

PvdA: Partij van de Arbeid, Labour Party; Until nov 2012: CDA: Christen Democratisch Appel, Christian Democrats, and PVV:

Partij voor de Vrijheid, Freedom Party

In only a third of the cases references to specific reports can be found, 27 in total. These include the

Keuzes in Kaart 2013-2017 (the analysis of ten political programs for the election for traffic congestion),

Monitor Duurzaam Nederland (Monitor Sustainable Netherlands), Vergroenen en Verdienen (Green

Gains), and the Ex Ante Evaluation of Green Deals Energy as the more frequently mentioned reports. The

number of references to specific reports is higher than in the previous period (27 present to 14 former).

The mentioning of Vergroenen en Verdienen is particularly to be noted. It is a signals report, a type of

reports written specifically to offer wide ranging policy perspectives, or in the words of PBL “to indicate

either changing circumstances relevant for policy makers or the desirability for change in policies”.

Most speakers have a high appreciation of PBL reports and outcomes, which is apparent from the way

they refer to reports and outcomes. In the majority of cases, PBL is referred to as an authoritative source

of information. This is clear from many contributions in which facts and figures from PBL reports are used

to underline the point of view of the speaker (table 3). In a number of cases of authoritative quotes (21)

PBL is also explicitly cited for expressing views about expected policy effects with critical remarks about

the effects of policies and policy measures in view of outcomes of studies or international agreements.

Criticizing government policies by MPs also occurred in the mentions of the signals report Vergroenen en

Verdienen.

PBL reports and outcomes are also discussed when MP’s and cabinet members differ in opinion about

how to interpret PBL reports. These include mentions of different interpretations, either by one MP in

one contribution opposing the view of another MP, or in debating an MP or cabinet member (19 cases).

An example is a debate about sustainable energy and climate, in which one party (GL) insist on renewable

energy whereas the other party (PvdA) quotes PBL for a statement about a priority for CO2 reduction. In

other cases, the discussion is about the perceived lack of understanding or regard of PBL outcomes (9



cases) or the priority to be given to a particular view or standpoint over another view (9 cases) or refer in

the debate to different valuations of PBL methods (3 cases, referring to the use of models in the

prediction of the development of traffic congestion).

The parliamentary mentions of PBL are not restricted to facts and outcomes of reports but also relate to

the wider perspective of the policy making process, as is noted above. In its publications and also in the

direct communication with policy makers and members of the government, PBL expressly takes this wider

perspective as one of its tasks. At instances this role of PBL is approvingly noted, also by members of the

government:

“When talking with PBL, you can learn to be much more effective if you take the other measure”5

However, the policy insights and perspectives offered by PBL may also lead to remarks of policy research

with having a specific and ultimately limited role in policy making. This is done in particular by members of

the cabinet, sometimes by referring to a limitation to the authoritativeness in view of policy concerns, at

other instances with reference to the specific role of PBL in the policy process, for instance with regard to

the kind of authority it will have when monitoring effects of policy making by lower authorities such as

provinces and local authorities.

The four more explicit critical remarks found in the sample are expressed in relation to topics regarding

agriculture, in particular with regard to manure policies (Mestbeleid), both by MPs (CDA) and the then

acting secretary of State, and about traffic congestion, and concern a different choice of policies. (Table 3)

Table 3 Parliamentary Appreciation of PBL reports and points of views 2012-2017

w/o

furth

er

classification

PB

L cited

as

critical ab

ou

t

po

licies

oth

er

total

Authoritative 152 21 4 177 Discussion 19 19 Limited authority attributed, Role 6 11 17 Critical 4 4 Other 12 1 13 Total 193 21 16 230

This table is based on all contributions of all speakers, including multiple mentions for each speaker

The high number of instances in which PBL has been cited as an authoritative source of information, in

comparison to the much lower numbers for critical remarks, expressions of limited authority and

discussions, should not mask the incisive and urgent nature with which MPs address issues on climate

change, sustainability and energy. MPs, in particular from a number of opposition parties, display in-depth

knowledge of PBL reports over a very long period of time, citing and comparing outcomes of various

5 “Als je spreekt met het PBL, hoor je dat je veel effectiever bent als je niet de ene maar de andere maatregel neemt. De

vraag is dus niet zozeer hoeveel hectare je realiseert. Dit is wat we nu doen, nog los van wat provincies verder zouden willen doen.” Staatssecretaris Dijksma, Debate on EHS, Handelingen TK, 1 October 2013, 30825 nr. 207.



reports related to the same or similar topics, while expressing their political views. Likewise, members of

the cabinet and government parties too, take position with the help of PBL information. Debating the

many PBL reports is therefore part and parcel of a process of political decision making where outcomes

and points of view are due to political interpretation in a sometimes highly politicized field.

Working in this context, also with the intention to inform policy makers both with factual information and

with policy insights and perspectives, the challenges for PBL are to maintain its position as an

independent policy research organization and to deliver its communication in a consistent manner also in

the collaboration with other institutes. This is clear from two examples taken from the sample of minutes,

in which PBL statements and publications have been reinterpreted in parliament.

The first example is a debate on suitable methods to retain the size of national (dairy) cattle population in

relation to manure policy which takes us back to 2011. In view of the coming abolishment by the EU of the

European system of Milk Quota, PBL performed two studies on the national size of the cattle population.6

Although both reports came to the same conclusion that some policies of curbing the size were an

advantage in terms of sustainability and living circumstances, the conclusions of the two publications

differed somewhat in wording. The subsequent governmental letter responding to the two reports gave

one MP (Van Veldhoven D66) the impression of a juxtaposition of outcomes, which led the then acting

secretary of state (Bleker) to the following statement:

“Scientists amaze you every day again. Even of scientist within one and the same bureau or of one

and the same discipline may come to different valuations.”7

The other example is from the same dossier, a few years later. With the abolishment of the European

system of Milk Quota under way, a new set of policy measures is elaborated in collaboration with the

related agricultural sector. At the start of this, PBL and LEI (Landbouw Economisch Institute) of the

University of Wageningen is asked to perform an Ex-Ante study Mest beleid (Manure Policy (2013) This in

relation to the national size of the dairy cattle population, as this is vital for meeting EU directives.8 The Ex

Ante study assumes that EU directives can be met if sufficient growth takes place in the capacity of

industrial processing of manure. The later Quickscan study of PBL (2015), investigates the two emerging

policy options. One option is a nationally maintain criterium (so called Forfeiture System), the other is a

system allowing individual diary firms to maintain levels of phosphate production in relation to size of

land, or to process manure or to export manure. Although in its Quickscan PBL remains very critical about

the effects of the policy based on the cooperation of individual firms, the government decides to this

system. However, only after a few month, the issue resurfaces in parliamentary debate as it becomes

clear that the expected growth of the cattle population exceeds by far the expectations.

6 Willems, J. & H. van Grinsven (2011), Afschaffen van productierechten in de veehouderij in 2015: gevolgen voor

veehouderij en leefomgeving. Achtergrondstudie bij de Evaluatie Meststoffenwet 2012, Den Haag: Planbureau voor de Leefomgeving. Van Grinsven, Hans et al. (2011), Welke veestapel past in Nederland? Inbreng voor de maatschappelijke discussie over begrenzing en sturing van de omvang van de veestapel, Den Haag: Planbureau voor de Leefomgeving. 7 Translation of: “Wetenschappers verbazen je elke dag weer. Zelfs van de wetenschappers binnen één en hetzelfde bureau of van één en dezelfde wetenschapper kunnen er verschillende taxaties komen. H. Bleker, Secretary of State for Agriculture, 25 januari 2012, Intensieve veehouderij, 28973 nr. 91 8 Rougoor, C., H. van Grinsven, J . van Dam (2015), Fosfaatrechten Voor Melkvee. Een quickscan naar hun effecten op de

leefomgeving en de sector, Den Haag: PBL Willems, J. et al. (2013), Ex ante evaluatie mestbeleid 2013. Gevolgen van de invoering van verplichte mestverwerking en het afschaffen van productierechten in de veehouderij, Den Haag/Wageningen: PBL/Wageningen UR.



In the subsequent debate, MPs and cabinet members discuss the cause of this development, also pointing

to these studies. While some raised the suggestion that it were the reports that could not be trusted, the

secretary of state was to point at the principle of independence of the role of PBL and LEI, while at the

same time holding the institutes partly responsible.

“I find it remarkable that Mr. Dijkgraaf talks about scientific reports. Isn’t it the scientists that said

the Milk Quota could be discarded?.... So I plead for not changing policies on the basis of a report

or an opinion and then find a leeway, while we have seen what a shambles and chaos it was.” 9

“It may also mean that the market has been behaving differently than scientists expected on the

basis of their knowledge and insights. That is, I think, what happened. LEI and PBL are

independent institutes, and must be allowed to advise in that manner. For them it can be of

interest to reconstruct how they came to that insight at that time and how things went differently.

For the issue before us, this is less relevant.” 10

PBL informs parliament and policy makers with policy relevant insights and perspectives as well as factual

information in a context in which many of the topics, such as climate change, sustainability and energy

and also agriculture. The topics are often politically sensitive and open to confronting political

interpretations. Even though it is seen as a bureau cooperating often with other policy research institutes,

PBL has a high profile also in relation to these topics. Nevertheless, PBL is mostly seen as authoritative and

independent both with regards to facts and policy relevant insights in these areas, both by MPs and

members of the government. In this context, the challenges for PBL are to maintain its position as an

independent policy research organization, its high profile and to communicate possibly critical policy

views also while collaborating with other institutes.

9 Ik vind het wel opmerkelijk dat de heer Dijkgraaf het heeft over die wetenschappelijke rapporten. Laten nu die

wetenschappers gezegd hebben dat het melkquotum eraf kon. ..Dus ik pleit ervoor dat we niet te snel op basis van een rapport of een mening het beleid moeten omgooien en dan weer allerlei ruimte moeten geven, terwijl we de afgelopen vijf jaar hebben gezien wat voor puinbak en chaos het was. Van Gerven SP, 1 dec 2016, Fosfaatrechten. Meeting 30, item 13. 10 Het kan ook betekenen dat de markt zich anders is gaan gedragen dan de wetenschappers op basis van al hun kennis en inzichten hadden verwacht. Dat is, denk ik, wat er is gebeurd. Het LEI en het PBL zijn onafhankelijke instituten. Die moeten ook onafhankelijk kunnen adviseren. Dat zij hierop terugblikken en zich afvragen hoe zij destijds tot die inschatting kwamen en waarom het anders is gelopen, lijkt mij voor hun werk heel interessant, maar voor de vraag waar wij voor staan, lijkt het mij minder relevant. . Secretary of State Van Dam, Government, 24 nov 2016, Budget Ministry of Economic Affairs, dep. Agriculture, Meeting 27, item 9.



3. General Societal Use Searching for the societal response to the 66 publications, with the help of Google and BING, resulted in

2076 websites, responding to one or more of the investigated reports. The twenty most used reports are

listed in table 4. The list is headed by the essay De Energieke Samenleving, a Signals Report of 2011, used

241 times, that is to say without the eleven references to a report with an almost similar name Energieke

Samenleving en Duurzame Mobiliteit. The essay also appeared in the previous report with 144 users.

(Table 4)

Table 4. Twenty reports most frequently used, by unique references

* minus the 11 (infrequent) users that referred to the almost similar title Energieke Samenleving en Duurzame Mobiliteit -

Verkenning van opgaven en aangrijpingspunten voor beleid (2014)

Two of the three Signals reports are among the most frequently used publications: De Energieke

Samenleving, and Wissels Omzetten. The other Signals report, Vergroenen en Verdienen has modest

numbers of users (48), yet if the number of users of its translation (Green Gains) are added, the title

would rank too among the most frequent publications, which underlines the importance of this type of

publications in the public domain.

Compared to the previous analysis of 2012, the degree of use of the reports is overall similar. The twenty

most frequently used reports in this analysis are on par with the twenty previous most frequently used

Title infrequent users frequent users Grand Total Top

publ.

De Energieke Samenleving* 120 121 241

Wissels omzetten 70 90 160 *

Nationale Energieverkenning 2016 62 97 159

Balans van de Leefomgeving 2016 91 63 154 *

De macht van het menu - opgaven en kansen voor

duurzaam en gezond voedsel

110 44 154 *

Kansrijk Mobiliteitsbeleid 70 81 151 *

Nationale Energieverkenning 2015 65 85 150

Nederland in 2030 en 2050: twee referentiescenario's 47 97 144 *

Balans van de Leefomgeving 2014 43 87 130 *

Vergrijzing en ruimte 60 64 124 *

De veerkrachtige binnenstad 53 71 124

Kansrijk woonbeleid (PVS-1837) 75 48 123

Grondstof voor een circulaire economie 49 71 120

IABR boek Slimme steden – de opgave voor de 21e

eeuwse stedenbouw in beeld

56 64 120 *

Green gains (vertaling "Vergroenen en verdienen" uit

2013)

33 72 105 *

Opties voor energie- en klimaatbeleid 32 68 100

Belastingverschuiving: meer vergroening en minder

complexiteit?

24 76 100

rapport De stad: magneet, roltrap, spons 34 65 99

Doorrekeningen SER energieakkoord 30 61 91

Belastingkortingen voor zuinige auto’s: afwegingen voor

fiscaal beleid

35 54 89



reports, with the exception of three most frequently used ones in the previous period with much higher

numbers of users. 11

Structural reports too, such as the Nationale

Energieverkenning 2016 and the similar report for 2015,

and Balans van de Leefomgeving 2016 and its similar report

of 2014 are among the most frequently used titles. The list

contains also reports on single topics. Particularly noted is

De macht van het menu, a topical report on sustainable

supply of healthy food, drawing also considerable attention

in printed media. This report seems especially popular

among non-frequent users, i.e. those that use less than

three publications.

Notable are also the reports related to urban planning, housing and mobility, such as Vergrijzing en

Ruimte (aging and urban planning), De Veerkrachtige Binnenstad, (the resilient inner city in relation to

demographic developments) the IABR book Slimme Steden, (about sustainable city planning in the future)

De Stad: magneet, roltrap, spons, (the city as a motor for social mobility) and Belastingkortingen voor

zuinige auto’s (Taxreducation for fuel economic cars). The present reports on urban planning, housing and

mobility are much more frequent used than similar reports in the 2012 analysis.

3.1. Characteristics of Users

Table 5 Numbers of unique referring websites according to the numbers of used PBL publications

# users in

2017

# users in 2012

One time 1383 1490

Two times 302 252

Three times 132 114

Four to seven 145 116

Eight to fifteen 90 77

Fifteen or more 41 16

In order to characterize the users of publications, all frequent users have been classified according to

social economic sector and their particular function or presentation in their sector. In this analysis, all

users that refer to two or more publications have been classified. Classifying users makes it possible not

only to exclude self-references (coming from the various websites of PBL) but also referring websites that

by their nature do not indicate substantial response. These include: libraries and repositories, booksellers

and publishers and referrals from automated websites such as internet news pages and browsers. The

sectors definitions are listed in table 6. If not specified otherwise, the analysis of the response from the

11 These were Quick Scan varianten EHS, (412 users) a politically topical study about the foreseeable consequences of drastic budget reductions in the long-term investments in ecological reconstruction, Milieubalans 2009 (212 users) and Rio+20, (237 users) a report also closely related to the political disputes on climate change. The Quick Scan varianten EHS report was in this respect an outlier, with many references found on websites of the local and regional authorities that were discussing the possible consequences of the suddenly proposed budget cuts.

Definitions:

1.1. Unique reference: website

referring to a specific report,

independent of how many

references the website contains

to that report.

1.2. Frequent user/organization: a

website of a user or

organization referring two times

to one or more PBL reports.



various sectors in the next sections exclude users in the category General and exclude the printed media

and radio and TV from Media General, which is analyzed in the subsequent section. (Table 6)

Table 6 User Categories according to Sector

Sector # Users definition

General 10 Not otherwise specified

Labour, Economy, Industry 44 Industry, Economy, Labor Unions, Unions of Employers

Culture 3 Debate Centers, Entertainment

Energy, inc. Sustainable Energy 37 Energy, incl sustainable energy

Environment, Sustainable

Development

88 Broadly defined sector of users working on sustainability and environmental

issues if not in Energy, Climate

Health Care 2 Health Care

Urban Planning, Housing 104 Urban Planning, Housing, Construction, Neighborhood associations

Climate 7 Users specifically focused on Climate issues

Argiculture, Food, Biodiversity 15 Agriculture, incl biological agr., food, biodiversity

Media, General 7 Media, incl media forums, excl specialized media for specific sectors

Unknown 2 Unknown, site not reachable or obsolete

Education and Research 44 Universities, Higher Education, Research Institutes (incl policy research)

Governments 77 (Inter)national governments, regional and local authorities, advisory and

supervision org., water authorities

Politics 11 Political parties, Parliament

Social Media 9 Twitter, Facebook, LinkedIn

Mobility 33 Road construction, Car Lease, Water transport, Public Transport, Planning,

Parking

Water 11 Water quality, Water management (drinking water)

Total number of frequent users 504

In comparison to the previous analysis, the total number of users is up, with the exception of the large

group of one time users, which are users mentioning only one publication. Excluding self-references from

PBL websites, printed news and not substantial websites, the frequent users, referring to two publications

or more increase from 235 users in 2012 to 504 in the present analysis. For comparison of the response

per sector, the numbers of users for 2012 and the present have been taken together in four large

categories: 1. Environment, Sustainability, Climate, Energy and Agriculture, 2. Urban planning, mobility, 3.

Governments, and 4. Users in other sectors. As it appears from table 7, the increase in numbers is in part

due to a steep rise of users working in the sectors of urban planning, housing and mobility, but it is also

clear that the numbers increase in each broad category of users. (Table 7) . 12

Table 7 Comparison of numbers of users in four broad categories of sectors, 2012-2017

# users, 2012 # users, 2017

Broad sector of environment, sustainability, climate, agriculture and energy 121 158

Governments 38 77

Broad sector of urban planning, housing and mobility 31 137

research 7 44

Other 31 88

Total 235 504

12 9



The growth of numbers of users, almost a doubling in five year, is comparable to the growth of response

found for SCP in the 2016 study. However, whereas in the case of SCP this growth in part is to be surmised

to come from the growth of internet, this seems less likely for PBL, as the growth of internet in The

Netherlands has diminished.13 The increase in numbers of users is therefore more likely to be linked to an

increasing response to PBL publications. The numbers of users have increased, also when correcting for

the higher numbers of analyzed publications.

The comparison of the two datasets of users should go with the warning that each set is a selection of

publications, which may mean that the sets not only differ in size but also in character also with respect to

the usefulness in specific sectors. Differences in characteristics of users can thus be related to the

differences between the datasets rather than being an expression of a changing landscape among users.

However, some differences are notable and noteworthy, in particular if we look at the characteristics of

users within each sector category with regard to their function.

In the previous analysis, knowledge platforms and interest groups were among the most frequent types of

users, when classified according to function. In the present dataset knowledge platforms too are the most

important way of use and dissemination, but are now closely followed by specialized media. (table 8)

The main sectors relevant as audience for PBL differ, however, since urban planning and mobility include

construction firms, transport organizations, and local and regional authorities involved in planning and

zoning, while the broad sector of environmental concerns, sustainability, climate, agriculture and energy

includes also includes a broad range of interest groups and public organizations.

Table 8 Comparison of numbers of users according to function, 2012 - 2017

# users, 2012 # users, 2017

knowledge platform 47 118

specialized media 29 105

interest group 40 59

research 24 40

consultancies 6 40

Politics, blogs 22 34

lower authorities 18 25

advisory and supervision 11 22

network organizations 2 16

governments 20 11

other 0 34

Total 235 504

In sectors characterized by many organizations involving both small and large in size communication

might involve to a somewhat higher degree intermediate forms of communication in order to reach

smaller organizations. For varied sectors with many smaller organizations this can take the form of

13 Dutch internet domains increased between 2008 and 2015 from 3,2 million to 5,7 million sites in the SCP case, whereas the increase in the current comparison of 2012 and 2017 is from 5,4 million to 5,7 million. Source: SIDN

https://www.sidn.nl/a/kennis-en-ontwikkeling/statistieken



knowledge platforms and specialized media, rather than direct use by organizations that by sheer size

must be considered able to keep up complicated websites rich with information. This is the case with both

broad sectors, which is visible in the higher numbers of users in the form of knowledge platforms.

However, for the broad sector of environment, sustainability, climate, agriculture and energy, the uses for

the present set of publications are very frequently also coming from specialized media, such as the

Nieuwsbrief Milieu en Economie (Newsletter Environment and Economics), and Duurzaam Nieuws

(internet site with news on sustainability), De Boerderij and AgriHolland – media in agriculture, news

media for water quality and water management, and for energy and sustainable energy.

Notable is also a difference in the response from governmental organizations. Whereas the number of

responding governments in 2017 is smaller than in 2012 – This may also have to do with the large

attention for Quick Scan varianten EHS in 2012, with no such report in this set. However, there is a small

rise in a specific type of organization related to governments, network organizations which intend to

cooperate and communicate among governments with other public organizations and firms. This

indicates the possibility that the landscape of relevant organizations using PBL publication today not only

differs from the previous analysis, but most likely also represents a change in audiences. (figures 1 and 2)



Figure 1 Users according to broad sectors plotted against function 2012

Figure 2 Users according to broad sectors plotted against function 2017



3.2. Response of Frequent Users to specific PBL publications

Most PBL publications investigated here are about complex topics, addressing policy issues pertinent to

wide varieties of users in national, regional or local governments and in other societal sectors. This is also

clear in the distribution of response to frequently used publications, which are clearly not responded to in

specialized audiences in a single sector but in variegated ranges of sectors.

Some degree of specialization of audiences can be found among some publications, in which the response

comes from audiences outside governments but expected to be interested in the topic of the publication.

This is obviously the case with publications with a prime and clear focus on energy and on housing and

urban planning. For instance, the two frequently used National reviews on energy, Nationale

Energieverkenning 2015 received attention from 19 different organizations working in the sector of

(sustainable) energy out of a total of 85 frequent users, and the Nationale Energieverkenning 2016 had 23

users in that sector out of a total of 97 frequent users. These include knowledge platforms and

specialized media but also specialized consultancies in the field and some industry. Likewise, publications

with specialized topics on housing or urban planning indeed find many users working in these fields, such

as the IABR book on smart cities Slimme Steden, or De Veerkrachtige Binnenstad.

An overview of such specialized or typical response to specific publications is found in table 9, with

typically very frequent response in a sector for a specific report is indicated in yellow, and frequent

response in a sector indicated in green. 14 The table also shows that frequent and also very frequent

response is found in other sectors as well, indicating that the degree of specialization is only partial and in

all cases not a one-on-one relation between publication and audience. (Table 9)

Five publications seem to defy this partial specialization with a response less characterized by frequent

take up by specific sectors. These include the two Signals reports, Energieke Samenleving and Wissels

Omzetten, Balans van de Leefomgeving 2016, and Natuurverkenning 2010-2040, and Monitor Duurzaam

Nederland. These five reports show a more even distribution of response over the various sectors, and

are apparently of interest to a wide variety of users among all sectors.

The varied distribution of response over different sectors is further to be demonstrated when focusing on

smaller samples of publications. In the next figures 3-8, 44 publications with a response of more than 25

users are compared in sets attributed to specific topics related to the main sectors relevant for PBL. The

relative response is calculated as a percentage of the total response for each report (100%).

14 Table 10 lists the 44 reports with a net response of more than 25 users. For each report the relative response from a sector is taken (share, as a % total for each publication, total is 100%). Standard deviations from the average of each sector indicate the characteristic response coming from the sector for the publication. Yellow: Typically Very Frequent for the sector’s response to the 41 publications. Green: Typically Frequent for the sector’s reponse, No Color: a percentage within the range of normal distribution for the sector’s response). Values are calculated as standard deviations from the sector’s average: yellow: equal or larger than two standard deviations from the sector’s average of all 41 publications, Green: 1,5 standard deviations from the sector’s average.



Table 9 Typically frequent use of specific publications in the most important sectors

Titles

Urb

an P

lann

ing

& H

ou

sing

Go

vernm

ents

Enviro

nm

ent

sustain

able

develo

pm

en

t

Edu

cation

and

Research

Energy, -

sustain

able

Labo

ur an

d

Econ

om

y

Med

ia

Social m

edia

Transp

ort

Gen

eral

Po

litics

Agricu

lture an

d

bio

diversity

Water

Clim

ate

Total R

espo

nse

(freq u

sers)

De Energieke Samenleving 23,24% 16,20% 15,49% 8,45% 7,75% 9,15% 2,11% 1,41% 4,93% 1,41% 3,52% 0,00% 2,11% 4,23% 142

Nationale Energieverkenning 2016 12,50% 12,50% 19,23% 3,85% 25,96% 11,54% 4,81% 1,92% 1,92% 1,92% 1,92% 0,00% 0,00% 0,96% 104

Nederland in 2030 en 2050 14,14% 17,17% 14,14% 8,08% 8,08% 7,07% 3,03% 5,05% 13,13% 3,03% 0,00% 3,03% 3,03% 0,00% 99

Nationale Energieverkenning 2015 9,28% 13,40% 16,49% 5,15% 28,87% 8,25% 6,19% 2,06% 1,03% 0,00% 3,09% 2,06% 0,00% 2,06% 97

Balans van de Leefomgeving 2014, 26,37% 12,09% 14,29% 6,59% 4,40% 4,40% 2,20% 7,69% 3,30% 5,49% 4,40% 4,40% 0,00% 1,10% 91

Kansrijk Mobiliteitsbeleid 14,94% 14,94% 8,05% 4,60% 2,30% 11,49% 11,49% 1,15% 25,29% 2,30% 2,30% 0,00% 1,15% 0,00% 87

Wissels omzetten 16,87% 19,28% 14,46% 12,05% 6,02% 6,02% 9,64% 2,41% 0,00% 3,61% 3,61% 1,20% 2,41% 1,20% 83

meer vergroening minder complexiteit? 15,71% 4,29% 24,29% 2,86% 11,43% 25,71% 5,71% 2,86% 1,43% 1,43% 0,00% 2,86% 0,00% 1,43% 70

Opties voor energie- en klimaatbeleid 5,71% 12,86% 21,43% 11,43% 17,14% 7,14% 8,57% 2,86% 4,29% 1,43% 1,43% 1,43% 0,00% 4,29% 70

Grondstof voor een circulaire economie 17,39% 15,94% 27,54% 5,80% 7,25% 11,59% 1,45% 2,90% 1,45% 5,80% 0,00% 1,45% 0,00% 0,00% 69

Green gains 7,69% 23,08% 16,92% 9,23% 12,31% 16,92% 3,08% 1,54% 3,08% 1,54% 4,62% 0,00% 0,00% 0,00% 65

Doorrekeningen SER energieakkoord 10,94% 17,19% 15,63% 3,13% 15,63% 14,06% 9,38% 1,56% 0,00% 1,56% 3,13% 0,00% 3,13% 3,13% 64

De veerkrachtige binnenstad 42,86% 20,63% 1,59% 1,59% 0,00% 6,35% 1,59% 3,17% 12,70% 6,35% 1,59% 0,00% 1,59% 0,00% 63

IABR boek Slimme steden 41,27% 9,52% 3,17% 11,11% 3,17% 1,59% 1,59% 4,76% 4,76% 4,76% 7,94% 3,17% 1,59% 0,00% 63

Balans van de Leefomgeving 2016. 10,00% 16,67% 16,67% 5,00% 8,33% 6,67% 8,33% 3,33% 3,33% 6,67% 3,33% 5,00% 0,00% 1,67% 60

Kansrijk woonbeleid 35,59% 3,39% 1,69% 8,47% 0,00% 11,86% 25,42% 5,08% 0,00% 6,78% 1,69% 0,00% 0,00% 0,00% 59

rapport De stad: magneet, roltrap, spons 32,76% 25,86% 0,00% 17,24% 0,00% 5,17% 3,45% 5,17% 8,62% 0,00% 1,72% 0,00% 0,00% 0,00% 58

Vergrijzing en ruimte 27,27% 32,73% 1,82% 10,91% 0,00% 7,27% 10,91% 0,00% 5,45% 0,00% 3,64% 0,00% 0,00% 0,00% 55

Aanpassen aan klimaatverandering 14,55% 14,55% 16,36% 5,45% 7,27% 1,82% 5,45% 1,82% 9,09% 3,64% 3,64% 5,45% 1,82% 7,27% 55

Monitor Duurzaam Nederland 12,73% 23,64% 18,18% 7,27% 10,91% 9,09% 9,09% 0,00% 0,00% 3,64% 3,64% 0,00% 1,82% 0,00% 55

Kiezen en delen 35,19% 24,07% 1,85% 7,41% 0,00% 3,70% 3,70% 3,70% 11,11% 3,70% 3,70% 0,00% 1,85% 0,00% 54

Belastingkortingen voor zuinige auto’s: 2,08% 16,67% 18,75% 10,42% 10,42% 10,42% 6,25% 4,17% 10,42% 4,17% 0,00% 4,17% 0,00% 0,00% 48

Waterkwaliteit nu en in de toekomst 10,87% 15,22% 30,43% 8,70% 2,17% 2,17% 2,17% 6,52% 0,00% 2,17% 0,00% 6,52% 13,04% 0,00% 46

Leren van het Energieke Platteland 25,00% 18,18% 13,64% 20,45% 4,55% 2,27% 2,27% 6,82% 0,00% 2,27% 0,00% 2,27% 2,27% 0,00% 44



Titles

Urb

an P

lann

ing

& H

ou

sing

Go

vernm

ents

Enviro

nm

ent

sustain

able

develo

pm

en

t

Edu

cation

and

Research

Energy, -

sustain

able

Labo

ur an

d

Econ

om

y

Med

ia

Social m

edia

Transp

ort

Gen

eral

Po

litics

Agricu

lture an

d

bio

diversity

Water

Clim

ate

Total R

espo

nse

(freq u

sers)

Natuurverkenning 2010-2040 30,23% 23,26% 20,93% 6,98% 0,00% 2,33% 4,65% 6,98% 0,00% 0,00% 2,33% 2,33% 0,00% 0,00% 43

thuismarkt & eco-innovaties 4,76% 9,52% 30,95% 16,67% 4,76% 7,14% 2,38% 9,52% 0,00% 4,76% 0,00% 7,14% 2,38% 0,00% 42

woningbouwmogelijkheden bestaande stad 60,00% 15,00% 0,00% 2,50% 0,00% 5,00% 5,00% 0,00% 2,50% 10,00% 0,00% 0,00% 0,00% 0,00% 40

Natuurlijk kapitaal: naar waarde geschat. 20,00% 15,00% 27,50% 5,00% 5,00% 12,50% 5,00% 5,00% 0,00% 0,00% 0,00% 2,50% 0,00% 2,50% 40

Betaalbaarheid van het wonen in de

huursector. 53,85% 20,51% 2,56% 2,56% 0,00% 2,56% 7,69% 5,13% 0,00% 5,13% 0,00% 0,00% 0,00% 0,00% 39

Monitor Infrastructuur en Ruimte 2016 18,42% 23,68% 10,53% 2,63% 0,00% 0,00% 5,26% 5,26% 18,42% 7,89% 2,63% 0,00% 5,26% 0,00% 38

rapport De economie van de stad 28,95% 28,95% 0,00% 23,68% 0,00% 2,63% 5,26% 5,26% 2,63% 0,00% 2,63% 0,00% 0,00% 0,00% 38

Plannen voor de stad 27,03% 24,32% 5,41% 16,22% 0,00% 2,70% 2,70% 0,00% 16,22% 5,41% 0,00% 0,00% 0,00% 0,00% 37

Vergroenen en verdienen 11,11% 5,56% 16,67% 0,00% 19,44% 19,44% 2,78% 11,11% 5,56% 0,00% 8,33% 0,00% 0,00% 0,00% 36

Gebiedsontwikkeling en commerciele

vastgoedmarkten 44,44% 22,22% 2,78% 2,78% 0,00% 8,33% 0,00% 8,33% 0,00% 5,56% 5,56% 0,00% 0,00% 0,00% 36

Dalende bodems, stijgende kosten 14,29% 22,86% 22,86% 0,00% 5,71% 2,86% 14,29% 5,71% 0,00% 2,86% 2,86% 0,00% 5,71% 0,00% 35

Verduurzaming van internationale

handelsketens 0,00% 29,41% 20,59% 23,53% 5,88% 5,88% 0,00% 2,94% 0,00% 2,94% 0,00% 2,94% 2,94% 2,94% 34

De ruimtelijke metamorfose van Nederland

1988-2015 47,06% 11,76% 8,82% 8,82% 2,94% 0,00% 2,94% 8,82% 2,94% 0,00% 0,00% 0,00% 2,94% 0,00% 34

De macht van het menu 0,00% 26,47% 26,47% 11,76% 0,00% 8,82% 5,88% 8,82% 0,00% 2,94% 0,00% 8,82% 0,00% 0,00% 34

Cahier Mobiliteit 16,13% 41,94% 0,00% 16,13% 0,00% 0,00% 0,00% 3,23% 19,35% 3,23% 0,00% 0,00% 0,00% 0,00% 31

Ex ante evaluatie mestbeleid 16,67% 26,67% 16,67% 13,33% 0,00% 0,00% 0,00% 0,00% 0,00% 0,00% 10,00% 16,67% 0,00% 0,00% 30

Klimaatverandering 6,67% 20,00% 13,33% 6,67% 16,67% 0,00% 13,33% 3,33% 0,00% 6,67% 0,00% 0,00% 6,67% 6,67% 30

Samenhang in de Zuidwestelijke Delta 7,41% 29,63% 14,81% 7,41% 0,00% 0,00% 7,41% 18,52% 0,00% 3,70% 0,00% 3,70% 7,41% 0,00% 27

Kleine kansen, grote gevolgen 14,81% 33,33% 3,70% 0,00% 3,70% 3,70% 14,81% 0,00% 0,00% 3,70% 3,70% 0,00% 14,81% 3,70% 27

De verdeelde triomf 42,31% 0,00% 0,00% 3,85% 3,85% 15,38% 11,54% 11,54% 0,00% 7,69% 0,00% 0,00% 0,00% 0,00% 26

Yellow: Percentage of use of the publication (row total is 100%) is >=2x st.dev. of average of each sector of share in reports. Green: => 1,5 st dev. of av. of each sector



In figure 3 the response from various sectors to the two signals reports Energieke Samenleving and

Wissels Omzetten and several structural reports such as the Monitor studies for Sustainability and for

Infrastructure as well as the two outlook reports on energy Nationale Energieverkenningen (2015 and

2016) and the two structural studies on quality of the environment, Balans van de Leefomgeving (2014

and 2016). Most of these publications are getting the attention from a broad range of users from almost

all sectors, although the two National Energy Reports draw particularly high attention from users working

in the Energy Sector.

Figure 3 Response from various sectors for two signal reports and structural reports

As these publications aim to address complex issues and developments, structural reports such as the

Balans van de Leefomgeving have different takes to the complexity of the issues in each edition. Is the

focus of the 2014 edition about future developments, the 2016 edition pays attention to the spatial

aspects of environmental developments, regarding planning, health and housing. This clearly is of interest

among users in the sector of Urban Planning and Housing, as figure 3 shows.

Notable is also the response to Natuurverkenning 2010-2040 (Nature Outlook 2010-2014), particularly for

the response drawn not only from users in the sector Environment & Sustainable Development, but also

from Government and Urban Planning & Housing. The publication analyzes the different perspectives and

visions taken at “nature” which apparently seem to concern the various users in these sectors as well.

(figure 3)

A different pattern of use can be discerned in figure 4 for the publications on Urban Planning and

Housing. Various reports on inner city development (De Economie van de Stad, De Veerkrachtige



Binnenstad), city developments in economic and demographic perspective (De Stad: Magneet, Roltrap,

Spons or Transformatiepotentie: woningbouwmogelijkheden in de bestaande stad and De Verdeelde

Triomf, Ruimtelijke Verkenning 2016), regional demographic change (schrinkage) (Vergrijzing en Ruimte),

Housing Policies (Kansrijk Woonbeleid), renting fees for social housing or urbanization (Betaalbaarheid

van het wonen in de huursector) and transport infrastructure (Kiezen en Delen, Plannen voor de Stad).

Most if not all of these publications are mainly taken up by users in the sector of Urban Planning &

Housing, with some exceptions for media attention for Kansrijk Woonbeleid, and the attention of regional

and local governments for regional demographic change (Vergrijzing en Ruimte). Striking is the absence of

response of users in the broad sector of environment and sustainable development, which is non-

appearing for all these reports but one, the publication on spatial change in historical perspective (De

Ruimtelijke Metamorfose van Nederland 1988-2015. (figure 4)

Figure 4 Response from various sectors to reports related to Urban Planning and Housing

Figure 5 shows the response to eight publications on issues of Environment and Sustainability. Much more

than in the case of the publications on Urban Planning and Housing, the response is varied among the

investigated sectors. Among governmental organizations, there is notable attention for sustainability of



international trade (Verduurzaming van internationale handelsketens), the monitor for sustainability

(Monitor Duurzaam Nederland) and for Green Gains (the translation of Groen Verdienen).

Green Gains is also used in the sector of Labour and Economy, including users in specialized media for

industry and business, interest organizations for SMEs and for larger companies (VNO-NCW) and banks. A

similar pattern of attention among users in Labour and Economy exists for Vergroenen en Verdienen, a

publication on the strengths and weaknesses of Dutch businesses regarding goals in sustainability and

climate. The tax report (Belastingverschuiving meer vergroening en minder complexiteit) also draws

attention from this sector, including interest organizations for SMEs, financial advisors and knowledge

platforms for sustainable production. (figure 5)

Figure 5 Sector Response to publications on issues of Environment and Sustainability

Publications on climate change draw a somewhat mixed response from diverse sectors, though not

necessarily from the same group of users. The frequently used publication on options for climate and

energy policies (Opties voor Energie en Klimaatbeleid) has the attention of users in Environment and

Sustainable Development, Education and Research and Energy as well as among government agencies.

Cahier Klimaat en Energie is frequently used in a variety of users of government agencies, in Energy and in

Education and Research and in Media. In some contrast, the publication on developments in

environmental and energy footprints in the Netherlands, Trends in Nederlandse Voetafdrukken is more



frequently used in Urban Planning and Housing, Governments and Environment and Sustainable

Developments, but does not attract the attention in media or by users in energy.

Notable is the attention given to two topical reports on climate change, the explanation of the policy

implications of the Paris agreements on climate and the evaluation of a nationwide economic and political

agreement on energy under the auspices of the governmental advisory board SER (Doorrekening SER

energieakkoord). These two reports are mainly taken up by users in Labour and Economy, Energy as well

as by Governments. Aanpassen aan Klimaatverandering, the publication about adapting to climate

change, has yet another group of users, among which users again in governments and Environment and

sustainability, but also in some range of other sectors, including climate and agriculture. (figure 6)

Figure 6 Sector response to publications on issues regarding Climate Change

According to figure 7, publications on agriculture and food also show diverse groups of users. Publications

on manure policies and nitrogen containment, (Ex ante evaluatie Mestbeleid and 2015 Beoordeling

Programmatische aanpak Stikstof) both also forcefully discussed in Parliament (see section…), are used by

government agencies and users in agriculture. However, related reports on food, the consumption of

agricultural products, such as Nitrogen on the table, and De Macht van Het Menu seem more to be taken

up among users in Environment and Sustainable Development, Education and Research and some

governmental agencies. (figure 7)



Lastly, a mixed set of publications on transport, water quality and water security indicate also attention

from users out of various sectors, including governmental users. This is especially the case with the

publications about the risks of flooding Kleine Kansen, Grote Gevolgen, including the advisory committee

on water and regional authorities (Provincies). Kansrijk Mobiliteitsbeleid, a report on the opportunities for

mobility. This report especially drew attention in the media and transport sector for the finding that more

road and rail capacity would not solve future mobility issues. (figure 8)

Figure 7 Sector response to publications on agriculture and food



Figure 8 Sector response to publications on transport and water

3.3. Findings for societal use

To sum up the findings for societal use, the numbers of users have increased which can be taken as a

growing response to PBL publications.

Several changes in user characteristics can be noted. Firstly, increasing numbers of users can be noted for

frequent users, especially those working in sectors of urban planning & housing or in education &

research. The growth of users in the broad sector of urban planning, housing & mobility, now on par with

the numbers for the broad sector of environment, sustainability, energy & climate is particularly

noteworthy since the numbers in the previous study were very low.

Knowledge platforms are among the largest groups of users in all sectors, as intermediates for users in

smaller organizations in each sector. However, communication patterns of users differ for the two broad

sectors of urban planning, housing & mobility and of environment, sustainability, energy & climate differ,

also in comparison with the previous analysis. Specialized media have become more important, in

particular, in the sector of environment, sustainability, energy & climate. Also, whereas the number of

users among governmental organizations are slightly diminishing, a small but new type of users emerges

in the form of network organizations in which governments and various societal partners collaborate. 15

The numbers of users for specific PBL publications remain stable. Structural publications, such as the

various recurrent publications of Balans van de Leefomgeving, Energieverkenning, and Monitors are

among the most frequently used. This is also the case with two signals reports De Energieke Samenleving

15 The 16 network organizations are present in many sectors, rendering a low visibility per sector in figure 2



and Wissels Omzetten, offering encompassing perspectives on broad policy issues. Both the structural and

signal publications show a large ambit of users from a broad and diverse set of sectors.

Other publications too address a large ambit of users of varying background, not only among users in

directly related sectors, but mostly also to a wider set of user interested in the specific topic. An exception

is the set of publications on urban planning & housing, which draws attention mainly from the related

sector of users but only a fair number of users from other directions.



4. PBL reports in printed news media Searching newspapers with the generic name of the agency, “Planbureau voor de Leefomgeving” and its

acronym “PBL” resulted in a list of more than 970 articles in which the agency was mentioned in the

period of beginning of May 2012 until the end of 2016, not counting press agencies and other not

relevant sources (such as mentions in parliamentary sources). All major newspapers and many regional

newspapers in the Netherlands are in this list. The list also carries specialized media, weekly's and

newspapers among the frequently referring media, notably in agriculture, such as De Boerderij and De

Boerderij Vandaag and construction (Cobouw).

In comparison with SCP and CPB, the numbers of articles in printed media are lower, as media coverage

for these institutes are 2961 and 2033 respectively for 2010-2015, which in turn is also lower than the

coverage of the Statistical Bureau of the Netherlands CBS, with 10269 mentions.

The newspaper articles refer only in a limited number of cases to specifically named publications. Most

articles often also select items of interests out of the more complex and rich content while omitting titles

or references of the publication. In many cases, however, a relation can be surmised of the newspaper

article with a specific title, on the basis of the publication date in combination with the subject. Linking

newspaper articles to publications is thus possible on the basis of inference, not on direct empirical

evidence. Also, some categorization of newspaper articles is possible, attributing articles to major PBL

themes.

As is clear in figure 9, most of the topics of the articles are on Urban Planning & Housing, Environment &

Sustainability. The articles mainly appear in daily newspapers, however, a good deal of the publications

for Urban Planning & Housing also appear in specialized media, in particular the journal for the

construction industry Cobouw. Similar frequent publication on PBL related subjects on Agriculture appear

in journals for agriculture, poultry, dairy cattle etc. in particular De Boerderij and De Boerderij Vandaag.

A notable difference occurs in the choice of topics among daily newspapers of national and regional

background. Whereas for most topics the number of articles on PBL in national newspapers exceeds by

far those in regional daily’s, the exception is the attention in the latter group for topics on Urban Planning

& Housing. Many of these articles are on the prospects of inner cities, demographic change and economic

perspectives of regional cities in the competition with larger cities. National newspapers pay relatively

more attention to issues regarding climate change. (figure 9)



Figure 9 News items per subject and type of outlet

The prominent attention for topics related to Urban Planning is also reflected in the reconstruction of the

relation of the topics of newspaper articles with PBL publications. As appears from table 10, many of the

publications are on Urban Planning & Housing, including De Veerkrachtige Binnenstad, Regionale

bevolkings- en huishoudensprognose 2016, Gebiedsontwikkeling en Commerciële Vastgoedmarkten, De

Verdeelde Triomf, etc.16 The signals report Wissels Omzetten also is mentioned or related to newspaper

articles, but mainly in national papers. (Table 10)

Table 10 PBL publications apparently related to articles in printed media

Natio

na

l da

ily’s

Pro

fessio

nal

ne

wsp

ape

rs and

magazin

es

Re

gion

al Da

ily’s

We

ekly’s

Total

De veerkrachtige binnenstad 7 2 11 20

Gebiedsontwikkeling en commerciele vastgoedmarkten 5 4 5 14

Regionale bevolkings- en huishoudensprognose 2016 4 1 9 14 Wissels omzetten 9 1 1 1 12

De verdeelde triomf 9 1 10

Vergrijzing en ruimte 2 3 3 8 Kansrijk woonbeleid (PVS-1837) 6 1 1 8

Belastingkortingen voor zuinige auto’s: afwegingen voor fiscaal beleid 8 8

Kansrijk Mobiliteitsbeleid 7 1 8

Nederland in 2030 en 2050: twee referentiescenario's 4 2 2 8 Arbeidsmarkt zonder grenzen 4 3 7

Doorrekeningen SER energieakkoord 5 2 7

Kleine kansen, grote gevolgen 2 1 4 7

Most of the articles report about PBL publications for facts and findings, while in a limited number of

cases also a PBL researcher or spokesperson is quoted (8,5 % of all cases).

16 Table 11 contains also publications not selected by PBL for this analysis.

0 20 40 60 80 100 120 140 160 180

Urban Planning & Housing

Climate

Environment and Sustainability

Agriculture

Energy, -sustainable

Transport & mobility

Other

Newsitems per subject in printed media

Other regional media specialized national newspapers



A concise content analysis may provide further insight in how PBL is quoted or used. The content analysis

is based on a selection of the 362 larger articles in newspapers for three broad topics, Urban Planning &

Housing, Climate and Environment & Sustainability. A stratified sample of 86 articles have been taken, the

numbers based on the distribution of these topics in the dataset. In most cases, PBL is quoted as a source

of information not disputed in the article, often with omission of the name of authors: “PBL says….”, “PBL

reports….”, “as a recent PBL publication points out….”. This is also the case with the eight opinion (Op-Ed)

articles, substantiating the opinion with information from PBL publications.

In a few cases, critical remarks are placed about PBL or PBL’s point of view. In most cases the criticism

relates to issues of Urban Planning and Housing. The critiques entail different points of view related to

PBL predictions of demand for new houses – based in part on demographic changes, and comes from

authors working as researcher on Urban Planning or in the construction industry17, or have to do with

predictions about developments in inner cities.18 Two articles in the sample raise some criticism related to

issues of Environment and Sustainability, one of which claiming that the housing industry now shows

developments towards more sustainability in a speed “that the PBL’s, ECN’s of this world cannot follow”,

the other mentioning PBL without further substantiation.19 (table 11)

Table 11 Reactions on PBL or PBL publications in larger newspaper articles

Urban

Planning

&

Housing

Climate Environment

and

Sustainability

Total

Cited as authoritative source of information 36 17 12 65

Op-Ed piece, cited for information 4 4 3 11

Critical about PBL information or method 5 0 1 6

Interview 0 0 1 1

Op-Ed by PBL authors 0 1 0 1

Other 1 0 1 2

Total # sample 46 22 18 86

Total # articles with larger size (>380 words) 247 44 71 362

There were no critical articles about PBL or its publications in the sample related to climate.20 Also, no

critical or negative appraisal or reaction can be found in relation to the one Op-Ed piece by a PBL

researcher on climate change.21 Apparently, public debate about PBL publications mainly concerns the

implications of its findings regarding urban planning and housing instead of climate or environment and

17 F de Zeeuw, “Juist nu moeten we volop gaan bouwen”, Volkskrant Opinie, 18 september 2014. T. van Hoek, Analyse

Woonbeleid Overtuigt Nog Niet, Reformatorisch Dagblad, 9 juni 2016 18 De Gelderlander, Veenendaal breidt juist verder uit, 17 juni 2015, A Walthaus, Investeren in Krimpgebieden is Zinloos, Leeuwarder Courant, 9 sept 2011. 19 Jan Willem van de Groep, , “Warmte dumpen of woonplezier creëren”, Cobouw section Duurzaam, 17 april 2015, F. van Beusekom, “Natuur wordt overgeleverd aan boeren en megalomanen”, Trouw Podium, 15 feb 2011 20 In an attempt to find critical comments in the full list of newspaper articles on climate, one article was found, which pointed at the difficulties of inferring from historical data future developments as far as to 2050, NRC-Next, NRC, “Wees niet weer Kodak!”, 3 dec 2015. 21 E. Buitelaar, “Stop meer energie in beleggen in een beter klimaat; Overheid kan helpen met deltaplan voor klimaat en energie als beleggingsalternatief voor commercieel vastgoed”, Het Financiëel Dagblad, Opinie, 12 juni 2015.



sustainability two areas that have been (and are) debated forcefully, but apparently in the printed media

of pas years not so much in relation to PBL climate research.



5. Scholarly response to PBL reports As for the internet searches, the 66 PBL reports were searched for in scholarly literature, using Google

Scholar as a source. In view of the response found on the internet, scholarly attention may seem modest.

However, many of the reports also have led to academic output in scientific journals. For an analysis of

the reception of this specific body of output, see the separate report on this subject provided by CWTS.

Also, reports are often written in Dutch, leading to a diminished coverage by Anglo-Saxon scientific

journals.

Table 12. Scholarly citations for frequently cited reports according to Google Scholar

Citations in

Google

Scholar1

Hans van Grinsven, Our Nutrient World: challenge to produce more food and energy with

less pollution, 2013

143

Maarten Hajer , De Energieke Samenleving, 2011 72

Gerbert Romijn, Gusta Renes, Algemene leidraad voor maatschappelijke kosten-

batenanalyse, 2013

22

Edwin Buitelaar, Niels Sorel, Femke Verwest, Frank van Dongen & Arjan Bregman,

Gebiedsontwikkeling en commerciele vastgoedmarkten, 2013

16

Hans Farjon & Rikke Arnouts, Leren van het Energieke Platteland, 2013 11

Koen Schoots; Michiel Hekkenberg; Pieter Hammingh, Nationale Energieverkenning 2016,

2016

9

Martijn Verdonk, Corjan Brink, Herman Vollebergh, Mark Roelfsema, Evaluation of policy

options to reform the EU Emissions Trading System, 2013

7

Westhoek H., Lesschen J.P., Leip A., Rood T., Wagner S., De Marco A., Murphy-Bokern D.,

Pallière C., Howard C.M., Oenema O. & Sutton M.A., Nitrogen on the table, 2016

7

1) corrected data for untraceable sources

Nevertheless, citation takes place for PBL reports. In total, 330 citations to the 66 publications were

found. Compared with the 155 references found for 41 publications in the earlier analysis, this is an

increase in visibility in the domain of scientific literature.

The analysis of visibility of external publications of PBL in scientific literature, part of the previous analysis,

is beyond the remit of the present one.



6. Comparison with other government agencies

For reports of two other government agencies, SCP (the Netherlands Institute for Social Research) and

CPB (Council for Societal Developments), data are available of similar contextual response analyses

conducted in 2016 and 2015 respectively.

In terms of societal response, PBL is smaller than SCP or CPB. This was the conclusion of the previous

analysis of 2012, in the comparison of PBL 2012 with the analysis of SCP of 2008. In this analysis too, the

use of PBL reports is smaller, based on the comparison of the averages of numbers of responses to the 20

most frequently used publications of the three institutes. (figure 10). As discussed also in the 2012

analysis, the difference might be attributed to general characteristics of the bureaus. This involves the

marked differences in scope of subject matter and policy areas and also the characteristics of both CPB

and SCP as established and well entrenched governmental institutions whereas PBL has been established

only in 2008 as a merger of two advisory agencies. These factors may contribute to different patterns of

use and of recognition, including a greater variety of users and a stronger entrenchment among users in

governmental institutions for SCP and CPB reports.

Figure 10 Averages of Societal Responses to PB, SCP and CPB most frequently used publications

Another finding in the analysis of 2012 was the relative high degree of infrequent users, in comparison

with SCP, with a lower degree of response of institutional responders. However, although the degree of

institutional response – expressed also in the volume of frequent use, remains the same, the degree of

infrequent use is diminishing. This shift seems characteristic for PBL as by contrast the numbers for

infrequent users of SCP publications doubled between 2008-2015.



Figure 11 Shifts in response in domains of science, press and society for PBL and SCP

Comparing the shifts in the various domains of response for SCP and PBL, both bureaus show an increase

in the visibility of their reports in scientific literature, as analysed on the basis of Google Scholar data.

However, as the visibility in newspapers for SCP publications remains equal, the visibility of PBL show an

increase.

Lastly, the comparison with SCP in the previous analysis showed a marked difference in audiences, in

particular with respect to interest groups and political opinion. This is also the case in the present analysis.

As is also clear from the sections about parliamentary response and societal response, PBL works in a

politically sensitive environment. The numbers of interests groups and the volume of response from these

groups and political opinion makers both in parties and via blogs is much higher for PBL than for the other

bureaus. In the previous analysis, about 20% of the volume of response to PBL was from interest groups

and political parties and blogs, even to a high percentage of 28% for response among users in

environment and sustainability. In comparison, interest groups and political response for SCP in the

previous period was 8%.

Figure 12 Volume of response for interest groups, compared over time and different bureaus



Currently, interests groups and political response for PBL publication in the societal domain is slightly

down to 12,4%, and 24,3% for the broadly combined sectors of environment, sustainability, climate and

agriculture. However, this is still more than a similar response for SCP and for CPB. Although the

politicized characteristics seems to diminish, which with regard to climate also is the impression in the

sections about parliamentary response and in newspapers, PBL still apparently operates in a more

politicized environment than the other two policy research bureaus. (figure 12)



7. Public engagement of PBL through social media This section is researched and written by Nicolas Robinson-Garcia (Ingenio, CSIC-UPV, Valencia)

7.1. Findings and recommendations from the analysis of response in social media This section represents a tentative effort at analyzing the outreach of scientific and social engagement

activities taken by the Netherlands National Environmental Agency. It is structured in three parts which

look at 1) reactions to PBL’s outreach efforts through Twitter, 2) reactions to PBL’s publications in social

media, policy and news media, and 3) reaction to a selection of PBL reports. Following we describe the

main findings for each of these sections.

The first part focuses on PBL’s efforts to outreach and socially engage through Twitter. Here we observe

that the Twitter account has a national and local outreach with most of its followers based in The

Netherlands. These followers have mixed interests for following PBL’s activities in Twitter and mixed

profiles. However, through the self-description provided in their Twitter account, we could discern three

types of interests: 1) related with environmental policy issues, 2) related with urban planning,

development and economics with a more academic profile, and 3) advocacy groups, lay people and

media. Distinct national figures can be identified within the many followers from political parties such as

D66, students and academics, other research and national institutes and environmental-related firms.

While PBL employs two Twitter accounts, one in Dutch and one in English, it is the former one which is

more active and engages with larger audiences. In fact, almost half of the users following the English

account also follow the Dutch one.

The second part of the report analyses coverage in social media, news outlets and policy documents to

scientific literature produced by PBL within the 2013-2016 period and indexed in Web of Science. Here

we find a global audience covering the research output of PBL. While most of the attention is

concentrated in few publications, the coverage is impressive. Twitter attention is derived mostly from the

United States and the United Kingdom, followed by The Netherlands, Australia and Germany. News

stories covering PBL scientific output come from media such as The Guardian, The New York Times, Le

Monde or Washington Post. In all cases, the publications covered are the results of institutional

collaboration and co-authorship efforts and of articles published in high impact journals such as Nature or

Nature Climate Change. However, some of the coverage is due to specific political circumstances and

cannot be fully attributed to PBL. For instance, the latest peak of attention is due to the recent triumph of

the Trump administration in the White House and their opposition to climate change policies. Still,

publications from PBL were used by institutions such as the EU, the FAO or the World Health Organization

and cited in policy documents, reflecting the societal relevance of PBL’s work in the international sphere.

The third and last part of this report analyses reactions in Twitter to a set of selected reports from PBL.

As it occurred with Twitter users following the activity of PBL, the outreach of these reports is largely

national or local, showing a very different type of engagement into local and national issues. This

underlines the diverselevels of engagement and outreach that PBL is simultaneously addressing and the

diverse societal missions PBL is undertaking (international influence, national impact, etc.). While not

much evidence of contested or disputed knowledge could be found, we did observe discussions related to

the role of The Netherlands in the international sphere and related to local initiatives contribute (or not)

towards reducing carbon emissions.



Data for this part were very hard to retrieve. , We suggest some recommendations to be able to better

monitor and assess the local and global impact of PBL. First, it would be of benefit for PBL to deposit their

reports in repositories that assign unique identifiers such as DOI numbers (e.g., Zenodo, Figshare or

BioArxiv). This would facilitate monitoring of their coverage in social media, news outlets and policy

documents a similar vein to that used to monitor scientific publications. It would also give more visibility

in search engines such as Google Scholar. Second, the topics on which PBL is currently working or has

worked are of a delicate nature due to their political and societal dimension. The audiences approach

performed using Twitter could be of relevance when focusing on particular topics rather than reactions to

specific reports. It would be of interest for further analyses to discuss and select specific topics of interest

to PBL and focus our social media analyses on these rather or as well as on reactions to PBL’s activity.

7.2. Objective, data and methods This report is a pilot study with which we intend to provide further understanding on the level of success

to which PBL fosters discussions in social media and, specifically in Twitter. As a first attempt on doing so,

the nature of the report is more exploratory than informative. It describes current observations on

Twitter discussions and provides a set of recommendations to 1) improve the use of social media to

outreach society and 2) better monitor and identify conversations and mentions to PBL’s work.

OBJECTIVES: Specifically, we aim at four specific goals:

• Describe the audiences PBL is reaching through its Twitter accounts in geographical and thematic

terms.

• Identify discussions related to scientific literature published by PBL within the 2013-2016 period.

• Qualitative analysis of reactions to PBL policy documents

DATA AND METHODS: We combine social network and qualitative analysis as well as descriptive analysis.

We have analyzed PBL’s presence in Twitter and its capability to engage with different audiences as well

as traces of evidence of discussions on the work made by PBL with an especial focus on a list of scientific

reports which were considered to contain contested or disputed knowledge. It is important to note the

explorative nature of any report using social media to analyze any type of phenomenon due to the

volatility of the data, its large volume of noise as well as conceptual limitations making the claim that

what is observed in such medium actually reflect what real perception. Next we describe the three data

sources employed.

Table 13 Description of data sources and limitations

Goal Data source Data Limitations

1 Twitter REST API Twitter users Information is self-reported and not always

complete.

2 Altmetric.com Users discussing PBL’s

pubs.

This analysis was made for all mention up to Jun

2016. Only records with a direct link (e.g., DOI) to a

publication were retrieved

3 Twitter web search Twitter discussions on

selected reports

Date restrictions are not as severe, but results are

noisier, more probability of false negatives

Data from Twitter REST API and Altmetric.com was downloaded in May 22, 2017, while searches on

specific reports took place in April 2017. Publications authored by members from PBL were retrieved from

Web of Science Core Collection using the following search query:



ADDRESS: ((pbl or "Netherlands Environmental Assessment Agency" or "planbureau voor de

Leefomgeving") and netherlands) AND YEAR PUBLISHED: (2013-2016)

Refined by: ORGANIZATIONS-ENHANCED: ( PBL NETHERLANDS ENVIRONM ASSESSMENT AGCY OR

NETHERLANDS ENVIRONM ASSESSMENT AGCY PBL )

All document types were included.

7.3. Audiences following PBL in Twitter

PBL has two Twitter accounts from which it outreaches to national (@Leefomgeving) and foreign

(@nlenvironagency) public created in 2011 and 2010 respectively. Overall, these two accounts are

followed by a total of 8995 accounts. 96% of these follow the Dutch account while 8% follow the English

one. Levels of activity also differ greatly, while @Leefomgeving tweets more than once per day,

@nlenvironagency tweets more than 8 times a month. However, the language distribution of accounts

shows consistent differences on the audiences following each account. Still, there is an overlap with

44.3% of followers of the English account who also follow the Dutch account. While some the overlapping

accounts belong to workers at PBL, most of them belong to Dutch individuals and organizations. The non-

overlapping accounts belong either to foreign individuals or foreign individuals or international

organizations based in The Netherlands. In this regard, it is important to mention that the activity of the

Dutch account compared to the English one is greater and questions to some extent the use of both.

To gain more insight as to who are the people and organizations behind those accounts and their

motivations for following PBL’s activity, we have performed an analysis of the profiles such individuals

based on the description they provide in Twitter. Bear in mind that personal description as most of the

rest of the information available in Twitter profiles is not required and hence, some accounts do not

include any. For the set of users following PBL’s Twitter accounts, 15.1% of the accounts did not include a

description. Based on those which did include a description and did so in English language, a term map

was created using VOSViewer22. The term map is shown in figure 13. This map uses natural language

processing techniques to extract terms from a text corpus. The distance between terms indicates the

number of times they co-occur, that is, they are present in the same description of a user (shorter

distances means larger co-occurrences).

Colors of nodes represent words showing different profiles, these are calculated using the VOS clustering

technique for community detection. As observed, three distinct groups of terms based on the self-

description included in their profile are observed. After manually analyzing each of the terms belonging

to each cluster, we observe the following pattern. Terms from the green cluster represent lay people

concerned with environmental issues and students. The second group (blue) is formed by academics from

fields of the social sciences such as economics, geography, communication studies, urban planning and

development or architecture. The last group (yellow) is that formed by professionals as well as academics

involved in environmental policy. Terms such as development, renewable energy, climate or policy arise.

But also, others such as personal view, Europe or endorsement are present, showing elements that point

towards advocacy work.

While the three clusters that emerge from the term map seem to represent three distinct groups, observe

that the layout of the map shows that they are intertwined, meaning that users following PBL’s Twitter

activity tend to have mixed components of each of these three groups.

22 Van Eck, N.J., & Waltman, L. (2011). Text mining and visualization using VOSviewer. ISSI Newsletter, 7(3), 50–54.



Figure 13 Co-word analysis of the description of PBL followers. Only terms in English are included

Based on these three communities we look into the most prominent accounts of followers, that is, those

with the largest follower-followee ratio.



Table 14 Top 20 following accounts with the highest follower/friend ratio. Colors represent types of audiences as defined in Figure 13.

Account Description Country Followers Fol./fri.

ratio Audience

1 Alexander Pechtold Group Chairman and

MP of Culture for D66 Netherlands 624677 1053,42

2 Duurzaam Ondernemen Netherlands Enterprise

Agency Netherlands 49032 231,28

3 UN Environment

United Nations

Environment

Programme

Kenya 617039 209,52

4 Ministerie IenM

Dutch Ministry of

Infrastructure and

Environment

Netherlands 29011 208,71

5 RVO.nl Netherlands Enterprise

Agency Netherlands 65620 147,46

6 Herman Wijffels Professor University of

Utrecht Netherlands 13881 133,47

7 Roger Scruton Professor from

Buckingham University UK 11784 109,11

8 Cycling Professor

Researcher from

University of

Amsterdam


9 Mathijs Bouman Journalist and

Economist France 55386 85,34

10 MinisterieEZ Dutch Ministry of

Economy Netherlands 39087 65,80

11 Duurzamestad Rotterdam

Environmental Center Netherlands 5834 64,82

12 ArchiNed

Foundation on urban

planning and

architecture

Unknown 7828 64,69

13 NatureTodayNL News media about

biology Netherlands 12907 62,96

14 De Groene News media Netherlands 51207 60,74

15 Trouw Groen! News media about

nature Netherlands 18588 54,83

16 RIVM

Dutch National Institute

for Public Health and

the Environment


17 Klimaatplein.com Green energy firm Netherlands 4922 51,27

18 AM Green energy firm Netherlands 5635 51,23

19 De Vlinderstichting

Foundation on

conservation of

butterflies


20 Vastgoedmarkt Real state firm Netherlands 12495 45,60



Table 14 shows the top 20 most influential followers PBL currently has. Based on the description of their

profile and the three clustering of terms observed above, we have manually assign a group or two to each

of these profiles. Note that this is a tentative classification based on the description of the profile and is

subject to discussion. For instance, Herman Wijfells is considered to belong to the blue and green clusters

as he has been involved both in academia and politics. On the other hand, Cycling Professor is the profile

of an academic involved in a graduate level programme of the University of Amsterdam entitled “Planning

the Cycling City”.

As observed, among these influential accounts we find national political figures such as Alexander

Pechtold as well as ministries and international organizations. The list is also formed by academics and

advocacy groups, as well as mixed profiles were academia, environmental policy and advocacy and

societal concerns intertwine.

7.4. Outreach of PBL output in social media We identified a total of 282 publications indexed in Web of Science Core Collection and published in the

2013-2016 period and searched their coverage in social media from 2013 to 2017. Figure 14 shows the

chronological distribution of mentions to research publications from different social media and sources.

As observed, most of the mentions come from Twitter, although there is a large presence of mentions

from news media, blogs and policy documents.

We observe five distinct peaks during the period of analysis. The first and smallest one takes place in

September, 2014 around the article entitled ‘Persistent growth of CO2 emissions and implications for

reaching climate targets’. This paper, published in Nature Geoscience, warns on the need to reduce

carbon emissions which have not diminished as promised by governments. This paper receives a large

media coverage in news press such as The Guardian, Le Monde, BBC News or Scientific American.

Figure 14 Chronology of mentions in social media to scientific publications authored by PBL between 2013

and 2016



The second observed peak takes place of December, 2015, where we highlight again the press coverage of

a single paper entitled ‘Biophysical and economic limits to negative CO2 emissions’ published in Nature

Climate Change, related with CO2 emissions. The Huffington Post, The Guardian, Yahoo News, The

Conversation or Business Insider are some of the media who refer to this paper. The largest peak takes

place in June 2016 due to a publication of a perspectives piece in Nature entitled ‘Paris Agreement

climate proposals need a boost to keep warming well below 2 °C’. This paper continues to be present in

many media stories due to the latest decision by president Trump to withdraw from the Paris Agreement.

The Washington Post, Business Standard, The Guardian, World Economic Forum, ABC, The Huffington

Post among others have reffered to this paper.

The fourth peak takes place a couple of months later, in September 2016. In this case, more than one

paper are being referred to in news media and blogs. The Nature piece is still being covered by the media,

but papers such as ‘Assessing the land resource–food price nexus of the Sustainable Development Goals’,

published in Science Advances, or ‘Similar estimates of temperature impacts on global wheat yield by

three independent methods’, published in Nature Climate Change, are also being covered in news outlets

and scientific blogs. Lastly, in November 2016 an outburst of news coverage takes place with 154 news

outlets and up to five policy documents referring to publications authored by researchers at PBL. This

time it is not related with the publication of a new study but with an interest on climate change and its

effects. For instance, George Monbiot talks at The Guardian about ‘The 13 impossible crises that humanity

now faces’ referring to the Nature paper aforementioned. Another important trend for explaining the

outburst is the winning of the US elections by Trump. Hence we observe that The New York Times,

Business Insider, Today and National Geographic warn about the new position the Trump administration

may take on Climate Change and cite the paper ‘Global carbon budget 2016’ published in Earth System

Science Data. Finally, we must note all of these papers mentioned are all in collaboration with many

countries, emphasizing the importance of international collaboration at PBL for outreaching the global

community.

The distribution of coverage of publication in social media is highly skewed. Seven documents represent

10% of the most mentioned publications from PBL within the period 2013-2016. A complete list of all

publications with mentions in social media is included as supplementary material. A selection of the most

relevant mentions is included. These are number of tweets, news stories, blog posts and citations from

policy documents.

Twitter users discussing scientific output from PBL originate from a total of 86 countries. Appendix A

includes the total number and share of tweets and Twitter accounts from each country. United States,

United Kingdom and the Netherlands represent 33% of the total audience. Figure 15 shows the

geographical distribution of Twitter users discussing PBL scientific publications.



Figure 15 Demographic distribution of tweets directed at publications from PBL. 2013-2016 period

Table 15 Frequently used publications in social media and mentions in most relevant sources

Publication News Blogs Policy Wikipedia Tweets

Global Carbon Budget 2016

145 11 2 0 123

Earth System Science Data, 2016

Paris Agreement climate proposals need a boost to

keep warming well below 2 °C 82 23 4 1 326

Nature, 2016

Reducing emissions from agriculture to meet the 2°C

target

31 14 2 0 297

Global Change Biology, 2016

Biophysical and economic limits to negative CO2

emissions

27 16 2 0 270

Nature Climate Change, 2015

Differences between carbon budget estimates

unraveled

36 13 1 0 77


Scientists' views about attribution of global

warming.

11 14 0 3 233

Environmental Science & Technology, 2014

Food choices, health and environment: Effects of

cutting Europe's meat and...

23 6 1 0 116

Global Environmental Change Part A, 2013

Persistent growth of CO2 emissions and implications

for reaching climate targets

16 14 0 0 80

Nature Geoscience, 2014

Similar estimates of temperature impacts on global

wheat yield by three...

23 6 0 0 79


Sharing a quota on cumulative carbon emissions 14 10 0 0 153


Greenhouse gas mitigation potentials in the

livestock sector

21 6 15 0 71



Publication News Blogs Policy Wikipedia Tweets


Transport: A roadblock to climate change

mitigation?

20 1 0 0 97

Science, 2015

A mid-term analysis of progress toward international

biodiversity targets

17 3 2 0 108

Science, 2014

National post-2020 greenhouse gas targets and

diversity-aware leadership

29 2 0 0 58


Table 15 shows the publications which have received the greatest attention according to different sources

of social media. As observed, issues such as the Paris Agreement or the need for greenhouse gas and

carbon emissions, which have been in the spotlight in the last few years seem to drive the large presence

of these publications in social media, news stories and other sources. Looking closely at the number of

counts by source, we observe that there is no clear relation between them, underpinning that there is a

large variety of motivations depending on the analyzed sources explaining the attention received by each

article according to each media of attention.

14 international and national institutions have made use of the scientific output generated by PBL to

develop new policies and guidelines. These organizations have produced a total of 71 documents in which

they have references to journal publications from PBL. Table 16 provides the institutions and number of

policy documents reported by each institution citing the work of PBL.

Table 16 Number of policy documents by organization citing the work of PBL

Organization # policy docs.

The Publications Office of the European Union 18

Food and Agriculture Organization of the United Nations 14

overheid.nl 9

Australian Policy Online 7

rijksoverheid.nl 7

National Academies Press 6

Intergovernmental Panel on Climate Change 2

Chatham House 2

UK Government (GOV.UK) 1

World Health Organization 1

World Bank 1

The International Fund for Agricultural Development 1

European Food Safety Authority 1

The Inter-American Development Bank 1

Table 17 focuses on the top cited publications in policy documents. It shows for each publication, the

number of policy documents citing such paper and the relation of policy document along with the

organization behind them. While international institutions are using the scientific output of PBL to



develop their own policies and guidelines, we observe that the EU is one of the main organizations looking

at the output of PBL and referring to it when developing new policies.

Table 17 Frequently used publications in policy documents and examples of citing documents

Policy documents (examples) #

A new scenario framework for climate change research: the concept of shared

socioeconomic pathways

Climatic Change, 2013

6

- Including Health in Global Frameworks for Development,

Wealth, and Climate Change: Workshop Summary

National Academies Press

- The State of Food and Agriculture 2016 (SOFA): Climate change,

agriculture and food security

FAO of the United Nations

- Human health and climate change in Pacific Island countries World Health Organization

- Review of the U.S. Global Change Research Program's Update

to the Strategic Plan Document


- Extension of the CAPRI model with an irrigation sub-module -

EU Law and Publications

Publications Office of the EU

- Science for disaster risk management 2017 - EU Law and

Publications


A blueprint for mapping and modelling ecosystem services

Ecosystem Services, 2013 5

- Soil threats in Europe Publications Office of the EU

- Mapping and assessment of forest ecosystems and their

services


- A Retrospective Stated Preference Approach to Assessment of

Coastal Infrastructure Investments: An Application to Barbados

The Inter-American

Development Bank

- An introduction to the digital observatory for protected areas

(DOPA) and the DOPA explorer (Beta)


- ESTIMAP Publications Office of the EU

Greenhouse gas mitigation potentials in the livestock sector

Nature Climate Change, 2016 3

- Sustainable Agricultural Development for Food Security and

Nutrition: What Roles for Livestock?


- Greenhouse Gas Mitigation Opportunities in Agricultural

Landscapes : A Practitioner’s Guide to Agricultural and Land

Resources Management

World Bank

- Action on the land: reducing emissions, conserving natural

capital and improving farm profitability - an issues paper

Australian Policy Online

Mapping and modelling ecosystem services for science, policy and practice

Ecosystem Services, 2013 3

- An introduction to the digital observatory for protected areas

(DOPA) and the DOPA explorer (Beta)


- Opportunities for the Gulf Research Program: Monitoring

Ecosystem Restoration and Deep Water Environments:

Summary of a Workshop


- Soil threats in Europe Publications Office of the EU



Assessing the impacts of livestock production on biodiversity in rangeland ecosystems

Proceedings of National Academy of Sciences, 2013 3

- A review of indicators and methods to assess biodiversity -

Application to livestock production at global scale


- Principles for the assessment of livestock impacts on

biodiversity


- Sustainable Value Chains for Sustainable Food Systems FAO of the United Nations

7.5. Outreach of selected publications

The searches in Google and BING (section 3) led to 34 reports for which URLs related to Twitter were

found. Of these 34 reports five were selected based on the numbers of retrieved retweets. It should be

noted that a full retrieval of all Twitter information is limited, however. The information below intends to

give an impression of Twitter conversations related to the five reports. (Table 18)

From the analyzed reports, ‘Wat betekent het Parijsakkoord voor het Nederlandse langetermijn-

klimaatbeleid?’ is the one which has raised more discussion in Twitter due to its political contents as it

mentions the commitment reached by the Dutch government to reduce carbon emissions.

Table 18 Main topics of discussion and attention received by selected reports from PBL

Report Motivation

for

discussion

User

first tweet

Retweets Responses

Nationale Energieverkenning 2016 Hashtag Jacques van de Worp 48 --

Wat betekent het Parijsakkoord voor

het Nederlandse langetermijn-

klimaatbeleid?

CO2 emissions Gerben Jan Gerbrandy 33 4

Aanpassen aan klimaatverandering Climate change PBL 30 1

Dalende bodems, stijgende kosten Environmental

policy

Paul van de Vijver 10 1

Grondstof voor een circulaire

economie

Scientific

outreach

Ron Nuwenhof 9 0

Nationale Energieverkenning 2016

This report was disseminated due to the following of the hashtag #Nev2016. This hashtag had been

previously used for other events by other communities. After removing non-relevant tweets, we observe

that it started to be used due to an event in which PBL participated. The first tweet referring to it dates

from October 6, 2016. However, tweets referring to it have been continuously published up to March

2017. Most of these tweets are reporting different excerpts of the report as well as broadcasting the

event with pictures from the presentations. The profile of users tweeting about it is basically of those

related with ecologists (e.g., there is a member from Greenpeace) as well as environmental researchers.

Wat betekent het Parijsakkoord voor het Nederlandse langetermijn-klimaatbeleid?

Tweets generated around this report have to do with the commitment reached by the Dutch government

to reduce carbon emissions after the Paris Agreement. Although first tweets are very similar, they are

posted independently raising parallel discussions in similar dates.



The first tweet took place on November 17, 2016 at 4.52pm. It was posted by Gerben Jan Gerbrandy,

member of the European Parliament and of the D66 political party. He refers to the commitment reached

by the dutch government. Leon de Graff, environment adviser, responds challenging such commitment

due to current policies taking place. Among retweets received by Gerben Jan Gerbrandy we find another

member of D66 as well as Floris van Hövell, from the Shell Oil company. Few minutes later, Henri

Botenbal tweets a similar messagealso linking to PBL's report. In this case two responses dismiss the news

with sarcastic comments on government policy favoring biomass. A day later the foundation Het Groene

Brein tweets a similar post. In this case, it receives five retweets and a like but no further responses.

However, the tweet which gains most attention is that by Jan Rotmans, professor of Transit Science and

opinion leader. He highlights the difficulties the country faces for such effort. He receives two responses.

Marc van Gemert seems positive with government's actions. Roel van Nieuwstadt sarcastically responds

that in Zeist more carbon emissions will be generated due to a new energy station being planned. This

engages on a discussion with Martijn Harman, from the Ministry of Defense, about the best options to

reduce emissions.

Aanpassen aan klimaatverandering

In this case, PBL is the one which first tweets about the report getting up to 30 retweets and seven likes.

Most of the people retweeting the report are either environmental advocates or people with some policy

position. For instance, Nicole Olland, politician from D66 or ‘Vrouwen In De Techniek’, a group of female

students are among the users retweeting the report. The only response received is actually a quote to the

original tweet.

Dalende bodems, stijgende kosten

In this case, the discussion takes place between two individuals and seems to be very localized. Paul van

de Vijver, advisor of the District Water Control Board of Leiden discusses the report and is replied by Paul

Vertegaal, who defines himself as someone who respects environment and nature. Retweets do not show

any particular pattern on the profile of users.

Grondstof voor een circulaire economie

This report was first tweeted simultaneously by Ron Nuwenhof, sector manager of High Tech for the

Development Agency East Netherlands, and Maarten Goorhuis, senior policy advisor of the Dutch Solid

Waste Association. Ron Nuwenhof receives a retweet and like from Stefan Nijwening, strategic advisor to

the Board of the Water Authority Vechtstromen.

The next day (29 sept, 2016) the report is tweeted by Menno Smit praising it, he is an ecologist who

received a tweet and two likes. In october 3, PBL tweeted the report and was further retweeted 7 times

and received two likes. The profile of these users was mainly from Economics.



Addendum I. Country list of tweets and Twitter accounts mentioning publications from PBL

COUNTRY TWEETS % TWEETS ACCOUNTS % ACCOUNTS

Not available 1517 37,00% 941 36,80%

United States 582 14,20% 407 15,90%

United Kingdom 541 13,20% 332 13,00%

Netherlands 251 6,10% 105 4,10%

Australia 223 5,40% 108 4,20%

Germany 118 2,90% 64 2,50%

Canada 109 2,70% 83 3,20%

Sweden 93 2,30% 45 1,80%

Spain 70 1,70% 54 2,10%

France 62 1,50% 39 1,50%

Norway 52 1,30% 32 1,30%

Italy 41 1,00% 25 1,00%

Ireland 37 0,90% 17 0,70%

Finland 35 0,90% 18 0,70%

Belgium 33 0,80% 29 1,10%

Switzerland 22 0,50% 16 0,60%

Austria 22 0,50% 7 0,30%

Indonesia 20 0,50% 13 0,50%

Brazil 19 0,50% 14 0,50%

South Africa 19 0,50% 16 0,60%

Mexico 16 0,40% 10 0,40%

Denmark 14 0,30% 12 0,50%

Kenya 12 0,30% 8 0,30%

India 11 0,30% 11 0,40%

New Zealand 11 0,30% 9 0,40%

Singapore 11 0,30% 8 0,30%

Japan 10 0,20% 7 0,30%

Other countries 153 3,60% 127 4,74%



Addendum II: lists of tables and figures

Table 1 Comparison of numbers of speakers in selected meetings for PBL, CPB and SCP ............................. 9 Table 2 Speakers mentioning PBL in parliament per party and subject 2012-2017 ..................................... 10 Table 3 Parliamentary Appreciation of PBL reports and points of views 2012-2017 ................................... 11 Table 4. Twenty reports most frequently used, by unique references ......................................................... 14 Table 5 Numbers of unique referring websites according to the numbers of used PBL publications .......... 15 Table 6 User Categories according to Sector ................................................................................................ 16 Table 7 Comparison of numbers of users in four broad categories of sectors, 2012-2017 .......................... 16 Table 8 Comparison of numbers of users according to function, 2012 - 2017 ............................................. 17 Table 9 Typically frequent use of specific publications in the most important sectors ................................ 21 Table 10 PBL publications apparently related to articles in printed media .................................................. 31 Table 11 Reactions on PBL or PBL publications in larger newspaper articles ............................................... 32 Table 12. Scholarly citations for frequently cited reports according to Google Scholar............................... 34 Table 13 Description of data sources and limitations ................................................................................... 39 Table 14 Top 20 following accounts with the highest follower/friend ratio. Colors represent types of audiences as defined in Figure 13. ................................................................................................................ 42 Table 15 Frequently used publications in social media and mentions in most relevant sources ................. 45 Table 16 Number of policy documents by organization citing the work of PBL ........................................... 46 Table 17 Frequently used publications in policy documents and examples of citing documents ................ 47 Table 18 Main topics of discussion and attention received by selected reports from PBL .......................... 48 Table 19. Sources used in social impact analysis .......................................................................................... 55

Figure 1 Users according to broad sectors plotted against function 2012 ................................................... 19 Figure 2 Users according to broad sectors plotted against function 2017 ................................................... 19 Figure 3 Response from various sectors for two signal reports and structural reports ............................... 23 Figure 4 Response from various sectors to reports related to Urban Planning and Housing ....................... 24 Figure 5 Sector Response to publications on issues of Environment and Sustainability .............................. 25 Figure 6 Sector response to publications on issues regarding Climate Change ............................................ 26 Figure 7 Sector response to publications on agriculture and food ............................................................... 27 Figure 8 Sector response to publications on transport and water ............................................................... 28 Figure 9 News items per subject and type of outlet ..................................................................................... 31 Figure 10 Averages of Societal Responses to PB, SCP and CPB most frequently used publications ............. 35 Figure 11 Shifts in response in domains of science, press and society for PBL and SCP ............................... 36 Figure 12 Volume of response for interest groups, compared over time and different bureaus................. 36 Figure 13 Co-word analysis of the description of PBL followers. Only terms in English are included .......... 41 Figure 14 Chronology of mentions in social media to scientific publications authored by PBL between 2013 and 2016 ........................................................................................................................................................ 43 Figure 15 Demographic distribution of tweets directed at publications from PBL. 2013-2016 period ........ 45



Addendum III: list of investigated reports

• Nationale Energieverkenning 2016, (2016), Koen Schoots; Michiel Hekkenberg; Pieter Hammingh

• Opties voor energie- en klimaatbeleid, (2016), J. Ros en K. Schure

• Global and regional abatement costs of INDCs and of enhanced action to levels well below 2 °C and 1.5 °C,

(2016), Andries Hof; Michel den Elzen; Annemiek Admiraal; Mark Roelfsema

• Wat betekent het Parijsakkoord voor het Nederlandse langetermijn-klimaatbeleid?, (2016), Detlef P. van Vuuren,

Pieter Boot, Jan Ros, Andries Hof en Michel den Elzen

• Kansrijk woonbeleid (PVS-1837), (2016),

• Transformatiepotentie: woningbouwmogelijkheden in de bestaande stad, (2016), Lianne van Duinen, Bart Rijken,

Edwin Buitelaar

• Samenvatting voor Beleidsmakers van de IPBES assessment van scenario’s en modellen voor biodiversiteit en

ecosysteemdiensten , (2016), Marcel T. J. Kok (PBL), Kasper Kok (WUR), Garry D. Peterson (Stockholm University),

Rosemary Hill (James Cook University), John Agard (University of the West Indies), Stephen R. Carpenter

(University of Wisconsin-Madison)

• Belastingverschuiving: meer vergroening en minder complexiteit?, (2016), Herman Vollebergh; Justin Dijk; Eric

Drissen; Hans Eerens; Gerben Geilenkirchen

• Grondstof voor een circulaire economie, (2016), Trudy Rood en Aldert Hanemaaijer

• Het belang van een thuismarkt voor de export van eco-innovaties, (2016), Alexander van Vooren; Bart Wesselink

• Monitor Infrastructuur en Ruimte 2016, (2016),

• Balans van de Leefomgeving 2016. Zie ook website www.pbl.nl/balans, Richting geven - Ruimte maken (2016),

Melchert Reudink, Niels Sorel, Kees Schotten

• Natuurlijk kapitaal: naar waarde geschat. Zie ook website: http://themasites.pbl.nl/natuurlijk-kapitaal-nederland/,

(2016), Petra van Egmond, Arjan Ruijs

• De verdeelde triomf, Verkenning van stedelijk-economische ongelijkheid en opties voor beleid. Ruimtelijke

Verkenningen 2016 (2016), Edwin Buitelaar, Anet Weterings, Otto Raspe, Olaf Jonkeren (allen PBL) & Willem

Boterman (UvA)

• Kansrijk Mobiliteitsbeleid, (2016), Annemiek Verrips (CPB), Anco Hoen (PBL)

• Betaalbaarheid van het wonen in de huursector. Verkenning van beleidsopties, (2016), Carola de Groot, Femke

Daalhuizen, Frans Schilder, Joost Tennekes

• Koers houden in de delta, Ontwerp van een monitorings- en evaluatiekader voor het Deltaprogramma (2016),

Willem Ligtvoet, Eva Kunseler en Ron Franken

• Rapport UNEP IRP: Food systems and Natural Resources, (2016), Henk Westhoek

• Waterkwaliteit nu en in de toekomst, Eindrapportage ex ante evaluatie van de Nederlandse plannen voor de

Kaderrichtlijn Water (2016), Frank van Gaalen, Aaldrik Tiktak, Ron Franken, Erwin van Boekel, Peter van

Puijenbroek en Hanneke Muilwijk

• Dalende bodems, stijgende kosten, (2016), G.J. van den Born, F. Kragt, D. Henkens, B. Rijken, B. van Bemmel en S.

van der Sluis

• Nitrogen on the table, The influence of food choices on nitrogen emissions and the European environment (2016),

Westhoek H., Lesschen J.P., Leip A., Rood T., Wagner S., De Marco A., Murphy-Bokern D., Pallière C., Howard

C.M., Oenema O. & Sutton M.A.

• rapport De stad: magneet, roltrap, spons, (2015), Frank van Dam, Gijs Beets, Andries de Jong, Dorien Manting

• rapport De economie van de stad, (2015), Otto Raspe, Peter Zwaneveld en Sara Delgado

• Nationale Energieverkenning 2015, (2015), Koen Schoots (ECN) en Pieter Hammingh (PBL)

• Implications of long-term scenarios for medium-term targets, (2015), Detlef P. van Vuuren; Mariësse van

Sluisveld; Andries F. Hof

• Arbeidsmarkt zonder grenzen, (2015), Anet Weterings (PBL), Gusta van Gessel-Dabekaussen (CBS)

• 2015 Beoordeling Programmatische Aanpak Stikstof: De verwachte effecten voor natuur en vergunningverlening,

(2015), Dirk-Jan van der Hoek, Rob Folkert en Rikke Arnouts

• Trends in Nederlandse voetafdrukken, (2015), Harry Wilting, Aldert Hanemaaijer, Mark van Oorschot en Trudy

Rood

• De veerkrachtige binnenstad, (2015), David Evers, Joost Tennekes, Frank van Dongen

• Wetenschap en beleid verbinden, Een terugblik op de eerste zeven jaar van het PBL (2015), Jacqueline

Timmerhuis



• Klimaatverandering, Samenvatting van het vijfde IPCC-assessment en een vertaling naar Nederland (2015),

Willem Ligtvoet & Bram Bregman (redactie), Rob van Dorland (KNMI), Wilfried ten Brinke (Blueland), Rolf de Vos

(Ecofys), Arthur Petersen (University College London), Hans Visser (PBL)

• Nederland in 2030 en 2050: twee referentiescenario's, Toekomstverkenning Welvaart en Leefomgeving (WLO)

(2015), Ton Manders

• Cahier Regionale ontwikkelingen en verstedelijking, WLO (2015), Ton Manders

• Cahier Mobiliteit, WLO (2015), Ton Manders

• Cahier Klimaat en energie, WLO (2015), Ton Manders

• Balans van de Leefomgeving 2014 - UK versie, (2015), Olav-Jan van Gerwen, Nico Hoogervorst, Gert Eggink,

Laurens Brandes, Guus de Hollander

• De ruimtelijke metamorfose van Nederland 1988-2015, Het tijdperk van de Vierde Nota (2015), Ries van der

Wouden

• Adaptation to climate change in the Netherlands, Studying related risks and opportunities (2015), Willem

Ligtvoet, Rijk van Oostenbrugge, Joost Knoop, Hanneke Muilwijk and Marijke Vonk

• Aanpassen aan klimaatverandering, Kwetsbaarheden zien, kansen grijpen (2015), Rijk van Oostenbrugge, Joost

Knoop, Hanneke Muilwijk, Marijke Vonk, Willem Ligtvoet

• Belastingkortingen voor zuinige auto’s: afwegingen voor fiscaal beleid, (2014), Gerben Geilenkirchen (PBL),

Robert Kok (PRC), Anco Hoen (PBL), Fabian van der Linden (PRC), Hans Nijland (PBL)

• How Sectors can contribute to sustainable use and conservation of biodiversity, (2014), Marcel Kok, Rob

Alkemade (eds), Michel Bakkenes, Eline Boelee (Water Health), Villy Christensen (UBC), Martha van Eerdt, Stefan

van der Esch, Jan Janse (PBL), Sylvia Karlsson-Vinkhuyzen (WUR), Tom Kram, Tanya Lazarova, Vincent Linderhof

(WUR), Paul Lucas, Maryia Mandryk, Johan Meijer, Mark van Oorschot, Louise Teh (UBC), Luc van Hoof (IMARES),

Henk Westhoek, Roderick Zagt (Tropenbos International).

• Monitor Duurzaam Nederland - Verkenning, Uitdagingen voor adaptief energie-innovatiebeleid. (2014), CPB

(Centraal Planbureau), PBL (Planbureau voor de Leefomgeving), SCP (Sociaal en Cultureel Planbureau)

• Green gains (vertaling "Vergroenen en verdienen" uit 2013), In search of opportunities for the Dutch economy

(2014), Aldert Hanemaaijer, Ton Manders, Otto Raspe, Martijn van den Berge, Laurens Brandes, Stefan van Esch,

Jos Notenboom and Melchert Reudink

• Kiezen en delen, Strategieën voor een betere afstemming tussen verstedelijking en infrastructuur (2014), David

Hamers, Daniëlle Snellen, Marnix Breedijk, Hans Hilbers, Kersten Nabielek, Joost Tennekes

• IABR boek Slimme steden – de opgave voor de 21e eeuwse stedenbouw in beeld, (2014), Maarten Hajer, Ton

Dassen

• Balans van de Leefomgeving 2014, zie ook website www.pbl.nl/balans, De toekomst is nú (2014), Olav-Jan van

Gerwen, Nico Hoogervorst, Gert Eggink, Laurens Brandes, Guus de Hollander

• Kleine kansen, grote gevolgen, Slachtoffers en maatschappelijke ontwrichting als focus voor het

waterveiligheidsbeleid (2014), Joost Knoop, Willem Ligtvoet

• Doorrekeningen SER energieakkoord, (2013), PBL en ECN

• The Netherlands in 21 infographics (vertaling van "Nederland Verbeeld" uit 2012), (2013),

• Algemene leidraad voor maatschappelijke kosten-batenanalyse, (2013), Gerbert Romijn, Gusta Renes

• Evaluation of policy options to reform the EU Emissions Trading System, (2013), Martijn Verdonk, Corjan Brink,

Herman Vollebergh, Mark Roelfsema

• Vergroenen en verdienen, Op zoek naar kansen voor de Nederlandse economie (2013), Aldert Hanemaaijer, Ton

Manders, Otto Raspe, Martijn van den Berge, Laurens Brandes, Stefan van Esch, Jos Notenboom en Melchert

Reudink

• Plannen voor de stad, Een multidisciplinaire verkenning van de effecten van verstedelijkingsprojecten op het

functioneren van een stad (2013), Gerbert Romijn, Gusta Renes

• Changing track, changing tack (vertaling van "Wissels omzetten"), Dutch ideas for a robust environmental policy

for the 21st century (2013), Nico Hoogervorst, Maarten Hajer, Frank Dietz, Jacqueline Timmerhuis, Sonja

Kruitwagen

• Wissels omzetten, Bouwstenen voor een robuust milieubeleid voor de eenentwintigste eeuw (2013), Nico

Hoogervorst, Maarten Hajer, Frank Dietz, Jacqueline Timmerhuis, Sonja Kruitwagen

• Leren van het Energieke Platteland, (2013), Hans Farjon & Rikke Arnouts



• Verduurzaming van internationale handelsketens, Voortgang, effecten en perspectieven (2013), Mark van

Oorschot, Marcel Kok, Johan Brons, Stefan van der Esch, Jan Janse, Trudy Rood, Edward Vixseboxse, Harry Wilting

(PBL) & Walter Vermeulen (Universiteit Utrecht)

• Gebiedsontwikkeling en commerciele vastgoedmarkten, een institutionele analyse van het (over)aanbod van

winkels en kantoren (2013), Edwin Buitelaar, Niels Sorel, Femke Verwest, Frank van Dongen & Arjan Bregman

• Vergrijzing en ruimte, (2013), Frank van Dam, Femke Daalhuizen, Carola de Groot, Manon van Middelkoop,

Pautie Peeters

• Samenhang in de Zuidwestelijke Delta (Hoofdrapport), Naar een vitale, veerkrachtige en veilige Zuidwestelijke

delta (2013), Rienk Kuiper, Marijke Vonk, Dirk-Jan van der Hoek

• De macht van het menu - opgaven en kansen voor duurzaam en gezond voedsel, (2013), Henk Westhoek, Trudy

Rood, Martha van Eerdt, Marjolein van Gelder, Hans van Grinsven, Melchert Reudink & Henk van Zeijts

• Ex ante evaluatie mestbeleid, Gevolgen van de invoering van verplichte mestverwerking en het afschaffen van

productierechten in de veehouderij (2013), Jaap Willems, Hans van Grinsven, Jaap Schröder, Willy Baltussen,

Tanja de Koeijer, Ferry Leenstra, Gerard Velthof en Nico Verdoes

• Our Nutrient World: challenge to produce more food and energy with less pollution, (2013), Hans van Grinsven

• Nederland verbeeld , (2012),

• Natuurverkenning 2010-2040, (2012), Rijk van Oostenbrugge, Petra van Egmon, Ed Dammers, Arjen van Hinsberg

- Dick Melman, Janneke Vader, Wim Wiersinga

• De Energieke Samenleving, (2011), Maarten Hajer



Addendum IV. Analyzing PBL reports This study is based on the recommendations of the ERiC and the EU FP7 SIAMPI-projects, concerning the presentation of evidence of societal impact and relevance in research evaluation. Following ERiC and SIAMPI it is crucial that diversity of use is shown, that users are identified and that, where possible, users are involved in the evaluation. The results of the ERiC and SIAMPI projects also led to the recommendation that societal results are regarded as productive outcomes of interaction, i.e. indications of active use.23 Contextual Response Analysis, a method developed in the context of these projects, aims to identify active users, to analyse the diversity of the social domains of use, including science, parliament, press and society. Contextual Response Analysis uses several sources, listed in table 19. In the investigation of societal use, search results are obtained using two search engines. Outcomes of

research, such as publications, are searched for, based on search strings taken from titles, subtitles, press

releases etc. Each search string is checked for specificity, by investigating the appropriate linkages of

search results at the end of the search list. There are two phases in the analysis.

• In phase one each report is individually investigated both on how many times it is used and also in

which social domain it is used. In internet searches, all search results are included to avoid ranking

biases.

• Secondly, each user found in phase one is classified according to various characteristics. This is the

most elaborate phase in the analysis which is necessary to verify the results of phase one. This

results also in detailed information of who is using a specific report or book, and in which social

domain use occurs. As the method traces users that are yet unknown to researchers, the outcome

can also provide valuable information about (desired or hitherto unknown) stakeholders.

Table 19. Sources used in social impact analysis

Source Domain of Response

Zoek.Officiëlebekendmakingen.nl Parliament

Google Scholar Scientific Literature

Lexis Nexis Printed media (newspapers)

Google, BING General societal use, domains to be specified

The Contextual Response Analysis uses data that can be found and verified in various databases including

the internet. However, researchers and representatives of the agency are often also involved in direct

interaction with policy makers and interested others, in meetings, conferences and advisory relationships.

Although direct interaction is undoubtedly a very substantial task of the agency and an effective way to

bring results to the attention of relevant audiences, the analysis provided here makes no claims about use

that results from it.

23 Spaapen, Jack, Huub Dijstelbloem, and Frank Wamelink. "Evaluating research in context." A method for comprehensive

assessment, 2nd edition, The Hague: COS (2007). Spaapen, J. B., & Drooge, L. van (2011). Introducing'productive interactions' in social impact assessment. Research Evaluation, 20. Zie ook: Cozzens, S & Snoek, M, Knowledge to Policy, Contributing to the Measurement of Social, Health, and Environmental Benefits, Workshop on the Science of Science Measurement, Washington, DC, 2 –3 December 2010; Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. Sage. Gibbons, M., & Nowotny, H. (2001). The potential of transdisciplinarity. In Transdisciplinarity: joint problem solving among science, technology, and society (pp. 67–80). Birkhäuser Basel; Nowotny, H., Scott, P., Gibbons, M., Introduction; “Mode 2” revisited: The New Production of Knowledge, Minerva, 41, 179–294, 2003.



PBL has provided 66 titles of its most important

reports to be investigated with the method of

Contextual Response Analysis. This is a selection of

the reports and other documents published by the

agency in the period 2011-2016/2017. (see

Addendum II for a full list of investigated reports.)

In this analysis, search strings are composed, based

on the titles of the publication, the subtitle or the

texts available at the website of PBL, in particular the

its press releases. Publications may have more than

one search string. In total, 130 search strings were used to identify the response to 66 titles.

The search strings were identical while searching via Google, BING and Scholar. The parliamentary

database and Lexis Nexis are searched with the terms "PBL" and "Planbureau AND Leefomgeving" in order

to include results for the agency in which there is possibly no direct reference to the titles that are

investigated.

Data collection took place between 20 February and 6 March 2017 for internet data, 10 – 20 February

2017 for Parliamentary data, and 13-15 April 2017 for data on newspapers.

Websites may contain several references to the same report. Not every instance of such multiple

references is substantial. Therefore, the unit of analysis chosen here is the unique reference, which is the

instance of the website referring to a specific report, independent of how many references more the

website may contain to that report. It is assumed that each website represents an organization or a

person, the total number of unique references to a report represents also the number of users of that

report. In total 4832 unique references were found to the 66 PBL studies, whereby the 2076 different

websites referred to one or more reports. About 25% of these unique references were found in Word-

documents, Excel sheets or PDFs, most often produced by the user referring to the PBL report.

Definitions:

1.3. Unique reference: website

referring to a specific report,

independent of how many

references the website contains

to that report.

1.4. Frequent user/organization: a

website of a user or

organization referring two times

to one or more PBL reports.

Contextual Response Analysis of Publications of PBL ... · PBL publications. A co-word analysis...

Documents

Transcript of Contextual Response Analysis of Publications of PBL ... · PBL publications. A co-word analysis...