Verification of systems biology research in the age of collaborative competition

May 7th, 2015

Dr Julia Hoeng, Philip Morris International

Prof. Manuel Peitsch, Philip Morris International

2

Outline

• 21st Century Science

• sbv IMPROVER methodology

• sbv IMPROVER challenges

• Tools and strategies for Open Innovation

3

Outline

• 21st Century Science – Need for Open

Innovation and Method Verification

• sbv IMPROVER methodology

• sbv IMPROVER challenges

• Tools and strategies for Open Innovation

4

Scientific

knowledge

Biological

insight

Toxicity

testing

Mode of

action

identification

Pharmacological

assessment

Biomarker

discovery

Patient

stratification

Biological Network Models to Empower ‘Systems’ -Biology, -Toxicology, and -Medicine

Scientific

data

Structured knowledge

in biological network

models

Tailored algorithms

55

Research Verification in the Age of Collaborative-competition

• Rigorous scrutiny of scientific

research based on communities

involvement

• A crowd sourcing approach of

challenge-based evaluation of

scientific methods could be a

potential alternative to the peer

review system.

• Our goal is to develop a robust

methodology which verifies

systems biology-based

methods and results in the

context of industrial and

academic research

6

Develop a robust methodology that verifies systems biology-based approaches

Meeting the needs of industry –The role of sbv IMPROVER

The self-assessment trap: can we all be

better than average?

Researchers wishing to publish their methods are

usually required to compare their methods against others

Authors’ method tends to be the best in an unreasonable

majority of cases

Selective reporting of performance: inadvertent or

disingenuous

Choice of only one, best metric

Mol Syst Biol. 2011 Oct 11;7:537. doi: 10.1038/msb.2011.70.

7

Concepts of Community-based Efforts

• Crowd-sourcing: A natural evolution of web technologies led to the

development of distributed problem-solving. Challenges are broadcasted

to potential interested stakeholders (solvers). The winning participants are

rewarded either with monetary awards, prizes, certificates, or with

recognition.

• Collaboration by Competition: The scientific community sought to

understand the limitations and comparative advantages of their methods

by challenging model developers to make blind predictions on previously

unseen data in a competitive framework.

• The community appreciates the successful methods which grow in

credibility. Therefore, consideration of the scientific community is one of

the forces that shape what is currently considered as the way to do the

science right

8

Outline

• 21st Century Science

• sbv IMPROVER methodology

• sbv IMPROVER challenges

• Tool and strategies to support crowd

engagement

9

Community-Wide Prediction & Verification Projects in Science

10

Analysis of Community Approaches

Common principles in community

efforts

• Assessment of a prediction by an

impartial, outside party is a more

rigorous model verification than self-

assessment.

• The responses of the community to a

prediction challenge can build

consensus in the community regarding

the most constructive methods for the

task.

• Voluntary participation in a prediction

challenge is driven by a number of

incentives.

• Harvest the knowledge and intellectual

power of the community to improve

methods, algorithms and experimental

design.

11

Companies Offering Services as Crowdsourcing Brokers

Type of

work

Company Description of company Project types Industry clients

Res

earc

h a

nd

Des

ign

InnoCentive Provides an open innovation

marketplace that matches a global

network of crowdworkers with

research and development

challenges submitted by

organisations.

Research and

development problems in

engineering, computer

science, mathematics,

chemistry, life sciences,

physical sciences and

business.

Commercial, government and non-

profit organisations; e.g., Procter

& Gamble, Dow AgroSciences, the

Air Force Research Lab, NASA, and

the Rockefeller Foundation.

Presans Provides an open innovation

marketplace that matches expertise

and business via problem-solving

competitions in various industrial

sectors.

Research and

development problems in

defence, space and

aeronautics, car

manufacturing, advanced

manufacturing and

materials, energy and

resources, and food and

agriculture.

Mostly large organisations,

universities and research

organisations; e.g., École

Polytechnique and CNRS.

IdeaConnection Provides an open innovation

marketplace that matches a global

network of crowdworkers with

research and development

challenges submitted by

organisations.

Research and

development problems in

various industries.

Small and large enterprises.

Mic

ro

-tas

ks Amazon

Mechanical Turk

Provides a crowdsourcing Internet

marketplace. Works on commission.

Tasks that computers are

currently unable to do,

such as transcribing,

rating and image tagging.

Individuals and businesses

12

Crowdsourcing Advantages

• Many contributors with independent methods / knowledge

• Different solutions tackle various aspects of a complex problem

• The combination of solutions often outperforms the best performing

submissions and is extremely robust “Wisdom of Crowds”

• Nucleates a community around a given scientific problem

• Allows for unbiased benchmarking

• Establishes state-of-the-art technology and knowledge in a field

• Complements the classical peer-review process

13

sbv IMPROVER Fills a Gap in Research Quality Assessment

Leverages the collective expertise of the scientific community to provide the best answers

Anonymous as

Known

14

Divide a Research Workflow into Verifiable Building Blocks

Building blocks support each other towards a final goal

Each building block is verifiable by a challenge

Example of a Research Pipeline – In vivo Inhalation Study

sbv IMPROVER team. 2011. Verification of systems biology research in the age of collaborative competition.

Nature Biotechnology 29: 811-815.

15

Participation and Collaboration with Experts

• The development of a Systems Biology Verification (SBV) method is a major scientific

challenge which requires participation and collaboration with experts from academia,

industry and other interested parties.

• To secure a wide application of the SBV method and to meet the complex requirements of

developing and applying a standard scientific method, we have built a multidisciplinary

research team, made-up of leading scientists and experts across academia and from

multiple industries

Project

Team

Special

Interest Group

Participants

Subject Matter Expert

(SME)

16

Benefits of Strategic Crowd Engagement

Enhanced dialogue

with the scientific

community

New standards for

validating and

publishing big datasets

in Toxicology and

Biology

Benchmarking

methods and results

with the community

of expertsPublications in

strategically important

journals, usually

inaccessible to individual

articles

Publications and

presentations

Full transparency of

the research process

Scientific

credibility

and

confidence

Joint discoveries and

open innovation

17

Outline

• 21st Century Science

• sbv IMPROVER methodology

• sbv IMPROVER challenges

• Tools and strategies for Open Innovation

18

Training

datasets

Test

datasets

Symposium

Crowd sourcing brings new ideas

and leverages the wisdom of crowds

Clear challenge description and forum

discussion in a user friendly website

The Elements of a Challenge

Challenge

Define the

Question

Score Challenge and Present Results

Engage Crowds to Solve Challenge

Collect Data and

‘Gold Standard’

Narrow the

Scope

19

The Diagnostic Signature Challenge

20

Outcome of the Diagnostic Signature Challenge

21

The Species Translation Challenge

The objective of the Species Translation Challenge was to:

• Identify a function which maps measurements derived from systematic perturbations in

one species to another

• Understand the system boundaries of the translatability concept

• Quantify the translatability between species

22

Outcome of the Species Translation Challenge

• To learn more about the outcome of the Species Translation Challenge the following articles have been

published in the Bioinformatics Journal:

– Understanding the limits of animal models as predictors of human biology: lessons learned from the

sbv IMPROVER Species Translation Challenge

(Bioinformatics Overarching Paper, 17 September 2014)

– Inter-Species Pathway Perturbation Prediction via Data Driven Detection of Functional Homology

(Bioinformatics, 4 August 2014)

– Predicting protein phosphorylation from gene expression: Top methods from

the IMPROVER Species Translation Challenge

(Bioinformatics, 23 July 2014)

– Inter-species prediction of protein phosphorylation in the

sbv IMPROVER Species Translation Challenge

(Bioinformatics, 3 July 2014)

• To learn more about the data set used during the Species Translation Challenge the following article has

been published in Scientific Data:

– The species translation challenge - A systems biology perspective on human and rat bronchial

epithelial cells. (Scientific Data, 10 June 2014)

This work constitutes a proof of principle that the molecular responses

induced by active substances in an in vitro system are to some extend

predictive of the responses observed in the same system of another species

23

sbv Improver team. 2013. On Crowd-verification of Biological Networks. Bioinformatics and biology insights 7:

307-325.

Challenge 3 – Biological Networks Verification Challenge

24

Network Verification Challenge

• The first NVC took place October 2013 –

February 2014

– Participants reviewed biology in networks

and approved or rejected evidence

– Participants added biology to the networks

in the form of additional evidence or new

edges

– The aim is to build a consensus around what

parts of the networks are accurate, incorrect

or incomplete.

• NVC2 is ongoing: started February 2014

and closes April 2015

https://sbvimprover.com/challenge-3/challenge

25

The sbv IMPROVER project team (2013). On Crowd-verification of Biological Networks. Bioinformatics and Biology Insights 2013:7 307-325.

NVC2 (ongoing): February 2014 April 2015 Mid-2015NVC (finished):

Steps in the sbv IMPROVER Network Verification Challenge

26

Number of Publications that Mention the Word “Jamboree”.

0

10

20

30

40

50

60

70

80

90

1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 2020

Nu

mb

er

of

Pu

blic

atio

ns

add

ress

ing

"jam

bo

ree

"

Scoutsjamborees

Technicaljamborees

Rock'n rolljamboree

Gene findingjamboree

Genome annotation jamborees

Undergradjamboree

Networkreconstruction

jamborees

27

Outcome of The first Network Verification Challenge in numbers

150 participants

from 18 countries

Number of Participants1 10 20

• The sbv IMPROVER project team and the

Challenge Best Performers. Enhancement of

COPD biological networks using a web-based

collaboration interface. F1000Research. 2015; 4:

32.

• Binder J, Boue S, DI Fabio A, et al. Reputation-

based collaborative network biology. Pacific

Symposium on Biocomputing Pacific Symposium

on Biocomputing. 2014, p. 270-81.

2 publications

451 new edges

2,456 votes

885 new pieces of evidence

Activity during the open phase

(10/2013 – 02/2014)

28

Outline

• 21st Century Science

• sbv IMPROVER methodology

• sbv IMPROVER challenges

• Tools and strategies for Open Innovation

29

The Network Verification Challenge Principles

Motivation

Jamboree

Marketing

Platform

Crowd-

sourcing

System

Biology

Scientific community to verify

biological network built by PMI

2000 nodes (lung)

8000 edges

80000 pieces of evidence

Leaderboard & reputation

Travel bursaries

Scientific event

Academic articles

Scientific event

Used to “curate”

activities

Top speakers

Networking opportunities

Ads

Conferences

Videos & webinars

Publications

Ambassadors

Intuitive

Interactive

Social

Network

Verification

Challenge

30

What Motivates Scientists to Join sbv IMPROVER?

Be part of a pioneering community, working together to improve how scientific

research is verified

Drive innovation in science with creative crowd-sourced solutions

Create smarter solutions to complement peer review with collaborative crowd-sourcing

Get peer recognition and self-esteem

Engage closely and network with experts around the world

Contribute and work towards consensus in the scientific community

31

Geographical Tracking to Engage More Participants

32

A Platform to Easily Verify and Refine the Networks and Supporting Evidence

33

Display News Feed to Encourage Participation

34

Gamification Principle

• Used widely to support user engagement and enhance positive patterns in

service use

• Recently also reflected in academic use

• Results in increased motivation and engagement in the learning and

enjoyment over the tasks

Hamari J, Koivisto J, Sarsa H (2014) Does Gamification Work?--A Literature Review of Empirical Studies on Gamification. In:

System Sciences (HICSS), 2014 47th Hawaii International Conference on. IEEE, p 3025-3034

Domínguez A, Saenz-De-Navarrete J, De-Marcos L et al. (2013) Gamifying learning experiences: Practical implications and

outcomes. Computers & Education 63:380-392

35

Gain Points and Increase Your Rank in the LeaderBoard

36

Marketing & Advertising - How Wise is the Crowd?

Promotional materials were

translated into different languages:

Spanish, Chinese, Russian,

Japanese

37

Compete. Collaborate. Contribute. Join us in Barcelona in June 2015.

https://sbvimprover.com/discover

38

Jamboree Meeting in Montreux, Switzerland

As published in Nature, 8 May 2014, page 127

39

sbv IMPROVER: a Brand and a Community

40

Summary - sbv IMPROVER at a Glance

Aims to provide a measure of quality control in R&D by identifying

the building blocks that need verification in a complex industrial

research pipeline

Is a robust, respected and recognized methodology which verifies

systems biology-based methods and results

Complements the classical peer review system

Has potential application in the areas of risk assessment where a

more profound and insightful mechanistic understanding (e.g.,

biomarkers) would make product development and assessment

more efficient and reliable

41

sbv IMPROVER: Conference Attendance 2015

42

Acknowledgements to a Global Team

ProtAtOnce

OrangeBus

PMI

IBM GBS

Advantage Integral

IBM

Research

ADS

Selventa

IBM

43

The sbv IMPROVER project, the website and the Symposia are part of a

collaborative project designed to enable scientists to learn about and

contribute to the development of a new crowd sourcing method for

verification of scientific data and results. The current challenges, website

and biological network models were developed and are maintained as part

of a collaboration among Selventa, OrangeBus and ADS. The project is led

and funded by Philip Morris International. For more information on the focus

of Philip Morris International’s research, please visit www.pmi.com

Thank you for your Attention

https://sbvimprover.com/discover

To learn more, visit

Questions? Contact Us

sbvimprover.RD@pmi.com>

44

BACKUP

45

Why do we need to verify that it is possible to infer clinical phenotype from genomics data?

A few success stories of gene expression based biomarkers in clinical use

MammaPrint (breast cancer recurrence assay)

70-gene profile; requires fresh tissue

Oncotype Dx (breast cancer recurrence assay)

21-gene profile; works on both fresh and fixed tissue

Counter-balanced by a few failure stories of gene expression based biomarkers in

clinical use

Potti et al, Nat Med (2006) claimed to identify genomic signatures for drug response.

Three clinical trials begun in 2007, 2008 for lung and breast cancer. The research

was later deemed statistically flawed and at least 10 high profiled publications

were retracted and the clinical trials stopped.

Amgen scientists tried to confirm 53 landmark papers in pre-clinical oncology

research: Only 6 (11%) were confirmed.

Bayer HealthCare reported that only about 25% of published preclinical studies

could be validated.