Zen and the Art of Research Assessment

Stephen Curry Imperial College

R2R Conference | 20 Feb 2017

AnInquiryintoValues

Whydopeoplebecomeresearchers?

Tounderstandtheworld

Tochangetheworld

Toearnaliving

Toberemembered

Wikim

ediaCom

mon

s(Do

naEidam

/USG

S)

Howdopeoplebecomeresearchers?

h"p://www.nature.com/new

s/science-pu

blish

ing-the-go

lden-club-1.13951 Tounderstandtheworld

Tochangetheworld

Toearnaliving

Toberemembered

EasiertoachieveifpublishingisopenandevaluaHonmorebroadlybased

Publishinahighimpactfactorjournal

Measurementhasitsuses…

h"ps://www.premierleague.com/tables?co=1&se=42&mw=-1&ha=-1

EnglishPremierLeagueTable(2015-16)

…butwherearethelimits?

TheMetricTide(2015):acallforresponsiblemetrics

• Indicators,notmetrics?

• Numbersinformbutcannotreplacejudgement

• Users&providersneedtobetransparent

The Metric TideReport of the Independent Review of the Role of Metrics in Research Assessment and Management

July 2015

h"p://www.ascb.org/dora/

h"p://www.hefce.ac.uk/pubs/rereports/Year/2015/metricMde/Title,104463,en.html

SwimmingagainstthemetricHde:becarefulwhatyouwishfor

“Don’taimatsuccess[…]forsuccess,likehappiness,cannotbepursued;itmustensue,anditonlydoessoastheunintendedside-effectofone’sdedicaMontoacausegreaterthanoneself…”

ViktorFrankl

Onyourdeathbed,willyourbibliographyoryourh-indexbeuppermostinyourmind?

SwimmingagainstthemetricHde:behumble

Ficklenatureofhumanjudgement…

FallacyofWhatYouSeeIsAllThereIsHaloeffectsAnchoringeffectsHindsightbiasLossaversion

“Ihaveyettomeetasuccessfulscientistwholackstheabilitytoexaggeratetheimportanceofwhatheorsheisdoing.”

DanielKahneman

SwimmingagainstthemetricHde:accoun;ngfortheunexpectedApollo8MissionTranscript(24Dec1968)

Anders:"Oh,myGod,lookatthatpictureoverthere.There'stheEarthcomingup.Wow,isthatpre`y!"

Anders(toLovell):"Yougotacolorfilm,Jim?Handmearollofcolor,quick,wouldyou?"

Lovell:"Oh,man,that'sgreat!Whereisit?"

Anders:"Hurry.Quick."

Lovell:"Downhere?"

Anders:"Justgrabmeacolor.Acolorexterior.Hurryup.Gotone?"

Lovell:"Yeah,I'mlookin'forone.C368."

Anders:"Anythingquick."

Anders:"Ithinkwemissedit."

Withinseconds,Lovellseestheshotagaininanotherwindow.Heasksforthecamera.

Anders:"Waitaminute,justletmegettherightseingherenow,justcalmdown.Calmdown,Lovell!"

h"p://www.npr.org/secMo

ns/thetw

o-way/2013/12/23/256605845/on

-an

niversary-of-apo

llo-8-how

-the-earthrise-ph

oto-was-m

ade

Focusingresearcherassessmentonpublishedoutputsisproblema;c

h"p://journals.plos.org/plosmedicine/arMcle?id=10.1371/journal.pmed.1001747

h"p://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/

h"p://dx.doi.org/10.1016/j.cub.2007.06.014

h"p://www.palgrave-journals.com/arMcles/palcomms2016105h"p://nuffieldbioethics.org/project/research-culture/

CantheopennessofourscienHficheritagehelpustodobe`erresearch?

Istheamateur(andopen)ethossMlloneofthenormsoftheacademy?

Someoftheworkistechnical(andrequirescollaboraHon…)

Canopennesshelptochangebehaviour?

Cita%onDistribu%onsRoyalSocietyJournalsEMBOJournalPLOSPNASNatureNatureCommunicaHonsNatureChemistryScienHficReportsActaCryst.(A-F)…

Nopromo;onofJIFsPLOSeLifeASMjournals…

AcademicleadersandinsHtuHonshavetodotheirbit…

Vale,R.D.(2012)MolBiolCell23,3285–3289.

h"p://www.nature.com/new

s/fewer-num

bers-be"

er-science-1.20858

1. Research,publicaHons,grants2. Managerial&academicduHes3. Mentoring&teaching4. Clinicalwork(ifapplicable)5. Entrepreneurship&communityoutreach

ResearcherassessmentatUMCUtrecht

h"ps://emckiernan.wordpress.com/pledge/

Apublicgood:howopensciencecanbebe`erscience

Preprints:fastercommunicaHonFocusonthecontent,notthecontainerLargestpossibleaudience(sharing&scruHny)

Datasharing:re-use&scruHnybenefitsBe`erforchangingtheworld

Apublicgood:howopensciencecanbemoreinclusive

CiHzenscienceteachesresearchersabout:new(non-tradiHonal)audiences&scienHsts

CommunicaHon+ParHcipaHon=PublicTrust

PoliHcalrealiHes:wehavetobeopenwiththepublic

“Peopleinthiscountryhavehadenoughofexperts.”MichaelGove,MP

“…wehavetodomoretoshowhow,andwhy,universiHesareoneofsociety’sgreatachievementsandbesthopes.”

ChrisHusbands(VC,Sheffield-HallamUniversity)

Thankyou

AlternaHveMetrics?

CitaHoncountsareeasyandnotuninteresHng,but…

Howreliableisyourresearch?

Howeasyisitforotherresearchertoaccessyourdata?

Howmanyliveshasyourworksaved?

Howmanypeoplehasyourresearchhelpedtoheal?

Howmanypeoplehavebeenlivedfrompovertythroughyourpublishedoutput?

Howmucheconomicgrowthhasyourworkgenerated?

HowmuchhasyourresearchaddressedsignificantnaHonalandinternaHonalchallenges?

Howmuchhasyourresearchreducedinequalityintheworld?

Howmanyyoungmindshasyourworkinspired?

HowwellhasyourmentoringhelpedtonurturethenextgeneraHonofresearchers?