Information Hazards

7/31/2019 Information Hazards

1/35

INFORMATIONHAZARDS:ATYPOLOGY

OF POTENTIAL HARMS FROM

KNOWLEDGE(2009) Draft 1.11Nick Bostrom

Future of Humanity Institute

Faculty of Philosophy & James Martin 21st Century School

Oxford University

hethatincreasethknowledgeincreasethsorrow

Ecclesiastes1:18

AbstractInformationhazardsarerisksthatarisefromthedisseminationorthepotentialdisseminationoftrue

informationthatmaycauseharmorenablesomeagenttocauseharm. Suchhazardsareoftensubtler

thandirectphysicalthreats,and,asaconsequence,areeasilyoverlooked. Theycan,however,be

important. Thispapersurveystheterrainandproposesataxonomy.

1.IntroductionThereis,thesedays,acommonlyheldpresumptioninfavorofknowledge,truth,andtheuncovering

anddisseminationofinformation. Itisraretofindsomebodywhoselfidentifiesasanobscurantistor

whoopenlyespousesobscurantismasalegitimatepolicyinstrumentofwideutility.

Evenreactionariesrarelyobjecttothisgeneralfavoringofinformation. Considersomeparticularly

intransigentcreationistwhoopposestheteachingofevolutiontheoryinpublicschools. Hedoesnot

constituteacounterexample. Forhedoesnotbelievethatevolutiontheoryisatruthtobeconcealed.

Rather,hebelievesevolutiontheoryanerrorthatoughttobereplacedwithmoreaccurate

information. Therefore,althoughhehappensunwittinglytostandinthewayoftruth,heneednot

disagreewiththeclaimthatthetruthshouldbepromoted. Thecreationist,too,isatruthlover,albeit

onewhoseaffectionsareunreciprocated.

Althoughnobodymakesabriefforignorancegenerally,therearemanyspecialcasesinwhichignorance

iscultivatedinorder,forexample,toprotectnationalsecurity,sexualinnocence,juryimpartiality;to

preserveanonymityforpatients,clients,reviewers,andvoters;tocreatesuspenseinfilmsandnovels;

toprotecttradesecrets;tomeasuretheplaceboeffectandavoidvariousresearchbiases;andtocreate

1


2/35

mentalchallengesforgamingandstudy. Thesecasesarecommonlyacceptedexceptionstothegeneral

ruleofknowledgefavoring.1

Inthispaper,wewillnotbeconcernedwithpostmodernistcritiquesoftheideaofobjectivetruthnor

withskepticaldoubtsaboutthepossibilityofknowledge. Ishallassumesomebroadcommonsensical

understandingaccordingtowhichtherearetruthsandwehumanssometimesmanagetoknowsomeofthesetruths.

Thispaperwillalsonotdiscussthewaysinwhichharmcanbecausedbyfalseinformation.Manyof

thosewaysareobvious.Wecanbeharmed,forinstance,byfalseinformationthatmisleadsusinto

believingthatsomecarcinogenicpharmaceuticalissafe;or,alternatively,thatsomesafepharmaceutical

iscarcinogenic.Wewilllimitourinvestigationtothewaysinwhichthediscoveryanddisseminationof

trueinformationcanbeharmful.

Letusdefine

Informationhazard: Ariskthatarisesfromthedisseminationorthepotentialdisseminationof

(true)informationthatmaycauseharmorenablesomeagenttocauseharm.2

Relativetotheirsignificance,andcomparedtomanydirectphysicaldangers,sometypesofinformation

hazardareundulyneglected. Itwillthereforebeusefultodistinguishsomedifferenttypesof

informationhazard. Thiswillservetodrawattentiontosomeeasilyoverlookedrisksandwillhelpus

createavocabularyfordiscussingthem.

Theaimofthispaperistocataloguesomeofthevariouspossiblewaysinwhichinformationcancause

harm.Wewillnothereseektodeterminehowcommonandserioustheseharmsare,norhowthey

stackupagainstthemanybenefitsofinformationquestionsthatwouldneedtobeengagedinorder

toreachaconsideredpositionaboutpotentialpolicyimplications. Itmaybeworthstressing,however,

thatevenifonehasanextremelystrongintellectualcommitmenttotruthseekingandpubliceducation,

onecanstilllegitimatelyandingoodconscienceexplorethequestionofhowsomeknowledgemightbe

harmful. Infact,thisverycommitmentdemandsthatonedoesnotshyawayfromsuchanexploration

orfromreportingopenlyonthefindings.

2.Sixinformationtransfermodes1

Thenotionofdangerousorforbiddenknowledgeisalsoacommontropeinliteratureandinmanymythologies;seee.g.Shattuck1996.

2Wesetasidethetrivialwayinwhichutterancescancauseharmviatheirperformativefunctions;cf.Austin1962.

Thus,adictatorwhoproclaimsInvadethatcountry!disseminatesinformationthancanobviouslycausealotof

harm,butweshallnotcountthispossibilityasaninformationhazard.

2


3/35


4/35

adversarymayinvestmoreinunderstandingandexploringanideathatheknowsthathisenemyis

tryinghardtokeepsecret. Abookthatiscensoredoftenbecomesmorewidelyread.4

Itispossiblethateffortstocontemplatesomeriskareasay,existentialriskwilldomoreharmthan

good. Onemightsupposethatthinkingaboutatopicshouldbeentirelyharmless,butthisisnot

necessarilyso. Ifonegetsagoodidea,onewillbetemptedtoshareit;andinsodoingonemightcreateaninformationhazard. Still,onelikestobelievethat,onbalance,investigationsintoexistentialrisksand

mostotherriskareaswilltendtoreduceratherthanincreasetherisksoftheirsubjectmatter.

Sometimesitisrighttoharm. Ifinformationenablesanagenttoharmrightly,thiscanbeagoodthing;

andthepossibilityofthathappeningshouldthereforenotbeclassifiedasarisk. Forexample,the

policesobtainmentofcertaininformationmightharmsomecriminalbyenablingthepolicetoarrest

him;andthiscanbegood. However,wecouldsaythatfromthecriminalspointofviewthereisan

informationhazard. Hefacesariskthathiswhereaboutswillbereported.

Notalltypesofinformationtransferarebestthoughtofintermsofdata,ideas,orattention.Wecan

alsoconsiderimplicitformsofinformation,suchasprocessesororganizationalstructures,whichcan

giveonefirmanadvantageoveranother,andwhichmightbeimitatedorreplicatedbyacompetitor.5

Similarly,individualsoftenlearn,andshapetheirownpersonality,byemulatingsomeotherperson.

Suchemulationcanhappenunintentionallyandevenwithoutawarenessthatemulationistakingplace.

Abadrolemodelcanposeatemplatehazard.

Templatehazard: Thepresentationofatemplateenablesdistinctivemodesofinformation

transferandtherebycreatesrisk.

Wecanalsoregisterasadistinctmodeofcommunicationsocialsignaling,wherethefocusisnotonthe

content

that

is

being

transmitted

but

on

what

this

contentand

the

fact

that

it

is

being

communicatedsaysaboutthesender. Nonverbalactionscanalsoserveasocialsignalingrole,ifthey

areaimednotonlyatdirectlyachievingthesomepracticaloutcometowhichtheyarenominallygeared

butalsotosignalsomehiddenqualityoftheagentthatperformstheaction. Forexample,onecould

givealmstothepoornotonlybecauseonewishestosatisfytheirneedsbutalsobecauseonewishesto

beseenbyonespeersasakind,caring,andgenerousperson. Onemightutterpatrioticstatementsnot

onlytoconveytothelistenerinformationaboutvariousattributesofonesnation,butalsotosignal

onesstatusasaloyalcitizen,oronesaffiliationwithsomeparticularpoliticalgrouping.

Signalinghazard:Verbalandnonverbalactionscanindirectlytransmitinformationaboutsome

hiddenqualityofthesender,andsuchsocialsignalingcreatesrisk.

4AgoodexampleofthisistheratherdullSpycatcherbyPeterWright,whichbecameabestsellerinthe1980safter

Thatchertriedtobanit,seeZuckerman1987.

5NelsonandWinter1990andPorter2004.

4


5/35

Sometopicsareespeciallyattractivetocrackpots. Seriousacademicsmightshyawayfromsuchtopics

becausetheyfearthatworkingonthosetopicssignalsintellectualflakiness. Atleasttwosignaling

hazardsariseinthiscontext. Oneistherisktoindividualthinkerswhomightsufferundeserved

reputationaldamagemerelyforworkinginanareawhichalsohappenstoattractlotsofcrackpots.

Anotheristherisktosocietythatimportantareasofresearchwillremainuncultivatedbecausethe

ablestresearchers(andtheirsponsors)protecttheirreputationseitherbyshunningthoseareasinfavor

ofmoresociallyacceptable,highstatusfieldsofstudyorbyadoptingrelativelylesseffectivemeansof

exploration,suchashypertrophicformalismandexpensivetechnicalapparatus,whichareharderfor

crackpotstomimic.

Finally,wealsocountasadistinctmodethetransferofinformationcontainedintheparticularway

somecontentisformulatedandexpressed. Avividdescriptionofsomeevent,forexample,canactivate

psychologicalprocessesthatliedormantwhenthesameeventisrecountedindryprose.

Evocationhazard:Therecanbeariskthattheparticularmodeofpresentationusedtoconvey

somecontentcanactivateundesirablementalstatesandprocesses.

Eachoftheseinformationtransfermodesdata,idea,attention,template,signaling,andevocation

canplayaroleincreatingvariouskindsofrisk. Thelatterfour,especially,areeasilyoverlooked.

Thefollowingfivesectionsintroduceanothercategorizationschemewhich,whensuperimposedupon

thedivisionintoinformationtransfermodes,rendersamorefinegrainedpictureofthewaysinwhich

informationcanbehazardous(summarizedintable1,below).

3.AdversarialrisksPreviousexamplesfocusedonadversarialsituationsinwhichsomefoeisintentoncausingusharm. A

burglarwhoknowswherewekeepourmoneyandwhenwewillreturnhomeisinastrongerpositionto

succeedwithhiscrime.

Enemyhazard: Byobtaininginformationourenemyorpotentialenemybecomesstrongerand

thisincreasesthethreatheposestous.

Oneparadigmaticcontextforthistypeofhazardisnationalsecurity.Withinthedefensesector,

activitiesaimedatreducingenemyinformationhazardrangefromcounterintelligenceworktothe

applicationofcamouflagetoconcealtroopsinthefield.

Enemyhazarddependsontheexistenceofvaluableinformationthatanenemymightobtain. Indirectly,

therefore,ourownactivitiescanbehazardousiftheycontributetotheproductionofsuchinformation.

Militaryresearchoffersmanyexamples.Weinvestinresearchanddevelopmentofnewweaponsand

newtactics. Thisactivityproducesinformationthatisvaluabletoourenemy. Theenemyobservesour

improvedtactics. Hisspiesobtaintheblueprintsofourimprovedtechnology. Ortherelevant

informationleaksoutinotherways,perhapsintheformofideas,attention,ortemplates. Asa

5


6/35

consequence,ourenemysoonreplicatesourachievement.Whenhostilitieserupt,webattleourown

inventions.

Rationalstrategyformilitaryresearchwouldgivesignificantconsiderationtosucheffects. TheUnited

States,forexample,mightrefrainfromaggressivelypursuingdevelopmentofelectromagneticpulse

weapons. Becauseofthecountrysunusuallyheavyrelianceonelectronics,theexistenceofeffectiveEMPweaponswoulddifferentiallybenefititsadversaries.

Conversely,anaggressiveapproachtoresearchcouldhelpprotectacountryagainstitsenemies. A

countrymightpursuemilitaryresearchtocatchupwiththetechnologicalleader. Aleaderinmilitary

technologymightinvestinresearchinordertomaintainitslead. Theleadermightreasonthat,whileits

discoverieswilleventuallyleakoutandbenefititsenemies,itcanproduceasteadystreamof

discoveriesandcontinuallykeepafewstepsahead.

Therearesituations,though,inwhichacountryisbetteroffretardingitsprogressinmilitary

technology,evenwhenthenewtechnologieswouldnotdifferentiallybenefititsenemiesandevenif

considerationssuchascostandforegoneopportunitiesforbuildinginternationaltrustaresetaside.

Supposesomecountryhasgreatsuperiorityinmilitarypowerandmilitarytechnology,andasa

consequencefaceslittleseriousthreat. Byinvestingheavilyinmilitaryresearch,itcouldincreaseits

leadandtherebyfurtherenhanceitssecuritysomewhat. Yetsuchinvestmentmightunderminesecurity

inthelongerterm. Therateofinformationleakagemightbeafunctionofthesizeofthetechnological

gapsuchthatincreasingthegapincreasestherateoftheleakage. Perhapsweaponssystemscomein

generationsanditmaybeinfeasibletokeepsecretmorethanaboutonegenerationbeyondthe

enemyslevel. Ifso,introducingnewgenerationsatafasterratemightnotincreasethetechnological

lead,butserveonlytoacceleratebothcountriesascentofthemilitarytechnologytree,perhapsto

levelswherewarsbecomemoredestructive. Ifyouarealreadyutterlysuperiorinconventional

weapons,thenrushingtoinventafissionbomblongbeforeyourenemiescouldhavegottheremightbe

counterproductive. Similarly,hasteningtheintroductionofthefusionbombmightbebadstrategyif

youcouldhavebeenconfidentofremainingsuperiorinfissionbombs.

Acceleratingtheascentofthetechnologytreecouldalsobebadifthetreeisoffiniteheight,sothatat

somepointtheleaderrunsoutofopportunitiesforinnovation. Someweaponssystemsmightreacha

levelofperfectionfromwhichfurtherimprovementsaredifficultorimpossible. (Inthecategoryof

weaponsofmassdestructionfordeterrentuse,forinstance,thehydrogenbombmightrepresenta

nearmaximum.) Eventuallyeverybodymayplateauatthislevel,andthepreviousleaderwillloseits

advantage. Inordertomaintainatechnologicalleadforaslongaspossible,theleadermightwishto

pushthetechnologicalfrontierattheslowestpossiblepacethatisconsistentwithmaintainingan

acceptableleadateverypointintimeuntiltechnologicalmaturityisreached.

Themilitaryillustrationshowshowinformationhazardsariseinsomesituationsfromonepartys

(potential)intenttoinflictharmonanother. However,informationrisksstemmingfromanadversarial

relationshiparisemuchmorewidely. Incompetitivesituations,onepersonsinformationcancause

6


7/35

harmtoanotherevenifnointentiontocauseharmispresent. Example:Therivaljobapplicantknew

moreandgotthejob.

Competivenesshazard: Thereisariskthat,byobtaininginformation,somecompetitorofours

willbecomestronger,therebyweakeningourcompetitiveposition.

Exclusivepossessionofinformationiscentraltothebusinessmodelofmanyfirms. Acompetitorcan

gainvaluableinformationbyobservingtheproductionandmarketingmethodsofarivalfirm,reverse

engineeringitsproducts,orheadhuntingitsemployees.6 Firmsgotogreatlengthstoprotecttheir

intellectualcapital,relyingonawidevarietyofmethodsincludingpatenting,copyright,nondisclosure

agreements,physicalsecurity,inhouseproductioninsteadofoutsourcing,compensationschemesthat

discourageemployeeturnover,andsoforth.Wecanidentifythreattointellectualpropertyasaspecial

caseofcompetitivenesshazard:

Intellectualpropertyhazard:AfacestheriskthatsomeotherfirmBwillobtainAsintellectual

property,therebyweakeningAscompetitiveposition.

Anothertypeofadversarialhazardariseswhenanagentsownpossessionofsomeinformationhasthe

potentialtorenderherlessabletoprevailinsomecompetitivesituation. Inorderforablackmailerto

besuccessful,histargetmustbelievethathehassomeincriminatingorembarrassinginformation,

informationthathecouldthreatentorelease. Solongasthetargetremainsunaware,noblackmailcan

takeplace.Whenshelearnsaboutthethreat,sheisdeliveredintotheextortionistsclutches. Similarly,

inthegameofchicken:Twodriversspeedtowardsoneanotherfromoppositedirections;thefirstto

swerveloses. Ifonedrivercouldcrediblycommittonotswerving,hewouldwin,sincehisopponent(it

isassumed)wouldratherlosethegamethancrash. Agametheoristengaginginthispastimecould

havehimselfblindfolded,preventinghimselffromacquiringinformationabouttherelativedistanceof

thetwocars,thusrenderinghimselfincapableofreliablyswervingatthelastpossiblemoment;andtherebyconvincinghis(hopefullyrational)counterparttoyieldtheroad.

Commitmenthazard: Thereisariskthattheobtainmentofsomeinformationwillweakenones

abilitycrediblytocommittosomecourseofaction.

Commitmenthazardsaresometimesinstancesofenemyhazardandsometimesofcompetitiveness

hazards. (Commitmenthazardscanalsoariseinasingleagentcontext,asweshallseelater.)

Insomesituationsitcanbeadvantageoustomakeaprobabilisticthreat,athreatthatleaves

somethingtochanceintheterminologyofThomasShelling.7 Athreat,tobeeffective,mustbe

credible. Yetthereasonforissuingathreatisdeterrenceexante,notrevengeex

post;andcarryingout

athreatisoftencostly. Considersomepossiblepunitiveactionthatissocostlytocarryoutthatathreat

todosowouldscarcelybecredible,suchasanuclearfirststrikebyonemajorpoweronanother. A

6Porter2004.

7Schelling1981.

7


8/35

nucleararsenalcouldneverthelessbeusedtomakeathreat. SideAcanthreatenthatunlessthesideB

makessomeconcession,Awilltakesomeactionthatincreasestheriskofnuclearwar. Forinstance,A

couldthreatentoinitiateskirmisheswithconventionalweaponsthatwouldcreatesomeriskof

escalation. Alternatively,Acouldthreatentomakeitsownnuclearcontrolandcommandsystemless

safeagainstaccidentallaunch,forinstancebycreatingacrisisandputtingitsforcesonhighalert. The

ideahereisthatitismuchlesscostlyforAtocarryoutathreattomoderatelyincreasetheriskofa

nuclearwarthanitwouldbeforAtoactuallylaunchanuclearattack. Theprobabilisticthreatcan

thereforebemorecredibleandmoreeffective.

If,however,newinformationcametolightthatdispelledtherequisiteuncertaintyuncertainty,for

example,abouthowkeyactorswouldreactduringacrisisthentheabilitytomakeprobabilisticthreats

wouldbeundermined. Thepossibilityofsuchinformationbeingreleasedcanthusconstituteakindof

informationhazard. DuringtheColdWar,kindredconsiderationsmayhaveledthesuperpowersto

maintainsomeambiguityintheirstrategicpostures. Thiskindofinformationhazardmightinvolvea

combinationofenemyhazardandcommitmenthazard.8

Wecanalsoidentifyanothertypeofriskthatcanarisefromourownknowledgewhenthereisa

possibilitythatsomebodyelsewillbecomeouradversarybecauseofthisknowledge:

Knowingtoomuchhazard:Ourpossessingsomeinformationmakesusapotentialtargetor

objectofdislike.

NadezhdaSergeevnaAlliluyeva,Stalinssecondwife,wasknowntohavemisgivingsaboutthe

CommunistPartypurgesandtheconcomitantfamine. FollowingaspatwithStalinin1932,shewas

founddeadinherbedroom,apparentlyhavingtakenherownlife.9 Thesuicidecouldbeinterpretedas

akindofdesperaterebukeofStalinandhispolicies;andsincethatwouldbepoliticallyembarrassing,

thedeathwasofficiallyattributedtoappendicitis. Theindividualswhohaddiscoveredthebodyandwhoknewtherealcauseofdeathfoundthemselvesingravedanger. Inalaterallegedlyunrelatedtrial,

twodoctorswhohaddeclinedtosignthefalsedeathcertificateweresentencedtodeathandexecuted.

InthewitchhuntsoftheEarlyModernperiodinEurope,awomansallegedpossessionofknowledgeof

theoccultorofbirthcontrolmethodsmayhaveputheratincreasedriskofbeingaccusedof

witchcraft.10 InthegenocideperpetratedbyPolPotandtheKhmerRougeregime,theentire

8IfsideAknewhowAwouldbehaveinacrisis;andifsideB,whilenotknowinghowAwouldbehavebutknowing

thatAknewhowAwouldbehave;thenAcouldbecomelessabletoissueaneffectiveprobabilisticthreat. Bcould

reasonthatifAknewthatAwouldlaunchanuclearattackinacrisisthenAwouldbelesslikelytothreatento

createacrisis(assumingthatBknewthatAwasuncertainastowhetherBwouldyieldtoAsthreat). Thus,B

couldinferthatifAdoesinfactissueathreattocreateacrisis,itwouldlikelybebecauseAknewthatacrisis

wouldnotescalateintoanuclearwar. ThiswouldmakeBlesslikelytoyieldtothethreat.

9Montefiore2005.

10Levack1987.

8


9/35

intellectualclasswasslatedforextermination.11 Someculturesplaceahighvalueonsexualinnocence,

particularlyingirls,andayoungwomanmightfindhermarriageprospectsdimmedifsheappearsto

knowtoomuchaboutsexorifhergeneraleducationintimidatesprospectivehusbands.

contexts.

4.Riskstosocialorganizationandmarketsation

sinamilitaryattack;orindirectlyandperhapsunwittinglybyweakeningourcompetitive

position.

gearmsome

agentswithoutnecessarilystrengtheningorbenefittingtheiradversariesoranybodyelse.

bjects;andariskisposedbyinformationthatcoulddisrupttheseexpectationsforthe

worse.

normofdrivingontherightside,

justasthereisanormofextendingtherighthandinahandshake.

uilibria

re

omone

12 Inmany

schools,nerdychildrenwhoappeartohavetoomuchacademicknowledgeareostracizedand

bullied. Knowingtoomuchhazards,ofvaryingdegreesofseverity,seemtoariseinmanydifferent

Wehavesofarfocusedonthepossibilityofanadversarygaininganadvantageasaresultofinform

obtainedbyeithertheadversaryorourselves. Theadversarymightthenharmusdeliberatelyand

directly,a

Yettherearemanyothertypesofinformationhazard. Inparticular,informationcansometimesdamapartsofoursocialenvironment,suchascultures,norms,andmarkets. Suchdamagecanh

Normhazard: Somesocialnormsdependonacoordinationofbeliefsorexpectationsamong

manysu

Behaviorinsomegivendomaincanbeguidedbydifferentnorms,withtheresultofdifferentsocial

equilibriabeinginstantiated. Normsaresometimesformulatedandimposedfromabove,withlegal

backing;forexample,anormthatonemustdriveontherightsideoftheroad. Buteveniftherehad

beennosuchlaw,motoristsmighthavespontaneouslydevelopedthe

Withregardtowhichsidetodriveon,thereisnointrinsicbenefittoleftorright,solongasthereis

someclearrulethateverybodyfollows. Inotherdomains,however,differentpossiblesocialeq

canhavewidelydivergentconsequencesforhumanwelfare. Inasocietywithlowcorruption,

individualsmightprospermostbybeinghonest,trusting,andplayingbytheofficialrules;whileina

highcorruptionsociety,individualsfollowingthosestrategieswouldbesuckers. Theoptimalstrategy

foroneindividualdependsonthestrategiespursuedbyotherswhochosetheirstrategiesonthebasis

theirexpectationsabouthowotherswillreact. Informationthatalterstheseexpectationscantherefo

changebehavior. Thiscanleadtoachangeinnormsthatmovesagrouporawholesocietyfr

11FawthropandJarvis2005. Inanyactuallargescalehistoricalepisode,ofcourse,multiplecausesarelikelyto

havebeeninvolved,possessionofdangerousknowledgebeingatmostonecontributingfactor.

12Schlegel1991.

9


10/35

equilibriumstatetoanother. Thepossibilityofmovingtoaworsesocialequilibriumcanbe

characterizedasanormhazard.

Locallysuboptimalpoliciesaresometimesjustifiedfromawiderperspectiveasapriceworthpayingto

protectnormsthatservetoblockaslideintoaworsesocialequilibrium. Asociallyconservativeou

mightbebasedonthebeliefthatsuchslidesareamajordangerandthatstrictenforcementofexistingnormsisanecessarycountermeasure.

tlook

all

andtherebyweakentheoverall

structureofextantnorms,increasingtheriskofmoraldecayorsocialunraveling.14 Similarly,onecan

tainexpectations

aboutotherpeoplesbeliefsandattitudestowardsmoneyanditsspecificforms,suchascash.

Informationcascadescanarisewhenagentsmakechoicessequentially,andeachagenthas,inaddition

tosomenoisyprivateinformation,theabilitytoobservethechoices(butnottheinformation)ofsome

ntofherinthequeue.16 Ithasbeensuggestedthatinformationcascadesplayan

13 Evencallingintoquestionaparticularnorm,ormakingsm

adjustmentsofsomenorm,mightunderminetheauthorityof

objecttosomejudicialdecisionsbecauseofthelegalprecedentstheyset;tosomeforeignpolicy

decisionsbecauseoftheireffectoncredibility;andsoforth.15

Ifwetakethewordnorminitswidesense,wecanalsothinkofmoneyasanormorabundleof

norms. Thefunctionsthatmoneyservesintheeconomydependonpeoplehavingcer

Counterfeitingandexcessivemoneyprintingcanundermineacurrency,destroyingitsabilitytoserveas

amediumofexchangeandastoreofvalue. Thisisanotherexampleofnormhazard.

Itisobvioushowsomekindsoffalseinformationcandamagebeneficialnorms. Butnormscanalsobe

damagedbytrueinformation.Wehavealreadyalludedtothephenomenonofselffulfilling

prophesiespeopleactingmorehonestlyiftheybelievethemselvestobeinalowcorruptionsociety,

andviceversa;driversdrivingontherightsideiftheybelievethatotherswillmakethesamechoice.

Anotherphenomenoninwhichtrueinformationcandamagenormsisinformationcascades.

oftheagentsinfro

chelling1960).

towhether

rong(1/3)thanthatboththeprecedingcarswentthewrongway(1/9);soheturnsrightaswell.

Asimilarcalculationisperformedbyeachsubsequentdriverwhocanseeatleasttwocarsahead. Everycarends

13Hirschman1991.

14Cf.Schellingsconceptofafocalpoint(S

15RizzoandWhitman2003;Volokh2003.

16SupposethathundredsofrockfansaredrivingtotheGlastonburymusicfestival. Atsomepointeachdriver

reachesanintersectionwheretheroadsignshavebeenvandalized. Asaresult,thereisuncertaintyas

toturnleftorright. Eachdriverhassomeprivateinformation,perhapsadimdrugcloudedrecollectionfromthe

previousyear,whichgiveshera2/3chanceofpickingthecorrectdirection. Thefirstcararrivesatthe

intersection,andturnsright. Thesecondcararrives,andalsoturnsright. Thedriverinthethirdcarhasseenthe

firsttwocarsturnright,andalthoughhisprivateintuitiontellshimtoturnleft,hefiguresitismorelikelythathis

ownintuitionisw

upturningright.

Inthisscenario,thereisa1/9chancethatalltherockfansgetlost. Letussupposethatifthathappens,the

festivaliscancelled. Hadtherebeenadensefog,preventingeachdriverfromseeingthecarinfront(thus

reducinginformation),then,almostcertainly,approximately2/3ofallthefanswouldhavereachedGlastonbury,

10


11/35

importantexplanatoryroleinaccountingforfaddishbehaviorinmanydomains,includingfinance,

zoology,politics,medicalpractice,peerinfluenceandstigma.17 Informationalcascadingmightalso

contributetoaMatthew(therichgetricher)effect. Forexample,eminentscientiststendtogetmore

creditthanunknownresearchersforsimilarcontributions.18 Partofthereasonmightbethatwhen

thereisuncertaintyastowhomadethebiggercontribution,itis,ceterisparibus,morelikelytohave

beenmadebythemoreeminentscientist,whoconsequentlygetsthecredit;butwiththeresultthatthe

fameofthealreadyslightlyfamouscansnowballtoundeservedproportionswhileothersareunfairly

ignored.

Anotherimportantwayinwhichtrueinformationcandamagesocialorganizationisthroughinformation

asymmetries.Whenonepartyhasinformationthatotherslack,theinformationasymmetrysometimes

preventsmutuallybeneficialtransactionsfromtakingplace.

Informationasymmetryhazard:Whenonepartytoatransactionhasthepotentialtogain

informationthattheotherslack,amarketfailurecanresult.

Economicmodelsofadverseselectionandmoralhazardillustratesomeofthepossibilities. Inthe

marketforusedautomobiles,theselleroftenhasmoreinformationaboutthequalityofthecarthanthe

prospectivebuyer. Ownersofbadcars,lemons,aremorewillingtopartwiththeirvehiclethan

ownersofgoodcars. Buyers,knowingthis,suspectthatthecarbeingofferedthemisalemon,andare

willingtopayaccordingly. Thisbuypriceistoolowtointerestpotentialsellersofgoodcars,withthe

resultthathighqualityusedcarsarewithheldfromthemarket,leavingpredominantlylemons. The

informationasymmetryinhibitsthemarketinhighqualityusedcars. Thishelpsexplainwhythevalueof

abrandnewvehicleplummetsthemomentitdisembarksthedealership.19

Insuranceoffersmanyillustrationsofthepotentialfornegativeeffectsofinformationasymmetry. For

example,incountrieswithprivatehealthcare,considerascenarioinwhichtheavailabilityofgenetictestingcombinedwithvastlyimprovedknowledgeabouthowinterpretthetestsprovidebuyersof

healthinsurancewithawealthofnewinformationabouttheirpersonalriskprofile. Ifprivacylegislation

prohibitedinsurancecompaniesfromaccessingthesameinformation,theresultinginformation

asymmetrycouldunderminetheinsurancemarket. Adverseselectionwouldleadthesubjectswiththe

highestriskprofilestobuymoreinsurance. Insurancecompanies,anticipatingthis,wouldraise

enablingthefestivaltotakeplace. Oncethefestivalstarts,anylostfancanhearthemusicfromafarandfindtheir

waythere.Wecouldthushaveasituationinwhichreducinginformationavailabletoeachdriverincreasesthe

chancethathewillreachhisdestination. Clearweathercreatesaninformationalcascadethatleadstoan

inefficientsearchpattern.

17Bikhchandani,HirshleiferandWelch1992.

18Merton1968.

19Akerlof1970. Here,asthroughoutthispaper,wearenotconcernedtogiveadetailedaccountofsome

particularempiricalphenomenon;ourgoalistoilluminatesomefeaturesoftheconceptuallandscape.

11


12/35

premiums. Thehigherpremiumswoulddetermorelowrisksubjects,amplifyingtheadversesele

effectuntil,inanextremescenario,thehealthinsurancemarketcollapses.

ction

n.

rmationtoonepartyofapotentialtransactioncanthus,under

certaincircumstances,beahazard.

e

itknowsthat

youwillincuratsomeknowndate;thepremiumwouldhavetoexceedthecoverage.

it

nate.

Selfishagentsjointheschemebecausetheydonotknow,exante,towhichgrouptheybelong.

the

e

ecutedminority

andthattheiropinionswouldneverbeamongtheonesthatthecensorswouldsilence.

Thepossibilityofsucheffectsofsymmetricinformationcanbeviewedasarisk:

20 Relativetosucha

scenario,bothbuyersandsellersofinsurancemightbetteroffifneitherobtainstheextrainformatio

Thepossibilityofreleaseofnewinfo

Althoughasymmetricinformationisparticularlycorrosive,insurancemarketscanalsocollapsebecaus

ofsymmetricinformation,informationthatissharedbetweenallparties. Insuranceispredicatedon

uncertainty. Itmakesnosenseforyoutoinsureagainstalossthatyouarecertainyouwillnotincur,

anditmakesnosenseforaninsurancecompanytoofferyouinsuranceagainstalossthat

Iftheonlyusefulroleofinsuranceweretoreduceuncertaintyaboutfuturerevenueorwelfare,then

informationthatincreasedpredictabilitywouldremovetheneedforinsuranceatthesametimeas

removedthepossibilityofinsurance:noharmwouldbedone. However,insuranceservesother

functionsaswell. Oneisredistributivejustice. Ininsurance,thefortunatesubsidizetheunfortu

Entirepoliticalphilosophieshavebeenconstructedaroundthenotionofinsurance. Forexample,in

JohnRawlstheoryofjustice,thejustsocialorderisdefinedwithreferencetowhatpeoplewould

hypotheticallychoosefrombehindaveilofignorance,i.e.iftheywereignorantaboutwhichsocial

roletheythemselvesoccupy.21 ARawlsianmightattributemanyofthepracticaldifficultiesingetting

thisconceptionofjusticeimplementedtothefactthatvotersandpoliticaldecisionmakersareinreality

notbehindaveilofignorance. Selfishpeoplewhoknowtheirowncircumstancestheirsocioeconomic

class,race,occupation,andsoforthmayfavorpoliciesthatpromotetheirselfinterestratherthan

allegedlyfairerpoliciesthattheywouldhavechosenhadtheybeenignorantabouttheirownactual

situation. Knowledgeofonespresentandfuturesituation,though,isamatterofdegree. Onecan

thinkofscenariosinwhichincreasingtheavailabilityofinformationaboutthesethingswouldmaketh

implementationofajustsocialordermoredifficult. Forinstance,elitesupportforasocialsafetynet

mightslackenifelitescouldknowwithcertaintythatneithertheynortheirchildrenorgrandchildren

wouldeverneedtouseit.22 Supportforprotectionoffreedomofspeechandminorityrightsmight

weakenifmostindividualscouldbesurethattheywouldneverfindthemselvesinapros

20Itisunrealistictosupposegeneticinformationtoproducesuchanextremeconsequencesincemuchofthe

varianceinhealthoutcomesisduetonongeneticvariablesandchance.

21Rawls2005.

22AsimilarpointismadeinKavka1990. Kavkaalsoarguesthatintensesocialconflictwouldariseifthose

individualsandgroupsthatwouldsuffer(possiblynoncompensable)harmfromsomeproposedpolicycouldknow

thisexante.

12


13/35

Unveilinghazard: Thefunctioningofsomemarkets,andthesupportforsomesocialpolicies,

dependsontheexistenceofasharedveilofignorance;andtheliftingofwhichveilcan

underminethosemarketsandpolicies.

Thisphenomenoncanalsobeinstantiatedintheiteratedprisonersdilemma,whereagentsfacea

choicebetweencooperatinganddefectinginanunknownnumberofrepeatencounters. Agentsmightcooperateineachroundinordertosecuretheotherplayerscooperationinthefollowingrounds. Yet

cooperationcanunravelifplayersknowhowmanyroundstherewillbe.Whentheyknowtheyarein

thefinalroundhencefacingtheequivalentofatraditionaloneshotprisonersdilemmatheyboth

faceincentivestodefect.Worse,inthepenultimateroundtheycanforeseethattheywillnextbeinthe

finalroundinwhichtheywillbothdefect;soincentivesfavordefectinginthepenultimateroundtoo

andsoon,allthewaybacktothefirstround. Theopportunityforlongtermmutuallybeneficial

cooperationcouldthusberuinedthroughthelossofignoranceaboutthefuturedurationofthe

relationship.

Wehavealreadydiscussedintellectualpropertyhazardasanexampleofadversarialrisk. Intellectual

propertytheft,though,isaproblemnotonlyforindividualfirmsthatrisklosingouttotheir

competitors. Threatstointellectualpropertycanundermineentiresectorsoftheeconomicsystemby

makingitharderforfirmsandindividualstointernalizethebenefitsoftheirresearchandproduct

development. Thelegalsystemprovidesonlypartialprotectionandimposesbigadministrative,

transaction,andenforcementcostswhichcanthemselvesimpedeinnovation. Defenseofintellectual

assetsthereforetendstodependalsoonvariousformsofsecrecyandphysicalbarrierstoaccessand

copyingofsensitivedata. Thepotentialfordevelopmentsthatwouldreducetheseobstacles,whenthat

wouldhavenegativeconsequences,constitutesanunveilinghazard.23

Consider,finally

Recognitionhazard: Somesocialfictiondependsonsomesharedknowledgenotbecoming

commonknowledgeornotbeingpubliclyacknowledged;butpublicreleaseofinformationcould

ruinthepretense.

Twogentlemen,AandB,areinasmallroomwhenAbreakswind. Eachknowswhathashappened.

Eachmightalsoknowthattheotherknows. Yettheycancolludetopreventanembarrassingincident.

First,Bmustpretendnottohavenoticed. Second,Amight,withoutlettingonthatheknowsthatB

knows,provideBwithsomeexcuseforescapingoropeningthewindow;forexample,Acouldcasually

remark,afterashortdelay,thattheroomseemstoberatheroverheated.24 Therecognitionhazard

consistsinthe

possibility

of

dissemination

of

some

information

that

would

constitute

or

force

apublic

acknowledgement;onlythenwouldtheflatusbecomeasociallypainfulfauxpas.

23Theclaimhereisnotthattheeasieritistoprotectintellectualassets,thebetter. Insomeareastheremightfor

examplebeaninefficientlyhighleveloflegalprotection. Developmentsthatmakeintellectualpropertytheft

easiertocarryout,andhardertodetectandpunish,couldthenbesociallybeneficial.

24Goffman1959.

13


14/35

5.RisksofirrationalityanderrorThenthereareinformationhazardswhich,bycontrasttothosementionedabove,dependoneither

irrationalityorfalsebeliefs. Thisdependency,ofcourse,doesnotconsignthecorrespondinghazardsto

amarginalstatus.

Consider

Ideologicalhazard: Anideamight,byenteringintoanecologypopulatedbyotherideas,

interactinwayswhich,inthecontextofextantinstitutionalandsocialstructures,producea

harmfuloutcome,evenintheabsenceofanyintentiontoharm.

SupposethatBobbelievesthatscriptureSconsistsofexclusivelyliteraltruths,andthatheiscommitted

todoingwhateverSsaysoughttobedone. Suppose,furthermore,thatScontainsthe(presumably

false)moralstatementThoushaltdrinkseawater,butthatBobisunawareofthis. Thepotential

disseminationtoBobofthispartofthecontentofSconstitutesaninformationhazard. Theinformation

couldharmBobbyinducinghimtodrinkseawater. (Notethattheconveyanceoftrueinformation

harmsBobhere;inthiscase,thetrueinformationthatScallsfordrinkingseawater.)

Intheprecedingexample,thehazardposedbytherelevantinformationistightlycoupledtoBobs

idiosyncraticbeliefsystem. Itistruethattheideaofanuclearbombisalsoahazardonlywhencoupled

withalargerbeliefsystemforinstance,beliefsaboutphysicsandtechnologyrequiredtobringabomb

intoexistence. Yetitseemspossibleandusefultodistinguishthiskindofinstrumentalinformation

hazardfromideologicalinformationhazard. Ideologicalhazard,wemightsaybywayofexplication,

referstothepossibilitythatthatsomebodywillbemisleadtoheadinsomebaddirectionbecauseofthe

waythatsomeinformationinteractswithfalsebeliefsorincompleteknowledge.

Thatbadideologiescanbeextremelydangerousisamplyevidencedbytwentiethcenturyhistory.What

islessclearishowideologicalhazardcanbestbereduced. Partofthereasonwhythisisadifficult

problemisthatideologycanalsobeaforceforgood. TheideologyoftheAmericancivilrights

movement,forexample,helpedpushbackracialdiscriminationintheU.S. Inawidesense,ideologyis

perhapsaninevitablepartofthehumancondition,andtheproblemofdistinguishinggoodfrombad

ideologymaybenoeasiertosolvethantheproblemofdistinguishinggoodfrombadpolicy:nosimple,

generallyacceptablealgorithmexists.Moreover,whileradicalideologiesmaybeespeciallydangerous,

theymayalsodependingonwhatthestatusquoisrelativetowhichthealternativestheypresentare

radicalbeespeciallyappropriateforthesituation. Ifthestatusquoisslaveryandreligious

prosecution,thenitwouldbearadicalideologythatproposesnotmerelyameliorationoftheworking

conditionsforslavesandreductionofthepenaltiesforheresy,butcompleteabolitionandunlimited

religiousfreedom.

Nextweturntothefactthathumanbeingsarenotperfectlyrationalnordowehaveperfectself

control.Wecanbedistractedagainstourwillandwecansuccumbtotemptationagainstourbetter

14


15/35

judgment. Exposuretoinformationcanhaveeffectsonusotherthansimplyimprovingtheaccuracyof

ourrepresentationsoftheworld.

Someinformationisdistracting. Itinvoluntarilydrawsourattentiontosomeideaorthemewhenwe

wouldprefertofocusourmindselsewhere. Anadvertizingjinglemightloopinourmindsanddistract

usfromsomethingwewouldratherbethinkingabout. Onetechniqueweusetofighttemptationistoputsomethingoutofourmind;yetinformationaboutthetemptingobjectcanundermineoureffort

andmakeusmorelikelytocave. Arecoveringalcoholiccanbeharmedbyexposuretoavividaccount

oftheattributesofChateauPetrusPomerol1990.

Distractionandtemptationhazards: Informationcanharmusbydistractingusorpresentingus

withtemptation.

Inmostindividualcasesthedamagedonebydistractingortemptinginformationisperhapsminor. Yet

itisnotunreasonabletowonderwhetherthereadyavailabilityofcertainkindsofinformationmight

potentiallycausedamageonawiderscale. Perhapsitcouldbearguedthattelevisionhasanaggregate

effectonthecontemporaryhumanconditionnottoodissimilarfromthatwhichwouldbeproducedby

thewidespreadrecreationaluseofopiatedrugs. Inthefuture,evenmorecompellinglypresented

informationandhyperstimulimightbecomeavailableandproveenormouslyaddictive;forexample,

newformsofhighlyimmersiveorinteractivevirtualrealityenvironments. Druglikeeffectsonour

psychescanbeproducednotonlythroughinjection,ingestion,andinhalationbutalsothroughthe

intakeofinformationpresentedincertainmannerstooursenses.

Wecanalsobeharmedbyexposureto(thetemplatehazardof)badrolemodels. Evenwhenweknow

thatamodelisbad,andwewouldprefernottobeinfluencedbyit,prolongedexposurecan

neverthelessbedetrimentalbecauseofakindofsocialosmosis. Someonewhoaspirestoagood

writingstylemightbewelladvisedtoavoidreadingtoomuchtrash. Onewhoseekstocultivatealoftysentimentmightwanttoavoidthecompanyofthemeanandpetty. Andthosewhohopethattheir

childrenwillbecomeuprightcitizensshouldkeepthemawayfromdelinquentpeers.25 Recentstudies

indicatethatsubjectivewellbeingandevenbodymassaresignificantlyinfluencedbyourassociates.26

Thus,

Rolemodelhazard:Wecanbecorruptedanddeformedbyexposuretobadrolemodels.

OneexampleofthisistheWerthereffect,namedafterthewaveofsuicidesamongyoungmenwhich

sweptEuropeafterthepublicationin1774ofGoethesnovelDieLeidendesjungenWerthers. Several

25Otherthingsbeingequal,thatis;whichofcoursetheyseldomare.Whendecidingwhattodo,oneshouldalso

takeintoaccountthatexposuretoawiderangeofrolemodelscouldprovidemoreopportunitiesforchoice;and

thatonecanbecomewiserbyalsoknowingsomethingaboutthedarkside.Whenexcessive,thefearof

contaminationbybadinfluencesisstultifying. Initsextremeforms,aloveofpuritycanproducedangerous

intoleranceandbigotry.

26HallidayandKwak2007.

15


16/35

studieshavecorroboratedtheexistenceofsuchaneffect,findingalinkbetweenmediareportingof

highprofilecasesandensuingcopycatsuicides.27

Informationrisksariseoutofoursusceptibilitytovariouscognitivebiasesthatcanbeaggravatedbythe

provisionofcertainkindsofdata. Anchoringbiasresultsfromapplicationoftheanchoringand

adjustmentheuristicinwhichpeopleestimatesomeunknownquantitybyfirstanchoringonsomefigurethathappenstocometomindandthenadjustingthispreliminaryestimateeitherupordownin

anattempttoreflecttheirtotalinformation. Thisleadstobiaswhenpeopleinitiallyanchoronan

irrelevantquantityandthenunderadjustintheadjustmentphase. Inonestudysubjectswereaskedto

estimatethenumberofcountriesinAfrica. Beforeproducingtheirestimate,awheeloffortunewas

spun. Subjectswhoobservedalargernumberonthewheeltendedtogiveahigherestimateofthe

numberofAfricancountries,despitethetransparentirrelevanceoftheformerfact. Theextrapieceof

trueinformationaboutthenumberonthefortunewheeldiminishedtheaccuracyofgeographical

judgment.28

Manypeopleoverestimatetheirownvirtuesandabilities. Supposesuchapersonreceivessome

additionalweakcueoftheirsupposedexcellence,suchasagoodscoreonatriviaquiz. Thisbitof

evidence,whichwecansupposetobetrueandinaverylimitedwayinformative,couldaggravatetheir

selfoverestimationandconceitedness.29

Evenknowledgeofhumanbiasesandcriticalphilosophycanleadtheunwarydeeperintoerror,and

reducehisabilitytolearn,byarminghimwithcleverargumentswithwhichtorebutobjectionsand

rationalizeinconvenientfacts.30 Aspecialkindoffoolisbornwhenintelligencethusoutwitsitself.

Biasinghazard:Whenwearebiased,wecanbeledfurtherawayfromthetruthbyexposureto

informationthattriggersoramplifiesourbiases.

Methodology,suchasdoubleblindingindrugtrials,canhelpreducetheriskofbiasesentering

uninvitedintoourthinkingandacting. Forsimilarprecautionaryreasons,thegullibleoftenhavereason

toavoidthehighlypersuasive. Andifoneplanstoexperiencetransportsandecstasiesthatwill

temporarilyincreaseonessusceptibilitytodangerousillusionsandimpulses,oneshouldfirsthave

oneselftiedtothemast.

Conversely,informationcouldalsoharmusbyreducingourbiasesinsofarasourbiasesservesome

usefulpurpose. Forexample,atendencytooverestimateourownabilitiesmightnotonlymakeusfeel

happierandconfident;astrongbeliefinourownabilitymightalsosignalcompetenceandleadothersto

27Seee.g.Phillips1982;Stack1996;Jonas1992.

28TverskyandKahneman1974.

29Dittoforthosewhounderestimatetheirownvirtuesandabilities:feedbackthatconfirmsthistendstobe

internalizedwhilefeedbackthatcontradictsittendstobeignored(Brown,Duttonetal.2007).

30Yudkowsky2008.

16


17/35

allywithus,promoteus,orvoteforus. Informationthathelpsusseeourselvesforwhatwereallyare

coulddepriveusofthesebenefits. Itisalsopossiblethatsocietybenefitsfromexcessindividualrisk

takinginsomedisciplines;forexampleifentrepreneurs,inventors,andyoungacademicsoverestimate

theirownchancesofsuccess. Iftheseoccupationshavenetpositiveexternalities,itcouldbebeneficial

thatbiasesandunrealisticexpectationsoffame,fortune,orhighachievementseduceadditional

entrantsintothesefields.

Debiasinghazard:Whenourbiaseshaveindividualorsocialbenefits,harmcouldresultfrom

informationthaterodesthesebiases.

Thereisalsoawiderphenomenonofwhichrolemodelinfluenceisbutaspecialcase. Ourbrainsare

constantlyreshapedbywhatwelearnandexperience. Informationgleanedisnotsimplystoredaway

asinertdatapackages,asthoughitwerenewvolumessuperaddedtosomeinternalbookshelf. Rather,

theincominginformationinteractscomplexlywithpreexistingcognitivestructuresinwaysthatarenot

alwayseasytocharacterizeinfolkpsychologicalterms. Newconceptsmightform;boundariesofextant

conceptsmightchange;neuronalwiringpatternsarealtered;somecorticalareasmightexpand,causing

otherareastocontract;andsoforth. Thereisariskthatsomeofthesechangeswillbefortheworse.

Neuropsychologicalhazard: Informationmighthavenegativeeffectsonourpsychesbecauseof

theparticularwaysinwhichourbrainsarestructured,effectsthatwouldnotariseinmore

idealizedcognitivearchitectures.

Toomuchknowledgecanbebadforsometypesofmemory.31 Perhapssomementalillnessesresult

frominappropriatecrosstalkbetweencognitivemodulesdesignedtooperateasmoreencapsulated

unitsakindofundesirableinternalinformationdissemination. Arecurringideainliteratureand

mythologyisthemotifofharmfulsensation,whereapersonsuffersmentalorphysicalharmmerely

byexperiencingwhatshouldnormallybeabenignsensation(themythofMedusa,beliefsabouttheevileyeetc.). Arealworldexampleofharmfulsensationisphotosensitiveepilepsywhichcanbe

triggeredinsomesensitiveindividualsflickeringlightsorspecificgeometricpatterns.32

Irrelevantinformationcanmakevaluableinformationhardertofind. Thisfactisusedinsteganography,

thecryptographictechniqueofhidingsecretmessageswithinrepresentationsthatappeartobeof

somethingelsesothateventheexistenceofcoverttextisconcealed. Forexample,someofthepixelsin

animagefilecanbesubtlymodifiedsoastoencodeaverbalmessageinwhatlookslikeanordinary

touristpicture. Inasimilarvein,legaldefenseteamssometimesconcealincriminatingdocumentation

31

RobinsonandSloutsky2007.

32WhenthecartoonepisodeDenn SenshiPorygonairedinJapanin1997,onescenefeaturinganexplosion

renderedwithstrobelightingeffectcausedmildsymptomsin510%oftheviewers(thoughsomeofthesemight

insteadhavebeenafflictedwithepidemichysteria)andsent685childrentohospitalinambulance. Nolongterm

damagewasreported. Therehasalsobeenatleastonemaliciousattempttodeliberatelytriggerphotosensitive

epilepsyonline,butitappearsnottohavebeenverysuccessful. SeeRadfordandBartholemew2001;Takada,Aso

etal.1999.

17


18/35

thathasbeensubpoenaedbytheprosecutionbyoverwhelmingitwithsuchmassiveamountsof

archivalmaterialthattherelevantdocumentscannotbelocatedintimeforthetrial.

Informationburyinghazard: Irrelevantinformationcanmakerelevantinformationharderto

find,therebyincreasingsearchcostsforagentswithlimitedcomputationalresources.33

Onagranderscale,anoverabundanceofinformationalaffordancesmightdeflectourthinkingfrom

topicsthataremorecentraltousandrelativelymoreworthyourcontemplation,sothatweshalllive,as

inT.S.Eliotscharacterizationofthemodernpredicament,Distractedfromdistractionby

distraction.34 Thiskindofpossibilityleadsustothenextsection.

6.RiskstovaluablestatesandactivitiesWehavelookedathowinformationcancauseharmbyaffectingbehavior. Informationcanalsoharm

throughitsdirectpsychologicaleffects,forexamplebycausingdisappointment.Moreover,accordingto

atleastsomeaccountsofwellbeing,informationcancauseharmevenasidefrompsychological

spillovereffectsbyaffectingsomepartofsomeepistemicorattentionalstatethatplaysaconstitutive

roleinsomesubjectswellbeing. Thuswecandefine

Psychologicalreactionhazard: Informationcanreducewellbeingbycausingsadness,

disappointment,orsomeotherpsychologicaleffectinthereceiver.

Andwecandistinguishthisfromthefollowingmorephilosophicallyintricatenotion:

Beliefconstitutedvaluehazard: Ifsomecomponentofwellbeingdependsconstitutivelyon

epistemicorattentionalstates,theninformationthataltersthosestatesmighttherebydirectly

impactwellbeing.

Considerfirsttheobviousexampleofapsychologicalreactionhazard:badnews,thereceiptofwhich

makesussad.

Disappointmenthazard: Ouremotionalwellbeingcanbeadverselyaffectedbythereceiptof

badnews.

Insomecases,ifsomethinggoeswrong,weareboundtohearofiteventually. Insuchcases,the

disappointmentisinasensealreadycommittedwhentheadverseeventtakesplace,eventhoughit

might

take

a

while

before

the

effect

is

known

and

felt.

33Andpotentiallyresultinworsesolutions;foradiscussionofhowexcessiveknowledgecandegradeperformance

insomeartificialintelligencesystems,seeMarkovitchandScott1988.

34Eliot2001.

18


19/35

Inothercases,however,thereisarealchancethatifasubjectavoidshearingofhermisfortunenow,

shewillremainignorantandwillbesparedthedisappointmentthatthebadnewswouldoccasion. Such

casesmakeiteasiertodisentanglethedisappointmenthazardfromotherpossibleharmsthatmightbe

involved. Takethecaseofamotheronherdeathbed,whoseonlysonisfightinginsomefarawaywar.

Themotherfacesatleasttwodistinctrisks. First,thereistheriskthathersonwillbekilledorinjured;

thisisnotnecessarilyaninformationrisk. Supposethatthesonisinfactkilled. Thenthereisasecond

risk,whichisthatthemotherwillfindoutaboutherloss. Supposethatthenewsiscontainedinaletter,

whichmightreachherquicklyoritmightbedelayed. Ifitreachesherquickly,shewillspendherlast

daysinagonyanddespair;ifitisdelayed,shewilldieinpeace. Herewemightsaythatthemotheris

exposedtoaseveredisappointmenthazard.

Spoilersconstituteaspecialkindofdisappointment.Manyformsofentertainmentdependonthe

marshallingofignorance. Hideandseekwouldbelessfuniftherewerenowaytohideandnoneedto

seek. Forsome,knowingthedayandthehouroftheirdeathlonginadvancemightcastshadowover

theirexistence.

Beforehisretirement,myfatherwouldsometimesmissapivotaltelevisedsoccergamethattookplace

duringworkinghours. Planningtowatchtherepriselater,hewouldmeticulouslyavoidanynewssource

thatmightdisclosetheresults. Hisdesign,however,wasthwartedbymygrandfather,whohad

watchedthegameliveandwhoinvariablyfoundhimselfunabletorefrainfrommakingnotquite

subtleenoughallusionstothematch,enablingmyfathertoguesswhohadwon.

Spoilerhazard: Funthatdependsonignoranceandsuspenseisatriskofbeingdestroyedby

prematuredisclosureoftruth.

Knowledgecanalsoexertmoregeneraleffectsonourpsychesandpersonalities. Perhapsanunwanted

cynicismispromotedbyanexcessofknowledgeaboutthedarksideofhumanaffairsandmotivations.Nietzschewarnedofthemisusesofhistory:howhistoricalknowledge,approachedandvaluedina

certainway,cansapourzestforlifeandinhibitartisticandculturalauthenticityandinnovation. The

dangerNietzschepointedtowasnottheeffectsofanyoneparticularpieceofinformationbutrather

theconsequencesofacertainexcessofhistorywhichcancauselivingtocrumbleaway:esgibteinen

Grad,Historiezutreiben,undeineSchtzungderselben,beiderdasLebenverkmmertundentartet

(thereisadegreeofdoinghistoryandvaluingofitthroughwhichlifeatrophiesanddegenerates).35 If

Nietzscheisrightaboutthis,andifthedisseminationof(variouskindsof)informationaboutthepast

can,underunfavorablecircumstances,contributetosuchanatrophyofspirit,thenwehaveherean

exampleofanothertypeofpsychologicalreactionhazard,namely

Mindsethazard: Ourbasicattitudeormindsetmightchangeinundesirablewaysasa

consequenceofexposuretoinformationofcertainkinds.

35GermanquotationtakenfromNietzsche1984;EnglishtranslationtakenfromNietzsche2007.

19


20/35

Alongsimilarlines,somepeopleworrythatscientificreductionism,akintostripmininginanancient

forest,despoilslifeofitsmysteryandwonder.

Letusturntobeliefconstitutedvaluehazard. Inpractice,thiscategorycanbedifficulttodistinguish

frompsychologicalreactionhazard.

Consideragaintheexampleofthemotheronherdeathbedwhoriskshearingthathersonhasbeen

killed. Thereisclearlyonerespectinwhichhearingthiswouldbebadforher:itwouldcauseher

extremepsychologicaldistress. Thisissufficientfortheretobeapsychologicalreactionhazard. Itdoes

notrequirethatitwouldbebadforthemother,allthingsconsidered,tohearofhersonsdeath.

Thereareseveralreasonsforthis. First,thereareofcoursevariouspracticalmattersthatwouldneedto

beconsideredinanactualsituationlikethis: Perhapsthemotherneedstoknowsothatshecanalter

herwill? Perhapsconcealmentofunpleasantnewswouldtendtoerodesocialtrust? Butevenaside

fromsuchpragmaticconsiderations,thereisasecondtypeofreasonwhyitmightbebetterforthe

mothertoknowofhersonsdeathdespitethesufferingthisknowledgewouldcauseher. Such

knowledge,accordingtosomemoraltheories,canbeacomponentofapersonswellbeing(thegood

foraperson)evenifitaffectsthesubjectivecomponentofwellbeingfortheworse.Onemighthold

thatalifeismadeworse,otherthingsequal,byitsbeinglivedinignoranceofimportantfactsaboutthe

centralconcernsofthatlife. Lifeinafoolsparadise,evenifitscoreshighonthehedonicdimension,

mightneverthelessscorequitelowinoverallwellbeingonsuchatheory.

Justasonemightholdthatthereissomeknowledgethepossessionofwhichisanimportantconstituent

ofagoodlife,onemightalsoholdthatthereisknowledge(atleastforsomepeople,insome

circumstances)thatmakesadirectnegativecontributiontotheirwellbeing. Thiscanmostobviouslybe

seentobethecaseaccordingtoapreferencesatisfactionaccountofwellbeing;fortherewecan

generateexamplestriviallysimplybysupposingsomebodytohaveabasicpreferenceagainstknowingaboutsomeparticularsubjectmatter. Butmanyotheraccountsofwellbeingmightalsopermitof

examplesofsuchdirectlyburdensomeknowledge. Innocencemightbevaluedforitsownsakeand

mightberuinedbyknowledge.Wemighttreasureourprivacyandfinditinfringedbyotherpeoples

knowingthingsaboutusthatwewouldratherhavekepttoourselvesorsharedexclusivelywithchosen

intimates. Orwemightbebetteroffnotknowingsomepersonaldetailsaboutothers,notjustbecause

suchknowledgemightexposeustoriskofworsetreatmentfromothers(asinknowingtoomuch

hazard)butalsobecauseourawarenessofthesedetailswouldstandinthewayofourconceivingof

othersinmannersthataremoreappropriateormoretoourliking.Withregardtoourfriendsbowels

andourparentsbedrooms,thelessweknowthebetter.36

Onecommonlyfearedriskfrominformationis

Embarrassmenthazard:Wemaysufferpsychologicaldistressorreputationaldamageasaresult

ofembarrassingfactsaboutourselvesbeingdisclosed.

36AndofcourseBismarckclaimedoflawsandsausagesthatitisbetternottoseethembeingmade.

20


21/35

Embarrassmenthazards(whichoftentaketheformofsignalinghazard)commonlycombineelementsof

psychologicalreactionhazard,beliefconstitutedvaluehazard,andcompetitivenesshazard.Wemay

evenfeartoembarrassourselvestoourselves,perhapsbecauseselfesteemisnotawhollyprivate

matterbutisalsoasocialsignalthatinfluencesothersopinionsofus.37 Somepsychologistsbelievethat

aconcerntoprotectselfesteemfromunderminingbyselfrelevantfailurescanleadindividualsto

engageinselfhandicappingbehavior.38 Thiscouldhelpaccountforsomeinstancesofphenomenasuch

asprocrastination,hypochondria,substanceabuse,andpracticeavoidance.39 Supposethatthinkingof

yourselfasintelligentisimportantforyourselfesteemandthatyouhaveanimportantexamcoming

up. Ifyoupracticehardandfailontheexam,yoursenseofselfcompetencewilltakeahit. Butifyou

putoffpracticinguntilthenightbeforetheexam,yourriskisreduced;evensmartpeoplecandopoorly

onexamswhentheyhavenotstudiedenough. Andifdespitethehandicapofinsufficientpreparation

youstillmanagetogetahighmark,whythenyoumustbetrulybrilliant. Suchperceptionmanagement

canimposesignificantcosts.

Riskofembarrassmentcansuppressfrankdiscussion. AstudyondeliberationintheFederalReserves

FederalOpenMarketCommitteefoundevidencethatanewlyadoptedpolicyoftransparencyinvolvingthepublicationofdetailedtranscriptsfrommonetarypolicymeetingsstifledthevoicingofdissenting

opinionsandseemedtoreducethequalityofdebate.40

Intangibleassets,suchasreputationandbrandname,constitutealargepartofthevalueofmanyfirms.

Embarrassmentsthatnegativelyimpacttheseassetscancausebillionsofdollarsindamage. Foran

exampleonanevengranderscale,considertheColdWarsuperpowerrivalry,inwhichbothcontenders

wereengagedinstatuscontestaswellasamilitaryarmsrace. TheApolloprojectwasadirectresponse

totheembarrassmentcausedtotheUnitedStatesbytheSovietUnionslaunchofSputnik1,an

accomplishmentthatchallengedtheAmericasclaimtotechnologicalsuperiority. TheVietnamandthe

Afghanwars

were

both

prolonged

because

of

reluctance

to

suffer

the

reputational

damage

that

leaders

believedwouldresultfromadmittingdefeat.

7.RisksfrominformationtechnologysystemsItisnotonlyanimatebeingsthatprocessanddisseminateinformation;ourinformationtechnological

systemsdosoaswell. Distinctiveinformationhazardsariseinrelationtoourcomputersandnetworks.

Informationtechnologysystemsarevulnerabletounintentionallydisruptiveinputsequencesorsystem

interactionsaswellastoattacksbydeterminedhackers. Hereweconsideronlyriskoccasionedby

37Hobden1997.

38BerglasandJones1978.

39Smith,SnyderandPerkins1983;Stone2002;ThompsonandRichardson2001.

40MeadeandStasavage2008.

21


22/35

informationaleffectsunanticipatedsysteminteractions,worms,viruses,Trojanhorses,denialof

serviceattacks,andsoforth. Thismeansweexcluderisksarisingfromthepossibilityofflooding,power

outages,andsomebodyattackingyourcomputerwithasledgehammerexceptinsofarasarisk

consistsinthepossibilityofinformationalamplificationoftheeffectsofsomesuchnoninformational

trauma. Thus,theriskthatyoumightdropyourbrandnewlaptoponahardfloorsothatitbreaksand

youincurthecostofbuyingareplacementisnotaninformationhazard. Noristheriskthatsome

criticalinformationsystemmightgodownnecessarilyaninformationhazardasdefinedhere. Themere

cessationoffunctioningofsomeusefulinformationprovidingsystemisnotenoughunlessthecauseof

thecessationisinformationalortheharmarisesfromsomekindofundesirablepropagationof

information.

Informationsystemhazard: Thebehaviorofsome(nonhuman)informationsystemcanbe

adverselyaffectedbysomeinformationalinputsorsysteminteractions.

Thiscategorycanbesubdividedinvariousways:onecould,forexample,distinguishcomputerhazards

fromnetworkhazards;orsingleoutthreatstocriticalinformationinfrastructure;oronecouldmakea

separationbetweenscenariosinvolvinglossofdata,corruptionofdata,disseminationofdatatothe

wrongparties;andsoforth. Quitealotofattentionisalreadygiventoinformationsystemhazards,and

muchofthisattentionisfocusedonwhatwemayterm

Informationinfrastructurefailurehazard: Thereisariskthatsomeinformationsystemwill

malfunction,eitheraccidentallyorasresultofcyberattack;andasaconsequence,theowners

orusersofthesystemmaybeinconvenienced,orthirdpartieswhosewelfaredependsonthe

systemmaybeharmed,orthemalfunctionmightpropagatethroughsomedependentnetwork,

causingawiderdisturbance.

Risksofthistypecanbequiteseverewhensomecomplexsystemornetworkisusedtocoordinateimportanthumanactivities. Forinstance,acorruptionofthesoftwarethatundergirdsimportant

financialsystemscouldhaveseriousconsequences.

Adifferenttypeofinformationsystemhazardisthatsomeinformationsystemwillinfactfunctionas

intended,butbydoingsoitwillcauseharmoramplifysomeriskofharm.

Aprivacyadvocatemightobjecttosomegovernmentdatabaseprojectthatwillamassvastquantitiesof

informationaboutthecitizenry,notonlybecauseoftheriskthatthesystemmightmalfunctionorbe

hacked,butalsobecauseoftheriskthatitwillperformtospecificationandtherebystrengthenthe

statesabilitytomonitortheactivitiesofitspeopleandshouldthegovernmentonedayseeaneedto

dosototakeactionagainstelementsdeemedundesirableordisloyal. Evenifitwereadmittedthat

thegovernmentthatbuildsthesystemcanbetrustedtouseitonlyforgood,onemightfearthatlater

governmentswhichinheritthesystemcannotbesotrusted,orthatsomemoreperniciousgovernment

elsewherewillseeinthesystemaninspiringprecedent(cf.,ideahazard,andattentionhazard)or

justification(cf.,normhazard)forbuildingitsowncomparablesystemandapplyingittoitsown

nefariousends.

22


23/35

Similarconcernscanapplytoprivatefirms,suchasGoogle,thatcollectpersonalinformationabout

hundredsofmillionsofusers. ConsiderhowusefulitwouldhavebeenforatwentyfirstcenturyStalin

tobeabletohavehissecurityservicedataminethecitizenrysemailcorrespondenceandsearchengine

queriesnotleasttextwrittenbeforehisascenttopowerandatatimewhenhisenemiesmighthave

communicatedtheirmostincriminatingthoughtsunguardedly.41

Informationinfrastructuremisusehazard: Thereisariskthatsomeinformationsystem,while

functioningaccordingtospecifications,willservicesomeharmfulpurposeandwillfacilitatethe

achievementofsaidpurposebyprovidingusefulinformationinfrastructure.

Asystemcanalsobedangerousbypresentinganeasyopportunityforunintentionalmisuse. Considera

poorlydesignedemailprogramthatmakesistooeasyfortheunwaryuseraccidentallytoforwardan

emailreplytoalltheaddresseesinhercontactlist;anembarrassmentwaitingtoblush. Thishazardlies

ontheborderbetweeninformationinfrastructurefailureandinformationinfrastructuremisuse,itbeing

unclearwhethersuchanemailprogramisfunctioningaccordingtoitsintendedspecificationsand

arguablehowapportionblamebetweenthesystemsdesignersanditsusers.

Forcomparison,wemayalsonotetwoothertypesofhazardpotentiallyarisingoutofinformation

technology(butwhicharetypicallynotinformationsystemhazards)wheretheharmisnotsomucha

consequenceofthegeneralinformationinfrastructureservicesthatasystemprovidesorfailstoprovide

butinsteadismoredirectlyrelatedtotheagencyofthesystemitself:

Robothazard: Therearerisksthatderivesubstantiallyfromthephysicalcapabilitiesofarobotic

system.

Anautonomousvehicle,loadedwithexplosivemissiles,andabletolaunchonitsowninitiative,could

constitute

a

robot

hazard.

We

can

contrast

this

with

Artificialintelligencehazard: Therecouldbecomputerrelatedrisksinwhichthethreatwould

deriveprimarilyfromthecognitivesophisticationoftheprogramratherthanthespecific

propertiesofanyactuatorstowhichthesysteminitiallyhasaccess.

Anartificialintelligencewouldneedtobeveryadvancedinordertoposeanysignificantthreatinvirtue

ofitsowningenuityandagency. Thecreationofartificialgeneralintelligence,withgeneralpowersof

reasoningexceedingthoseofhumanbeings,wouldbeassociatedwithaserious,indeedexistential,

risk.42 Asuperintelligence,evenifinitiallyrestrictedtointeractingwithhumangatekeepersviaatext

interface,mighthackortalkitswayoutofitsconfinement. Itcouldthengaincontrolovereffectorsto

conductoperationsintheexternalworldforexample,byusingitspowersofpersuasiontogethuman

beingstodoitsbiddings,orbyassumingcontrolofroboticmanipulators. Itcouldusetheseeffectorsto

41Ofcoursetherearebigpotentialupsidestoo;e.g.,agoodgovernmentcouldsubpoenathisinformationforuse

inagoodcause.

42Bostrom2002;Yudkowsky2008.

23


24/35

developnewtechnologiesandtosecureamorecomprehensivegraspofitsphysicalsurroundings. The

threatposedbyasufficientlyadvancedartificialintelligencemaydependmuchmoreonitscognitive

capabilitiesanditsgoalarchitecturethanonthephysicalcapabilitieswithwhichitisinitiallyendowed.

Notallrisksrelatedtorobotsorartificialintelligencesaretobeclassifiedasinformationsystem

hazards. Ariskwouldcountassuchahazardif,forexample,itarosefromthepossibilityofacomputervirusinfectingtheoperatingsystemforarobotoranAI. Butasidefromsuchspecialcases,weshallnot

countrobothazardsandartificialintelligencehazardsasinformationsystemhazards.43

Thereis,however,anotherwayforrobot andAIrelatedriskstoenterourinformationhazard

taxonomy. Theycanenteritinthesamewaysasanyriskrelatingtopotentiallydangeroustechnological

development.

8.RisksfromdevelopmentBothtechnologicalinnovationandeconomicdevelopmentmorebroadlyarisefromtheaccumulationof

information,ideas,andinsights;andthiscanresultinarangeofrisksthatwecangrouptogetherunder

therubricofdevelopmenthazards.

Developmenthazard:Progressinsomefieldofknowledgecanleadtoenhancedtechnological,

organizational,oreconomiccapabilities,whichcanproducenegativeconsequences

(independentlyofanyparticularextantcompetitivecontext).

WhenthemushroomcloudsroseoverHiroshimaandNagasaki,physicists,manyofwhomhadentered

theirprofessionforthesheerjoyofdiscovery,foundthemselvescomplicitinthedeathsofperhaps

200,000people.44

IfthecoldwarhadendedinanalloutnuclearexchangebetweenNATOandtheSovietUnion,asitmighteasilyhavedone,thenmorethanabillioncivilianscouldhavediedasafairly

directconsequenceofthedevelopmentofnuclearweapons.45

43Thereis,ofcourse,asenseinwhichbothrobotsandadvancedmachineintelligencesareinformationsystems.

Thereisalsoasenseinwhichthehumanbrainisaninformationsystem. Yettherisksthatarisefromintelligence

ingeneral,orfromthephysicalequipmentofsomerobot,areextremelyheterogeneous;whereforeitwouldseem

notveryilluminatingtoconstructaninformationsystemhazardcategorythatlumpedthemalltogether.

44TheAtomicArchiveestimatesthedeathsinHiroshimaandNagasakiimmediatelyfollowingthebombingsat

105,000,withafurther94,000injured(TheManhattanEngineerDistrict1946).Manyhavelaterdiedofcancerorbirthdefectscausedbyradiationexposure,buttheexactfiguresareasubjectofdebate.

45PresidentKennedyissaidtohaveatonepointduringtheCubanmissilecrisisestimatedtheprobabilityofa

nuclearwarbetweentheU.S.andtheUSSRtobesomewherebetweenoneoutofthreeandevenKennedy

1968;Leslie1996. JohnvonNeumann,whoaschairmanoftheAirForceStrategicMissilesEvaluationCommittee

wasoneofthearchitectsofearlyU.S.nuclearstrategy,isreportedtohavesaiditwasabsolutelycertain(1)that

therewouldbeanuclearwar;and(2)thateveryonewoulddieinit(Putnam1979,114). SeealsoCirincione2008.

24


25/35

RobertOppenheimer,thescientistwhohadspearheadedtheManhattanproject,acknowledged

afterwardsthatthephysicistshaveknownsin;andthisisaknowledgewhichtheycannotlose.46 Of

course,reachingamoralverdictonthescientistswhoworkedontheManhattanprojectisnotassimple

astotingupthenumberofdeathsthatwerelatercausedbytheweapontheyinvented.Manyofthese

scientistsdevotedthemselvestotheprojectbecausetheyfearedthatHitlermightgetthebombfirsta

fearwhich,althoughitturnedouttobeunfounded,wasreasonablegiventheinformationavailable

whentheprojectbegan. RichardFeynman,anotherphysicistwholaterreflectedonhisinvolvement,

regardedhisinitialdecisiontoparticipateasmorallyjustifiedforjustthisreason;buthethoughtthathe

hadfailedmorallyinnotreconsideringhisinvolvementafteritbecameclearthatHitlerhadbeenunable

toacquirethebombandthatGermanycouldbedefeatedwithoutit. Furthermore,thedecisiontouse

thebombwasnottakenbyphysicistsbutbyPresidentTruman,whomayhaveactedonavarietyof

motivesinacomplexstrategicsituation;andsoforth.47

Thepointhereisnotthatsomeparticularpastactionwasorwasnotjustified,butthatthiskindof

consequencecanresultfromtheinformationgatheringworkofphysicistsincludingbasicresearch

suchastheearlierworkinquantumandparticlephysicsthatlaidthetheoreticalfoundationsfortheManhattanproject. Toproceedblithelyandwithoutscruple,asthoughnothingverybadcouldcome

fromsuchresearch,wasperhapsexcusablenavetbackthen.48 Forourowngeneration,whichisable

toobservemorehistoricalprecedent,suchnegligencewouldmorelikelyamounttoculpableabrogation

ofmoralresponsibility.

WhatwastrueofphysicsinthedecadesleadinguptotheSecondWorldWarmaybetrueofother

academicdisciplinestoday. Biologyandbiotechnology,whileprovidingurgentlyneededmunitionsfor

useinourbattleagainstdisease,malnourishment,andagerelateddebility,alsothreatentoarmthe

humanspecieswithweaponsofmassdestructionthatmightbedeployedagainstourownkind.

Recentdevelopmentspointtodisturbingpossibilitiesdowntheroad. Considerthesteadilyimproving

capacityandavailabilityofDNAsynthesismachines. Thistrendisworrisomewhenoneconsidersthat

thegenomesofmanyextremelydangerouspathogensresideinthepublicdomain,includingEbola,

Marburg,smallpox,andtheSpanishfluvirus(believedtohavekilledmorethan50millionpeopleduring

19181919). Theknowledgeandtechnologyrequiredtogeneticallymodifymicroorganismssoasto

enhancetheirpathogenicityandtheirresistancetocountermeasuresarealsoadvancing. Technological

barrierstotheproductionofsuperbugsarebeingsteadilyloweredwhilebiotechnologicalknowhow

andequipmentdiffuseevermorewidely.49

46

Oppenheimer1947.

47Foroneattemptatamoralassessment,seeGlover2001.

48AlthoughLeoSzilardsexamplesuggeststhatmuchofthisnavetwasavoidableatleastasearlyas1933.

Rhodes1995.

49Seee.g.NouriandChyba2008. Ofcourse,therearealsoriskmitigatingbenefitsfromsuchresearch,for

examplebetterprophylacticsandtherapeutics,andbetterknowledgeofourownvulnerabilities.

25


26/35

Dangerousinformationcouldalsoarisefromotherfieldsofinquiry. Advancedfutureformsofmolecular

nanotechnologymightbeusedtobuildweaponssystemevenmorepowerfulthanhydrogenbombsand

supergerms.50 Artificialintelligencemightonedaysurpassbiologicalintelligenceandtherebybecome

extremelypowerful.51

Technologiesformonitoringandmodifyinghumanbehaviormightadvanceonseveralfrontssuchasubiquitoussurveillancesystems,automatedfaceandvoicerecognitionsoftware,effectiveliedetection,

psychopharmacology,geneticengineering,orneuralimplants. Socialsciencemightmakeprogresson

understandingandpredictingthetriggersofsocialunrestandinsurrection. Suchcapabilitiescouldbe

usedforgoodorill. Inaworstcasescenariotheycouldfacilitatetheemergenceofnewandpermanent

formsoftotalitarianisms,possiblyonaglobalscale.

Thepossibilitiesreferredtoaboveconstitutesomeofthemostsignificantexistentialrisksthatmay

confronthumanityinthefuture.52 Otherpotentialtechnologicaldevelopmentssomeforeseeable,

othersperhapsnotmayalsocreateexistentialrisks. Becauseoftheextremevaluesatstakein

existentialrisks,theycandeservesubstantialconcerneveniftheycouldbeshowntobebothvery

unlikelyandveryremoteneitherofwhichisclearlythecasefortherisksjustmentioned.53

Thesetechnoscientificareasdonotfunctioninisolation. Bioweaponsengineerswoulddrawondata

andtechniquesdevelopedbyawiderangeofresearchersinfieldssuchasvirology,medicine,genetics,

andbiochemistry. Nanotechnologistsdrawondrawonfieldssuchasmaterialsscience,chemistry,

proteinengineering,biotechnology,andsystemsengineering. Artificialintelligencepioneersmay

benefitfromadvancesinneuroscience,cognitivescience,computerscience,foundationsofprobability

theory,andsemiconductorphysics. Furthermore,alloftheseareasareinfluencedtosomeextentby

generaleconomicgrowth,whichtendstoleadtoincreasedfundingforresearch,bettersupporting

infrastructure,andamoreeducatedworkforce.

Developmenthazardsthusariseinmanyareas,andtheyrangeinseverityfromtrivialtoexistential. Itis

importanttorecognizethatdevelopmenthazardsarenotconfinedtoespeciallysinisteror

Prometheantechnologicalbreakthroughs. Evensomethingasinnocentasmedicaloragricultural

advancesthathelpreduceinfantmortalitycanposesignificantdevelopmenthazards,suchastheriskof

overpopulationandpotentiallynegativeknockoneffectsonconflict,percapitaincome,andthe

environment. (Obviously,thefactthatsomepotentialdevelopmentisassociatedwithsomeriskdoes

50Drexler1987;Freitas2000;Gubrud1997.

51Moravec2000;Bostrom1998;Vinge1993;Kurzweil2006;BostromandSandberg2008. Aselfenhancing

generalintelligencethatbecamesuperintelligentwouldbecomeextremelypowerfulandwould,unlessrightly

designed,constituteaseverethreattohumanity. Bostrom2003;Yudkowsky2008.

52AnexistentialriskisonethatthreatstocausetheextinctionofEarthoriginatingintelligentlifeortopermanently

anddrasticallydestroyitspotential;seeBostrom2002;Rees2004.

53Bostrom2003;Matheny2007;Leslie1996;Posner2005.

26


27/35

notentailthatthisdevelopmentwouldonbeonbalancebadorthatitoughtnotbevigorously

pursued.)

Thedistinctionbetweendevelopmenthazardandthevarioushazardslistedaboveasadversarialrisksis

somewhatvague. Developmenthazards,bycontrasttoadversarialrisks,arenottiedtoanyparticular

extantcompetitivecontext. Forexample,ariskofsometechnologicaldevelopmentthatconsistsinthepotentialharmtousthatcouldresultfromthedifferentialstrengtheningofourenemyorrivalshould

beclassifiedasanenemyhazardoracompetitivenesshazardratherthanadevelopmenthazard. Buta

riskofsometechnologicaldevelopmentthatconsistsinthepotentialforharmthatarisesfromthefact

thatthistechnologywouldbelikelytocausesomesevereaccidentorwouldgenerallylenditselfto

abusesbyawiderangeofindividuals,groups,orstateswouldposeadevelopmenthazard. Some

technologicaldevelopmentsmightposebothadversarialanddevelopmentalrisks.

9.DiscussionTheconsiderationsadducedabove,althoughnotontheirowndeterminativeofwhatistobedonein

anyparticularactualcase,canhelpinformourchoicesbyhighlightingthesometimessubtlewaysin

whicheventrueinformationcanhaveharmfulaswellasbeneficialeffects(table1).

27


28/35

TYPOLOGYOFINFORMATIONHAZARDS

I.Byinformationtransfermode

Datahazard

Ideahazard

Attentionhazard

Templatehazard

Signalinghazard

Evocationhazard

II.Byeffect

TYPE SUBTYPE

Enemyhazard

Intellectualpropertyhazard

Commitmenthazard

ADVERSARIALRISKS Competivenesshazard

Knowingtoomuchhazard

Informationasymmetry

hazard

Unveilinghazard

RISKSTOSOCIAL

ORGANIZATONAND

MARKETS

Normhazard

Recognitionhazard

Ideologicalhazard

Distractionandtemptation

hazards

Rolemodelhazard

Biasinghazard

Debiasing

hazard

Neuropsychologicalhazard

RISKSOFIRRATIONALITY

ANDERROR

Informationburyinghazard

Disappointmenthazard

Spoilerhazard

Psychologicalreaction

hazard

Mindsethazard

Beliefconstitutedvalue

hazard

RISKSTOVALUABLESTATES

ANDACTIVITIES

(mixed) Embarrassmenthazard

Informationinfrastructure

failurehazard

Informationinfrastructure

misusehazard

RISKSFROMINFORMATION

TECHNOLOGYSYSTEMS

Informationsystemhazard

Artificialintelligencehazard

RISKSFROMDEVELOPMENT Developmenthazard

Table1

28


29/35

Therearemanywaysofrespondingtoinformationhazards. Inmanycases,thebestresponseisno

response,i.e.,toproceedasthoughnosuchhazardexisted. Thebenefitsofinformationmaysofar

outweighitscoststhatevenwheninformationhazardsarefullyaccountedfor,westillunderinvestin

thegatheringanddisseminationofinformation.Moreover,ignorancecarriesitsowndangerswhichare

oftentimesgreaterthanthoseofknowledge. Informationrisksmightsimplybetolerated. Insome

contextstheycouldbeinsuredorhedgedagainstusingavarietyoffinancialinstruments.54

Whenmitigationiscalledfor,itneednottaketheformofanactiveattempttosuppressinformation

throughmeasuressuchasbans,censorship,disinformationcampaigns,encryption,orsecrecy. One

responseoptionissimplytoinvestlessindiscoveringanddisseminatingcertainkindsofinformation.

Somebodywhoisworriedaboutthespoilerhazardoflearningabouttheendingofamoviecansimply

refrainfromreadingreviewsandplotsummaries.

Sometimes,suchasinthecasesofsomeideologicalhazardsandsomeinformationasymmetryhazards,

thedangerliesinpartialinformation. Thebestresponsemaythenbetoprovidemoreinformation,not

less. Someproblemscanbesolvedthroughpolicymeasurestheproblemofasymmetriesinhealth

informationcanbeobviated,forexample,byinstitutingpubliclyfundeduniversalhealthcare. Inother

cases,suchasdistractionhazardandsomebiasinghazards,thesolutionmaybetocarefullyselectan

appropriateformatandcontextfortheinformationthatistobepresented.

Whencontemplatingtheadoptionofsomepolicydesignedtorestrictinformation,itisworthreflecting

thathistoricallysuchpolicieshaveoftenservedspecialinterests. InTheWeaponofOpenness,ashort

essayonroleofsecrecyandopennessinnationalsecurity,ArthurKantrowitzwrote:

[S]ecrecyinsiderscomefromaculturewhereaccesstodeepersecretsconveyshigherstatus.

Thosewhogetaheadinthecultureofsecrecyunderstanditsusesforpersonaladvancement.

Knowledgeispower,andformanyinsidersaccesstoclassifiedinformationisthechiefsourceoftheirpower. Itisnotsurprisingthatsecrecyinsidersseethepublicationoftechnological

informationasendangeringnationalsecurity.55

Outsidersoftenhavereasonforskepticismwheninsidersinsistthattheirinnerdealingsmustbe

protectedfrompublicscrutiny. Secrecybreedscorruption. Kantrowitzarguedthatevenwithrespectto

thenarrowcriterionofmilitarystrength,aunilateralpolicyofopenness(atleastinpeacetime)leadsto

betterresults.

Atthesametime,however,weshouldrecognizethatknowledgeandinformationfrequentlyhave

downsides. Futurescientificandtechnologicaladvances,inparticular,maycreateinformationwhich,

misused,wouldcausetremendousharmincluding,potentially,existentialcatastrophe. Ifweaddin

themanylesserhazardsthatcanbecreatedbysuchadvances,forexamplebytechnologiesthat

facilitatecommercialfraudorthatintroduceinsidiousnewchemicalsintothehumanbody,therange

54Seee.g.Petratos2007.

55Kantrowitz1992.

29


30/35

andcomplexityofpotentialinformationhazardsgrowsevengreater. Ifwefurtherexpandourpurview

andconsiderthemanyindirectandreciprocalinfluencesbetween,forinstance,scientificinformation

andeconomicgrowth,andif,moreover,wealsogiveattentiontothenumerousways,cataloguedin

precedingsections,inwhichinformationoutsidetherealmsofscienceandtechnologycancause

harmthenweshallcometoappreciatethatinformationhazardsareubiquitous,potentiallyserious,

andoftennonobvious.

Ananalysisofthepolicyimplicationsofthisresultisbeyondthescopeofthispaper.56 Bywayof

conclusion,though,wemayadumbratetwocontrastingpotentialresponses. Giventhecomplexityof

theissuesinvolved,andtheirentanglementwithmanystrategic,philosophical,andpolitical

considerations,itisnottrivialtoascertainwhichoftheseresponseshasthemosttorecommendit.57

Onepossibleresponse,then,wouldbetotaketoheartthemanifoldwaysinwhichthediscoveryand

disseminationofinformationcanhavenegativeeffects.58Wecouldaccepttheneedtoqualifythe

fawningadmirationandunquestioningcommitmenttothepursuitoftruththatcurrentlyconstitutes

officialpolicyifnotalwaysconsistentpracticeinmostuniversitiesandlearnedbodies. Amottolike

HarvardsVeritas!couldbeviewedasnaveandreckless. Instead,onemightconclude,weoughtto

thinkmorecarefullyandopenmindedlyaboutwhichparticularareasofknowledgedeservetobe

promoted,whichshouldbeletbe,andwhichshouldperhapsevenbeactivelyimpeded.

Sincescholarsareverylikelytobebiasedinfavorofthinkingthattheirownfielddeservestobe

promoted,outsiderswhoarelessprejudicedshouldbebroughtintoparticipateinthesedeliberations.

TheoldEnlightenmentmodelofscientificresearch,whichpicturesscienceasagoosethatlaysgolden

eggsbutonlyifallowedfullautonomyandifshieldedfromexternalsocialcontrol,wouldperhapshave

tobereplacedwithadifferentmodelinwhich,forexample,democraticprocessesandpreferencesare

allowedgreaterinfluenceoverresearchdirectionsandpriorities.

Anotherresponsewouldnotethegreatbenefitsthathistoricallyhavecomefromthepursuitof

knowledgeandenlightenment,andfastenonthedangersinherentinanyattempttocurtailfreeinquiry

ortoyokescientificresearchtosomepreconceivednotionofthesocialgood. Thoseinclinedtogivethis

responseneednotdenythattrueinformationcaninmanyinstancesbeharmfulorhazardous;they

needonlymaintainthatonbalancewearebetteroffasloyalsubjectstothecauseofenlightenment. It

canalsobehopedthatnewinformationtechnologieswillbringaboutavastlymoretransparentsociety,

inwhicheverybody(thewatchmenincluded)areunderconstantsurveillance;andthatthisuniversal

56Ihopetoaddresssomeoftheseissuesinacompanionpaper.

57Wemaybelikelytooverlookatleastonecrucialconsideration;seeBostrom2006.

58OnefairlyrecentandwellknownattempttoarguethisisBillJoysarticleinwhichheadvocatesselective

relinquishmentofresearchincertainfieldswithinartificialintelligence,nanotechnology,andbiotechnology

becauseofdangersheforeseesinthefutureifsuchresearchispursued(Joy2000).

30


31/35

transparencywillpreventtheworstpotentialmisusesofthenewtechnologicalpowersthathumanity

willdevelop.59

Evenifourbestpolicyistoformanunyieldingcommitmenttounlimitedfreedomofthought,virtually

limitlessfreedomofspeech,anextremelywidefreedomofinquiry,weshouldrealizenotonlythatthis

policyhascostsbutthatperhapsthestrongestreasonforadoptingsuchanuncompromisingstancewoulditselfbebasedonaninformationhazard;namely,normhazard:theriskthatpreciousyetfragile

normsoftruthseekingandtruthfulreportingwouldbejeopardizedifwepermittedconvenient

exceptionsinourownadherencetothemoriftheirviolationwereingeneraltooreadilyexcused.

Itissaidthatalittleknowledgeisadangerousthing. Itisanopenquestionwhethermoreknowledgeis

safer. Evenifourbestbetisthatmoreknowledgeisonaverageagoodthing,weshouldrecognizethat

therearenumerouscasesinwhichmoreknowledgemakesthingsworse.60

References1. Akerlof,G.A.1970.TheMarketforLemons:QualityUncertaintyandtheMarketMechanism.

TheQuarterlyJournalofEconomics84(3):488500.

2. Austin,J.L.1962.InJ.O.Urmson,ed.HowtoDoThingswithWords:TheWilliamJamesLecturesDeliveredatHarvardUniversityin1955.Oxford:Clarendon.

3. Berglas,S.andE.E.Jones.1978.Drugchoiceasaselfhandicappingstrategyinresponsetononcontingentsuccess.JournalofPersonalityandSocialPsychology36(4):405417.

4. Bikhchandani,S.,Hirshleifer,D.andWelch,I.1992.ATheoryofFads,Fashion,Custom,andCultural

Change

as

Informational

Cascades.

The

Journal

of

Political

Economy

100

(5):

992.

5. Bostrom,N.andA.Sandberg.2008.WholeBrainEmulation:ARoadmap.Oxford: FutureofHumanityInstitute.

6. Bostrom,N.2006.TechnologicalRevolutions:EthicsandPolicyintheDark.InNanoscale:IssuesandPerspectivesfortheNanoCentury,eds.N.M.D.S.CameronandM.E.Mitchell,129152.

Hoboken,N.J.:Wiley.

7. Bostrom,N.2003.AstronomicalWaste:TheOpportunityCostofDelayedTechnologicalDevelopment.Utilitas15(3):308314.

8. Bostrom,N.2003.EthicalIssuesinAdvancedArtificialIntelligence.Cognitive,EmotiveandEthical

Aspects

of

Decision

Making

in

Humans

and

in

Artificial

Intelligence2:1217.

59Cf.Brin1999.

60ForcommentsanddiscussionsIamgratefultoWalterSinnottArmstrong,AllenBuchanan,TylerCowen,Robin

Hanson,TobyOrd,PythagorasPetratos,RebeccaRoache,AndersSandbergandNickShackel. Iwouldalsoliketo

thankNancyPatelandRachelWoodcockforassistance.

31


32/35

9. Bostrom,N.2002.ExistentialRisks: AnalyzingHumanExtinctionScenariosandRelatedHazards.JournalofEvolutionandTechnology9.

10.Bostrom,N.1998.HowLongbeforeSuperintelligence?InternationalJournalofFutureStudies2.11.Brin,D.1999.TheTransparentSociety:WillTechnologyForceUstoChooseBetweenPrivacyand

Freedom?NewYork:PerseusBooks.

12.Brown,J.D.,K.A.DuttonandK.E.Cook.2001.FromtheTopDown:SelfEsteemandSelfEvaluation.CognitionandEmotion15:615631.

13.Cirincione,J.2008.TheContinuingThreatofNuclearWar.InGlobalCatastrophicRisks,ed.N.BostromandM.Cirkovic,381401.Oxford: OxfordUniversityPress.

14.Drexler,E.1987.Chapter3:11EnginesofDestruction.InEnginesofCreation:TheComingEraofNanotechnology,171190.GardenCity:Anchor.

15.Eliot,T.S.2001.FourQuartets.London:FaberandFaber.16.Fawthrop,T.andH.Jarvis.2005.GettingAwayWithGenocide:CambodiasLongStruggle

AgainsttheKhmerRouge.Sydney:UNSWPress.

17.Freitas,R.2000.SomeLimitstoGlobalEcophagybyBiovorousNanoreplicators,withPublicPolicyRecommendations.ForesightInstitute.http://www.foresight.org/nano/Ecophagy.html.

18.Glover,J.2001.Humanity:AMoralHistoryoftheTwentiethCentury.NewHaven:YaleUniversityPress.

19.Goffman,E.1959.ThePresentationofSelfinEverydayLife.GardenCity:DoubledayAnchor.20.Gubrud,M.1997.NanotechnologyandInternationalSecurity.ForesightInstitute.

http://www.foresight.org/Conferences/MNT05/Papers/Gubrud/index.html.

21.Halliday,T.J.andS.Kwak.2007.IdentifyingEndogenousPeerEffectsintheSpreadofObesity,WorkingPaper727.UniversityofHawaiiatManoa,DepartmentofEconomics.

http://www.economics.hawaii.edu/research/workingpapers/WP_0727.pdf.

22.Hirschman,A.O.1991.TheRhetoricofReaction:Perversity,Futility,Jeopardy.Cambridge,M.A.:BelknapPress.

23.Hobden,K.L.1997.BehaviouralVersusClaimedSelfHandicapping:UnderlyingMotivationsandAttributionsFollowingFailure.Toronto:UniversityofTorontodoctoralthesis.

24.Hurka,T.1993.Perfectionism.Oxford:ClarendonPress.25. JonasK.1992.Modellingandsuicide:atestoftheWerthereffect.BrJSocPsychol.,Dec31(Pt

4):295306.

26.Joy,B.2000.WhytheFutureDoesn'tNeedUs.Wired8.04,April:238245,248263.27.Kahneman,D.,Slovic,P.andA.Tversky,eds.1982.JudgmentunderUncertainty:Heuristicsand

Biases.NewYork:CambridgeUniversityPress.

32


33/35

28.Kantrowitz,A.1992.TheWeaponofOpenness.InNanotechnologyResearchandPerspectives,eds.B.C.CrandallandJ.Lewis,303311.Cambridge,M.A.:MITPress.

29.Kavka,G.S.1990.SomeSocialBenefitsofUncertainty.MidwestStudiesinPhilosophy15(1):311326.

30.Kennedy,R.1968.13Days.London:Macmillan.31.Kurzweil,R.2006.TheSingularityIsNear:WhenHumansTranscendBiology.NewYork:Penguin.32.Leslie,J.1996.TheEndoftheWorld:TheScienceandEthicsofHumanExtinction.London:

Routledge.

33.Levack,B.P.1987.TheWitchHuntinEarlyModernEurope.London:Longman.34.Markovitch,S.andP.D.Scott.1988.KnowledgeConsideredHarmful,ResearchPaper#030788.

AnnArbor,Michigan:CenterforMachineIntelligence.

35.Matheny,J.G.2007.ReducingtheRiskofHumanExtinction.RiskAnalysis27(5):13351344.36.Meade,E.E.andD.Stasavage.2008.PublicityofDebateandtheIncentivetoDissent:Evidence

fromtheUsFederalReserve.EconomicJournal118(528):695717.

37.Merton,R.K.1968.TheMatthewEffectinScience.Science159(3810):5663.38.Montefiore,S.S.2005.Stalin:TheCourtoftheRedTsar.NewYork:Vintage.39.Moravec,H.2000.Robot:MereMachinetoTranscendentMind.NewYork: OxfordUniversity

Press.

40.Nelson,R.andG.Winter.1990.AnEvolutionaryTheoryofEconomicChange.Cambridge,M.A.:Harvard

University

Press.

41.Nietzsche,F.1984.VomNutzenundNachteilderHistoriefrdasLeben.Zurich:DiogenesVerlag.42.Nietzsche,F.2007.OntheUseandAbuseofHistoryforLife.SiouxFalls,S.D.:NuVision

Publications.

43.Nouri,A.andC.F.Chyba.2008.BiotechnologyandBiosecurity.InGlobalCatastrophicRisks,eds.N.BostromandM.Cirkovic,450480.Oxford: OxfordUniversityPress.

44.Oppenheimer,R.1947.PhysicsintheContemporaryWorld,LectureatM.I.T.,November25.45.Petratos,P.2007.Weather,InformationSecurity,andMarkets.IEEESecurityandPrivacy5(6):

5457.

46.Phillips,D.1982.TheImpactofFictionalTelevisionStoriesonU.S.AdultFatalities:NewEvidenceontheEffectoftheMassMediaonViolence".TheAmericanJournalofSociology87(6):1340

59.

47.Porter,M.E.2004.CompetitiveAdvantage.NewYork:FreePress.48.Posner,R.2005.Catastrophe:RiskandResponse.NewYork:OxfordUniversityPress.

33


34/35

49.Putnam,H.1979.Thepla

Information Hazards

Documents

Transcript of Information Hazards