1

Echo Chambers and Epistemic Bubbles

C. Thi Nguyen (Thisisthepenultimatedraft.Pleasecitethepublishedversion,publishedinEpisteme:https://doi.org/10.1017/epi.2018.32. Anearlierandmuch-shortenedversionofthisessay,writtenforageneralaudience,originallyappearedinAeonMagazineasEscapetheEchoChamber.)

AbstractDiscussionofthephenomenaofpost-truthandfakenewsoftenimplicatestheclosedepistemicnetworksofsocialmedia.Therecentconversationhas,however,blurredtwodistinctsocialepistemicphenomena.Anepistemicbubbleisasocialepistemicstructureinwhichotherrelevantvoiceshavebeenleftout,perhapsaccidentally.Anechochamberisasocialepistemicstructurefromwhichotherrelevantvoiceshavebeenactivelyexcludedanddiscredited.Membersofepistemicbubbleslackexposuretorelevantinformationandarguments.Membersofechochambers,ontheotherhand,havebeenbroughttosystematicallydistrustalloutsidesources.Inepistemicbubbles,othervoicesarenotheard;inechochambers,othervoicesareactivelyundermined.Itiscrucialtokeepthesephenomenadistinct.First,echochamberscanexplainthepost-truthphenomenainawaythatepistemicbubblescannot.Second,eachtypeofstructuresrequiresadistinctintervention.Mereexposuretoevidencecanshatteranepistemicbubble,butmayactuallyreinforceanechochamber.Finally,echochambersaremuchhardertoescape.Onceintheirgrip,anagentmayactwithepistemicvirtue,butsocialcontextwillpervertthoseactions.Escapefromanechochambermayrequirearadicalrebootingofone’sbeliefsystem.

Keywords:Epistemology,socialepistemology,echochambers,testimony,philosophyoftestimony,trust,expertise,filterbubbles,politicalpolarization,post-truth,socialmedia.

Somethingseemstohavegonewrongwiththeflowofinformation.Recentanalysesof

FacebookfeedsandTwitternetworksrevealthattheiruser’sinformationalinputisbeing

radicallyfiltered,thatusersarebeingexposedlargelytoargumentsandviewswithwhich

theyalreadyagree(An,QuerciaandCrowcroft2014;Saez-Trumper,CastilloandLalmas

2013).What’smore,wholesegmentsofthepopulationhavedismissedthemainstreammedia

2

ascorruptanduntrustworthy.Manyofushavestartedtowonder:arewetrappedinecho

chambersofourownmaking?1

Therecentconversation,however,hasblurredtwodistinct,butinterrelated,social

epistemicphenomena,whichIwillcallepistemicbubblesandechochambers.Bothare

problematicsocialstructuresthatleadtheirmembersastray.Bothreinforceideological

separation.Buttheyaredifferentintheirorigins,mechanismsforoperation,andavenuesfor

treatment.Botharestructuresofexclusion–butepistemicbubblesexcludethrough

omission,whileechochambersexcludebymanipulatingtrustandcredence.However,the

modernconversationoftenfailstodistinguishbetweenthem.

Loosely,anepistemicbubbleisasocialepistemicstructureinwhichsomerelevantvoices

havebeenexcludedthroughomission.Epistemicbubblescanformwithnoillintent,through

ordinaryprocessesofsocialselectionandcommunityformation.Weseektostayintouch

withourfriends,whoalsotendtohavesimilarpoliticalviews.Butwhenwealsousethose

samesocialnetworksassourcesofnews,thenweimposeonourselvesanarrowedandself-

reinforcingepistemicfilter,whichleavesoutcontraryviewsandillegitimatelyinflatesour

epistemicself-confidence.

Anechochamber,ontheotherhand,isasocialepistemicstructureinwhichother

relevantvoiceshavebeenactivelydiscredited.MyanalysisbuildsonKathleenHallJamieson

andFrankCapella’swork,withsomephilosophicalaugmentation.AccordingtoJamiesonand

Capella,anechochamber’smemberssharebeliefswhichincludereasonstodistrustthose

outsidetheechochamber.Echochambersworkbysystematicallyisolatingtheirmembers

1 An earlier version of this article appeared as ‘Escape the Echo Chamber’ in Aeon Magazine (Nguyen 2018b). That earlier version was written with a general audience in mind. The present article contains fuller versions of the core definitions and arguments, and some additional material, especially on the topics of responsibility and escape.

3

fromalloutsideepistemicsources(JamiesonandCappella2008,163-236).Thismechanism

bearsastrikingresemblancetosomeaccountsofcultindoctrination.Bydiscrediting

outsiders,echochambersleavetheirmembersoverlydependentonapprovedinsidesources

forinformation.Inepistemicbubbles,othervoicesaremerelynotheard;inechochambers,

othervoicesareactivelyundermined.(Thisisaconceptualdistinction;acommunitycan

practicebothformsofexclusiontovaryingdegrees.)

Thecontemporarydiscussionhasbeenmostlyfocusedonthephenomenonofepistemic

bubbles.CassSunstein’sfamousdiscussionsofgrouppolarization,extremism,andthe

Internetlargelyfocusonmattersofconstrictedinformationflowandomittedviewpoints

(Sunstein2001,2009b,2009a).EliPariser’sTheFilterBubblefocusesentirelyonfiltration

effectsfrompersonalizationtechnology,asinGooglesearches,andself-selected

informationalnetworks,asinFacebook(Pariser2011).Popularconversationhastendedto

followPariser’sfocusontechnologically-mediatedfiltration.Theterm“echochamber”has,in

recentusage,beenreducedtoasynonymforsuchbubblesandtheirconstrictedinformation

flow.Whenthespecificallytrust-orientedmanipulationsofechochambersarediscussed,

theyareusuallylumpedinwithepistemicbubblesaspartofoneunifiedphenomenon.

Thisisamistake;itisvitaltodistinguishbetweenthesetwophenomena.Anepistemic

bubbleisanepistemicstructureemergingfromtheinformationalarchitectureof

communities,socialnetworks,media,andothersourcesofinformationandargument.Itisan

impairedinformationaltopology—astructurewithpoorconnectivity.Anechochamber,on

theotherhand,isanepistemicstructurecreatedthroughthemanipulationoftrust;itcan

existwithinahealthyinformationaltopologybyaddingasuperstructureofdiscreditand

authority.Ihopetoshow,contratherecentfocusonepistemicbubbles,thatechochambers

poseasignificantanddistinctivethreat–perhapsevenamoredangerousone–thatrequires

4

averydifferentmodeofrepair.Furthermore,echochamberscanexplainwhatepistemic

bubblescannot:theapparentresistancetoclearevidencefoundinsomegroups,forexample,

climatechangedeniersandanti-vaccinationgroups.

Itmaybetemptingtotreatmembersofechochambersasmeresheep,andaccusethem

ofproblematicacquiescencetoepistemicauthority.Butthataccusationreliesonan

unreasonableexpectationforradicalepistemicautonomy.Contemporaryepistemology,

especiallysocialepistemology,hastaughtusthattrustinothersisnecessaryand

ineradicable.Weare,asJohnHardwigsays,irredeemablyepistemicallydependentoneach

other(Hardwig1985,1991).Echochamberspreyonourepistemicinterdependence.Thus,in

somecircumstances,echochambermembersdonothavefullepistemicresponsibilityfor

theirbeliefs.Onceoneistrappedinanechochamber,onemightfollowgoodepistemic

practicesandstillbeledfurtherastray.Andsomepeoplecanbetrappedinechochambers

becauseofcircumstancesbeyondtheircontrol—forexample,theycanberaisedinthem.

Whichleadstothemostimportantquestions:howcanonetellifoneisinanechochamber?

Andhowcanoneescape?Iwillarguethat,forthosetrappedwithinanechochamber,

prospectsfordetectionarepoorandtheescapepathdaunting.Detectingandescapingfrom

echochamberswillrequirearadicalrestructuringofamember’srelationshiptotheir

epistemicpast,whichmaybemorethanwecanreasonablyexpectofoneanother.

EpistemicBubbles

Let’sstartwithasimplepictureofhowmanyofusconductourepistemiclivesrightnow.

IgetmuchofmynewsviaFacebook.IhaveselectedmostofmyFacebookfriendsforsocial

reasons;theyaremyfriendsandcolleagues.Asignificantconduitformylearningabout

5

eventsintheworldisbypeoplere-postingarticlesthattheyhavefoundnewsworthyor

interesting.WhenIgooutsideofFacebook,Iusuallyturntosourceswhich,byandlarge,are

affiliatedwithmyownpoliticalbeliefsandintellectualculture.

Thisprocessimposesafilteronmyuptakeofinformation.Selectionandexclusionarenot

badinandofthemselves—theworldisoverstuffedwithsupposedsourcesofinformation,

manyofthemterrible.Thebettermyfilter,themoreitwillfocusmyattentiononrelevant,

useful,andreliableinformation.Butthebarefactthateachindividualmemberofthesystem

isthemselvesreliablewillnotguaranteeanybroadnessorcompletenessofcoverage.

Suppose,forexample,thatmysocialnetworkwascomposedentirelyofintelligent,reliable

professorsofaestheticswhoseinterestswerelargelyfocusedonnewdevelopmentsinopera,

ballet,andavant-gardecontemporaryart.Throughthissystem,Imightlearnaboutallthe

excitingnewdevelopmentsintheNewYorkArtscene,butentirelymiss,say,relevant

developmentsinrap,orthefactthatmycountrywasslowlyslidingintofascism.Thesystem

lackswhatGoldbergcallscoverage-reliability—thecompletenessofrelevanttestimonyfrom

acrossone’swholeepistemiccommunity(Goldberg2011,93-4).Badcoveragecannotonly

leaveoutrelevantfactsandevidence;itcanalsofailtobringrelevantargumentstoour

attention.Thus,badcoveragecanstarveusofadequateexposuretorelevantarguments.

Noticethatbadcoverageisanepistemicflawofepistemicsystemsandnetworks,notof

individuals.

Icannowspecifymyuseof“epistemicbubble”withgreaterprecision.Anepistemic

bubbleisasocialepistemicstructurewhichhasinadequatecoveragethroughaprocessof

exclusionbyomission.Epistemicbubblesformbyleavingoutrelevantepistemicsources,

ratherthanactivelydiscreditingthem.Thereareattwoprimaryforcesencouragingthis

omission.First,thereisanepistemicagent’sowntendencytoseeklike-mindedsources.This

6

phenomenonissometimescalled“selectiveexposure”bysocialscientists(Nelsonand

Webster2017).Inmanycontemporarycases,suchasFacebook,theprocessofomissioncan

occurinadvertently.ItypicallyputpeopleinmyFacebookfeedforsocialreasons—becauseI

likethemorIfindthemfunny.Butsocialselectiondoesnotguaranteegoodcoverage

reliability;infact,thetypicalbasesofsocialselectionareinimicaltogoodcoverage

reliability.2Weusuallylikepeoplewhoaresimilartous,andsuchsimilaritymakescoverage

gapsmorelikely.Friendsmakeforgoodparties,butpoorinformationnetworks.Wenow

haveastraightforwardaccountofonewayinwhichepistemicbubblescanform.Wecan

buildastructureforonesetofpurposes—maintainingsocialrelations,finding—andthen

proceedtouseitforanentirelydifferentpurpose,forwhichitfunctionsbadly:information

gathering.

Second,therearetheprocessesbywhichanepistemicagent’sinformationallandscapeis

modifiedbyotheragents.Thismightinclude,say,systematiccensorshipormediacontrolby

thestateorotheractors.Themostworrisomeoftheseexternalforces,atthemoment,seems

tobethealgorithmicpersonalfilteringofonlineexperiences(Pariser2011;Watson2015).

Internetsearchengines,forexample,willtrackpersonalinformationforeachparticularuser,

andadapttheirsearchresultstosuiteachuser’sinterest.Certainly,newspapersandother

traditionalmediatechnologiesdoplaceexternalfiltersontheirreaders,butthemodern

instantiationisparticularlypowerfulandtroubling.AsBoazMillerandIsaacRecordargue,

Internettechnologiescreatehyper-individualized,secretfilters.Thesecrecyisparticularly

threatening.Manyusersdonotknowabouttheexistenceofalgorithmicpersonalfiltering.

Evenamongstthosethatdo,mostdonothaveaccesstotheparticularitiesofthealgorithms

2 For an overview of empirical research on personal similarity and polarization, see (Sunstein 2009a, 83-5). Curiously, Sunstein notes the group polarization literature has thought relatively little about the impact of personal similarity.

7

doingthefiltering;thus,theveryopacityoftheprocessmakesitharderforauserto

successfullyevaluateandepistemicallycompensateforsuchfiltering(MillerandRecord

2013).Thus,mostuserssignificantlyunderestimatethedegreetowhichtheirexposureto

information,viasearchresults,hasalreadybeentailoredtopresentsearchresultstowhich

theuserwillalreadybeamenable.

Boththeagent-drivenprocessofselectiveexposure,andtheexternalitiesofalgorithmic

filtering,contributetothecreationofepistemicbubbles.Iintroducetheterm“epistemic

bubble”heretoindicateabroadersetofphenomena.Pariserintroducedtheterm“filter

bubble”toreferspecificallytotechnologicallymediatedfiltering,especiallyviaalgorithmic

matching.Epistemicbubblesarethosestructureswhichomitrelevantvoicesbyanymeans,

technologicalorotherwise.Epistemicbubblesincludefilterbubbles,butalsonon-

technologicalselectionprocesses,suchasphysicallysortingoneselfintoneighborhoodsof

like-mindedpeople(Bishop2009).

TheaccountI’vegivenofepistemicbubblesfocusesonthewaytheyomitrelevant

information,butthatomissioncanalsothreatenuswithbootstrappedcorroboration.Usersof

socialnetworksandpersonalizedsearchtechnologieswillencounteragreementmore

frequentlyandsobetemptedtoover-inflatetheirepistemicself-confidence.Thisdanger

threatensbecause,ingeneral,corroborationisoftenaverygoodreasontoincreaseone’s

confidenceintherelevantbeliefs(Nguyen2010,2018a).Butcorroborationoughttoonly

haveweightifitaddssomethingepistemically,ratherthanbeingamerecopy.Toborrowan

examplefromWittgenstein:imaginelookingthroughastackofidenticalnewspapersand

treatingeachnextnewspaperheadlinesayingpasareasontoincreaseyourbeliefthatp

(Wittgenstein2010,100).Thisisclearlyamistake;thefactthatanewspaperclaimsthatp

hassomeepistemicweight,butthenumberofcopiesofthatnewspaperoneencounters

8

oughtnotaddanyextraweight.Similarly,imaginespeakingtoabunchofacolytesofGuru

Jane,whorepeatanythingthatGuruJanesayswithoutanyfurtherthought.Thefactthatall

theseacolytesrepeatGuruJane’stestimonyshouldaddnoextraweight.Solongasthe

disciplinesrepeatanythingGuruJanesays—solongastheyaremereconduitsfor

information,ratherthansourcesofinformation—theyaresimplyanothersortofcopy.

Butcopyingisn’ttheonlyroutetoaproblematicformofnon-independence.SupposeI

believethatthePaleodietisthebestdiet.IproceedtoassembleabodyofpeerswhoItrust

preciselybecausetheyalsobelievethatPaleoisthebestdiet.Inthatcase,theexistenceof

perfectagreementonPaleo’samazingnessthroughoutthatgroupoughttocountforfarless

thanitmightforothergroupsthatIhadnotassembledonthatbasis.Evenifallthegroup

membersarrivedattheirbeliefsindependently,theiragreementisalreadyguaranteedbymy

selectionprinciple.TothedegreethatIhavepre-selectedthemembersinmyepistemic

networkfromagreementwithsomesetofbeliefsofmine,thentheiragreementwiththatset

ofbeliefsandanyotherbeliefsthatitentailsoughttobeepistemicallydiscounted.3Ifwefail

tosodiscount,weareignoringapernicioushiddencircularityinourcorroborativeprocess.It

iseasytoforgettodiscountbecausethebootstraphereisobscuredbytimeandinterface.But

wemustactivelyadjustfortheincreasedlikelihoodofagreementinsideourbubbles,orwe

willunwarrantedlybootstrapourconfidencelevels.4

3 This is a relative of the problem, from statistics, of failing to identify dependent variables.

4 I am not claiming, as Alvin Goldman does, that all cases of non-independent testimony should be discounted Goldman 2001, 98-104). Goldman’s analysis, and the principle of non-independence, have been significantly challenged (Coady 2006; Lackey 2013). David Coady and Jennifer Lackey have demonstrated that we can trust the weight of non-independent agreement when we have good reason to think that the non-independent agreers had good epistemic meta-reasons for agreeing. For example, suppose that all scientists agreed that climate change was coming. Their agreement is non-independent — the majority of these scientists have not analyzed the data for themselves, but trust the expert specialists in climate change. But still, the weight of numbers matters here because the trusting scientists have good epistemic reasons for picking who to trust. But in the case I’ve described, the weight of numbers here does not emanate from a good epistemic process. In other words, my claim doesn’t depend on the claim that all forms of non-independence are problematic, only that some are, for the reasons I adduce.

9

Tosummarize:anepistemicbubbleisanepistemicnetworkthathasinadequatecoverage

throughaprocessofexclusionbyomission.Thatomissionneednotbemaliciousoreven

intentional,butmembersofthatcommunitywillnotreceivealltherelevantevidence,norbe

exposedtoabalancedsetofarguments.

EchoChambers

Luckilyforus,epistemicbubblesarerelativelyfragile.Relevantsourceshavesimplybeen

leftout;theyhavenotbeendiscredited.Itispossibletopopanepistemicbubblebyexposing

amembertorelevantinformationorargumentsthattheyhavemissed.Echochambers,on

theotherhand,aresignificantlymorerobust.

MyanalysisherecombinesempiricalworkandanalysisfromJamiesonandCappellaon

thenatureofright-wingecho-chamberswithrecentinsightsfromsocialepistemology.

JamiesonandCappellastudiedechochambersbuiltaroundparticularcharismatic

personalities—RushLimbaughandthenewsteamofFoxNews,andcertainothermembers

ofconservativetalkradio.TheirdataandanalysissuggestthatLimbaughusesmethodsto

activelyisolatehiscommunityoffollowersfromotherepistemicsources.Limbaugh’s

consistentattacksonthe“mainstreammedia”servetodiscreditallpotentialsourcesof

knowledgeortestimonybesidesLimbaughandaselectinner-cadreofotherapproved

sources(JamiesonandCappella2008,140-76).Limbaughalsodevelopswhattheycalla

privatelanguage,fullofalternatemeaningsforfamiliartermsandnewjargon(forexample,

“SJWs”),inordertoexaggeratetheinsularityandseparatenessofthein-group.Finally,

Limbaughprovidescounter-explanationsofallcontraryviews,intendednotonlytoattack

eachparticularview,butalsotounderminethegeneraltrustworthinessandintegrityof

10

anybodyexpressingacontraryview.Theresultingworld-viewisoneofhighlyopposed

forces;onceonehassubscribedtoLimbaugh’sview,onehasreasontothinkthatanybody

whodoesnotalsosubscribeisactivelyopposedtothesideofright,andtherebymorally

unsoundandsogenerallyuntrustworthy(177-90).JamiesonandCappellasuggestthatthis

makesafollowerdependentonasinglesourceorgroupofsources,andmakesthemhighly

resistanttoanyoutsidesources.Theyofferthefollowingdefinitionofanechochamber:an

echochamberisaboundedandenclosedgroupthatmagnifiestheinternalvoicesand

insulatesthemfromrebuttal(76).

Iwillusetheterm“echochamber”herefollowingtheiranalysis,butIadaptthedefinition

slightlyforphilosophicaluse.Iuse“echochamber”tomeananepistemiccommunitywhich

createsasignificantdisparityintrustbetweenmembersandnon-members.Thisdisparityis

createdbyexcludingnon-membersthroughepistemicdiscrediting,whilesimultaneously

amplifyinginsidermembers’epistemiccredential.Finally,echochambersaresuchthatin

whichgeneralagreementwithsomecoresetofbeliefsisapre-requisiteformembership,where

thosecorebeliefsincludebeliefsthatsupportthatdisparityintrust.

By“epistemicdiscrediting”,Imeanthatnon-membersarenotsimplyomittedornot

heard,butareactivelyassignedsomeepistemicdemerit,suchasunreliability,epistemic

maliciousness,ordishonesty.By“amplifyingepistemiccredentials”,Imeanthatmembersare

assignedveryhighlevelsoftrust.Ofcourse,thesetwoprocessescanfeedbackintoone

another.Solongasanechochamber’strustedinsiderscontinuetoclaimthatoutsidersare

untrustworthy,thentheinnertrustwillreinforcetheoutwarddistrust.Andsolongas

outsidersarelargelydistrusted,thentheinsiderswillbeinsulatedfromvariousformsof

counter-evidenceandrebuff,thusincreasingtheirrelativecredence.Onceasufficient

disparityincredencebetweeninsidersandoutsidershasbeenestablished,solongastrusted

11

insiderscontinuetoholdandespouseepistemicallydismissivebeliefstowardsoutsiders,

thentheechochambers’beliefssystemwillbeextremelydifficulttodislodge.

Comparethisprocessofcredencemanipulationtotheprocessofomissionfoundin

epistemicbubbles.Inonestandardscenario,Iaddothersastrustedmembersofmy

epistemicnetworkbasedonagreement.Iamthenlesslikelytoencounteranoutsidevoice—

butwhenIdoactuallyhavesuchanencounterwithanoutsider,Ihavenobackgroundreason

todismissthem.Bubblesrestrictaccesstooutsiders,butdon’tnecessarilychangetheir

credibility.Echochambers,ontheotherhand,workbyofferingapre-emptivediscredit

towardsanyoutsidesources.5

Theresultisaratherstrikingparalleltothetechniquesofisolationtypicallypracticedin

cultindoctrination.Thestandardtechniquesofcultindoctrination,byatraditionalaccount,

aretheaggressiveemotionalisolationofcultmembersfromallnon-cultmembers,which

amplifiesindoctrinatedmember’sdependencyonthecult(Singer1979;Langone1994;

Lifton1991).6Newcultmembersarebroughttodistrustallnon-cultmembers,which

providesanepistemicbufferagainstanyattemptstoextracttheindoctrinatedpersonfrom

thecult.Thisisnothinglikehowepistemicbubbleswork.Epistemicbubblesmerelyleave

5 Note that the kind of echo chambers here are different from those I described in (Nguyen 2018a). There, I explored what I called “personal” echo chambers, which can be generated through entirely good-intentioned individual action under very special epistemic conditions, which I called a “cognitive island”. A cognitive island is any cognitive domain in which there is no possible empirical method to check whether a purported expert is really an expert. For example, the moral domain and the aesthetic domain might be cognitive islands. What I’ve discussed in this paper is something different – what I’ve called “social” echo chambers. They are malicious and essentially social structures which can exist in any empirical domain, not just on a cognitive island. I will offer a synthesis of these two conceptions of echo chambers, with an eye towards elucidating the similarities in their underlying structures, in a future work. 6 Note that this view of cult indoctrination is standard among mental health professionals and social workers, but has been resisted by some theorists, especially from the fields of religious studies and sociology, as a method of de-legitimizing and repressing minority religions (Robbins and Anthony 1982). For a good overview of the debate, see (Coates 2010). For the purposes of my argument, it does not matter if cults actually work this way; what is interesting is that echo chambers work by the mechanism that is, under a traditional account, attributed to cults.

12

theirmembersignorant,butignorancecanbefixedwithsimpleexposure.Thefunctionofan

echochamber,ontheotherhand,istocredentiallyisolateitsmembersbyamanipulationof

trust.Bythis,Imeanthatmembersarenotjustcutoff,butareactivelyalienatedfromanyof

theusualsourcesofcontraryargument,consideration,orevidence.Membershavebeen

preparedtodiscreditanddistrustanyoutsidesources;thus,mereexposuretorelevant

outsideinformationwillhavenoeffect.

Infact,echochamberscanavailthemselvesofanotherepistemicprotectivemechanism:

theycancontainwhatI’llcalladisagreement-reinforcementmechanism.Memberscanbe

broughttoholdasetofbeliefssuchthattheexistenceandexpressionofcontrarybeliefs

reinforcestheoriginalsetofbeliefsandthediscreditingstory.Atoyexample:supposeIama

cultleader,andIhavetaughtmyfollowerstobelievethateveryhumanexceptthemembers

ofourgrouphasbeeninfestedandmind-controlledbyalienghostsfromMars.Ialsoteach

myfollowersthatthesealienghostsfromMarshateourgroupforknowingthetruth,andso

willconstantlyseektoundermineourknowledgeoftheirexistencethroughmechanismslike

callingusa‘cult’andcallinguslunatics.EndreBegbyhasofferedacarefulanalysisofthis

particularsortofdisagreement-reinforcementmechanism,whichhecalls“evidential

preemption.”SupposethatItellmyfollowerstoexpectoutsiderstofalselyclaimthatthere

arenoghostsfromMars.Whenmyfollowersdoconfrontsuchcontraryclaimsfrom

outsiders,thosecontraryclaimsareexactlywhattheyexpectedtohear.Thus,newcontrary

testimonyisneutralized,becauseitwaspredictedbypastbeliefs.This,saysBegby,functions

asakindofepistemicinoculation.Thereisalsoasecondaryeffect.Whenmyfollowershear

exactlywhatIpredicted,thenmyclaimshavebeenverified,andsomyfollowerswillhave

somereasontoincreasetheirtrustinme.Thus,theechochamber’sbeliefsystemnotonly

neutralizestheepistemicimpactofexposuretooutsiderswithcontrarybeliefs;theexistence

13

ofthosecontrarybeliefswillactivelycorroboratethepre-emptorandsoincreasethe

credenceleveloftheentireechochamber(Begby2017).Thiscreatesafeedbackmechanism

withintheechochamber—inmakingunderminingpredictionsaboutcontrarytestimony,

insideauthoritiesnotonlydiscreditthatcontrarytestimony,butincreasetheir

trustworthinessforfuturepredictions.

Oncesuchasystemofbeliefsissetup,itcanbeverydifficulttodislodge—itisself-

reinforcing,bounded,andbuilttodiscountanycontraryinput.Infact,thoughmydefinitionof

echochambersisconceptuallyseparablefromsuchadisagreement-reinforcement

mechanism,everyplausiblereal-worldcandidateforanechochamberI’veeverencountered

includessomeversionofadisagreement-reinforcementmechanism.Foradepressingreal-

worldexample,considerPizzagate.Pizzagateisaconspiracytheorythatboiledoutofaright-

wingonlineforumonReddit,whichincludedbeliefsthatCometPingPong,apizzarestaurant,

wasthesiteofachildsextraffickingringownedbyaliberalconspiracyinvolvingHillary

ClintonandBarackObama.Eventually,EdgarWelch,amemberofthatforum,forcibly

enteredthepizzaparlorarmedwithanassaultrifletoinvestigate;whenhesatisfiedhimself

thattherestaurantcontainednochildslaves,hegavehimselfuptothepolice.Theonline

forum,however,didnottakethisascontraryevidence.Instead,theyleanedontheirbelief

thattheliberalconspiracyhadtotalcontrolofthemainstreammedia,andwaswillingto

stagefakeeventstodiscredittheright-wing.TheforumtookWelch’sclaimsthattherewasno

sextraffickingringasevidencethatWelchwasapaidactor,andthusasfurtherconfirmation

oftheexistenceofapowerfulcabalofliberalchildsextraffickers(Mengus2016;Vogtand

Goldman2016).

Conspiracytheoriesfunctionhereinafascinatinginversiontocorroborative

bootstrapping.Incorroborativebootstrapping,themistakeistotreatproblematically

14

dependentlyselectedinsidersasiftheywereindependent,andthusoverweighttheir

testimony.Whenanechochamberusesaconspiracytheoryinthismanner,theyare

attributingaproblematicformofnon-independencetooutsiderswhoareactually

independent,andtherebyunderweightingoutsidetestimony.Anechochamberhereworks

bydiscreditingtheapparentindependenceof,say,differentclimatechangescientistsby

claimingthatalltheirvarioustestimoniesareproblematicallyderivedfromasinglesource.

Incidentally,Iamnotclaimingherethatconspiracytheoriesarealwaysornecessarily

incorrectortheproductofepistemicvice.Asothershaveargued,believinginconspiracy

theoriesisn’tbadperse,becausesomeconspiracytheoriesaretrue(Coady2012,110-137;

Dentith2017).Butthefactthatconspiracytheoriescanfunctiontoreinforcetheboundaries

ofechochambers—thoughtheydonotnecessarilydoso—mightexplainpartofconspiracy

theories’badrap.Becauseoftheireffectivenessinsettingupdisagreement-reinforcement

mechanisms,conspiracytheoriesareoftenconscriptedasapowerfultoolinthebad

epistemicbehaviorofcertaingroups.

Itisimportanttonotethattheepistemicmechanismsbywhichechochamberswork,

thoughproblematic,arenotsuigeneris.Theyareperversionsofnatural,useful,andnecessary

attitudesofindividualandinstitutionaltrust.Theproblemisn’tthatwetrustanddistrust

groupsandinstitutions.Infact,wemustdoso.EljiahMillgramcallsittheproblemofhyper-

specialization.Humanknowledgehassplinteredintoavastsetofspecializedfieldsthat

dependoneachother.Noonehumancanmanagethatinformation;weareforcedtotrust

eachother(Millgram2015,27-44).7Noneofusisinapositiontoreliablyidentifyanexpertin

7 Though this paper relies on insights from modern work in the epistemology of testimony, I have tried to rely only on uncontroversial claims from that field, and not on the technical details of any particular view. In particular, I have attempted to construct my analysis so as to be independent of the debate on whether or not testimony is a basic source of knowledge. I have also attempted to make the paper compatible with the major accounts of trust.

15

mostspecialistfieldsoutsideofourown.Iam,onmyown,helplesstoevaluatethevirtuesof

antibioticsortheexpertiseofaparticulardoctororsurgeon.Iam,instead,reliantonavast

networkofinstitutionallicensingpracticesinordertochoosemyhealthcaresources—

includingjournalpeerreview,medicalboardexams,universityhiringpractices,andthelike

(Nguyen2017a).Often,ItrustviawhatPhilipKitchercallsindirectcalibration—Itrust

mechanicalengineersbecausetheymakethingsthatwork,butIknowthatmechanical

engineerstrustappliedphysicists,andIknowthatappliedphysiciststrusttheoretical

physicists,soIacquiretrustthroughalongchainoffield-widelinks(Kitcher1993,320-3).I

evenuselitmustests:thefactthatanypersonorgroupisinfavorof,say,sexualorientation

conversiontherapyisenoughformetodiscreditthemonanysocialormoraltopics.Wemust

resorttosuchtacticsinordertonavigatethehyper-specializedworld(Nguyenforthcoming).

Echochambersfunctionparasiticallyonthesepracticesbyapplyingdiscreditswithout

regardfortheactualepistemicworthofthediscreditedinstitutionsorindividuals.The

discreditisinsteadappliedstrategicallyanddefensively,towardsalloutsiderssolelyonthe

basisoftheirbeingoutsiders.Oncethediscreditingbeliefsareinplace,theensuingbeliefs

andactiontheechochambers’membersaresurprisinglyclosetorational.Infact,wecan

easilyimaginealternativescenariosinwhichaverysimilarsetofbeliefswereappropriate

andveristic.IfIwasananti-NaziinGermanyduringtheriseoftheNaziparty,Iwoulddowell

tomaintainthebeliefsthatthemostpeoplewerecorrupt,untrustworthy,andoutto

maliciouslyunderminemyowntruebeliefs.Butifsuchbeliefsbecomeimplantedinan

inappropriatecontext,theycanleadtheirbelieversentirelyastray.

Epistemicbubblescaneasilyformaccidentally.Butthemostplausibleexplanationforthe

particularfeaturesofechochambersissomethingmoremalicious.Echochambersare

excellenttoolstomaintain,reinforceandexpandpowerthroughepistemiccontrol.Thus,itis

16

likely(thoughnotnecessary)thatechochambersaresetupintentionally,oratleast

maintainedforthisfunctionality.Myaccountthusbearssomeresemblancetosomeworkon

testimonialinjusticeandtheepistemologyofignorance,butitisimportantlydistinct.

MirandaFrickerhasarguedforakindoftestimonialinjustice,basedonagapbetweenactual

reliabilityandperceivedcredibility.Forexample,saysFricker,beingwhiteandbeingmale

arebothbonusestocredibility.Sincecredibilityisasourceofpower,anybodywith

credibilitywillseektoincreaseit,usingthatverycredibility.Thus,saysFricker,credibility

gapscanbeturnedintoepistemictoolsofsocialoppression(Fricker2011).Similarly,Charles

Millsarguesthatthereisanactivepracticeofignoranceamongmembersofoppressive

groups,suchaswhiteAmericans.Itistothebenefitofthoseinpowertoactivelyignoremany

aspectsoftheexistenceofoppressedgroups(Mills2007;Alcoff2007,47-57).Myaccountis

compatiblewith,butindependentfrom,Fricker’sandMills’accounts.Echochamberscanand

surelyareusedtomaintainsocialoppressionthroughenhancingcredibilitygapsand

supportingpracticesofactiveignorance.Thesystematicmistrustofanechochambersisa

powerfultoolforperpetuatingepistemicinjusticeandactiveignorance.However,the

conceptofanechochamberdoesnotrequirethattheybedeployedonlyinpoliticalcontexts,

nordoesitrequirethattheyonlybedeployedonlyintheserviceofoppression.Echo

chamberscouldpotentiallyexistamongtheoppressed,andsurelyexistinapoliticalcontexts.

IbelieveIhavewitnessedechochambersformingaroundtopicssuchasanti-vaccination,

multi-levelmarketingprograms,particulardiets,exerciseprograms,liberalactivism,

therapeuticmethodologies,philosophiesofchild-rearing,particularacademicsub-

disciplines,andCrossfit(Weathers2014).

17

‘Post-truth’andthepowerofechochambers

Ithasoftenbeenclaimed,duringandaftertheAmericanpoliticalseasonof2016,thatwe

haveentereda‘post-truthera’.Notonlydosomepoliticalfiguresseemtospeakwithablatant

disregardforthefacts,theirsupportersseemunswayedbyreasonorcontraryevidence.To

many,itseemsasifavastswathoftheelectoratehasbecomeentirelyunmooredfromany

interestinfactsorevidence.Callthisthe“totalirrationality”explanationofthepost-truth

phenomenon.

Butechochambersofferanalternativeexplanationfortheapparentpost-truthmood.It

seemslikelythatthereisatleastonevastpartisanecho-chamberpresentinthepolitical

landscape.JamiesonandCappella’sstudyisadecadeold,butsourceslikeBreitbartandAlex

Jones’Infowarsseemlikeclearextensionsofthesameright-wingechochamber.(Otherecho

chamberssurelyexistelsewhereonthepoliticalspectrum,though,tomymind,theleft-wing

echochambershavebeenunabletoexertasimilarlevelofpoliticalforce.)Inthatcase,the

accountofechochambersI’veofferedhassignificantexplanatoryforce.Theapparent“post-

truth”attitudecanbeexplained,atleastinpart,astheresultofcredencemanipulations

wroughtbyechochambers.Inhealthyepistemiccommunities,thereissomethinglikean

upperceilingonthecredencelevelaccordedtoanyindividual.Ahealthyepistemicnetwork

willsupplyasteadystreamofcontraryevidenceandcounterarguments;thus,nosingle

individualorgroupwillevergounchallenged.Epistemicbubblesmakethediscoveryof

mistakessignificantlylesslikely,andsotendtoexaggeratethecredencelevelsofepistemic

sourcesinsidethebubble.Butwhenanechochamberisinplaceandalloutsidesourceshave

beeneffectivelydiscredited,thatceilingdisappearscategorically.Echochamberscancreate

runawaycredencelevelsforapprovedindividuals.Byremovingdisconfirmationsand

discorroborationfromthesystemthroughthesystematicdiscreditingofoutsiders,echo

18

chamberscancreateexceptionallyhigh—oneistemptedtosayunnaturallyhigh—levelsof

trust.Thatpotentialforrunawaycredenceisbuiltrightintothefoundationsofanyecho

chamber,andarisesfromaninteractionbetweenthetwomaincomponentsofanyecho

chamber.First,anechochamberinvolvesasignificantdisparityoftrustbetweentheinsiders

andtheoutsiders.Second,anechochamberinvolvesbeliefs,espousedbytheinsiders,

reinforcingthatdisparity.Theessentialfeaturesofechochambersseemdesignedtoself-

reinforcetheirpeculiararrangementoftrust.

Noticethatepistemicbubblesalonecannotexplainthepost-truthphenomenon.Since

epistemicbubblesworkonlyviacoveragegaps,theyofferlittleinthewayofexplanationfor

whyanindividualwouldrejectclearevidencewhentheyactuallydoencounterit.Coverage

gapscannotexplainhowsomebodycould,say,continuetodenytheexistenceofclimate

changewhenactuallyconfrontedwiththeoverwhelmingevidence.Onewouldbetempted,

then,toaccuseclimatechangedeniersofsomekindofbruteerror.Butechochambersoffer

anexplanationofthephenomenonwithoutresortingtoattributionsofbruteirrationality.

Climatechangedeniershaveenteredanepistemicstructurewherebyalloutsidesourcesof

evidencehavebeenthoroughlydiscredited.Enteringthatepistemicstructuremightitself

involvevariousepistemicmistakesandvices—butherethestorycanbeoneoftheslow

accumulationofminormistakes,whichgraduallyembedthebelieverinaself-reinforcing,

internallycoherent,butultimatelymisleadingepistemicstructure.

Similarly,somehavesuggestedthat,inthepost-truthera,manypeople’sinterestinthe

truthhasevaporated.Onceagain,thisaccountofechochamberssuggestsalessdamningand

moremodestexplanation.Anechochamberdoesn’terodeamember’sinterestinthetruth;it

merelymanipulatestheircredencelevelssuchthatradicallydifferentsourcesandinstitutions

willbeconsideredpropersourcesofevidence.Thisphenomenonstandsinstarkcontrastto

19

accountsofobfuscatoryspeech.Take,forinstance,Orwelliandoublespeak—deliberately

ambiguous,euphemism-filledlanguage,designedtohidetheintentofthespeaker(Orwell

1968).Doublespeakisapracticethatevincesnointerestincoherence,clarity,ortruth.But

bymyaccount,weshouldexpectdiscoursewithinechochamberstobeentirelydifferent—

weshouldexpectsuchdiscoursetobecrispandclear,andtopresentunambiguousclaims

aboutwhatisthecase,whatsecretconspiraciesareinplace,andwhichsourcesaretobe

entirelydistrusted.Andthisispreciselywhatwefind(JamiesonandCappella2008,3-41,140-

176).Consider,forexample,Breitbart’sattacksonothermediasources.Onearticlebegins:

“MainstreammediaoutletscontinuetoprintfalseanddefamatorydescriptionsofBreitbart

Newsinanakedlypoliticalefforttomarginalizeagrowingcompetitor”(Pollak2017).Thisis

notthedouble-speakofadministratorsandbureaucrats—thisisaclear,strident,and

unambiguouslywordeddiscredit.

Onemightbetemptedtosay:butjustgivethemtherealevidence!Youcan’tdiscredit

neutralevidence!Butthisresponseradicallyunderestimatesthedegreeoftrustandsocial

processinginvolvedinmostpresentationsofevidence.ExceptforempiricalevidenceImyself

havegathered,allotherpresentationsofevidencerelyontrust.Mybeliefintherealityof

climatechangedependsonenormousamountsofinstitutionaltrust.Ihavenotgatheredthe

climatechangeevidencemyself;Imostlyjusttrustsciencejournalistswho,inturn,trust

institutionalcredentialingsystems.EvenifIhadbeenon,say,acoresamplingexpeditionto

theArctic,Iwouldbeunabletoprocessthatinformationformyself,orevenvetwhether

somebodyelsehasproperlyprocessedit.Eventheclimatologistwhoactuallyprocessesthat

informationmustalsodependontrustingavastarrayofotherexperts,includingstatisticians,

chemists,andtheprogrammersoftheirdataanalysissoftware.Mostso-called“neutral

20

evidence”dependsonlongandrobustchainsoftrust(Millgram2015,27-44).Membersofan

echochamberhaveacquiredbeliefswhichbreaktheusualarrangementsoftrust.

Butdespitetheirevidentexplanatoryforce,echochambershavebeenlargelyneglected

byrecentempiricalresearch.Muchoftherecentresearchoncausesofbeliefpolarization

focusesonthecausalroleofindividualpsychology,suchasthetendencytowardslazinessin

thescrutinyofone’sownbeliefs(Troucheetal.2016).Similarly,recentstudiesonclimate

changedenialfocusonstudyingtherelationshipbetweenanindividual’sstatedpolitical

beliefsandtheirreactionstoclimatechangeinformation,withoutinquiringintothesocial

epistemicstructuresinwhichtheindividualsareembedded(Corner,WhitmarshandXenias

2012).Famously,DanKahanandDonaldBramanarguefortheculturalcognitionthesis—

thatis,thatculturalcommitmentsarepriortofactualbeliefs,andthatnon-evidentially

formedculturalvaluesinformwhichfuturepresentationsofevidencewillbeadmittedas

weighty(KahanandBraman2006).Thoughthevaluesmayoriginallycomefroman

individual’sculture,KahanandBramanfocustheiranalysisonhowthoseacquiredvalues

functioninindividualreasoningtocreatepolarization.Theypaylittleattentiontothe

continuingroleofthecontingentsocialstructuresinwhichtheindividualisembedded.

Thedirectliteratureonechochambersandepistemicbubblesisnewandrelativelysmall,

comparedtothesizableliteratureonindividualbeliefpolarization.Unfortunately,evenin

thatliterature,echochambersandepistemicbubbleshaveoftenbeenconfused.Theyare

usuallyaddressedinthepopularmediatogether,andtheterms‘epistemicbubble’and‘echo

chamber’aretypicallyusedinterchangeably(El-Bermawy2016).Thesameblurringhas

occurredinthetreatmentofthephenomenainacademicepistemologyinthesurprisingly

smallliteratureonechochambers.Forexample,BertBaumgaertner,inhisanalysisofecho

chambersviacomputermodeling,lumpstogetherundertheheading‘echochamber’both

21

JamiesonandCappella’saccountofrightwingechochambersandEricGilbertetal’s

treatmentofblogsasechochambers(Baumgaertner2014).ButwhereJamiesonand

Cappella’ssubjectofstudyisechochambers,Gilbert’sdiscussionconcernshowblog

communitieshavemerelyexcludedrelevantvoicesthroughsocialpracticesofconnecting

withlike-mindedindividuals—clearlywhatI’mcallingepistemicbubbles(Gilbert,

BergstromandKarahalios2009).Inbothpopularandacademiccases,furtheranalysishas

focusedhasbeenonepistemicbubbles;thestructuresofdiscreditinvolvedinJamiesonand

Cappella’sechochambershavebeenlargelyneglected.8

Foraparticularlyprominentexample,considerSunstein’sinfluentialaccountofthe

relationshipofInternettechnologies,grouppolarizationandtheriseofpoliticalextremism.

Thoughheprofessestocoverbothfilterbubblesandechochambers,hisworkfocusesalmost

entirelyonepistemicbubbleeffects:constrictedinformationflow,lackofexposureto

alternatearguments,andbootstrappedcorroboration(Sunstein2009b,xi,19-06,2009a,1-

98).Thepointhereisaboutmorejustthanhischoiceofwords:hissubjectsofanalysis

include,amongotherthings,Facebookfriendgroups,hategroups,extremistonlinepolitical

forums,conspiracytheorists,andterroristgroups(99-125,2009b,46-96).Clearly,thislist

includesprimecandidatesforbothepistemicbubblesandechochambers.Buthisanalysis

focusesalmostentirelyontheeffectsofbootstrappedcorroborationandlackofexposure.For

Sunstein,theprimarymechanismdrivingpolarizationandextremismisthelossoftruly

publicforums,becausetechnologyhasover-empoweredpeople’stendencytoself-select

sourcesofferingfamiliarviews.Thus,hissolutionistore-create,inthenewmedia

environment,thekindofgeneralpublicforumswherepeoplemightbemorelikelyto

8 One exception is the careful treatment of aesthetic echo chambers in (Robson 2014), which brought Jamieson and Cappella’s work to my attention, and was instrumental in this paper’s development.

22

serendipitouslyencountercontraryviewsandarguments.Hissolutionsincludegovernment-

fundedpublicnewswebsiteswithdiversecoverageandvoluntaryworkbycorporationsand

individualstobursttheirbubbles.Hisrecommendationsforrepairlargelyhavetodowith

increasingexposure(Sunstein2009a,135-48,2009b,19-45,190-211).But,again,ifwhat’s

goingonisactuallyanechochambereffect,exposureisuselessorworse.

Theblurringofthetwoconceptshasalsoleadtosomeproblematicdismissalsofthe

wholeclusterofphenomena.Anumberofrecentarticlesinsocialscience,communications,

andmediastudieshavearguedthatthewholesetofworriesaboutbubblesandecho

chambersiswildlyoverstated.Thesearticlessharethesameargumentativepattern.First,

theyusetheterms“filterbubble”and“echochamber”interchangeably,andaddress

themselvestothesameclusterofphenomenaasSunstein,treatingthemassingular.Infact,

JamesNelsonandJamesWebsterconflateJamiesonandCappella’sanalysisofechochambers

andPariser’sanalysisoffilterbubbles,anderroneouslyattributetoJamiesonandCappella

theviewthatpoliticalpartisansonlyseekoutandencountermediafromsourceswith

matchingpoliticalalignments—thatis,NelsonandWebsterattributetoanepistemic

bubblesaccounttoJamiesonandCappella,whereJamiesonandCappella’sactualtextis

clearlyanechochambersaccount(NelsonandWebster2017,2).Moreimportantly,these

recentarticlesproceedtoargueagainsttheexistenceoffilterbubblesandechochambersby

demonstratingthat,throughtheanalysisofempiricaldataaboutmediaconsumption,most

peopleinfactexposethemselvestomediafromacrossthepoliticalspectrum.Nelsonand

Webster,forexample,argueagainstJamiesonandCapella,claimingthatfilterbubblesand

echochambersdon’texist.NelsonandWebstersupporttheirclaimwithdatashowingthat

bothliberalsandconservativesvisitthesamemediasitesandspendcomparableamountsof

timeatthosesites(6-7).Again,thismissesthemark—thisisevidenceonlyagainstthe

23

existenceofepistemicbubbles,andnotagainsttheexistenceofechochambers.Similarly,Seth

Flaxmanetalseekstoproblematizetheexistenceoffilterbubblesandechochamberswith

datathatsocialmediaplatformsseemtoactuallyincreasepeople’sexposuretomediafrom

acrossthepoliticaldivide(Flaxman,GoelandRao2016).Again,thesedataonlyconcernthe

exposureandomission,andonlyweighagainsttheexistenceofepistemicbubbles.Theysay

nothingaboutwhetherechochambersexist.Echochambers,recall,arestructuresofstrategic

discrediting,ratherthanbadinformationalconnectivity.Echochamberscanexistevenwhen

informationflowswell.Infact,echochambersshouldhopethattheirmembersareexposedto

mediafromtheoutside;iftherightdisagreementreinforcementmechanismsareinplace,

thatexposurewillonlyreinforcetheechochambers’members’allegiance.Weoughtnot

concludethen,fromdatathatepistemicbubblesdonotexist,thatechochambersalsodonot

exist.

Wecanseenowcrucialitistokeepthesetwocategoriesdistinct.Epistemicbubblesare

ratherramshackle—theygoupeasily,buttheyareeasytotakedown.Sincethereisno

systematicdiscreditingofoutsiders,simpleexposuretoexcludedvoicescanrelievethe

problem.Echochambers,ontheotherhand,aremuchhardertoescape.Echochamberscan

starttoseemalmostlikelivingthings—thebeliefsystemsprovidestructuralintegrityand

resilience.Mereexposurewillbeineffective.JamiesonandCappellaofferevidenceofthis

effect:oncelistenersarecaughtinRushLimbaugh’slanguage,framing,anddiscredentialing

ofthemainstreammedia,theirbeliefscansurvivefrequentcontactwithcontraryviewpoints.

Limbaugh’stechnique,sayJamiesonandCappella,servestoinsulateandinoculatehis

audiencefrombeingaffectedbyexposuretocontraryviewpoints(JamiesonandCappella

2008,163-190).Infact,iftheappropriatedisagreement-reinforcementmechanismsarein

place,exposurewillsimplystrengthentheattackedbeliefsystems.Thus,anoutsider’s

24

attempttobreakanechochamberasifitwereamerebubbleislikelytobackfireand

reinforcetheechochamber’sgrip.9

Responsibilityandthepossibilityofescape

Sowhat,then,aretheepistemicresponsibilitiesofanagenttodiscoverwhethertheyare

inoneofthesesocialepistemictraps,andwhataretheirprospectsforactuallydiscovering

theirpredicamentandsuccessfullyescaping?Toanswerthis,wemustconsidertwodistinct

questions:

Theescaperoutequestion:Isthereanywayoutofanechochamberorepistemic

bubble?

Theescaperesponsibilityquestion:Couldonebehaveepistemicallyvirtuously,andyet

stillremaincaughtwithinanechochamberorepistemicbubble?Inotherwords,towhat

degreeisanepistemicagentembeddedwithinsuchastructureblameworthy,orblameless,

forthefaultinessoftheirbeliefs?

Thefirstquestionasksaboutthepossibleexistenceofanescaperoute.Thesecondasks

whetherthereisanescaperoutethatwemightreasonablyexpectanepistemicallyvirtuous

agenttodiscoverandenact.Thesearedistinctquestions,becauseanescaperoutemightturn

9 Sunstein does briefly note the empirical data for the disagreement-reinforcement effect in passing, but then seems to ignore it in proposing his solutions (Sunstein 2009a, 54-5)

25

outtobepossible,butsodifficulttodiscoverorusethatitwasbeyondwhatwemight

reasonablyexpectofanagentofmoderateepistemicvirtue.

Forepistemicbubbles,theanswersarestraightforward.AsI’veargued,epistemicbubbles

arequiteeasytoshatter.Onejustneedsexposuretoexcludedinformation.Insofarasthat

informationisavailable,butsimplynotpartofone’sstandardnetwork,thenmembersof

epistemicbubblesarefailingtoliveuptotheirepistemicduties,whichincludeproactively

gatheringrelevantdata.Totranslateintocontemporaryterms:ifyou’resubjecttoan

epistemicbubblebecauseyougetallyournewsfromFacebookanddon’tbothertolookat

othersources,youare,indeed,blameworthyforthatfailure.Ifonefindsthelanguageof

epistemicvirtuesandvicesappealing,thenwecansaythatmembersofepistemicbubbles

arecommittingtheviceofepistemiclaziness.

Answeringthesetwoquestionsismuchmoredifficultforechochambers.Recall:where

encounteringexcludedvoicesandevidencewillshatteranepistemicbubble,suchencounters

arelikelytoreinforceanechochamber.Let’sgrantthatintentionallyconstructinganecho

chamber,asJamiesonandCappellaclaimthatRushLimbaughdid,isepistemically(and

morally)blameworthy.Furthermore,activelyenteringanechochamberseemsepistemically

blameworthyinmanycircumstances.Foragentinfullpossessionofawiderangeof

informationalsources,toabandonmostofthemandplacetheirtrustinanechochamberfor,

say,anincreasedsenseofcomfortandsecurity,issurelysomeformofepistemicvice.There

issomeevidencethatthismaybethecase;JamiesonandCappellasuggestthatpeopleenter

echochambersforthesakeofthecommunitybondsandthesenseofbelongingtoanin-

group(JamiesonandCappella2008,180).

Buttherearemanycasesinwhichtheagentseemsplausiblyblameless.Imagineaperson

raisedinanechochamber.Theirearliestepistemiccontacts—let’ssaytheirparents,

26

relatives,andclosefamilyfriends—areallfirmlycommittedmembersoftheechochamber.

Supposethatthechildiseitherhome-schooledbythoseechochambermembersorsenttoa

schoolthatreinforcesthebeliefsofthatparticularechochamber.Itakeitthatitisreasonable

forachildtotrusttheirparentsandthoseofseemingepistemicauthority,andthatachildis

epistemicallyblamelessforhavingdoneso(Goldberg2013).Thus,whenthatchildeventually

comesintocontactwiththelargerepistemicworld—say,asateenager—theecho

chamber’sbeliefsarefullyinplace,suchthattheteenagerdiscreditsallsourcesoutsideof

theirechochamber.

Itseems,atfirstglance,thatourteenagercouldbeactingverymuchlikeareasonable

epistemicagent.Theycould,infact,beepistemicallyvoracious:seekingoutnewsources,

investigatingthem,andevaluatingthemusingtheirbackgroundbeliefs.Theyinvestigatethe

reliabilityofpurportedexpertsanddiscreditexpertswhentheyhaveapparentlygoodreason

todoso,usingtheirbackgroundbeliefs.Ourteenagerseems,infact,tobebehavingwith

manyepistemicvirtues.Theyarenotatalllazy;theyareproactiveinseekingoutnew

sources.Theyarenotblindlytrusting;theyinvestigateclaimsofepistemicauthorityand

decideforthemselves,usingalltheevidenceandbeliefsthattheypresentlyaccept,whether

toacceptordenythepurportedexpertiseofothers.Theyhavetheories,whichtheyhave

acquiredbyreasonablemethods,predictingthemaliciousnessofoutsiders;theyincrease

theirtrustinthosetheorieswhentheirpredictionsareconfirmed.10

Theworryhereisthatagentsraisedwithinanechochamberare,throughnofaultoftheir

own,epistemicallytrapped—theirearnestattemptsatgoodepistemicpracticesare

transformedintosomethingepistemicallyharmfulbythesocialstructureintowhichthey

havebeenembeddedandwhichtheyhaveingested.PaulSmarthasarguedforthepossibility

10 For a parallel argument about the invidiousness of background prejudicial beliefs, see (Begby 2013).

27

ofatransformativesocialepistemicphenomenonwhichhedubs“mandevillianintelligence,”

inhonorofBernardMandeville.Mandevillearguedthat,intherightsocialcontext,individual

vicescouldleadtocollectiveeconomicprosperity.Foracertainkindofeconomictheorist,

capitalismissuchatransformativestructure—individualsactselfishly,butthestructureof

themarkettransformsthatselfishnessintovirtuouscollectiveaction.AccordingtoSmart,

thereisanepistemicanalog:themandevillianintelligence,whichtransformstheindividual

epistemicvicesofitsmembersintoacollectiveepistemicvirtuebyvirtueofthesocial

structureintowhichtheyareembedded(Smart2017).Intellectualstubbornness,for

example,mightbeanintellectualviceforindividuals.Butsetthosestubbornindividualsina

properlyarrangedsocialstructure(like,perhaps,academia)andyoumightgetacollective

systemthatproperlyexploreseveryrelevantnookandcrannywithoptimalthoroughness.

Butechochambersaretheveryopposite;theyarereverse-mandevillianintelligences.Echo

chambersaresocialepistemicstructureswhichconvertindividuallyepistemicallyvirtuous

activityintocollectiveepistemicvice.Infact,thereverse-mandevilliannaturecontributesto

thestickinessoftheechochambertrap.Ifourteenagerself-reflectsontheirepistemic

practices,whattheywillseemightberathergratifying.Theirepistemicbehaviormightvery

wellbeearnest,vigorous,andengaged.Itistheirexternalcontext—thesocialepistemic

systemintowhichtheyhavebeenunluckilyraised—whichmakessuchbehavior

problematic.

ContrastthisaccountwithQuassimCassam’streatmentofOliver,hisfictional9/11

conspiracytheorist.Oliverbelievesthatthecollapseofthetwintowerswasaninsidejob,and

heishappytoprovidereasonsandpointtosupportingevidencefromagreatmany

conspiracytheoristwebsites.SaysCassam:Oliverisobviouslymistaken—Oliverrelieson

outrageous,baselessclaimsfromclearlydiscreditedsources.Thebestexplanationfor

28

Oliver’sbeliefsisintermsofepistemicvice—thatis,intermsofOliver’sbadintellectual

charactertraits.Oliveris“gullible,cynical,andprejudiced,”saysCassam.Oliverisgullible

withregardtohisconspiracytheoristsites,cynicalwithregardtothemainstreammedia,and

hisprejudiceconsistsof,amongotherthings,intellectualpride,wishfulthinking,closed-

mindedness,andalackofthoroughness(Cassam2016,162-4).AndIcertainlygrantthat

suchepistemicvicescanleadtothesesortsofbeliefs.Butthestoryofourhaplessteenager

offersanalternateepistemicpathtosuchbeliefsandsuchnarrowcastedtrust—onein

whichepistemicallyvirtuouscharactertraitshavebeenwrong-footedbythesocialepistemic

structureinwhichtheagenthasbeenembedded.Thecrucialdifferencebetweenthereverse-

mandevillianaccountandCassam’saccountiswherethebruntoftheresponsibilitylies.In

Cassam’saccount,theresponsibilitylieswiththeindividual,andtheirownintellectualhabits

andpractices.11Inareverse-mandevillianaccount,asignificantpartoftheresponsibilitylies

withthesocialstructureinwhichtheactorsareembedded.Theepistemicviceisafeatureof

thecollectiveintelligence,ratherthanoftheindividual.Or,ifoneisaversetothinkingin

termsofcollectiveintelligences,here’saconceptuallyminimalwaytoputtheclaim:echo

chambersarelocalbackgroundconditionsthatturngenerallygoodepistemicpracticesinto

locallyunreliableones.

Butthepossibilityofatrulyfaultlessepistemicagent,whollymisledbyanechochamber,

alsodependsonthelackofanaccessibleescaperoute.So:arethereescaperoutesfroman

11 Note, however, that Cassam distinguishes between epistemic responsibility and epistemic blameworthiness, and does not take blameworthiness to necessarily follow from responsibility (168-9). Cassam leaves room for the view that the individual’s intellectual vices were epistemically responsible for their bad beliefs, but that the individual wasn’t blameworthy for those vices, because the vices were inculcated in them at an early age. However, my complaint still stands, for I contest Cassam’s claim that the responsibility is in the individual.

29

echochamber,andhowreasonableisittoexpectechochambermemberstodiscoverand

makeuseofthem?

Hereisonepossibleescaperoute.ConsiderThomasKelly’sdiscussionofbelief

polarization.Beliefpolarizationisthetendencyofindividuals,oncetheybelievethatp,to

increasetheirbeliefthatp.Kellyarguesthatbeliefpolarizationworksbythemechanismthat,

onceanagenthasacquiredabelief,theytendtosubjectcounter-argumentstogreater

scrutinyandgivesupportingargumentsarelativepass.Thus,theircriticalreflectionislikely

toreinforcepreviouslyheldbeliefs.Kellynotesthatthebeliefpolarizationviolatesthe

CommutativityofEvidencePrinciple:

TheCommutativityofEvidencePrinciple:totheextentthatwhatitisreasonableforonetobelievedependsonone'stotalevidence,historicalfactsabouttheorderinwhichthatevidenceisacquiredmakenodifferencetowhatitisreasonableforonetobelieve.(Kelly2008,616)

Inshort,beliefpolarizationmakesitmatterwhatordertheyreceivedtheevidence,but

thehistoricalorderingofevidenceoughtnotmatter.Notethatourepistemicallyhapless

teenagerhasalsoviolatedtheCommutativityofEvidencePrinciple.Forthem,itmattersvery

muchwhatorderthattheyreceivedtheevidence.Iftheyhadbeenraisedoutsidetheecho

chamberandfedabroaderdietofepistemicsourcesbeforeencounteringtheechochamber,

thentheywouldlikelyhavefoundtheechochamber’sworld-viewtobeproblematic.But

sinceourteenagerencounteredtheechochamberandassimilateditsbeliefsfirst,theiruseof

backgroundbeliefstovetnewsourcesleadsthemtocontinuallyincreasetheirtrustinthe

echochamberandtheirdistrustofoutsiders.Evenifourechochamberedteenagereventually

cametoencounterallthesameevidenceastheirepistemicallyfree-rangecounterpart,their

earlyeducationwithintheechochamberwouldbedecisive.Solongaseachnewpieceof

30

evidenceisassessedusingthecurrentlyheldsetofbeliefs,thenearlyeducationinanecho

chamberbecomesdomineeringlypowerful.

However,theCommutativityofEvidencePrinciplesuggestsawayout.Inordertofree

themselvesoftheechochamber’sgrip,ourteenagerneedstoundotheinfluenceofthe

historicalorderingoftheirencounterswiththeevidence.Howcouldtheypossiblydothis?

Ourteenagerwouldhavetosuspendbeliefinalltheirparticularbackgroundknowledgeand

restarttheknowledge-gatheringprocess,treatingalltestimonyasequallyviable.Theywould

needto,inasense,throwawayalltheirbeliefsandstartoveragain.Thissuggestsaprocess

that,initsoutlines,mightsoundawfullyfamiliar.Ourescaperouteturnsouttobea

somethinglikeamodifiedversionofDescartes’infamousmethod.

Whatproceedsfromthispointisadmittedlysomethingofafantasy,butperhapsitisa

fantasyfromwhichwecaneventuallydrawsomesortofmoral.Thestoryofthehistoryof

Westernepistemologymightbecartoonishlysummarizedthusly:Descarteshadadreamof

radicalintellectualautonomy.Byhisaccounting,hecametorealizethatmanyofthebeliefshe

hadacquiredinhisearlylifewerefalse,andthatthoseearlyfalsebeliefsmighthaveinfected

anynumberofotherbeliefs.Hisresponsewasthatfamedmethod:togetridofhisbeliefsand

startoveragain,trustingno-oneandnothingandonlypermittingthosebeliefsofwhichhe

wasentirelycertain.CallthistheCartesianepistemicreboot.Butifrecentepistemologyhas

taughtusanything,it’sthatthistotalrebootisnothingbutapipedream.Anysortof

reasonableepistemiclifeisessentiallyimpossiblewithouttrustingthetestimonyofothers

(Burge1993;Faulkner2000;Goldberg2010;Zagzebski2012;Hardwig1985,1991).

ButrecallthatthereasonDescarteswantedtodiscardeverythingandstartoverfrom

scratch—themotivationforhisproject,andnotthemethod—wasexplainedinthevery

firstlineof“Meditation1”:Hewasworriedbythefalsehoodshehadlearnedinchildhoodand

31

theshakinessoftheedificethathadbeenbuiltfromthosefalsehoods(Descartes1984,24).

Ourteenagerfacesaproblemquitesimilarinstructure.Thecredentialingstructureoftheir

upbringingisflawed;thatcredentialingstructurehasinfluencedanynumberoftheirother

beliefs,andthedegreeofthatinfluenceisimpossibletotrack.Furthermore,theselater

beliefs,approvedbytheechochambers’credentialedsources,willoftenservetoreinforce

thatcredentialingstructure.Theperniciouseffectofanechochambercannotbeattackedone

beliefatatime.Anysinglebeliefthatourteenagerre-consideredwouldcomeunderthe

influenceofthenetworkoftheflawedbackgroundbeliefsthatsustainsanechochamber.

Whattheyneedissomewaytostartover.Inordertoundotheinfluenceofhistorical

ordering,anepistemicagentwillhavetotemporarilysuspendbeliefinalltheirbeliefs,in

particulartheircredentialingbeliefs,andstartfromscratch.Butwhentheystartfromscratch,

theyneednotdisregardthetestimonyofothers,norneedtheyholdtoDescartes’stringent

demandforcertainty.Let’scallthisprocedurethesocialepistemicreboot.Inthesocial

epistemicreboot,theagentispermitted,duringthebeliefre-acquisitionprocess,totrustthat

thingsareastheyseemandtotrustinthetestimonyofothers.Buttheymustbeginafresh

socially,byre-consideringalltestimonialsourceswithpresumptiveequanimity,without

deployingtheirpreviouscredentialingbeliefs.Furthermore,theymustdiscardalltheirother

backgroundbeliefs,becausethosepotentiallyarosefromtheflawedcredentialstructureof

theechochamber,andverylikelyhavebeendesignedtosupportandreinforcethatvery

credentialstructure.Ourrebootermusttakeonthesocialepistemicposturethatwemight

expectofacognitivenewborn:oneoftentative,butdefeasible,trustinallapparent

testimonialsources(Burge1993)(Nguyen2011).Thismethodwill,ifsuccessfullyapplied,

undothehistoricaldependenceofourepistemicagentandremovetheundueinfluenceofthe

32

echochamber.Thesocialepistemicrebootis,theoreticallyatleast,theescaperoutewe’ve

beensearchingfor.12

Thisreboot,describedinsuchclinicalterms,mightseemratherfantastical.Butitisnot,I

think,utterlyunrealistic.Considerthestoriesofactualescapeesfromechochambers.Take,

forexample,thestoryofDerekBlack,whowasraisedbyaneo-Nazifather,groomedfrom

childhoodtobeaneo-Nazileader,andwhobecameateenagedbreakoutstarofwhite

nationalisttalkradio.WhenBlackleftthemovement,hewentthroughyears-longprocessof

self-transformation.Hehadtocompletelyabandonhisbeliefsystem,andhespentyearsre-

buildingaworld-viewofhisown,immersinghimselfbroadlyandopen-mindedlyin

everythinghe’dmissed—popculture,Arabicliterature,thepronouncementsofthe

mainstreammediaandtheUSgovernment,rap—allwithanoverallattitudeoftrust(Saslow

2016).

Ofcourse,allwehaveshownsofaristhatthesocialepistemicrebootwould,ifpulledoff,

undotheeffectsofanechochamberedupbringing.Whetherornotanepistemicagentmight

reasonablybeexpectedtoreboot,orblameworthyforfailingtoreboot,isaseparateand

significantlymoredifficultquestion.First,asocialepistemicrebootmightbepsychologically

impossible,oratleastbeyondwhatwecouldreasonablyexpectofnormalepistemicagents.

Second,whatreasonwouldanepistemicagenthavetoundertakeasocialepistemicreboot?

Suchanundertakingwouldbejustifiedonlyiftheagenthadasignificantreasontothinkthat

theirbeliefsystemwassystematicallyflawed.Butechochambermembersdon’tseemlikely

tohaveaccesstoanysuchapparentreason.Afterall,theyhaveclearandcoherent

explanationsforalltheevidenceandtestimonytheyencounter.Ifthisisallrightthenwe

12 Note that the social epistemic reboot would also undo the effects of Begby’s evidential preemption, since that preemption also depends on the historical ordering of received data.

33

arriveataworryingconclusion:thatechochambersmay,theoretically,beescapable,butwe

havelittlereasontoexpectmembersofechochambersrealizethattheyaremembersof

somethingthatneedsescaping.

Whatcouldhopedowehave,then,ofmotivatingareboot?DerekBlack’sownstorygives

usahint.Blackwenttocollegeandwasshunnedbyalmosteveryoneinhiscollege

community.ButthenMatthewStevenson,aJewishfellowundergraduate,begantoinvite

BlacktohisShabbatdinners.Stevensonwasunfailinglykind,open,andgenerous,andhe

slowlyearnedBlack’strust.ThiseventuallyleadtoamassiveupheavalforBlack—aslow

dawningrealizationofthedepthstowhichhehadbeensystematicallymisled.Blackwent

throughaprofoundtransformation,andisnowananti-Nazispokesperson.

TheturningpointseemstobepreciselythatStevenson,anoutsider,gainedBlack’strust.

Andthisisexactlywhereweshouldexpecttheturningpointtobe.Sinceechochamberswork

bybuildingdistrusttowardsoutsidemembers,thentheroutetounmakingthemshould

involvecultivatingtrustbetweenechochambermembersandoutsiders.Inordertomotivate

thesocialepistemicreboot,anechochambermemberneedstobecomeawareofhowinthe

echochamber’sgriptheyare,andformingatrustrelationshipwithanoutsidermightcould

mediatethatawareness.Buthowthattrustcouldbereliablycultivatedisaverydifficult

matter,andatopicforfutureinvestigation.Wehave,however,arrivedatatentativemoralof

thestory.Echochambersworkbyamanipulationoftrust.Thus,theroutetoundoingtheir

influenceisnotthroughdirectexposuretosupposedlyneutralfactsandinformation;those

sourceshavebeenpreemptivelyundermined.Itistoaddressthestructuresofdiscredit--to

34

worktorepairthebrokentrustbetweenechochambermembersandtheoutsidesocial

world.13

Bibliography

Alcoff,LindaMartín.2007.Epistemologiesofignorance:Threetypes.InRaceandEpistemologiesofIgnorance,Ed.ShannonSullivanandNancyTuana.Albany:SUNYPress.

An,Jisun,DanieleQuercia,andJonCrowcroft.2014.Partisansharing:Facebookevidenceandsocietalconsequences.InProceedingsoftheSecondACMConferenceonOnlineSocialNetworks.http://cosn.acm.org/2014/files/cosn033f-anAembTS.pdf.

Baumgaertner,Bert.2014.Yes,no,maybeso:Averitisticapproachtoechochambersusingatrichotomousbeliefmodel.Synthese191(11):2549-2569.

Begby,Endre.2013.Theepistemologyofprejudice.Thought:AJournalofPhilosophy2(2):90-99.

———.2017.Evidentialpreemption.AmericanPhilosophicalAssociationPacificDivision2017Meeting.

Bishop,Bill.2009.TheBigSort:WhytheClusteringofLike-mindedAmericaIsTearingUsApart.Boston:HoughtonMifflinHarcourt.

Burge,Tyler.1993.Contentpreservation.ThePhilosophicalReview102(4):457-488.Cassam,Quassim.2016.Viceepistemology.TheMonist99(2):159.Coady,David.2006.Whenexpertsdisagree.Episteme:AJournalofSocialEpistemology3

(1-2):68-79.———.2012.WhattoBelieveNow.WestSussex:Wiley-Blackwell.Coates,DominiekD.2010.Post-involvementdifficultiesexperiencedbyformermembers

ofcharismaticgroups.JReligHealth49(3):296-310.Corner,Adam,LorraineWhitmarsh,andDimitriosXenias.2012.Uncertainty,scepticism

andattitudestowardsclimatechange:Biasedassimilationandattitudepolarisation.ClimaticChange114(3-4):463-478.

Dentith,MatthewR.X.2017.Theproblemofconsparicism.Argumenta5:1-16.Dentith,MatthewRX.2015.Wheninferringtoaconspiracytheorymightbethebest

explanation.SocialEpistemology16(5-6):572-591.Descartes,René.1984.ThePhilosophicalWritingsofDescartes,Vol.2,TranslatedbyJohn

Cottingham,RobertStoothoff,andDugaldMurdoch.Cambridge:CambridgeUniversityPress.El-Bermawy,Mostafa.Yourfilterbubbleisdestroyingdemocracy.

https://www.wired.com/2016/11/filter-bubble-destroying-democracy/(accessedFebruary2,2017).

Faulkner,Paul.2000.Thesocialcharacteroftestimonialknowledge.TheJournalofPhilosophy97(11):581-601.

Flaxman,Seth,SharadGoel,andJustinM.Rao.2016.Filterbubbles,echochambers,andonlinenewsconsumption.PublicOpinionQuarterly80:298-320.

13 I’d like to thank Kara Barnette, Endre Begby, Anthony Cross, Melissa Hughs, Eric Stencil, Matt Strohl, Shannon Mussett, Bekka Williams, the anonymous reviewers, and many others for their assistance with this paper.

35

Fricker,Miranda.2011.Rationalauthorityandsocialpower:Towardsatrulysocialepistemology.InSocialEpistemology-EssentialReadings.Ed.AlvinGoldman.NewYork:OxfordUniversityPress.

Gilbert,Eric,TonyBergstrom,andKarrieKarahalios.2009.Blogsareechochambers:Blogsareechochambers.InSystemSciences,2009.HICSS'09.42ndHawaiiInternationalConferenceon.

Goldberg,Sanford.2010.RelyingonOthers:AnEssayinEpistemology.OxfordUniversityPress.

———.2011.IfthatweretrueIwouldhaveheardaboutitbynow.InTheOxfordHandbookofSocialEpistemology.Oxford:OxfordUniversityPress.

———.2013.Epistemicdependenceintestimonialbelief,intheclassroomandbeyond.JournalofPhilosophyofEducation47(2):168-186.

Goldman,AlvinI.2001.Experts:Whichonesshouldyoutrust?PhilosophyandPhenomenologicalResearch63(1):85-110.http://www.jstor.org/stable/3071090.

Hardwig,John.1985.Epistemicdependence.TheJournalofPhilosophy82(7):335-349.———.1991.Theroleoftrustinknowledge.TheJournalofPhilosophy88(12):693-708.Jamieson,KathleenHallandJosephNCappella.2008.EchoChamber:RushLimbaughand

theConservativeMediaEstablishment.Oxford:OxfordUniversityPress.Kahan,DanMandDonaldBraman.2006.Culturalcognitionandpublicpolicy.YaleLaw&

PolicyReview24(1):149-172.Kelly,Thomas.2008.Disagreement,dogmatism,andbeliefpolarization.TheJournalof

Philosophy105(10):611-633.Kitcher,Philip.1993.TheAdvancementofScience.NewYork:OxfordUniversityPress.Lackey,Jennifer.2013.Disagreementandbeliefdependence:Whynumbersmatter.In

TheEpistemologyofDisagreement:NewEssays.Ed.JenniferLackeyandDavidChristensen.Oxford:OxfordUniversityPress.

Langone,MD.ReflectionsonPost-CultRecovery.http://www.icsahome.com/articles/reflections-on-post-cult-recovery-langone(accessedMarch15,2017).

Lifton,Robert.1991.Cultformation.CulticStudiesJournal8(1):1-6.Mengus,Bryan.2016.PizzagatersAren'tGivingThisShitUp.Gizmodo,Millgram,Elijah.2015.TheGreatEndarkenment:PhilosophyforAnAgeof

Hyperspecialization.Oxford:OxfordUniversityPress.Miller,BoazandIsaacRecord.2013.Justifiedbeliefinadigitalage:Ontheepistemic

implicationsofsecretinternettechnologies.Episteme10(02):117-134.Mills,Charles.2007.Whiteignorance.InRaceandEpistemologiesofIgnorance.Albany:

StateUniversityofNewYorkPress.Nelson,JacobL.andJamesG.Webster.2017.Themythofpartisanselectiveexposure:A

portraitoftheonlinepoliticalnewsaudience.SocialMedia+SocietyJuly-September:1-13.Nguyen,C.Thi.2010.Autonomy,understanding,andmoraldisagreement.Philosophical

Topics.38(2):111-129.--.2011.Anethicsofuncertainty:moraldisagreementandmoralhumility(PhD

dissertation).UniversityofCaliforniaatLosAngeles.RetrievedfromProQuestDissertationsandThesesdatabase.(UMINo.3532448)

--.2018a.Cognitiveislandsandrunawayechochambers:Problemsforexpertdependence.Synthese.https://doi.org/10.1007/s11229-018-1692-0

--.2018b.Escapetheechochamber.AeonMagazine.https://aeon.co/essays/why-its-as-hard-to-escape-an-echo-chamber-as-it-is-to-flee-a-cult

36

--.Forthcoming.Hyper-specializationandthefragmentationofintellectualautonomy.PhilosophicalInquiries.

Orwell,George.1968.Politicsandtheenglishlanguage.InTheCollectedEssays,Journalism,andLettersofGeorgeOrwell.1ed.Ed.SoniaOrwellandIanAngos.NewYork:Harcourt,Brace,Javanovich.

Pariser,Eli.2011.TheFilterBubble:WhattheInternetIsHidingFromYou.PenguinUK.Pollak,Joel.2017.#FakeNews:MainstreamMediaContinuetoSlanderBreitbart.

Breitbart,January30.http://www.breitbart.com/big-journalism/2017/01/30/fakenews-mainstream-media-continue-slander-breitbart/.

Robbins,ThomasandDickAnthony.1982.Deprogramming,brainwashingandthemedicalizationofdeviantreligiousgroups.SocialProblems29(3):283-297.

Robson,Jon.2014.Asocialepistemologyofaesthetics:Beliefpolarization,echochambersandaestheticjudgement.Synthese191(11):2513-2528.

Saez-Trumper,Diego,CarlosCastillo,andMouniaLalmas.2013.Socialmedianewscommunities:Gatekeeping,coverage,andstatementbias.InProceedingsofthe22ndACMInternationalConferenceonConferenceonInformation&KnowledgeManagement.https://dl.acm.org/citation.cfm?id=2505623

Saslow,Eli.2016.ThewhiteflightofDerekBlack.TheWashingtonPost,October15.https://www.washingtonpost.com/national/the-white-flight-of-derek-black/2016/10/15/ed5f906a-8f3b-11e6-a6a3-d50061aa9fae_story.html?utm_term=.67f00ca25ce9.

Singer,MargaretT.1979.Comingoutofthecults.PsychologyToday12(8):72-82.Smart,PaulR.2017.Mandevillianintelligence.Synthese.https://doi.org/10.1007/s11229-017-1414-z

Sunstein,CassR.2001.Republic.Com.InRepublic.com.Princeton:PrincteonUniversityPress.

———.2009a.GoingtoExtremes:HowLikeMindsUniteandDivide.OxfordUniversityPress.

———.2009b.Republic.com2.0.PrincetonUniversityPress.Trouche,Emmanuel,PetterJohansson,LarsHall,andHugoMercier.2016.Theselective

lazinessofreasoning.CognSci40(8):2122-2136.Vogt,PJandAlexGoldman.2016.VoyageIntoPizzagate.

https://gimletmedia.com/episode/83-voyage-into-pizzagate/(accessedFebruary20,2017).Watson,JamieCarlin.2015.Filterbubblesandthepublicuseofreason:Applying

epistemologytothenewsfeed.InSocialEpistemologyandTechnology:TowardPublicSelf-AwarenessRegardingTechnologicalMediation.Ed.FrankScalambrino.London:Rowman&Littlefield.

Weathers,Cliff.2014.CrossFitIsaCult:WhySoManyofItsDefendersAreSoDefensive.Salon.https://www.salon.com/2014/10/22/crossfit_is_a_cult_why_so_many_of_its_defenders_are_so_defensive_partner/

Wittgenstein,Ludwig.2010.PhilosophicalInvestigations.Hoboken:JohnWiley&Sons.Zagzebski,Linda.2012.EpistemicAuthority.NewYork:OxfordUniversityPress.