Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Amplifying the Wisdom of the Crowd, Building and Measuring Expert Consensus for
Nonproliferation Issues
James CarrollINMM Taos Technical Meeting,
May 31, 2012LA-UR-12-21784
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Abstract
Scientific and cultural progress mostly takes place in the marketplace of ideas. The internet has largely changed how this marketplace functions. Although it has brought many improvements, it has also created a new set of problems. We propose a web service that can be used to crowdsource the process of knowledge acquisition and of knowledge summary, mediate disagreements, and improve discourse on controversial issues, thereby improving the efficiency of the marketplace of ideas. Our approach involves a specific technique for integrating: a forum; a wiki organized into separate camps, with camps organized into a hierarchical structure; a survey system based upon camp support; a mechanism for dynamically reorganizing the structure of the camp hierarchy while mediating user disagreements; and a customizable mechanism for determining how votes are weighted based on credentials or expert assignment. A version of this approach has been implemented, released as open source, and a beta test has gone live. Initial use seems to validate the merits of this approach. (U)
Slide 2
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Outline
1. Epistemology and the Marketplace of Ideas2. Inefficiencies in the Marketplace of Ideas
(In the internet age)3. A Potential Partial Solution
Slide 3
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
1. Epistemology and the Marketplace of Ideas
Slide 4
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Logic, the quest for truth:
Socrates Plato Aristotle
Slide 5
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Logic, the quest for truth:
Socrates Plato Aristotle
Slide 6
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
The Destruction of Objectivity
David Hume (1711-1776) The Problem of Induction a question of how things behave
when they go “beyond the present testimony of the senses, and the records of our memory” [Hume1748].
The problem of Generalization (no-free-lunch).
Kurt Gödel (1906-1978) for any self-
consistent recursive axiomatic system powerful enough to describe the arithmetic of the natural numbers (for example Peano arithmetic), there are true propositions about the naturals that cannot be proved from the axioms.
Slide 7
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
No Free Lunch Thomas Mitchell (The Futility of Bias Free Learning):
• There are as many hypotheses consistent with past data that predict one outcome as that predict another off training set (when generalizing).
David Wolpert (The Supervised Learning No Free Lunch Theorem):• When all hypotheses are equally likely, there is no “best” supervised learning
algorithm off training set, for the 0-1 loss function.
James Carroll (No-Free-Lunch and Bayesian Optimality):• When all hypotheses are equally likely, then the “best” learning algorithm will report
that all outputs are equally likely off training set.
Slide 8
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Empirical Falsifiable You can’t prove which theory is true, so stop
trying. Theories should be used that are falsifiable, but
which have not yet been falsified after concerted effort.
Such theories will be useful.
Slide 9
Karl Popper (1902-1994)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
The Marketplace of Ideas
“Ideas that can get themselves accepted in a competitive market of ideas will tend to be of better quality than other ideas. The marketplace of ideas therefore improves the quality of our ideas and our thinking. If different
Slide 10
ideas are presented in an “ideas-market”, and if that market is populated by a maximum number of free agents expressing themselves freely, then those competing ideas will be exposed to a maximum number of supporting and dissenting arguments, and the balance of arguments in favor of or against an idea will be compared to the same balance for counter-ideas. The idea with the best balance will “survive”, because alternative ideas will be seen as comparatively defective, given the fact that the arguments in favor of them are weaker or the arguments against them are stronger.”
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
The Marketplace of Ideas This doesn’t just lead to ideas that are true, but to hypotheses that are
useful, in the sense of making the most general predictions about the most things, with the most accuracy.
These survive in the marketplace.
Slide 11
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Another Rout to the Marketplace: Rout 2… Induction…. Occam’s razor solves the problem of inference.
We compute p(h|data)…. But we didn’t personally collect all the data, ourselves Nor can we all do the complex inference for every problem There are more problems you could be an expert on than there are
seconds in your lifetime. SO… we have to specialize and trade. But some people lie, so… We have to determine who to trust, and what data to trust, and
ultimately what to believe. Result: The marketplace of ideas.
Slide 12
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
I, Pencil, My Family Tree as told to Leonard E. Read
Slide 13
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Collective Intelligence
Slide 14
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Inefficiencies in the Marketplace of Ideas:
1) the limited access to information, 2) the un-manageable volume of information, 3) the reliability of information, 4) the need to determine who to consider an expert, 5) the need to determine what the experts believe, 6) the difficulty in changing popular notions that are resistant to change, 7) the incivility, repetition, and lack of focus in discourse, and 8) the difficulty in overcoming confirmation bias.
Slide 15
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Current Cultural and Internet Mechanisms to Improve Marketplace Functionality:
1. Search Engines2. Peer Review3. Wikis4. Forums, Bulletin Boards, and Email Lists5. Survey Systems
Slide 16
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
The Canonizer Solution A wiki to leverage the wisdom of the crowd for knowledge collection and
summary• Wiki pages are organized into camps with differing opinions, instead of single pages
for each topic• Camps are organized into a hierarchical structure to encourage agreement
A mechanism for dynamically reorganizing the structure of the camp hierarchy while mediating user disagreements to encourage agreement and highlight areas of continuing disagreement
A survey system based upon camp support for determining consensus, and A mechanism for customizing how votes are weighted based on credentials or
expert assignment to allow users to explore different demographic effects, or to allow users to give extra weight to experts in different fields if they so choose.
A forum which also functions as an email list for discussion
Slide 17
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D Slide 18
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D Slide 19
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Weighting Votes (hierarchy of theories of consciousness):
Theories of Mind and
Consciousness
Approachable Via Science
Representational Qualia Theory
Property Dualism
Representational Functionalism
MisIdeal Monism
MisMis
Slide 20
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Weighting Votes (1 person 1 vote)
Theories of Mind and
Consciousness (39)
Approachable Via Science (36)
Representational Qualia Theory
(26.25)
Property Dualism (20)
Representational Functionalism
(4.75)
Mis(1.5)
Ideal Monism (2.25)
Mis (7.5)Mis (3)
Slide 21
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Weighting Votes (PhDs)
Theories of Mind and
Consciousness (3)
Approachable Via Science (3)
Representational Qualia Theory (2)
Property Dualism (1)
Representational Functionalism
(1)
Mis(0)
Ideal Monism (0)
Mis (1)Mis (3)
Slide 22
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Weighting Votes (“expert” philosophers of mind)
Theories of Mind and
Consciousness (10.41)
Approachable Via Science (10.41)
Representational Qualia Theory
(10.41)
Property Dualism (9.46)
Representational Functionalism
(0.94)
Mis(0.01)
Ideal Monism (0)
Mis (0)Mis (0)
Slide 23
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Weighting Votes (“expert” computer scientists)
Theories of Mind and
Consciousness (1.65)
Approachable Via Science (1.65)
Representational Qualia Theory
(1.65)
Property Dualism (0)
Representational Functionalism
(1.65)
Mis(0.00)
Ideal Monism (0)
Mis (0)Mis (0)
Slide 24
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
Deliverables/Results
1. A history of the debate and conversation2. A hierarchy of opinions, driven towards consensus and points of
mutual agreement3. A single page for each “camp” summarizing the arguments for and
against each position. 4. A survey of participants, showing which view has the most support5. A mechanism for varying how the votes are weighted to determine who
believes what
Slide 25
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
U N C L A S S I F I E D
References Hume, D., 1747, “An Enquiry Concerning Human Understanding,” p. 108 Thomas Mitchell, 1980, “The Need for Biases in Learning Generalizations”,
Rutgers Computer Science. David Wolpert, 2001, “The Supervised Learning No Free Lunch Theorem”
NASA Ames Research Center. James Carroll, 2007, “No-Free-Lunch and Bayesian Optimality” in Meta-
Learning IJCNN Workshop 2007.
Slide 26
Top Related