Prediction Markets

Leighton Vaughan WilliamsProfessor of Economics and Finance

Nottingham Business School

Nottingham Trent University

Leighton.Vaughan-Williams@ntu.ac.uk

How Well Do Markets Aggregate Information?

• How wise is the crowd?

Galton’s Ox

• In 1906, Sir Francis Galton (1822-1911), the English explorer, anthropologist and scientist, visited the West of England Fat Stock and Poultry Exhibition, where he came across a competition in which visitors could, for sixpence, guess the weight of an ox.

• Those who guessed closest would receive prizes.

• 800 people entered.

Galton’s crowd

• “Many non-experts competed, like those clerks and others who have no knowledge of horses, but who bet on races, guided by newspapers, friends, and their own” (Brief paper by Galton in ‘Nature’, March 1907).

• Reference:

• F. Galton, Vox Populi, Nature, 75,

• March 7, 1907.

Galton’s findings

• Galton added the contestants’ estimates and calculated the average of the estimates.

• Using the mean, the crowd had guessed that the ox (slaughtered and dressed) would weigh 1,197 pounds. In fact, the ox weighed 1,198 pounds.

• The median estimate was 1,207 pounds, not as close but within 1% of the correct weight.

Treynor’s Jelly Beans Experiment

• Jack Treynor, in a classic experiment, asked his class of 56 students to guess the number of jelly beans in a jar. The mean guess was 871.

• The actual number was 850. Only one student guessed closer.

• Reference: Jack Treynor (1987), ‘Market Efficiency and the Bean Jar Experiment’, Financial Analysts Journal, 43, 50-53.

• See also: Kate Gordon’s seminal study of 200 students estimating the weights of items. The group (average) result was 94.5% correct: only 5 students were better than this.

• Kate H. Gordon (1921), ‘Group Judgements in the Field of Lifted Weights’, Psychological Review, 28 (6), November, 398-424.

Webinar on Forecasting Excellence and Prediction Markets, Sept. 15, 2007.

• Joe Miles, a mathematician employed at ‘eyepharma’ (a company offering services to the pharmaceutical industry) gave a presentation, with the following key points.

• 1. He relayed the results of a ‘M&Ms in a jar’ experiment he had conducted with a large group of conference delegates at a pharmaceutical forecasting conference earlier that year. The estimates ranged from 381 to over 40,000! The median estimate was 1,789. The actual number was 1,747, just 2.4% off. The middle estimate was closer than any individual estimate.

• 2. He relayed the results of an experiment conducted by ‘eyetravel’, a sister company, at a hotel industry conference. Delegates were asked to estimate the average price of a hotel room in Amsterdam that day. Estimates ranged by a factor of three, but the average estimate was just 0.5% off (Mean estimate = 117.8 Euro: Actual price = 118.4 Euro.

What destroyed the space shuttle ‘Challenger’?

• On January 28, 1986, the space shuttle Challenger lifted off from its launch pad at Cape Canaveral. Seventy-four seconds later, it blew up. Within minutes, investors started dumping the stocks of the four major contractors who had participated in the Challenger launch: Rockwell International, which built the shuttle and its main engines; Lockheed, which managed ground support; Martin Marietta, which manufactured the ship's external fuel tank; and Morton Thiokol, which built the solid-fuel booster rocket. Within minutes, trading in Thiokol was suspended and by the end of the day, Thiokol's stock was down nearly 12 percent. By contrast, the stocks of the three other firms each fell a little but soon started to creep back up, and by the end of the day had fallen only around 3 percent. The market was right. Six months later and after an extensive investigation, Thiokol was held liable for the accident. The other companies were exonerated

How do you find a missing submarine?

• On the afternoon of May 27, 1968, the submarine USS Scorpion was declared missing with all 99 men aboard. It was known that she must be lost at some point below the surface of the Atlantic Ocean within a circle 20 miles wide. This information was of some help, of course, but not enough to determine even five months later where she could actually be found.

• The Navy had all but given up hope of finding the submarine when John Craven, who was their top deep-water scientist, came up with a plan which pre-dated the explosion of interest in prediction markets by decades. He simply turned to a group of submarine and salvage experts and asked them to bet on the probabilities of what could have happened. Taking an average of their responses, he was able to identify the location of the missing vessel to within a furlong (220 yards) of its actual location. The sub was found!

•What are Prediction Markets?

Betting on the outcome

• Betting markets aggregate all available information to produce best estimate, not least because those who know, and are best able to process the information, bet the most. Based on the ‘Efficient Markets Hypothesis’, the idea that markets accurately incorporate all relevant information.

Prediction markets v. Betting markets

• The essential difference between prediction and betting markets is not an issue of structure.

• Rather, prediction markets, as usually termed, are distinct from betting markets in the purpose to which they are put.

• For example, when betting markets are used explicitly to forecast the outcome of any event, whether it is the World Cup or a rowing regatta, they are essentially acting as prediction markets.

• Even so, the term ‘prediction markets’ often implies that the markets are being used to produce information externalities that can inform business and policy decisions.

The Hayek Question

• How does one effectively aggregate disparate pieces of information that are spread among many different individuals, information that in its totality is needed to solve a problem?

• Hayek’s answer was that market prices are the means by which those disparate pieces of information are aggregated.

• “The mere fact that there is one price for any commodity ... brings about the solution which ... might have been arrived at by one single mind possessing all the information which is, in fact, dispersed among all the people involved in the process.”

• Source: F.A. Hayek, ‘The Use of Knowledge in Society’, American Economic Review, 35, 4, Sept. 1945: 520.

Speed of the market in processing new information

• Obama price spiked one day in August, 2008, despite the only obvious news being a relatively poor opinion poll.

• Why?

Warp Speed Market – Saddam capture or neutralize

• Date: 13 December, 2003

• Market moves from about 20 to 100.

• Next day: News of Saddam capture announced by US.

Predicting the outcome of rowing regattas

• Jed Christiansen (2007) reports on markets set up to predict the outcome of rowing regattas in the UK. Despite the small number of participants, and the absence of any incentives other than the challenge of getting it right, the predictions of the rowing events were highly accurate.

• Christiansen puts the success of the experiment down to the effects of community and uniqueness, which encouraged motivated participation.

• Source: Christiansen, J.J. (2007), Prediction markets: Practical experiments in small markets and behaviours observed’, Journal of Prediction Markets, 1, 17-41.

Polls or markets?

• Predicting the winner of an election!

Early prediction markets

• The earliest data we have from prediction markets are those from organized markets for betting on the US Presidential election between 1868 and 1940.

• Although there are reports that these markets date back to the election of George Washington, and even before, the market in 1868 seems to be the first we would call a prediction market in that its data was used to inform the public about the likelihood of a particular candidate winning and may have been used by financial asset traders.

• As an example, the New York Times reported that between $500,000 and $1 million was wagered on the Curb Exchange (the fore-runner to the AMEX) in one day on the 1916 election and that “oil stocks were almost forgotten”. The total amount wagered in these markets in 1916 was $165 million (at 2002 prices).

• In this period between 1868 and 1940, the market failed to predict the winner on just one occasion.

AN EARLY BRITISH PREDICTION MARKET

•Brecon and Radnor By-Election, 1985.

•Mori v. Ladbrokes

ELECTION EVE

•MORI: Labour to win by 18%.

•LADBROKES:

•Liberal candidate: 4 to 7

•Labour candidate: 5 to 4

WINNERS:

•The Liberal candidate.

•Those who ignored MORI and backed the market favourite.

Bush v. Gore, 2000

IG Index v. Rasmussen

Outcome forecasts

•IG Index:

•265-275 Bush

•265-275 Gore

•Rasmussen: Bush by 9%

Opinion Polls v. markets

• Opinion polls, like all market research, provide a valuable source of information, but they are ONLY ONE source of information.

• Other information includes:

• 1. Local canvass returns

• 2. On-the-ground inside information

• 3. Forecasting models

• 4. Opinions of professional ‘pundits’ (‘experts’)

• 5. Focus groups

• Betting markets aggregate all the available information

Producing an optimal forecast

• Because those who know the most, and are best able to process that information, tend to bet the most, this drives the market to produce an optimal forecast at any point in time.

• Moreover, unlike polls, which are snapshots of opinion, betting markets are all about forecasting the eventual outcome.

• Since the advent of zero-tax low-margin betting exchanges, the accuracy of these markets have improved yet further.

US Presidential Election 2004

• INTRADE state-by-state predictions: 50 out of 50.

British General Election, 2005

• Predicted Labour majority to within a handful of seats.

US Senate 2006Intrade: All correct

US Presidential Election 2008

• Prediction markets

• INTRADE state-by-state predictions: 49 out of 50 (called Missouri wrong).

• BETFAIR state-by-state predictions: 49 out of 50 (called Indiana wrong).

• Statistical Modelling Using Weighted Polling Data

• FIVETHIRTYEIGHT predictions: 48 out of 50 (called Missouri and North Carolina wrong).

Indiana

• Polls closed at 11.30 pm (UK time) in Indiana, a key state which McCain almost certainly needs to win to secure the Presidency.

• McCain favourite to win Indiana on Betfair.

• 11.45 pm: Obama becomes favourite to win Indiana, attracting significant sums to win from traders.

• By this time CNN was calling just 1% of precincts in Indiana.

• So what caused the shift to Obama on Betfair?

• In retrospect, it seems that professional traders had latched on to the detail in the few published results.

• Importantly, this shows the power of prediction markets in assimilating and processing new information very rapidly.

Early Precinct Results

• Stueben: Kerry 34%, Obama 42%

• De Kalb: Kerry 31%, Obama 38%

• Knox: Kerry 36%, Obama 54%

• Marshall: Kerry 31%, Obama 50%

• Only the most well-informed had accessed these results by 11.45, and knew what they meant, i.e. a big swing from Republican to Democrat since 2004, but Betfair traders were among them. Minutes later, the swing was confirmed in Vigo County. By 12.20, Obama was shorter than 1 to 2 on Betfair.

The election was called at 4am, but Betfair watchers knew before midnight!

• At 4am, California was declared, giving Obama the final few electoral votes required to win the Presidency.

• At 2.30am Ohio was called by most news networks.

• Before midnight, the knowledge that Indiana was going to Obama, or at least that McCain would at best claim a small win there, was enough to indicate to Betfair watchers that the election was all but over. At 12.23 am, McCain was available at 25 to 1.

• Meanwhile, Fox News declared that Indiana was over-polling for Obama because it shares a border with his home state of Illinois!

• It was well past 3am when Fox News called the election for Obama.

• Betfair 1, Fox News 0.

2010 UK General Election

• It was the debates that lost it for the Conservatives!!!

• Before the first debate, the markets all predicted a Conservative overall majority.

• After the first debate, none of the markets ever predicted anything other than a hung parliament!

• While the polls swung all over the place, the markets barely flickered after that first debate in predicting a hung parliament with the Conservatives the largest party with somewhere between 300 and 320 seats.

Could prediction markets have prevented 9/11?

• The 9/11 Commission Report stated the problem like this: "The biggest impediment to all-source analysis - to a greater likelihood of connecting the dots - is the human or systemic resistance to sharing information.

• "What was missing in the intelligence community ... was any real means of aggregating not just information but also judgements. In other words, there was no mechanism to tap into the collective wisdom of National Security nerds, CIA spooks, and FBI agents. There was decentralization but not aggregation ...“ (James Surowiecki, ‘The Wisdom of Crowds’)

• Can the market can help achieve this? Some people within the US Department of Defence had been working on just such an idea for several months when al Qaeda struck. Indeed, in May 2001 the Defense Advanced Research Projects Agency (DARPA) had issued a call for proposals under the heading of 'Electronics Market-Based Decision Support' (later 'Future Markets Applied to Prediction' (FutureMAP).

FutureMAP (cont.)

• The remit prescribed for FutureMAP was to create market-based techniques for avoiding surprise and predicting future events. It was not long, however, before the US media and key members of the Congress began to train their guns on the idea of such a market. After all, it isn't difficult to portray the market as no more than a forum for eager traders to profit from death and destruction. The populist arguments won the day and DARPA was forced to cancel the project.

While most of the arguments against the market were emotional rather than intellectual, there was nevertheless some genuine intellectual concern as to how effective it would be likely to be.

Was Stiglitz right?

• In particular, Prof. Joseph Stiglitz argued in an article published in the Los Angeles Times on 31 July 2003 ('Terrorism: There's No Futures in It'), that the market would be too "thin" (i.e. there would be too little money traded in the market) for it to be a useful tool for predicting events meaningfully. His argument was based on work he had previously published showing that markets can never be perfectly efficient when information is costly to obtain. The cost of obtaining and processing this information is, by implication, likely to act as a significant disincentive particularly in the context of a thin market (and hence low rewards).

• But is it obviously the case that a properly constructed market, populated by suitably motivated (and perhaps screened) players can be viewed in this way?

Can prediction markets be used to study climate change?

• 1) A properly constructed market might encourage climate change analysts to become more specific in their forecasts, and would encourage the development of new modelling techniques. 2) The markets could help to provide an assessment of the tangible impact upon climate change of various policies under consideration by governmental and international bodies.3) The market could potentially help to establish a price for carbon. 4) The markets could help to price in new information more quickly. 5) The market would help businesses and governments to hedge against both the dangers of climate change, and against costs of addressing it.

•There could be a series of contracts and perhaps options on, for example, temperature, CO2 emissions, precipitation, and tropical storms which expired at various intervals.

Can prediction markets help us make flight plans?

• Volcanic ash cloud

• BA Strike

• Can we construct a prediction market which can amass the collective wisdom of the informed crowd to help us plan our future schedule?

Prediction Markets in Public Authorities

• A notable feature of public policy in the UK over the past decade has been the imposition by central governments of performance targets as a means of evaluating the performance of local public organisations.

• Targets cover a huge range of activities ranging from those specific to health or education to those relating to more general local authority performance.

• Targets are used as a means of evaluating performance, improving standards and allocating resources. The significance of achieving or not achieving particular targets can be very high for local politicians as well as senior managers in local authorities and health organisations in terms of both resources, public image. At the same time, it is extremely difficult for politicians and central managers to be aware of, let alone to process, the complex streams of information that are available

PMs in public authorities (cont.)

• Within this context, prediction markets offer a potentially valuable tool that may be used to synthesize the specific knowledge of those directly involved with implementing policy at a lower level. The specific nature of targets relating to, e.g., waiting list times, educational outcomes, are both specific and quantifiable and, hence, ideal candidates for operating a trading market. Taking the example of health care targets, the numbers of people involved from nurses, doctors to administrators further suggest that the operation of markets in this context is feasible.

• The value of the information provided by prediction markets will come primarily from the advance warning that politicians and managers will be given of weak performance in particular areas. This has the potential to improve resource allocation to make it more likely that key targets are met.

PMs for Public Policy Decisions

• Example: Should policy A or policy B be undertaken to reduce waiting lists?

• Current waiting list for an appointment at the eye clinic = 30 days.

• Contract pays £1 for the length of the waiting list in days. And currently trades at £30 pounds.

• Participants in the market can BUY the contract at £30 if they think the waiting list will increase and SELL if they think it will decrease.

• E.g. If they SELL at £30 and the waiting list decreases to 25 days, they will £5 (30-25). But if the waiting list increases to 35 days, they lose £5.

• By comparing the ‘Waiting list with policy A’ contract with the ‘Waiting list with policy B’ contract, the policy maker has gained information on what the ‘market’ thinks about the relative impacts of introducing policy A and policy B on the length of the waiting list.

• If a policy is not implemented, the contract is declared void.

Using the power of prediction markets for disease surveillance

• http://iehm.uiowa.edu/iehm/index.html

• “Reporting speed is one of the most import aspects of any surveillance program; for seasonal influenza even two weeks advance notice can have dramatic results on the effectiveness of vaccinations.

• Although there are many existing strategies for gathering opinions about the future trends of infectious diseases, the resulting data are often difficult to interpret using standard epidemiological methods. Prediction markets, on the other hand, are well known for their ability to quickly collect and summarize information.

• The Iowa Electronic Health Markets is a research project at the University of Iowa exploring the use of prediction markets as a tool for disease surveillance. By combining the strengths of prediction markets with the knowledge of our trading community from around the world, our hope is that these markets will report future infectious disease activity quickly enough to be clinically useful”.

Limitations of crowd wisdom

• Can the crowd predict the lottery numbers?

• If not, why not?

• Because lottery numbers are drawn randomly, no model or individual or crowd or other means of aggregating information can predict them because random numbers are by definition unpredictable.

• If the lottery numbers were, for whatever reason, not drawn randomly, however, we have a different issue.

Is Manipulation Bad for Prediction Markets?

• Robin Hanson and Ryan Oprea, of George Mason University and the University of California, Santa Cruz respectively, co-authored a paper title, 'A Manipulator Can Aid Prediction Market Accuracy‘. A perspective on its basic message is offered by Alex Tabarrok at Marginal Revolution. Tabarrok was considering the impact of the clear attempt by at least one determined trader to manipulate one of the US election betting markets in favour of Senator John McCain. In particular, the John McCain contract was bought in the markets systematically every morning by one US-based trader for sizeable sums. In consequence, it was possible to arbitrage between McCain (on Intrade) and Obama (on Betfair) for a few weeks in the run-up to Election 2008.

• How much of a danger, Tabarrock asks, does this sort of activity pose for the whole concept of prediction markets? Not much, he argues, instead offering support for Hanson and Oprea's finding that manipulation can actually improve prediction markets, for the simple reason that manipulation offers informed investors a free lunch.

Manipulation (cont.)

• "In a stock market", Tabarrok writes, "... when you buy (thinking the price will rise) someone else is selling (presumably thinking the price will fall) so if you do not have inside information you should not expect an above normal profit from your trade. But a manipulator sells and buys based on reasons other than expectations and so offers other investors a greater than normal return. The more manipulation, therefore, the greater the expected profit from betting according to rational expectations.“

• For this reason, investors should soon move to take advantage of any price discrepancies thus created within and between markets, as well as to take advantage of any perceived mispricing relative to fundamentals. Thus the expected value of the trading is a loss for the manipulator and a profit for the investors who exploit the mispricing. Moreover, the incentive the activity of the manipulator gives for others to become informed, and to trade on the basis of this information, is valuable in itself in improving the efficiency of the market.

Worth manipulating?

• Tabarrok offers the additional observation that, considerations of predictive accuracy aside, there is one even more important lesson to be learned from the activities of the manipulators: "...that prediction markets have truly arrived when people think they are worth manipulating".

•But have they? What does the corporate sector think?

HOW CAN COMPANIES USE PREDICTION MARKETS?

• To take an example, a manufacturer of aero engines will seek good forecasts of future orders from plane manufacturers, which in turn will be contingent upon orders from airlines. Forecasts of future airline orders will be greatly assisted by the collation of a range of information from those involved in each of these sectors.

• It is important that the questions posed are in a form which is unambiguous and which can ultimately be quantified. This requires an assessment of who should be involved in responding, and ensuring that each of these contributors has an equivalent understanding of the meaning of what is being asked, and that these answers can usefully be pooled. The set-up will vary depending on the diversity of contributors, both geographically and functionally. There is also the issue of incentives and the number of markets to run, as well as the length of these markets and how often new markets should be introduced.

• But in principle markets should be able to help aggregate information.

But what’s the evidence? Can prediction markets actually help internal company forecasting?

• There is in fact plenty of published research showing how internal prediction markets have helped improve the ability of commercial organisations to structure and implement internal prediction markets to assist in forecasting.

• to predict key business variables

- e.g. when will a product launch, what will be the unit sales?

• broader-based prediction markets are a useful mechanism for predicting market-wide outcomes, e.g. box office receipts for a new film, success of a new video game, property prices.

Commercial examples

• Eli Lilly ran an experiment in which managers traded through an internal market the future monthly sales figures for three drugs. The market brought together all the information, from toxicology reports to clinical results, and produced more accurate forecasts than the official forecasts.

• Google have set up a market, in which any Google staff member could bet on the chances of an event coming true. The markets were used to forecast such things as product launch dates and new office openings. The results - based on the aggregated bets of thousands of Google staff members - were strong predictors of the actual outcomes.

How do these markets operate?

• Essentially, participants in the market exchange offers and counter-offers until they agree on a contract price.

• Trades are executed when two prices match.

• In describing how experimental internal-prediction market run by Eli Lilly (the pharmaceutical company) work, VP for Lilly Research Laboratories Alpheus Beingham noted:

• “When we start trading [in the drug] and I try buying your stock cheaper and cheaper, it forces us to a way of agreeing that never really occurs in any other kind of conversation”.

Corporate Applications of Prediction Markets: Special Issue of the JPM

• The Journal of Prediction Markets (2009)

• www.thejpm.com

• Guest editor: Prof. Koleman Strumpf, member of the Editorial Board of the Journal of Prediction Markets, and Koch Professor of Economics at the University of Kansas School of Business.

• Based on presentations at the ‘Conference on Corporate Applications of Prediction/Information Markets’, held at Kansas City’s Kauffman Foundation on 1 November, 2007.

What are Corporate Prediction Markets? Editorial Introduction

• ‘Prediction Markets use the knowledge of a pool of individuals to help forecast questions of importance to companies, such as whether a sales target will be reached or whether a project will be completed in a timely manner. A more recent development is the use of such markets to generate and evaluate new ideas, such as new products or cost saving procedures.

• Since first being applied within the corporate sector over a decade ago, over a hundred companies have run internal markets. These range in size from some of the largest in the world to those with only a handful of employees, and cover a broad range of sectors, from those whose products are abstract ideas to others which manufacture very low-tech products’.

Why Have Such a Broad Range of Firms Become Interested in Prediction Markets?

• ‘The answer lies in a common problem facing firms, namely the isolation of executives from the views and insights of the company’s workforce.

• Such seclusion is no accident but instead reflects one of the reasons companies are structured as they are in the first place, i.e. to avoid information overload for busy executives.

• To reach this goal firms developed a hierarchy structure, and assigned to middle management the task of deciding how much and what information was transmitted from employees to higher-level decision-makers. But the system has its costs, as potentially useful information may be filtered out if it reflects poorly on those who control the information flow. At the same time, lower-level employees have little incentive to make reports which conflict with their managers. The net result is that executives may only receive one-sided information, and flawed decisions may result’.

This is where Prediction Markets Come In!

• ‘Suppose the CEO must decide whether to continue funding a project, but is concerned that he has been receiving overly optimistic reports on its prospects from managers who will benefit from the project continuing.

• A market on the project’s prospects would allow front-line employees to convey more realistic information, and they could do so without fear of reprisal if the trading is anonymous.

• Prediction markets may also function better than other approaches currently in use. For example, group meetings are less likely to have frank discussions while suggestions boxes do not scale well - prediction markets tend to perform better when there are more participants.

• And while most workers may dread the thought of meetings, markets are often considered a fun activity and often do not require much in the way of incentives to generate active employee involvement’.

A Case Study of GE (JPM, 2009), by Brian Spears (Risk Manager, GE-Hitachi Nuclear Energy) et al.

• ‘1. Internal markets used to aggregate opinions are consistent with opinions collected via web surveys (Chan et al, 2002). Markets may in fact improve upon traditional survey methods by encouraging greater honesty from the participants, providing participants with valuable feedback from other participants, and offering participants the “joy of competitive play”.

• 2. GE’s markets are designed to help answer business questions such as ‘What new technology ideas should we be investing in?’ ‘What new products should we be developing?’

• 3. Market participants can submit their own ideas for entry into the market, and they can buy and sell shares of any idea in the market’.

GE Case Study (cont.)

• “GE’s interest in idea markets stem from our belief that innovative new product and service ideas can come from anywhere within an organization”.

• ‘Similar to most companies, GE uses a variety of methods to generate and down-select new ideas. Traditional means include suggestion boxes and brainstorming sessions.

• However, suggestion boxes often go unused because contributors receive little or no feedback about their idea or visibility into others’ ideas.

• Brainstorming sessions are often infeasible for soliciting ideas from large, globally distributed teams with potentially thousands of contributors.

• Hence the idea of a prediction market, which they call an ‘Imagination Market’ was born’.

Conclusions of Case Study

• “Overall, the GE Energy business was extremely pleased with the results of the Imagination Market.

• Funding was immediately provided to kick-start the two ideas tied for the top, and the business has decided to file patents for several others.

• GE Energy plans to continue use of markets in the future.

• The volume and quality of ideas compared favourably to brainstorming sessions, on-line suggestion boxes, and on-line discussion forums”.

• NB The Spears paper provides a wealth of information and detail about the GE markets, including a very detailed description of the mechanics of the markets, such as the incentives given to traders and to creators of new ideas.

Improving Forecasting Accuracy in Corporate Prediction Markets – A Case Study in the Austrian Mobile Communication Industry – JPM, 3, 2009, by Martin Waitz and Andreas Mild

• ABSTRACT

• Corporate prediction markets forecast business issues like market shares, sales volumes or the success rates of new product developments.

• The improvement of its accuracy is a major topic in prediction market research ...

• We propose a method that aggregates the data provided by such a prediction market in a different way by only accounting for the most knowledgeable market participants.

• We demonstrate its predictive ability with a real world experiment.

Why companies are well positioned to utilize the information generated from PMs

• ‘1. Company divisions often serve as standalone silos, and markets can be a means of integrating the pockets of information contained in each.

• 2. Executives may be interested not just in market aggregates, such as prices, but also the trades of particular groups of employees. For example, one could examine whether members of certain divisions are less prone to making biased forecasts.

• 3. Companies need real-time information about the many uncertain events surrounding their decision-makers.

• 4. Firms can internalize the informational benefits of the market. A company can profit from the information generated from prices, since the market can be kept private and outside of the purview of competitors.

• The last point is particularly important. Since the benefits of the markets largely accrue to the company, we might expect many prediction market innovations to first arise in a corporate setting’.

Key challenges facing PMs

• 1. Operators must overcome investor reluctance to a project with upfront costs and possibly delayed benefits.

• 2. There are impacts on employees, both detrimental (markets may distract staff away from their main responsibilities) and beneficial (there is often a gain in morale, as workers feel empowered because their market-mediated suggestions are impacting corporate decisions.

• 3. The markets may overwhelm executives with too much information.

• 4. Market organizers must allay concerns of middle management and those whose current role in the company is threatened by the market.

• 5. There may be systematic biases in some markets.

Optimism bias

• Optimism bias is the systematic tendency for people to be over-optimistic about the outcome of planned actions. This includes over-estimating the likelihood of positive events and under-estimating the likelihood of negative events".

• David Armor and Shelley Taylor highlight a number of examples of what they consider to be optimism bias in an interesting paper called 'When Predictions Fail: The Dilemma of Unrealistic Optimism', published in 2002.

• Examples include students' estimates of the likely starting salary of their first job in the graduate market and newlyweds' thoughts on how long their marriage will last. It is interesting, therefore, that evidence of the existence of this very same bias has been identified in 'internal' company prediction markets, notably in a 2008 paper co-authored by Bo Cowgill, of Google, Justin Wolfers of the Wharton School and Eric Zitzewitz, based at Dartmouth College.

Optimism bias (cont.)

• Cowgill, Wolfers and Zitzewitz examine the results generated by what they call the Google corporate prediction market experiment. The primary goal of these markets is, as they put it, to generate predictions that efficiently aggregate many employees' information and augment existing forecasting methods.

• In support of previous investigations into the value of internal prediction markets, they were able to confirm that prices in the Google markets closely approximated event probabilities, i.e. that the markets were reasonably efficient. Even so, they were not perfect, and one notable reason was an apparent 'optimism bias' which, according to their findings, "was more pronounced for subjects under the control of Google employees, such as whether a project would be completed on time or whether a particular office would be opened."

Optimism bias (cont.)

• Optimism bias was also found to be more evident in new employees and in the immediate few days following a good news day for the Google stock price. Still, what is a cost in terms of unadjusted predictive efficiency may be a benefit in terms of motivation and entrepreneurial zeal, a feedback mechanism the value of which it is perhaps easy to under-estimate.

• In any case, if we are able to identify and measure the source and extent of the bias, it should be possible to adjust and compensate for this particular inefficiency in generating the objective forecasts.

The Favourite-Longshot Bias

• Let the probability of an event occurring be 20%.

• Standard approach: Probability = 0.2

• Bookmaker’s approach: Odds = 4 to 1. This means you win £4 (net) from the bookmaker if your bet wins for every £1 staked (risked) with the bookmaker.

• Which yields the better expected return, a stake of £10 on a horse with odds of 2 to 1 or a stake of £10 on a horse with odds of 20 to 1?

• i.e. if Mr. A and Mr. B both start with £1,000. Now Mr. A places a level £10 stake on 100 horses quoted at 2 to 1 and Mr. B places a level £10 stake on 100 horses quoted at 5 to 1. Who is likely to end up richer?

Odds Backing – Ladbrokes Pocket Companion, Flat Edition, 1990, pp. 242-243

• Not one out of 35 favourites sent off at 1/8 or shorter (as short as 1/25) lost between 1985 and 1989. This means a return of between 4% and 12.5% in a couple of minutes, which is an astronomical rate of interest... The point being made is that broadly speaking the shorter the odds, the better the return. More broadly, the group of ‘white hot’ favourites (odds between 1 to 5 and 1 to 25) won 88 out of 96 races for a 6.5% profit. The following table looks at other odds groupings.

• Odds Wins Runs Profit %

• 1/5-1/2 249 344 +£1.80 +0.52

• 4/7-5/4 881 1780 -£82.60 -4.64

• 6/4 -3/1 2187 7774 -£629 -8.09

• 7/2-6/1 3464 21681 -£2237 -10.32

• 8/1-20/1 2566 53741 -£19823 -36.89

• 25/1 -100/1 441 43426 -£29424 -67.76

An Election Super-Bias?

The 2004 US Presidential state-by-state markets gave the equivalent of 50 successive winning favourites at the racetrack.

In 2008, the Betfair favourites won 49/50 states.

The Intrade favourites won 49/50.

Biases

• Do biases differ between different prediction market formats?

• Can we compensate for the biases to yield more accurate forecasts?

• Do some formats yield more volatile forecasts than others?

Some further examples of PMs

• Best Buy, the electronics retailer. Experimented with prediction markets on everything from demand for digital set-top boxes to store opening-dates.

• E.g. in Autumn 2006, the price in one of their PMs on whether a new store in Shanghai would open on time several weeks ahead dropped sharply from $80 a share to about $45. Players made yes-no bets, and the virtual dollar drop reflected increasing doubt that the store would open on time. The store opened a month late.

• Jeffrey Severts, a VP who oversees PMs at Best Buy, quoted in NY Times (April 9, 2008 – ‘Betting to Improve the Odds’):

• “The potential is that prediction markets may be the thing that enables a big company to act more like a small, nimble company again.”

• “It helps on two fronts, the speed and accuracy of information, so that management can move faster to deal with problems or exploit opportunities.”

Best Buy (cont.)

• “Severts invited several hundred employees to submit an estimate for sales for a one-month period. To help them calibrate their estimate, he provided monthly data from the past twelve months. 192 employees responded, including those on the store floor. The estimates were given equal weight and averaged. He found that the employees’ collective wisdom had an error of only 0.5% compared to an error of 5% by the traditional forecasting method.

• Severts went on to experiment with total sales over the 14 week holiday period. He provided last year’s sales figure from the holiday period and revenue growth for the first three months of the current fiscal year compared to previous years. The original 350 respondents predicted sales during the fourteen-week holiday period that was 99.9% accurate. The merchants themselves who were traditionally responsible for forecasting were 93% accurate”.

• (Hamel, 2007, The Future of Management, Boston: Harvard Business School Press).

PREDICTION MARKETS AS A MEDICALFORECASTING TOOL: DEMAND FOR HOSPITALSERVICES – David Rajakovich and Vladimir Vladimirov (JPM, 2009)

• This paper presents the outcome of a study conducted at the Royal Devon and Exeter Hospital in which a prediction market was established in order to forecast demand for services.

• The study was conducted over a period of one week, and involved 65 participants. Each was asked to provide an estimate for demand for services at the Royal Devon and Exeter Hospital. In each survey, each employee was asked to estimate the number of patients that would be admitted to each directorate, which meant that employees within each directorate were estimating the number of patients admitted to directorates other than their own.

Findings

• The overall results confirmed the effectiveness of prediction markets.

• The prediction for admittances was 1157.51 while the actual number of admittances was 1154, which is an error of only 0.3%.

• Market participants were almost exactly right in the Medicine directorate, predicting 353.38, while the actual was 353.

• Specialist Surgery’s prediction was almost as accurate with an actual number of admittances of 106 and an estimate of 107.75.

• However, the prediction market was less successful in predicting demand for services for each department, which the paper attributes to the small sample size and lack of diversity of participants.

Further examples

In 2007, a group in the purchasing unit at Hewlett Packard began prediction markets on the price of computer memory chips three and six months ahead.

Bernardo A. Huberman, director of the social computing lab at Hewlett-Packard: “The prediction markets were up to 70% more accurate than the company’s traditional forecasting model ... The more accurate predictions can be used to finesse purchasing, marketing and product pricing decisions.”

The Classic Study

• Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem, by Charles Plott (CalTech) and Kay-Yut Chen (Hewlett-Packard Laboratories), 2002.

• “Many business examples share the following characteristic: small bits and pieces of relevant information exists in the opinions and intuition of individuals who are close to an activity. Some examples are supply chain management issues, demand forecasting, new product introduction, and supply uncertainties. In many instances, no systematic methods of collecting such information exist. In these cases very little is known by any single individual but the aggregation of the bits and pieces of information might be considerable. For instance, it is extremely difficult to combine subjective information such as the knowledge of a competitor’s move with objective information such as historical data. In a perfect world, with unlimited time and resources, a user of such information could personally interview everyone that might have a

• relevant insight but such luxury does not exist”.

Chen and Plott (cont.)

• “Gathering the bits and pieces by traditional means, such as business meetings, is highly inefficient because of a host of practical problems related to location, incentives, the insignificant amounts of information in any one place, and even the absence of a methodology for gathering it. Furthermore, business practices such a quotas and budget settings create incentives for individuals not to reveal their information. The principles of economics together with new technologies that exist for creating markets and related mechanisms suggest that in might be possible to develop a new approach that avoids many of the practical problems”.

Some details of the experiment

• The experiments were conducted with three different HP divisions. Typically, around 20-30 people signed up for the experiments. Trading was done through at a web server located at Caltech. The subjects were geographically dispersed in California.

• Typically, the prediction was for monthly sales for a month three months in the future. The market mechanism employed to support the markets was the web based markets of the Marketscape software, which was

• developed at the Laboratory of Economics and Political Science at Caltech. All the markets for an event were organized on a single web page for easy access.

• A participant could enter a buy offer, a sell offer or acceptance of an offer through the web form on the page. Orders were compared to the other side immediately. The best offers were listed on the main market web page. The whole book of offers was available for each market at the click of a button.

Results

• Market predictions based on IAM (Information Aggregation Mechanism) prices outperformed official HP forecasts.

• In events for which official forecasts were available the IAM predictions were closer to the actual outcome than the official forecast 75% of the time. The absolute % errors of the official forecasts were also significantly higher than that of the IAM predictions.

Other general applications of PMs

• Examples of the applications of prediction markets range from traditional finance forecasting such as sales and costs, to product development support such as forecasting on-time project delivery or the likelihood of regulatory approval for new drugs, to innovative decision support such as evaluating the impact of switching advertising agencies or forecasting the market receptivity of new software releases

• Etc

• Etc

• Etc.

It’s not just about forecasting!

• A prediction-market pilot at Microsoft in 2005 was designed to forecast the probability of on-time release for several products.

• To management’s surprise, the stock price representing on-time release dropped to zero, despite the staff’s prior assurance that on-time release was likely.

• The ensuing conversation uncovered the true beliefs of the programmers, a result perhaps even more valuable than knowing whether the release would be missed.

• Source: Todd Proebsting, of Microsoft Research, in his presentation ‘Tee Time with Admiral Poindexter’, delivered at the DIMACS Workshop on Market as Predictive Devices (Information Markets), February, 2005, Rutgers University.

Conclusion

• Prediction markets offer major unexploited opportunities to aggregate information in a rapid and efficient manner. There are significant examples of success where they have been tried, although in some cases they have been tried and then discarded.

• In an interview with DIRECTOR magazine, I said this:

• “The real problem where these markets have not worked as well as management expected is that the companies have simply bought the technology and more or less expected it to take care of itself. I don’t want to stretch the point but this can be compared to buying a car and expecting it to drive itself. Of course you’d be disappointed with performance”.

• And yes – cars have design and performance glitches, as do PMs.

• But cars can be useful and wonderful things.

• The same, may I say it, goes for Prediction Markets.

Leighton.Vaughan-Williams@ntu.ac.uk