Supplementary Materials

Here we provide additional information about our experiment.

A. Forecasting Questions.

The 199 questions used in our experiment are shown below. Options are provided for all questions that were not binary - Yes or No - responses. 1001 Will the Six-Party talks (among the US, North Korea, South Korea, Russia, China, and Japan) formally resume in 2011? 1002 Who will be inaugurated as President of Russia in 2012? (a) Medvedev, (b) Putin, (c) Neither1003 Will Serbia be officially granted EU candidacy by 31 December 2011?(a) Yes, (b) No1004 Will the United Nations General Assembly recognize a Palestinian state by 30 September 2011?1005 Will Daniel Ortega win another term as President of Nicaragua during the late 2011 elections? 1006 Will Italy restructure or default on its debt by 31 December 2011? (a) Yes, (b) No1007 Will there be a lethal confrontation involving government forces in the South China Sea or East China Sea by 31 December 2011? (a) Yes, by 15 October 2011, (b) Yes, between 16 Oct and 31 Dec, (c) No1008 By 31 December 2011, will the World Trade Organization General Council or Ministerial Conference approve the "accession package" for WTO membership for Russia? (a) Yes, (b) No1009 By 1 January 2012 will the Iraqi government sign a security agreement that allows US troops to remain in Iraq? (a) Yes, by 15 October 2011, (b) Yes, between 16 October and 1 Jan, (c) No1010 Will the 30 Sept 2011 “last” PPB for Nov 2011 Brent Crude oil futures* exceed $115?1011 Will the Nikkei 225 index finish trading at or above 9,500 on 30 September 2011? 1012 Will Italy's Silvio Berlusconi resign, lose re-election/confidence vote, OR otherwise vacate office before 1 October 2011?1013 Will the London Gold Market Fixing price of gold (USD per ounce) exceed $1850 on 30 September 2011 (10am ET)? 1014 Who will win the September 2011 Guatemalan presidential election, or will a run-off be needed? (a) Otto Perez Molina (b) Rigoberta Menchu (c) Other or Run-Off1015 Will Israel's ambassador be formally invited to return to Turkey by 30 September 2011?1016 Will PM Donald Tusk's Civic Platform Party win more seats than any other party in the October 2011 Polish parliamentary elections? 1017 Will Robert Mugabe cease to be President of Zimbabwe by 30 September 2011?1018 Will Muqtada al-Sadr formally withdraw support for the current Iraqi government of Nouri al-Maliki by 30 September 2011?1020 Will peace talks between Israel and Palestine formally resume at some point between 3 October 2011 and 1 November 2011? 1021 Will the expansion of the European bailout fund be ratified by all 17 Eurozone nations before 1 November 2011?

1022 Will the South African government grant the Dalai Lama a visa before 7 October 2011?1023 Will former Ukrainian Prime Minister Yulia Tymoshenko be found guilty on any charges in a Ukrainian court before 1 November 2011?1024 Will Abdoulaye Wade win re-election as President of Senegal?1025 Will the Freedom and Justice Party win at least 20 percent of the seats in the first People's Assembly (Majlis al-Sha’b) election in post-Mubarak Egypt?1026 Will Joseph Kabila remain president of the Democratic Republic of the Congo through 31 January 2012?1027 Who will win the January 2012 Taiwan Presidential election?(a) Ma Ying-jeou (KMT candidate), (b)Tsai Ing-wen (DPP candidate), (c) Neither1028 Will Moody’s issue a new downgrade of the sovereign debt rating of the Government of Greece between 3 October 2011 and 30 November 2011?1030 Will the UN Security Council pass a measure/resolution concerning Syria in October 2011?1031 Will the U.S. Congress pass a joint resolution of disapproval in October 2011 concerning the proposed $5+ billion F-16 fleet upgrade deal with Taiwan?1032 Will the Japanese government formally announce the decision to buy at least 40 new jet fighters by 30 November 2011? 1033 Will the Tunisian Ennahda party officially announce the formation of an interim coalition government by 15 November 2011? 1034 Will Japan officially become a member of the Trans-Pacific Partnership before 1 March 2012?1035 Will the United Nations Security Council pass a new resolution concerning Iran by 1 April 2012?1036 Will media reports indicate that Iraq has experienced a military coup (or attempted coup) before 1 June 2012?1037 Will Greece remain a member of the EU through 1 June 2012? 1038 Will Hamad bin Isa al-Khalifa remain King of Bahrain through 31 January 2012?1039 Will Bashar al-Assad remain President of Syria through 31 January 20121040 Will Aleksandr Lukashenko remain president of Belarus through 30 June 2012?1041 Will Italy's Silvio Berlusconi resign, lose re-election/confidence vote, OR otherwise vacate office before 1 January 2012? 1042 Will Lucas Papademos be the next Prime Minister of Greece?1043 Will Lucas Papademos resign, lose re-election/confidence vote, or vacate the office of Prime Minister of Greece before 1 March 2012?1044 Will the United Kingdom's Tehran embassy officially reopen by 29 February 2012?1045 Will a trial for Saif al-Islam Gaddafi begin in any venue by 31 March 2012?1046 Will Australia formally transfer Uranium to India by 1 June 2012?1047 Will a foreign or multinational military force fire on, invade, or enter Iran before 1 September 2012?1048 Will S&P downgrade the AAA long-term credit rating of the European Financial Stability Facility (EFSF) by 30 March 2012? 1049 Will Asif Ali Zardari resign, lose re-election/confidence vote, or vacate the office of President of Pakistan before 1 June 2012?1050 Will the next Palestinian general election commence by 1 June 2012?

1051 Will Mario Monti resign, lose re-election/confidence vote, or vacate the office of Prime Minister of Italy before 1 January 2013?1052 Will Aung San Suu Kyi be sworn in or seated as a member of Parliament in Myanmar (Burma) before 1 January 2013?1052 By 31 December 2012, will the UK officially announce its intention* to withdraw from the EU?1054 Will a North Korean or multinational military force fire on, invade, or enter South Korea before 1 June 2012?1055 Will North Korea successfully detonate a nuclear weapon, either atmospherically, underground, or underwater, between 9 January 2012 and 1 April 2012?1056 Will €1 Euro buy less than $1.20 US dollars at any point before 1 January 2013?1057 Will the Russian military deploy* additional Iskander missiles before 1 February 2013?1058 By 1 April 2012, will Egypt officially announce its withdrawal from its 1979 peace treaty with Israel?1059 Will Kim Jong-un attend an official, in-person meeting with any G8 head of government* before 1 April 2012?1060 Will Christian Wulff resign or vacate the office of President of Germany before 1 April 2012?1061 Will Syria’s Arab League membership be reinstated* by 31 December 2012?1062 Will the daily Europe Brent Crude FOB spot price per barrel be greater than or equal to $150 before 3 April 2012?1063 Will the Taliban begin official* in-person negotiations with either the US or Afghan government by 1 April 2012?1064 Will Yousaf Raza Gillani resign, lose confidence vote, or vacate the office of Prime Minister of Pakistan before 1 April 2012? 1065 Will there be a significant* lethal confrontation involving government forces in the South China Sea or East China Sea between 23 January 2012 and 31 December 2012?1066 Will Yemen's next presidential election commence before 1 April 2012?1067 Will Traian Basescu resign, lose referendum vote, or vacate the office of President of Romania before 1 April 2012?1068 Will the UN Security Council pass a new measure/resolution directly concerning Syria between 23 January 2012 and 31 March 2012?1069 Before 1 April 2012, will South Korea officially* announce a policy of reducing Iranian oil imports in 2012?1070 Will Israel release Palestinian politician Aziz Duwaik from prison before 1 March 2012?1071 Will Iran and the U.S. commence official nuclear program talks* before 1 April 2012?1072 Will Serbia be officially granted EU candidacy before 1 April 2012? 1073 Will the IMF officially announce before 1 April 2012 that an agreement has been reached to lend Hungary an additional 15+ Billion Euros?1074 Will Libyan government forces regain control* of the city of Bani Walid before 6 February 2012?1075 Will a run-off be required in the 2012 Russian presidential election?1076 Will the Iraqi government officially announce before 1 April 2012 that it has dropped all criminal charges against its VP Tareq al-Hashemi?1077 Will Egypt officially announce by 15 February 2012 that it is lifting* its travel ban on Americans currently in Egypt?

1078 Will a Japanese whaling ship enter Australia's territorial waters between 7 February 2012 and 10 April 2012?1079 Who will win the 2012 Venezuela presidential primary for a unified opposition candidate to Hugo Chavez? (a) Pablo Perez, (b) Henrique Capriles Radonski, (c) Diego Arria, (d) Maria Corrina Machado, (e) Other1080 Will the Republic of Macedonia* be a NATO member before 1 April 2013?1081 Will the Nigerian government and Boko Haram commence official talks before 31 December 2012?1082 Will William Ruto cease to be a candidate for President of Kenya before 10 April 2012?1083 Will Marine LePen cease to be a candidate for President of France before 10 April 2012?1084 Who will win Venezuela’s 2012 presidential election?(a) Hugo Chávez, (b) Henrique Capriles Radonski, (c) Other1085 Between 21 February 2012 and 1 April 2012, will the UN Security Council announce any reduction of its peacekeeping force in Haiti?1086 Will Mohamed Waheed Hussain Manik resign or otherwise vacate the office of President of Maldives before 10 April 2012?1087 Will Amr Moussa win the next Egyptian presidential election? 1088 Will Japan commence parliamentary elections before 1 April 2012?1089 Will Iran successfully detonate a nuclear device, either atmospherically, underground, or underwater before 1 January 2013?1090 Before 13 April 2012, will the Turkish government officially announce that the Turkish ambassador to France has been recalled?1091 Will Standard and Poor's downgrade Japan's Foreign Long Term credit rating at any point between 21 February 2012 and 1 April 2012?1092 Will Myanmar release at least 100 more political prisoners between 21 February 2012 and 1 April 2012?1093 Will at least one Taliban representative be appointed to serve as a minister in the Afghan government before 1 January 2013?1094 Will Zimbabwe commence a presidential election before 1 January 2013? 1095 Will there be a significant* outbreak of H5N1 in China in 2012?1096 Will a civil war break out in Syria between 21 February 2012 and 1 April 2012?1097 Will Tunisia officially announce an extension of its current state of emergency before 1 April 2012?1098 Will a foreign or multinational military force fire on, invade, or enter Syria between 6 March 2012 and 31 December 2012?1099 Before 1 April 2012, will Al-Saadi Gaddafi be extradited to Libya?1100 Will the Colombian government and FARC commence official talks before 1 January 2013?1101 Will Laurent Lamothe be ratified as Haiti’s Prime Minister before 1 April 2012? (a) Yes, (b) No, but he will remain the nominee, (c) No, he will cease to be a nominee*1102 Will Ireland ratify the European Fiscal Compact in a referendum vote before 1 October 2012?1103 Before 1 April 2012, will the Sudan and South Sudan governments officially announce an agreement on oil transit fees?1104 Will Yemeni government forces regain control of the towns of Jaar and Zinjibar from Al-Qaida in the Arabian Peninsula (AQAP) before 1 April 2012?

1105 Will Standard and Poor's downgrade the United Kingdom’s Foreign Long Term credit rating at any point between 18 June 2012 and 1 April 2013?1106 Will Kim Jong-un resign or otherwise vacate the office of Supreme Leader of North Korea before 1 April 2013?1107 When will an Egyptian Referendum vote approve a new constitution? (a) Between 1 Jul 2012 and 30 Sep 2012, (b) Between 1 Oct 2012 and 31 Dec 2012, (c) Between 1 Jan 2013 and 31 Mar 2013, (d) Event will not occur before 1 April 20131108 When will Japan officially become a member of the Trans-Pacific Partnership? (a) Between 1 Jul 2012 and 30 Sep 2012, (b) Between 1 Oct 2012 and 31 Dec 2012, (c) Between 1 Jan 2013 and 31 Mar 2013, (d) Event will not occur before 1 April 20131109 Will any country officially announce its intention to withdraw* from the Eurozone before 1 April 2013?1110 When will North Korea successfully detonate a nuclear device, either atmospherically, underground, or underwater? (a) Between 1 Jul 2012 and 30 Sep 2012, (b) Between 1 Oct 2012 and 31 Dec 2012, (c) Between 1 Jan 2013 and 31 Mar 2013, (d) Event will not occur before 1 April 20131111 When will the UN announce that Iran has signed an official nuclear monitoring deal with the UN? (a) Between 1 Jul 2012 and 30 Sep 2012, (b) Between 1 Oct 2012 and 31 Dec 2012, (c) Between 1 Jan 2013 and 31 Mar 2013, (d) Event will not occur before 1 April 20131112 When will Bashar al-Assad resign or otherwise vacate the office of President of Syria?(a) Between 1 Jul 2012 and 30 Sep 2012, (b) Between 1 Oct 2012 and 31 Dec 2012, (c) Between 1 Jan 2013 and 31 Mar 2013, (d) Event will not occur before 1 April 20131113 When will Viktor Orbán resign or otherwise vacate the office of Prime Minister of Hungary? (a) Between 1 Jul 2012 and 30 Sep 2012, (b) Between 1 Oct 2012 and 31 Dec 2012, (c) Between 1 Jan 2013 and 31 Mar 2013, (d) Event will not occur before 1 April 20131114 Will Raja Pervez Ashraf resign or otherwise vacate the office of Prime Minister of Pakistan before 1 April 2013?1115 Will the UN Security Council pass a new resolution before 1 April 2013 that supports military intervention* in Mali?1116 Will Libya commence legislative elections before 8 July 2012?1117 Will Fayez al-Tarawneh resign or otherwise vacate the office of Prime Minister of Jordan before 1 January 2013?1118 Who will win Sierra Leone’s next Presidential election? (a) Ernest Bai Koroma, (b) Julius Maada Bio, (c) Other candidate1119 When will Libya name a new prime minister? (a) Between 16 Jul 2012 and 31 Jul 2012, (b) Between 1 Aug 2012 and 31 Aug 2012, (c) Between 1 Sep 2012 and 30 Sep 2012, (d) Event will not occur before 1 Oct 20121120 Will Spanish government generic 10-year bond yields equal or exceed 7% at any point before 1 September 2012?1121 Will the Palestinian group Islamic Jihad significantly violate its cease-fire with Israel before 30 September 2012?1122 When will Nouri al-Maliki resign, lose confidence vote, or vacate the office of Prime Minister of Iraq? (a) Between 16 Jul 2012 and 30 Sep 2012, (b) Between 1 Oct 2012 and 31 Dec 2012, (c) Between 1 Jan 2013 and 31 Mar 2013, (d) Event will not occur before 1 April 20131123 Will the Romanian people approve the removal of Traian Basescu from the office of President of Romania in a referendum vote before 1 August 2012?

1124 Will Israel officially announce that it recognizes the Armenian genocide before 1 April 2013?1125 When will South Korea and Japan sign a new military intelligence pact*? (a) Between 16 Jul 2012 and 30 Sep 2012, (b) Between 1 Oct 2012 and 31 Dec 2012, (c) Between 1 Jan 2013 and 31 Mar 2013, (d) Event will not occur before 1 April 20131127 Will Moody's issue a new downgrade on the long-term ratings for any of the eight major French banks between 30 July 2012 and 31 December 2012?1129 Will Syria use chemical or biological weapons before 1 January 2013? (a) If there is a substantial* lethal confrontation between Syrian forces and a foreign or multinational military force before 1 January 2013? (b) If there is not a substantial* lethal confrontation between Syrian forces and a foreign or multinational military force before 1 January 2013?1130 Will Moody’s issue a new downgrade of the long term debt rating of the Government of Germany between 30 July 2012 and 31 March 2013?1131 Will Victor Ponta resign or vacate the office of Prime Minister of Romania before 1 November 2012?1132 Will al-Shabaab commence official talks with the Somali government before 1 January 2013? (a) If Sharif Sheikh Ahmed wins re-election as President of Somalia?, (b) If Sharif Sheikh Ahmed does not win re-election as President of Somalia?1133 Will Israel officially establish a date for early elections before 6 November 2012?1134 When will the Free Syrian Army gain control of the city of Aleppo? (a) Between 13 August 2012 and 31 August 2012, (b) Between 1 September 2012 and 30 September 2012, (c) Between 1 October and 31 October 2012 , (d) Event will not occur before 1 November 20121135 Will the number of registered Syrian conflict refugees reported by the UNHCR exceed 250,000 at any point before 1 April 2013?1136 Will Kuwait commence parliamentary elections before 1 October 2012?1137 Will the United Kingdom’s Liberal Democrats and Conservatives remain in a coalition through 1 April 2013?1138 Will the Democratic People's Republic of Korea (North Korea) and the Republic of Korea (South Korea) commence official bilateral talks before 1 August 2013? "(a) If Park Geun-hye wins the next South Korean Presidential election* before official talks commence?(b) If Park Geun-hye does not win the next South Korean Presidential election* before official talks commence?1139 Will any government force gain control of the Somali town of Kismayo before 1 November 2012?1140 Will at least one individual be convicted of the July 2011 killing of Iranian nuclear physicist Darioush Rezaeinejad by an Iranian court of law before 1 January 2013?1141 Who will win Ghana's next Presidential election? (a) Nana Addo Dankwa Akufo-Addo (b) John Dramani Mahama (c) other1142 Will a foreign or multinational military force invade, enter or significantly* fire on Iran before 21 January 2013?1143 Will the IMF officially announce before 1 January 2013 that an agreement has been reached to lend Egypt at least 4 billion USD? (a) If the Egyptian pound to US dollar exchange rate falls below .16 USD at any point beforehand? (b) If the Egyptian pound to US dollar exchange rate does not fall below .16 USD at any point beforehand?1144 Will Mariano Rajoy resign or otherwise vacate the office of Prime Minister of Spain before 1 February 2013? (a) If S&P lowers its long-term sovereign credit rating for the Kingdom

of Spain beforehand, (b) If S&P does not lower its long-term sovereign credit rating for the Kingdom of Spain beforehand1145 Will the Yuan to Dollar exchange rate on 31 December 2012 be more than 5% different than the 31 August 2012 exchange rate? (a) If, before 31 December 2012, the Federal Reserve Board officially announces a third round of Quantitative Easing ("QE3"), (b) If, before 31 December 2012, the Federal Reserve Board does not officially announce QE31147 Before 1 April 2013, will the Egyptian government officially announce it has started construction of a nuclear power plant at Dabaa?1148 What change will the European Union Council ("EUC") make with respect to Bulgaria and Romania's inclusion in the Schengen area* before 1 February 2013? (a) The EUC will extend the Schengen area to include Bulgaria, but not Romania, (b) The EUC will extend the Schengen area to include Romania, but not Bulgaria, (c) The EUC will extend the Schengen area to include both Bulgaria and Romania, (d) The EUC will not extend the Schengen area to include either country before 1 February 2013.1149 Will the sentence of any of the three members of the band Pussy Riot who were convicted of hooliganism be reduced, nullified, or suspended before 1 December 2012?1150 Will Sudan and South Sudan sign a border security agreement before 1 December 2012?1151 Will Japan and North Korea announce an agreement to establish formal diplomatic relations before 1 April 2013? (a) If North Korea returns the remains of any additional Japanese who died in North Korea in World War II beforehand, (b) If North Korea does not return the remains of any additional Japanese who died in North Korea in World War II beforehand1152 Will the Vice President of Iraq, Tariq al-Hashimi's, death sentence be overturned before 1 November 2012? 1153 Before 1 December 2012, will Joseph Kony be *captured by a Ugandan, foreign or multinational military/law enforcement force? 1154 Before 1 April 2013 will the North Korean government officially announce it has invited UN nuclear inspectors to visit the country? (a) If the United States officially announces it has resumed its program to provide food aid to North Korea beforehand, (b) If the United States does NOT officially announce it has resumed its program to provide food aid to North Korea beforehand1155 What change will occur in the FAO Food Price index during September 2012? (a) Increase of >=3%, (b) Increase of <3%, (c) Decrease of <3%, (d) Decrease of >=3%, (e) No change1156 Will the new leader of Japan's Liberal Democratic Party (LDP) Shinzo Abe be declared Prime Minister of Japan before 1 October 2013? (a) If Japan commences snap parliamentary elections before 1 April 2013, (b) If Japan does not commence snap parliamentary elections before 1 April 20131157 Which political parties will be a part of the next Lithuanian coalition government?(a) The Homeland Union–Christian Democrats (TS-LKD) but not the Social Democrats (LSDP), (b) The Social Democrats (LSDP) but not the Homeland Union–Christian Democrats (TS-LKD), (c) Both the Homeland Union-Christian Democrats (TS-LKD) and the Social Democrats (LSDP), (d) Neither the Homeland Union-Christian Democrats (TS-LKD) nor the Social Democrats (LSDP) 1158 Will the Malian government and Ansar Dine commence official talks before 1 April 2013? (a) If the UN Security Council passes a new resolution that supports military

intervention* in Mali beforehand, (b) If the UN Security Council does not pass a new resolution that supports military intervention* in Mali beforehand1160 Will a significant* Turkish military force invade or enter Syria between 9 October 2012 and 30 November 2012?1161 If the International Monetary Fund (IMF) officially announces that Argentina has failed to provide the IMF with sufficient growth and inflation data before 20 December 2012: (a) Will the IMF officially announce sanctions on Argentina before 1 February 2013? 1162 Will the Canadian consulate in Tehran officially re-open at any time before 1 April 2013(a) If a UN nuclear inspector visits Iran’s Parchin nuclear site beforehand BUT Bashar al-Assad does not resign or otherwise vacate the office of President of Syria beforehand, (b) If Bashar al-Assad resigns or otherwise vacates the office of President of Syria beforehand BUT a UN nuclear inspector does not visit Iran’s Parchin nuclear site beforehand, (c) If BOTH a UN nuclear inspector visits Iran's Parchin nuclear site beforehand AND Bashar al-Assad resigns or otherwise vacates the office of the President of Syria beforehand, (d) If NEITHER a UN nuclear inspector visits Iran's Parchin nuclear site beforehand NOR Bashar al-Assad resigns or otherwise vacate the office of the President of Syria beforehand1163 Will Liu Yandong be selected as a member of the next Politburo Standing Committee of the Communist Party of China?1164 What will the number of registered Syrian conflict refugees reported by the UNHCR be as of 1 December 2012? (a) 250,000 or less, (b) 250,001-300,000, (c) 300,001-350,000, (d) 350,001 or more1165 Will Iran and the U.S. commence official nuclear program talks* before 1 April 2013?1166 Will Israel launch an airstrike against Sudan between 5 November 2012 and 31 December 2012?(a) If Israel officially acknowledges responsibility for the Khartoum bombing of 23 October 2012 beforehand, (b) If Israel does not officially acknowledge responsibility for the Khartoum bombing of 23 October 2012 beforehand1167 Will a banking union be approved in the EU council before 1 March 2013?(a) If the UK officially announces its intention to join the banking union beforehand, (b) If the UK does not officially announce its intention to join the banking union beforehand1168 Will the sentence of any of the seven Italian experts convicted of manslaughter for failing to “adequately warn” about the L'Aquila earthquake be reduced, nullified, or suspended before 1 April 2013?1169 Who will win the next South Korean presidential election? (a) Park Geun-Hye, (b) Moon Jae-In, (c) Ahn Cheol-Soo, (d) Someone else1170 Will S&P downgrade India's credit rating between 5 November 2012 and 31 January 2013?1171 Will the trial of Ahmed Shafik begin before 1 January 2013? (a) If he returns to Egypt beforehand, (b) If he does not return to Egypt beforehand 1173 Will the Chinese consumer confidence score for the month of November 2012 drop below 99?1174 Will the Turkish government release imprisoned Kurdish rebel leader Abdullah Ocalan before 1 April 2013?1175 Will a *significant Israeli military force invade or enter the Gaza strip between 19 November and 30 November 2012? (a) If at least one Hamas rocket explodes within Jerusalem’s city limits beforehand, (b) If no Hamas rockets explode within Jerusalem’s city limits beforehand

1177 Will Mohammed Morsi cease to be President of Egypt before 1 April 2013?1178 Will Benjamin Netanyahu resign or otherwise vacate the office of Prime Minister of Israel before 1 April 2013? 1179 Will opposition forces in Syria seize control of the Syrian city of Aleppo by 30 April 2013?1180 Will a significant* foreign or multinational military force invade or enter Iran between 17 December 2012 and 31 March 2013?1181 Will Iran sign an IAEA Structured Approach document before 1 April 2013?1182 Will Mahmoud Ahmadinejad resign or otherwise vacate the office of President of Iran before 1 April 2013?1183 Will the United Nations Security Council pass a new resolution directly concerning Iran between 17 December 2012 and 31 March 2013? 1184 Before 1 April 2013, will substantial* evidence emerge that Iran has enriched any uranium above 27% purity?1185 Will there be a substantial* lethal confrontation involving Iraqi government forces and Kurdish fighters before 1 April 2013? (a) If Jalal Talabani vacates the office of President of Iraq beforehand, (b) If Jalal Talabani does not vacate the office of President of Iraq beforehand1186 Who will be the next president of Cyprus? (a) Nicos Anastasiades, (b) Giorgos Lillikas, (c) Stavros Malas, (d) Other Candidate1187 Will M23 seize, recapture, or otherwise occupy the city of Goma at any time before 1 April 2013? (a) If the UK restores economic aid to Rwanda beforehand, (b) If the UK does NOT restore economic aid to Rwanda beforehand1190 Will Italian ten-year government bond yields be below 4% as of 31 March 2013? "(a) If Pier Luigi Bersani becomes Prime Minister of Italy beforehand, (b) If Pier Luigi Bersani does not become Prime Minister of Italy beforehand1191 Will the official US Dollar to Venezuelan Bolivar exchange rate exceed 4.35 at any point before 1 April 2013? (a) If Hugo Chavez vacates the office of President of Venezuela beforehand, (b) If Hugo Chavez does not vacate the office of President of Venezuela beforehand1193 What will the number of registered Malian conflict refugees reported by the UNHCR be as of 1 March 2013? (a) 100,000 or less, (b) 100,001-150,000, (c) 150,001-200,000, (d) 200,001-250,000, (e) 250,001 or more1194 Will a measurable* Syrian military force invade or enter Israel between 4 February 2013 and 1 April 2013? (a) If Israel launches an additional airstrike against Syria beforehand, (b) If Israel does not launch an additional airstrike against Syria beforehand1195 Will a measurable* Israeli military force invade or enter Syria between 4 February 2013 and 1 April 2013? (a) If Syrian government forces use chemical weapons beforehand, (b) If Syrian government forces do not use chemical weapons beforehand1196 Will any foreign or multinational military force significantly* attack North Korea before 1 April 2013? (a) If North Korea successfully detonates a nuclear device--either atmospherically, underground, or underwater--beforehand, (b) If North Korea does not successfully detonate a nuclear device--either atmospherically, underground, or underwater--beforehand1198 Will Egypt lift the state of emergency in Port Said, Suez, and Ismailiya before 25 February 2013? (a) If Mohammad Morsi and opposition leaders engage in official talks beforehand, (b) If Mohammad Morsi and opposition leaders do not engage in official talks beforehand

1199 Will Hamadi Jebali cease to be Prime Minister of Tunisia before 1 April 2013? 1200 Will Mali commence presidential elections before 1 January 2014? (a) If the UN Security Council passes a resolution authorizing a U.N. peacekeeping force in Mali beforehand, (b) If the UN Security Council does not pass a resolution authorizing a U.N. peacekeeping force in Mali beforehand1201 When will Mariano Rajoy vacate the office of Prime Minister of Spain? (a) Between 11 February 2013 and 24 February 2013, (b) Between 25 February 2013 and 10 March 2013, (c) Between 11 March 2013 and 24 March 2013, (d) Between 25 March 2013 and 7 April 2013, (e) Event will not occur before 8 April 2013.1202 Will the Malian government and National Movement for the Liberation of Azawad (MNLA) begin official talks before 1 April 2013? (a) If France ends Operation Serval beforehand, (b) If France does not end Operation Serval beforehand1203 Will a Zimbabwean referendum vote approve a new constitution before 1 April 2013?1204 Will Egypt commence parliamentary elections before 23 April 2013? 1205 Who will be the next Pope? (a) Cardinal Peter Turkson, (b) Cardinal Angelo Scola, (c) Cardinal Marc Ouellet, (d) Cardinal Gianfranco Ravasi, (e) someone else1206 When will Italy next form a new government? (a) Between 11 March 2013 and 17 March 2013, (b) Between 18 March 2013 and 24 March 2013, (c) Between 25 March 2013 and 31 March 2013, (d) Not before 1 April 20131207 Will France withdraw at least 500 troops from Mali before 10 April 2013?1209 Will Standard & Poor's improve Tunisia's sovereign credit rating or outlook before 10 April 2013? 1211 Will the Liberal Democratic Party (LDP) hold a relative majority of seats in the Japanese Parliament's upper house following the next elections? (a) If Prime Minister Shinzo Abe officially announces a decision to join Trans-Pacific Partnership negotiations beforehand, (b) If Prime Minister Shinzo Abe does not officially announce a decision to join Trans-Pacific Partnership negotiations beforehand1213 Will €1 Euro buy less than $1.27 at any point before 10 April 2013?1214 Will Standard & Poor's improve Cyprus' sovereign credit rating or outlook before 10 April 2013?

B. Instructions for All Forecasters

How Your Forecasts Will Be Scored for Accuracy

How do we calculate your forecasting score when we rank your performance relative to others on the leaderboard? We will use something called the Brier scoring rule, which we explain in detail below. But the bottom lines are these:

1. When the event in question does not occur, you are scored as more accurate when your forecasts are closer to 0%.

2. When the event does occur, you are scored as more accurate when your forecasts are closer to 100%.

To get a perfect score, you need to be omniscient and assign 0% to all non-occurrences of events and 100% to all occurrences of events. Just as it would be rash for a baseball player to try to hit

homeruns every time at bat, it would be risky for you to assign either 0% or 100% to all events. Why? You pay a steep penalty when you assign very high percentages to things that do not occur and very low percentages to things that occur. But you also do not want to be excessively cautious. You will pay a steep price for not making strong predictions when you have useful predictive knowledge. Strive for a balance between bold, but risky forecasts and cautious, low-payoff forecasts.

Your Brier score will be best when you report your true beliefs and make sure that irrelevancies, such as what you hope or fear, are kept at bay.

More Technical Discussion of the Brier Scoring Rule

Definition of Brier Scores. For any given forecasting problem, Brier scores range from 0 to 2, with the best possible Brier score being 0.

Suppose a problem asks whether it will rain tomorrow in Berkeley, CA. You report a percentage p that it will rain. For a dichotomous question, this means you believe the chances are 100% - p that it will not rain. We will represent the two possible states of the world, rain and no rain, as 100% and 0%, respectively. Now, let's suppose it actually rains.

Your Brier score would be (100% - p)2 + (0% - (100% - p))2. The first term is the squared difference between what actually occurred – rain – and your forecast for rain, and the second term is the squared difference between what didn't occur – no rain – and your forecast for no rain.

If you had said the chance of rain is 100%, your Brier score would be (100% - 100%)2 + (0% - 0%)2 = 0. Note that this is the best score. If you had said the chance of rain is 0%, your Brier score would be (100% - 0%)2 + (0% - 100%)2 = 2. (The answer is 2 because the Brier score is calculated using percentages in their decimal form, where the range 0%-100% is expressed as the interval 0-1.) This is the worst Brier score.

Your forecast will probably fall between 0% and 100%. Obviously, if you think rain is more likely than not, you want to assign a higher probability to rain. But since it might not rain, you want to hedge your bets. Errors in either direction have the same impact: being too high is bad, and being too low is equally bad. You get the best score when you truthfully report your best likelihood estimates.

We can't compute a Brier score for you on any given problem until the problem has closed and we know the "correct" answer. Therefore, we keep track of your predictions on each day, and calculate your daily Brier score after a problem has closed.

Your score for that forecasting question equals the unweighted average of your daily Brier scores over all days the problem has been open.

How do we determine your daily Brier score?

The first day a problem is posted, you have the opportunity to make a forecast. If you do not make one, we assign the then-current daily group-average daily Brier score to you for each day until you make your first forecast on that question. (In this context, "group" means everyone who uses the same forecasting website as you do.)

Once you make a prediction for a question, we presume that your answer remains the same each day until you explicitly update your forecast.

If you skip a question entirely, you will receive the group-average Brier score for that question.

Your accuracy will be assessed using two metrics: (1) calibration and (2) resolution. Understanding both metrics and how to improve each is essential to being a good forecaster.1. Calibration is your ability to assign probabilities to events that correspond to their true frequencies of occurrence. If you were perfectly calibrated, 60% of all events to which you assigned a probability of 60% would occur; 70% of events to which you assigned a probability of 70%, would occur; 100% of events to which you assigned a probability of 100% would occur, etc. In brief, good calibration means you know what you know, and what you don’t know.Suppose you are predicting the likelihood of rain for the next 365 days in Memphis–and all you know is that it rains there 30% of the time. An easy way to get a good calibration score is to predict a 30% chance of rain each day. Your average forecast would be 30%, and the overall chance of rain would be 30%. You’d have a perfect calibration score!

2. Resolution is your ability to assign higher probabilities to things that happen than to things that do not happen. Let’s return to the Memphis example. If you assigned a probability of 30% to rain every day, you would have zero ability to discriminate between rain and shine. Calibration, alone, is not enough. To improve resolution, you need to seek out more information about Memphis beyond the base rate. Suppose that data from the National Weather Service allow you to predict a 20% chance of rain on days without rain and an 80% chance of rain on rainy days. Your resolution score would be great. And if you could predict 0% on days without rainand 100% on rainy days, both your calibration and resolution would be perfect! In brief, use base rates to help with calibration – but seek more specific diagnostic information, whenever possible, to help with resolution.

C. Probability Training

Principles of Sound Probabilistic Reasoning for Guiding Your Forecasts

STEP #1. Comparison Classes and Base Rates

A comparison class is a group of people or things that share important attributes or characteristics. Step 1 is to seek out base rates from the best available comparison classes. Let’s see how to use comparison classes for the question: Will Kim Jong-il or his designated successor control North Korea at the end of 2012? Suppose you know nothing about Kim Jong-il. You are thus tempted to say 50% to express your maximum uncertainty about this two-possible-outcome forecast (another way of saying: “I am no more confident than I would be in guessing the outcome of the toss of a fair coin).

But it is a serious mistake to say 50% automatically when you feel deep uncertainty – a mistake because it is often (not always) easy to gain at least some predictive leverage. If the Kim Jong-il question were to be one of the actual forecasts in September, you could do a quick Internet search for basic information about historical and current political data for North Korea.

You could readily find the following information: Kim Jong-il has held power for roughly 15 years – and his father before him held power for 40-plus years. There is evidence of severe economic weakness and some unrest but the security forces have prevented serious threats to the regime.

This information points to a crude but relevant comparison class: dictators in power for 10 or more years. None will be exactly like Kim Jong-il. But exact replicas are not required. The base rate for dictator survival for one additional year in this comparison class is very high – hypothetically, say 95%.

If that were the only comparison class, you might stop here. But you could also readily locate the information that Kim Jong-il is ill and has been preparing a succession plan for his son who is presumed to be in his twenties. The base rate for dictator survival in this narrower and much smaller comparison class – dictator is ill and preparing a succession plan – will be lower (hypothetically, suppose it falls to 47%).

You now confront one of those tricky judgment calls: how much to lower your initial probability in response to this new information. In follow-up training, we will offer more technical guidance on how to make these judgment calls (using a mathematical framework known as Bayes Theorem). For now, we offer a rough approximation solution: when you are unsure which of two reference classes to assign more weight, simply average the estimates, which would yield an estimate of 71%.

Of course, this precision is misleading. You might argue, for instance, that much depends on how "ill" we believe Kim Jong-il to be. If you expect him to live several more years, you might give more weight in the averaging process to the 95% base rate than to the 47% base rate – and offer a probability judgment closer to 95%.

Regardless of the details, the key point is the process. We have identified a useful starting point for estimating an otherwise extremely-difficult-to-estimate number. And note the emphasis on starting point. Be ready to change your estimates in response to breaking news.

In sum, base rates are useful, but they are also crude estimates that hinge on tricky definitions such as " autocratic" and " regime change" – and that can change in response to breaking news in Syria or Libya or… And, regardless of your base-rate starting point, stand ready to change your mind whenever new evidence arises.

STEP #2. Averaging and the “Wisdom of Crowds”

Although experts are often wrong, expert guidance is unfortunately often the best we have. Note also that when experts disagree – as they often do – averaging expert opinions is usually wiser than picking a single opinion. It is a good idea to be skeptical of “super experts” or “gurus” who claim prescience. Such claims rarely hold up well under close inspection.

Averaging across experts often yields more accurate predictions because averaging cancels out the unusual errors each individual makes. One special type of crowd wisdom can be found in prediction markets in which people make bets about future events in finance, entertainment, sports, politics, technology, etc. In addition to prediction markets, there are many other places where you can find blends or composites of expert judgments. The influential magazine The Economist often publishes consensus forecasts – and so do many other outlets.

To illustrate the averaging of perspectives, consider the challenge of predicting whether Iran will conduct a successful nuclear test by the end of 2012. There are many school s of thought here – and unless you have access to highly specialized knowledge, you should consider relying on an average of the perspectives.

Suppose one school of thought believes that Iran attaches little credibility to American willingness and Israeli capability to destroy its nuclear program. Some even suspect that Iran might welcome an international crisis to defuse domestic opposition. These arguments suggest a higher probability of a successful nuclear test (you must do the difficult conversion of arguments into numbers in your head, but let’s say they range in your view from 40% to 90%).

Suppose another school of thought believes that Iran might be deterred by the threat of American or Israeli military action, or thwarted by technical problems (perhaps linked to sabotage) or seduced by promises to loosen economic sanctions. These arguments suggest a much lower range of probabilities to a successful nuclear test (say, between 1% and 25%).

The next step is to compute the midpoint of each range (65% and 13%, respectively) and, absent any other information, average those midpoints (39%). If you have reason to believe one set of assumptions more likely to be true than another, compute a weighted average. For instance, if you think it more likely that Iran is not ready to conduct a nuclear test in the specified period, you might give twice as much weight to the second range of probabilities – which would bring your estimate closer to 13% than to 39%. Again, this is one of those tough judgment calls.

STEP #3. Mathematical and Statistical Models – And the Variables that Go into Them

Predictions of statistical and computer models can be very useful, when treated with appropriate caution. For example, weather forecasters typically consult several computer models before combining them into an overall forecast. When models with good track records are available, take advantage of them.

Consider this question. Will Sarkozy win re-election in France in 2012?

The predictions of models are not yet available for the 2012 French presidential election. But you can consider the variables known to go into the models for other well-established democracies. Suppose that three variables often predict reelection: incumbency, approval ratings before the election, and GDP growth in the election year. Sarkozy is an incumbent, and incumbent presidents are a relevant reference class.

Suppose the probability of re-election is roughly 67%. What about Sarkozy’s approval ratings? In late Jul y, 2011, his ratings were 36%. There is uncertainty here (we are writing in August, 2011, and the election will occur on April 22, 2012) but by itself, the low approval rating implies a reelection probability well under 50% (say, 25%). And predictions for French GDP growth in

2012 are now 2%. Again, there is uncertainty, but by itself, this is not good news for reelection (say, a probability of 45%).

Now convert the information into a re-election probability. Lacking a formal model, you must do this conversion on your own. Fortunately, there is a shortcut known as equal weighting that often works well. You simply average your estimates based on incumbency, approval ratings, and economic growth rates. If you have no basis for treating one variable as more important than another, use equal weighting. This approach leads you to predict [(67% + 25% + 45%)/3] = approx. 46% chance of reelection – a number that could change fast.

You could use a strikingly similar method of reasoning to answer the question about the reelection prospects of President Obama. Start by estimating what the incumbency advantage is in the United States and then estimate prospects conditional on your view of his approval ratings and conditional on your views of likely future economic growth rates.

STEP #4. Continuous Historical and Time-Series Data: The Challenges of Forecasting Continuous Variables

Some questions ask about outcomes that are easy to quantify and that have long historical track records (time series data). The questions posed at the outset about the future price of Brent crude oil and about Chinese economic growth in 2012 both fall in this category.

In forecasting these sorts of outcomes, you can sometimes (only sometimes) do quite well by simply predicting continuation of the status quo (predict the most recent data point as likely to be the next data point in the future) or predicting continuation of the most recent rate of change (if economic growth was recently accelerating, predict continuation of acceleration; if growth was decelerating; predict continuation of deceleration).

That said, there are tricky judgment calls in dealing with time-series data. One difficult issue is how far back into the past you should search. One rule of thumb is to look back at least as far as you want to look forward. If you want to predict one year into the future (and that will be as far as most of our predictions go in this phase of the forecasting tournament), look back at least one year for informing your forecasts.

But this rule is too simple. Some time series are extremely noisy (there is so much variability that it is arguably impossible to identify any trend). A supplementary rule of thumb is that the noisier the data, the further back in time you should go to get a stable longer-term average to serve as your estimate of the likely future value of the variable.

We confront this problem in the next question, which asks about the price of Brent crude oil on 12/31/2012. Oil prices are notoriously noisy (some say just unpredictable) and the time series data for Brent crude oil has shown great variability. The variation in price over the last five years is greater than usual, but far from historically unprecedented.

Initial Point Estimate Guess: Begin by making an estimate of what you think the price will be on 12/31/2012 (the August 5, 2011 price was $109 but it has probably changed). Write your initial guess down.

Initial 90% Confidence Interval Guess: Now specify a 90% confidence interval—a lower-bound price and an upper-bound price far enough apart that you are 90% confident the true price will fall between these values.

A common bias here is overconfidence. When experts set a 90% confidence interval, the true outcome falls outside that interval far more than 10% of the time. The next steps illustrate how some simple rules of thumb can help to correct this bias.

1. Using Historical Data to Fine-Tune Your Initial Guess: Below is a graph of Brent crude prices from January, 2006 to May, 2011. You could fit a trend line and extrapolate to December, 2012. Depending on how the line is fit, estimates could range from $100 to $140. Perhaps the midpoint of $120 is a reasonable estimate.

2. Estimate the Lower Bound: Now estimate the lower bound of your confidence interval. You could compute the average downward change each year over the last 5 years. Let’s say it is $40. Then you could double that average change to estimate your lower bound. There is nothing magical about doubling the average downward change, but we do want something that is more extreme than the average. Let’s say our lower bound i s $80 less than our point estimate of $120. The lower bound is $40.

3. Estimate the Upper Bound: Now estimate the upper bound of your confidence interval. You could compute the average upward change each year over the last 10 years. Let’s say it is $50. You could double that estimate to create your upper bound for a 90% confidence interval. An upper bound might be $100 above the point estimate of $120 or $220.

Tell us your revised point estimate and 90% confidence interval: Note that this exercise can also help you answer many discrete-outcome types of forecasting questions. For instance, if asked for the probability of the price of Brent crude oil falling below $40, your analysis here implies an answer of 5%. If asked about the likelihood that the price of oil will go above $220, your analysis implies an answer of 5%. And you could treat the midpoint of your range – $120 – as a median (50% likelihood of the price being higher or lower).

Let’s examine one more continuous-variable problem. Will the 2012 GDP growth rate in China increase, decrease, or remain within plus or minus 1.5% of the GDP growth rate in 2011? Below is a graph of Chinese GDP rates from March, 2007 to March, 2010.

For Chinese GDP growth in 2012, a reasonable starting guess for your point estimate would be Chinese GDP growth in the previous year (it is estimated to be approximately 9% in August, 2011). And in setting your 90% confidence interval, you should consider using the same decision rules as employed in the Brent crude point estimate (a number for which it is 50% likely the true number is either higher or lower). Now set a 5% Bound (a number so low you think the chances are only 5% chance that the true number is lower) and a 95% Bound (a number so high you think the chances are only 5% that the true number is higher).

One judgment call in dealing with time-series data is the challenge of blending statistical information built into the time series with other types of evidence you deem important. For instance, we discussed earlier the value of averaging the forecasts of di verse experts (with special focus on Iran). There are also diverse points of view on the likely future of the Chinese economy. Some experts expect a host of factors to bring Chinese growth rates down either

gradually or precipitously: a slowing world economy that reduces demand for Chinese exports, rising inflation in China, huge debt in the Chinese financial system that could produce a cascade of bank failures, and the possibility of serious political unrest. Other experts expect continuation of robust growth and a few even expect acceleration of growth.

You can always supplement your time-series prediction with a prediction based on an averaging of expert perspectives. (Indeed, it is generally a good idea to do so.)

We recommend caution, however, when you are tempted to rely heavily on a single expert’s hunch that runs counter to a long-term historical trend – such as Chinese growth rates falling to zero or negative territory in the near-term future. Keep the following rule in mind. Predictions that depart sharply from long-term trends need to be backed up by very persuasive causal arguments for expecting countervailing forces to come into play in the specified period. You may be right when you bet against a long-term trend, but the odds will generally be against you.

STEP #5. Predictable Biases in Probabilistic Reasoning: Be Honest about Your Foibles

Research on forecasters has identified a number of common mistakes. The better you are at spotting and correcting these mistakes, the more likely you are to emerge as a top performer. Answer each of the questions to the best of your ability: Are you a risk seeker who swings hard for home runs by making extreme predictions? This strategy could result in good resolution, but could also reduce your calibration. Don't stray far from the historical base rates if you don' t have the information to back up your extreme judgments.

Are you risk averse and prone to make very cautious predictions? This strategy could result in good calibration, but degrade your resolution. Don't always take the easy way out by relying on historical base rates. Try to diagnose the specific problem.

Do you assign higher probabilities to events that you wish would happen (e.g., a candidate you like winning an election)? Separate your emotions from your beliefs. Don't let your wishes and desires distort your beliefs about the future. Do you assign higher probabilities to events that you think society should be concerned about (e.g., global warming; terrorists gaining access to Pakistan's nuclear arsenal)? Again, don't let emotions and personal priorities influence your beliefs. Preferences should be separated from facts.

Do you fail to update your beliefs in response to new information – to the point where others consider you stubborn? We all tend to look for information that confirms our beliefs. Be sensitive to information that you feel tempted to discount because it conflicts with your current assumptions. When dealing with binary predictions that you know little about, do you instinctively assign a 50% estimate even though their occurrence in the past has notbeen 50-50? Remember the 50% fallacy in two-outcome forecasts (and the 33/33/34 fallacy in the three-outcome case and the 25/25/25/25 fallacy in the four-outcome case). There isusually a way to get some predictive leverage even on topics about which you know little—and one of the best ways is by looking for instructive base rates.

Concluding Comments

We recognize how hard it will be to apply these probabilistic-reasoning guidelines to specific problems when we start forecasting in September. Our advice can be condensed into the following acronym: CHAMP.

1. Comparison classes are essential: Identify the best comparison class – and anchor your initial guesses on the base rate in that class.

2. Historical trends can help: When time series data are available, use them – and rely more on time series when there is little variability around those trends. You need a strong countervailing causal argument to go against a strong trend. When you confront noisy, high-variability time series with unclear trends, you need to go further back in time and average across larger numbers of observations.

3. Average across opinions: Expert-consensus data or prediction-market data are not infallible but they do tell you something important about the “wisdom of the crowd.” Take such data into account in making your own forecasts.

4. Mathematical models can offer useful guidance: Statistical models are not infallible, but they often capture key causal regularities from the past. When model -based predictions are available, you should take those predictions into account in making your forecasts. When such predictions are not available, you should still look at variables that go into the models and base estimates on unit weighting.

5. Predictable biases are a fact of life: Beware of the potential biases in your forecasting strategy and of how they can reduce your accuracy. Try to keep them in check.

In sum, every situation you confront is unique. But that does not mean there are no useful precedents. As Mark Twain said, history may not repeat itself but it does rhyme.

D. Scenario Training

In this session, we will (1) show you how to apply key principles of scenario thinking for improving your accuracy; and (2) test your understanding of these principles.To avoid misunderstandings, we emphasize at the outset that we are not pretending to offer mathematically precise solutions to messy real-world forecasting problems. Rather, we are trying to provide cost-effective analytical shortcuts for boosting accuracy—shortcuts that can be readily adopted by busy professionals, such as yourselves, who are juggling competing priorities and don’t have the spare time to carry out comprehensive analyses of all possible variables and their interactions. We believe that the principles of scenario thinking will serve you well in the upcoming forecasting tournament.

We will use seven examples of possible forecasting questions to illustrate applications of scenario thinking. These questions are:1. Will Kim Jong-Il or his designated successor be in charge of North Korea as of December 31,2012?2. Will the Saudi monarchy continue to rule Saudi Arabia as of March 1, 2012?3. Will President Barack Obama win reelection in 2012?4. Will Nicolas Sarkozy win the French presidential election in 2012?5. Will Iran conduct a successful nuclear test before December 31, 2012?

6. Will the price of a barrel of Brent crude oil fall below $40 per barrel before December 31, 2012? Will it rise above $240 per barrel?7. Will the Chinese GDP growth in 2012 be within plus or minus 1.5% of the 2011 growth rate?Below that range? Above that range?

Principles to Guide Your ForecastsResearch reveals that forecasters are often over-confident, holding too strongly to a single view of the issue at hand. One reason for overconfidence is that forecasters fail to challenge their assumptions and don’t explore how scenarios based on alternative assumptions might yield different predictions. They become locked into a narrow view of the future—a narrow view usually predicated on the tacit belief that the future will look like the past. Scenario thinking is designed to help you make more accurate forecasts and avoid the trap of overconfidence by assisting you in: (a) thinking more critically and flexibly about your underlying assumptions and learning to look at the future through alternative lenses; (b) identifying the underlying causal drivers of potential outcomes and learning to adjust in a disciplined way your probability estimates of these outcomes as a function of various combinations of causal drivers that might come into play. We will now show you how to apply the principles of scenario thinking to the seven forecasting problems listed earlier.

STEP 1: Explore Your Assumptions – and the Possibility You Might Be Wrong

Consider whether Kim Jong-Il (KJI) or his designated successor will control North Korea through 2012. Let’s say your initial answer is “very likely yes, a 90% chance of regime survival” because you assume that North Korean political elites will stand by the regime—and dictatorships that have been sustained elite support for over 10 years are highly likely to survive an additional year. (By implication: you see only a 10% chance of collapse as a result of a revolt from below or as a result of any other cause.)

Now, do something that people rarely do – question your assumptions. Ask: How likely is elite support for the regime to begin crumbling? On careful thought, your true belief is unlikely to be100%. Suppose, after considering the severe economic weakness in North Korea, you now see a20% chance of the regime losing elite support and an 80% chance of its retaining elite support.They sum to 100% because the possibilities are mutually exclusive and exhaustive. The percentages at the bottom (72% and 12%) are the probabilities of assumptions about both elite support and regime survival (80%*90% and 20%*60%). The sum of these probabilities (72% + 12% = 84%) gives us your revised belief that Kim Jong-Il will remain in power.

Note the subtle but real belief change. Your initial answer was 90%, but that was based on 100% confidence in continued elite support. But on second thought, you discover you are less confident: your estimate of the chances that Kim Jong-Il will stay in power is now 84%.In brief, people often lose confidence when they question their assumptions and acknowledge other scenarios. The more you engage in critical self-examination – scrutinizing assumptions and uncertainties – the less likely you are to be blindsided by unexpected turns of events.

Consider Question 2: Will the Saudi monarchy continue to rule by March 1, 2012? Suppose that your initial intuition is that leadership continuity is likely. You may be right, but you should still

explore your underlying assumptions about causal drivers—and whether you would change your mind if the most pivotal assumption were incorrect. Saudi Arabia is different from North Korea in many ways but, in both, there is an ailing leader and potential for disagreements within the elite to become sharper in the leadership succession process.

Now consider Question 3: Will President Obama be reelected in 2012? Suppose your initial hunch is that incumbent presidents have an advantage and the odds favor reelection. You may be right, but it is useful to unpack your assumptions about causal drivers – and to ask how much you would change your mind if your pivotal assumption about incumbency advantages were incorrect. Let's say that you recognize that the incumbency ceases to be an advantage when the economy is weak. This new causal argument might lead you to revise your probability downward.

Take Question 4: Will Nicolas Sarkozy win the French presidential election in 2012? Sarkozy, like Obama, has the advantage of the incumbency, but the disadvantage of a weak economy.Question 5 asks whether Iran will conduct a successful nuclear test by December 31, 2012.Suppose you think this is likely because the Iranian government sees possession of nuclearweapons as the best guarantee of its long-term security and status. Your hunch rests on the tacitbut pivotal assumption that Iran’s nuclear program is far enough advanced to conduct such a test– and there is controversy over that, which you should factor into your probability.

STEP #2: Exploring the Soundness of Your Assumptions About Key Causal Drivers

We have used scenario thinking to explore the implications of tweaking the correctness of a single pivotal assumption about causal drivers. But most forecasts rest on more than one pivotal assumption. To illustrate how to handle multiple assumptions, let's return to North Korea. Suppose that experts on North Korea disagree on two key issues: (1) continued elite support for the regime (keep or lose) and (2) changes in Chinese economic subsidies (falling, stable, or rising).

We can also assess the probability of each joint event; Keep support and falling subsides = 25%, Keep support and stable subsides = 45%, Keep support and rising subsides = 10%, Lose support and falling subsides = 5%, Lose support and stable subsides = 5%, and Lose support and rising subsides = 10%. The values must sum to 100% because we are treating the six possibilities as mutually exhaustive and exclusive. The next step is to consider the chances of regime survival if each scenario were to occur. Suppose your beliefs are 60%, 80%, 95%, 80%, 40%, and 40% respectively. To compute your estimates of regime survival and each scenario occurring, multiply the probability of each scenario by the probability of regime survival given each scenario. These probabilities are 15%, 36%, 10%, 4% 2% and 4%. The sum of the probabilities is 71%. This is our new estimate of regime survival – a judgment that is grounded in consideration of all six scenarios. It represents a further erosion of our initial confidence in regime survival (from 90% to 84% to 71%).

Scenario training highlights sources of uncertainty that can reduce false confidence in simple diagnoses of complex situations. Note however that all of the numbers are based on hunches, and there is no guarantee that scenario reasoning will lead to the right answer. All we can say is that

our judgment is based on a more open-minded consideration of the possibilities – and that reduces the likelihood of our being blindsided by unexpected changes.

Let’s consider a related problem. In the question about the Saudi monarchy, your views about leadership continuity were likely to hinge not only on your assumptions about the cohesiveness of the ruling elites, but also on your assumptions about resources that support the elite. In the North Korean case, the key source of resources was Chinese economic subsidies. In the Saudi Arabian case, the key source is the price of oil. It is instructive to ask how your probability of leadership continuity would change not only with your assumptions about elite cohesion, but also with your assumptions about the price of oil. If the price were to collapse (see later exercise on Brent crude prices), conflicts within the ruling elite over resources could destabilize the leadership. In the Saudi question, you might also consider that change could come not from the elite but from bottom up. The massive unrest that swept the Arab world in 2011 demonstrated the perils of simple extrapolation. It is unclear how much these new possibilities should reduce our confidence in Saudi stability, but good scenario thinkers reason through many combinations of possibilities.

Returning to Presidents Obama and Sarkozy, experts on electoral politics tell us that, although incumbency and the economy are key factors, there are others, including presidential approval ratings and the political skills of the opposing candidates. Again, how would your estimates change if your tacit assumptions about these additional causal drivers were incorrect?

STEP 3: Opening Your Mind to a Wide Range of Possibilities By Considering Worst andBest Case Scenarios – and the Challenge of Forecasting Continuous Variables.

Let's turn to the sixth and seventh forecasting questions listed at the outset: forecasting the price of Brent crude oil and forecasting Chinese economic growth in 2012. Unlike the earlier examples, these forecasts focus on continuous variables (dollars per barrel and percentage GDP growth), not discrete possibilities (such as whether a given leader will or will not be in power-- or which of the small set of leaders will be in power). Let’s start by predicting the price of a barrel of Brent crude oil on 12/31/2012.

Initial Point Estimate Guess: Begin by making an estimate (the August 5, 2011 price was $109 but it has probably changed by the time you are taking this training). Write your initial guess down.

Initial 90% Confidence Interval Guess: Now specify a 90% confidence interval—a lower-bound price and an upper-bound price far enough apart that you are 90% confident the true price will fall between these values.

A common bias is overconfidence. When experts set a 90% confidence interval, the true outcome falls outside that interval far more than 10% of the time. The next steps illustrate how scenario training can help to correct this bias.

1. Drivers of Initial Guess: Identify the two or three key drivers underlying your initial price estimate and confidence interval (e.g., possible turmoil in Middle East; global economic growth; discovery of new oil deposits; viability of alternative energy sources).

2. Factors Causing Lower Bound: Now sketch the combination of conditions most likely to cause an extremely low oil price (e.g., little Mideast turmoil, subsidies for domestic oil production, low economic growth); add other factors if you can (e.g., electric cars, new oil fields, no mega-spills). Write a few short paragraphs – we will ask for them later.

3. Estimate the Lower Bound: Now estimate what the oil price would likely be under this combination of extremely-low-price conditions. Your answer does not need to be the sameas the lower bound price you initially estimated. Scenario generation often changes minds.Let’s suppose your lower-bound estimate is now $40 per barrel).

4. Factors Causing Upper Bound: Next sketch the conditions most likely to cause an extremely high oil price, starting with the opposite of the above, plus, say, an economic boom in China and India or Iran closing the Straits of Hormuz. Write a few paragraphs. We will ask for them later.

5. Estimate the Upper Bound: Now estimate what the oil price would likely be in this combination of extremely-high-price conditions. Again, your answer does not need to be the same as the upper bound price you initially estimated. Let's suppose your upper bound estimate is now $220 per barrel.

6. Step Back and Look: Treat your new low and high scenario estimates as defining your new90% confidence ranged (we offered the guesses of $40 to $220 but your estimates certainly do not need to be the same).

In light of this exercise, tell us your revised point estimate and 90% confidence interval:Note this exercise can also help with discrete-outcome forecasts. If asked the probability of Brent crude falling below $40, your analysis implies an answer of 5%. If asked about the price going above $220, your analysis again implies 5%. And you could treat the midpoint of your range – $130 – as a median (50% likelihood of the price being higher or lower).

You can apply this mode of reasoning to other problems, such as Chinese GDP growth in 2012. As with the oil-price problem, estimate the confidence interval by using extreme scenarios. For instance, the extreme pessimists on Chinese economic growth generate gloomy scenarios that focus on the risks of massive debt within the Chinese financial system, growing inflation, growing political unrest within China, and the risk of sharp declines in Chinese exports as the global economy potentially slides into another major recession. You could use their views to anchor the lower end of your 90% confidence interval on likely Chinese economic growth.

By contrast, the extreme optimists on Chinese growth generate boomster scenarios that discount the risks above and stress the financial reserves of the central government, still massive untapped labor pools, gradual progress toward an independent judiciary and stable property rights, and other factors. Use their views to anchor the upper end of your 90% confidence interval.

STEP 4: Avoid Over-Correcting Biases

Beware of the law of the hammer: when children discover a hammer, everything becomes a nail.As with any tool, there is a danger of taking scenario thinking too far – and triggering new biases.We list the side effects worth worrying about below:1. Overprediction of Change: The scenario cure for overconfidence in the status quo couldcause us to over-predict dramatic change. As noted above, most of the forecasting problems presented in the first year of the tournament will be short-term questions, asking you to predict outcomes no more than 12 months out. So, even if you have good grounds for predicting long-term change, there may be good grounds for predicting the status quo in the short term. Be careful not to underestimate the inertial forces in political and economic systems that make the status quo a good bet.

2. Probabilities that Don’t Sum Properly : Spinning off too many scenarios can lead to probabilities that sum to over 100%. For this reason, generate a limited number of scenarios.It is easy to be logically consistent with a small number of scenarios. This exercise is designed to broaden your analysis by highlighting the complexity and unpredictability of the world. This knowledge can make it easy to become befuddled. Once you have seen many trees, you need to zoom out again, recalibrate your perspective, and size up the forest.

3. Incoherent Scenarios: Spinning off too many scenarios can make you too open to far-fetched ideas. Let’s revisit forecasting the price of Brent crude. Check that your extreme scenarios are internally consistent by considering countervailing forces that might be activated if prices fall “too low” or “too high.” If the price of oil rose high enough, it would stimulate alternative sources of energy and modes of transportation. Good scenario planners think “systemically” – and recognize interactions and feedback mechanisms among variables.

Scenario training is designed to protect you from being blindsided by an unconsidered or discounted outcome because you were locked into a narrow set of assumptions about the future.Getting the benefits while minimizing the risk of overcorrecting biases requires careful thought about how to check the unwelcome side effects outlined in this section.

Concluding CommentsOur advice can be condensed into the following acronym, QUEST:

Question views: Beware of the dangers of being locked into too narrow a view of the future – and into the assumption the future will simply look like the past;

Use plausible worst-case and best-case scenarios to expand your conception of the possible.

Explore assumptions: Beware of the dangers of excessive confidence in the assumptions underlying your estimates. Be open to the possibility your assumptions might be wrong. Estimatethe likelihood of the outcome not just on the basis of your favorite assumptions, but also on the basis of alternative assumptions being true – and then combine the estimates.

Several assumptions should be considered: Use multiple assumptions, but remember there is no magical formula. You need to exercise judgment about when to use scenario tools (as is true for all tools). There are downside risks of analysis-paralysis, confusion, and probabilities adding tomore than 100%.

Take heed of biases: Keep in mind potential biases in your own forecasting strategy – and how they can reduce your accuracy scores. Make active efforts to keep your biases in check.

Testing Your Understanding of the Scenario Training MaterialsThe exercises that follow test your understanding of the scenario training guidelines and yourwillingness to apply the guidelines to actual forecasting exercises.

Proper Implementation of Scenario Training

What were the best arguments you generated for the set of scenarios most conducive to a sharp increase in the price of Brent crude oil at the end of 2012? What were the best arguments you generated for the set of scenarios most conducive to a sharp decrease in the price of Brent crude at the end of 2012?

There are no absolute right or wrong answers here, but higher-quality answers are likely to take into account the following sorts of factors (organized into factors that could affect price either via supply or demand):

What "unknowns" could cause us to conclude that our current estimates of the potential supply of oil/fossil fuels are serious underestimates (e.g. discovery of massive new deposits; development of new technologies for the cost-effective extraction of previously inaccessible deposits)?

What unknowns could cause us to conclude that our current estimates of potential supply are serious overestimates (terrorism, revolutions, and wars in key oil-producing countries cause massive supply disruptions; discovery of exaggerations in oil-reserve estimates of major oil companies)?

What unknowns could cause us to conclude that our current estimates of potential demand for oil/fossil fuels are serious underestimates (even more dynamic growth in India, China, and other rapidly developing countries boosts demand for automobiles and other fossil-fuel intensive technologies; disappointing progress in developing alternative fuels and electric cars)?

What unknowns could cause us to conclude that our current estimates of potential demand are serious overestimates? (More rapid progress and development of alternative energy technologiesthan previously supposed; dramatic slowdowns in the growth rates of rapidly growing countries)?

Questions About Potential Positive/Intended and Negative/Unintended Effects of ScenarioTraining

Before scenario training, a forecaster believed that the likelihood of Québec seceding from

Canada in the next 10 years was 20%. After training, the forecaster believed that at least three scenario pathways for secession were plausible – and raised the probability of Québec secession to 60%. The forecaster was also impressed by three scenario pathways via which the Canadian government could neutralize the secessionist threat – and assigned probabilities of 20% to each (leading to an overall probability of 60% of Canada remaining intact).

Explain what is wrong with this forecaster's probabilistic reasoning.

A forecaster is impressed by scenarios that highlight causal forces that could dramaticallya. The forecaster did not develop enough scenario possibilitiesb. The forecaster's probabilities for the scenarios leading to Québec secession shouldhave added up to 100%c. The forecaster's probabilities for the two logically exclusive and exhaustive possibleoutcomes – Canada remaining intact versus Canada not remaining intact – should nothave added up to 120%d. None of the above

The correct answer is c: The forecaster's probabilities for the two logically exclusive and exhaustive possible outcomes – Canada remaining intact versus Canada not remaining intact – should not have added up to 120%.

A forecaster is impressed by scenarios that highlight causal forces that could dramatically slow Chinese economic growth and produce massive instability over the next 10 years. The forecaster applies these long-range projections to a short-term prediction of Chinese growth in the next year. The forecaster assigns the highest possible probability, 100%, to zero growth – even though the most recent data indicate a GDP growth rate of roughly 9.5% per year and there is a trend going back 20 plus years of high growth rates. Explain what is wrong with this forecaster's reasoning.

a. Given the power of the statistical trends, the forecaster should have given at least someweight to them and struck more of a compromise between the time series trend and thegloomy-growth scenarios which he finds convincing.b. The forecaster should have given greater weight to even more pessimistic scenariosabout Chinese economic growth in the next year.c. The forecaster should have ignored the scenarios and simply relied on the time-seriestrend.d. All of the abovee. None of the above

The correct answer is a: Given the power of the statistical trends, the forecaster should have given greater weight to the trends and struck more of a compromise between the time-series trend and the causal forces brought to light in the scenarios. The forecaster is probably giving too little weight to the time-series trend and also probably blurring the distinction between short and long term predictions. Furthermore, the forecaster should almost never use 100% or 0% because very little in life is absolutely certain. It is almost always a good idea to give yourself a little wiggle room.

A forecaster is asked to predict the likelihood of massive violence between North and South Korea in the next year. The forecaster assigns a low probability: 10%. The forecaster is later asked to predict the likelihood of the following event: an attempted coup leads to a civil war in North Korea and the South Korean army become involved in that civil war. The forecaster assigns a probability of 20% to the civil war scenario. Explain what is wrong with this forecaster's probabilistic reasoning.

a. The forecaster's initial probability estimate, 10%, was too low because it failed to recognize how close to death Kim Jong-Il is and how likely further violent provocations along the border are (the new leadership will use an external crisis to solidify its legitimacy).b. The forecaster's later probability estimate, 20%, was too high because it failed to recognize how unlikely it is that South Korea would ever intervene in a military struggle forpower in North Korea (given the nuclear weapons that North Korea possesses).c. The forecaster did not develop enough scenarios that focused on prospects for stabilityand preservation of the status quo.d. The forecaster is enmeshed in a logical contradiction. He/she assigned a probability of .10 to the set of all possible pathways leading to massive violence between North and South—and a probability of 0.2 to a subset of those pathways (a civil war in the North that South Korea becomes involved in). The larger set of possibilities must be as large or larger than any subset derived from the set.e. None of the above.

The correct answer is d: The forecaster is enmeshed in a logical contradiction. He/she assigned a probability of .10 to the set of all possible pathways leading to massive violence between North and South—and a probability of 0.2 to a subset of those pathways (a civil war in the North that South Korea becomes involved in). The larger set of possibilities must be as large or larger than any subset derived from the set.

Thank you for working through these training exercises. We believe the QUEST guidelines will help you in the forecasting tournament. Unfortunately, research suggests that training exercises of this sort often don’t “stick” as long as intended. It will often be hard to apply these somewhat abstract lessons to the specific forecasting problems you’ll soon be grappling with. For these reasons, we will periodically be in touch with reminders and tips about how to apply the QUEST guidelines.

Please keep the QUEST acronym in mind whenever you make forecasts.1. Question your views.2. Use plausible worst-case and best-case scenarios.3. Explore your assumptions.4. Several assumptions should be considered.5. Take heed of biases.

Finally, we close on the note of humility. As we noted at the outset, we do not claim to know all the answers and we recognize that our participants have a great deal of expertise. We welcomesuggestions for how to improve our training guidelines – and, once again, want to encourage you

to send constructive comments to feedback@goodjudgment.info.

E. Instructions for Teams

Training Module for Collective Intelligence

You will not be working in isolation. You will be part of a team and working with other forecasters on the problems to come (exact membership of teams yet to be determined). We will be asking all team members to achieve two equally important goals: to make accurate forecasts yourself AND to help your fellow forecasters be accurate – and thus win this tournament! The best way to accomplish these two goals is to hold yourself and fellow team members accountable to rigorous standards in generating and evaluating forecasts.

For each question, you will be given the opportunity to explain the reasons for your forecasts. You are welcome to explain all of your forecasts, but you don’t have to do so. We do, however, require forecasters to explain a minimum of 30 forecasts (15 chosen by you; 15 chosen by the system administrator to ensure coverage of all questions) AND to comment on at least 20 explanations offered by others. You will receive reminders if you don’t make progress toward this goal in a reasonable period of time.

NOTE: Although there might appear to be a conflict, there is no conflict between striving to be the most accurate forecaster and striving to be a great team player who shares your information with other team members. The reason is simple: whenever you have a fresh insight into a problem~the first thing you should do is revise your own forecast (we keep track not only of the accuracy of forecasts, but also of when those forecasts are made – and forecasters get more credit for getting closer to reality sooner rather than later). The next thing you should do is share that information with others (you get more credit for sharing valuable information sooner rather than later). In short, there is no conflict between cooperation and competition here.

Four Steps for Enhancing Your Forecasting Accuracy

We now offer guidance on how to achieve the two goals mentioned earlier – (1) maximizing your accuracy and (2) helping other forecasters to maximize their accuracy by explaining the reasons for your predictions and offering constructive critiques of the reasons that others give for their forecasts. Note that we will later be testing your understanding of these materials.

Step 1: Understand the scoring rules for assessing accuracy

Remember, your personal accuracy and your team’s accuracy are equally weighted. You want to do well, and you want your team to do well. As you may have read earlier, the accuracy scores for you and your team are based on calibration and resolution.

Calibration captures your skill in assigning probabilities that correspond to true frequencies of occurrence. If you were perfectly calibrated, then 60% of all events you assigned a probability of 60% would occur; 70% of all events you assigned a probability of 70% would occur; 100% of events you assigned a probability of 100% would occur; and so forth.

Suppose you are predicting the likelihood of rain every day next year in Memphis – and all you know is that it rains on 30% of the 365 days. An easy way to get a good calibration score is to

predict a 30% chance of rain each day. Your average forecast would be 30% and the overall chance of rain would be 30%. But with this strategy, your forecasts would never discriminate between rain and shine. Your predictions would be the same every day.

Resolution captures your ability to assign higher probabilities to things that happen than to things that do not happen. To improve resolution, you need more specific information about Memphis beyond the base rate. Suppose you link your forecasts to the weekly National Weather Service forecasts which, on average, assign a 20% chance of rain on days without rain and an 80% chance of rain on rainy days. Your resolution score would now be very good. And if your Weather Service predictions were 0% on days without rain and 100% on rainy days, both calibration and resolution scores would be perfect.

To maximize your forecasting score, take into account base-rate probabilities. But go beyond base rates by seeking out credible case-specific information about the key drivers of events.

Step 2: Understand the reasons forecasters under-predict and over-predict change – and be ready to challenge such thinking in yourself and others

Forecasters are at risk of under-predicting change (assigning too low probabilities to change and too high probabilities to no change) whenever:

a. Things don’t change much from year to year and we are lulled into false complacency. For instance, governments often survive many years. It is easy to assume the future will be like the past and to fail to consider new developments that imply potential for change (such as violent unrest or a coup attempt or ill health of the dictator).

b. Forecasters subscribe to a strong theory about why things will not change (or strongly prefer that things not change). Beware of the confirmation bias – accepting arguments in favor of your views uncritically but applying hypercritical standards to arguments that challenge your views.

c. Forecasters automatically use “50%” for two-outcome forecasts as a way of saying “I have no idea” without considering available base-rate information. For instance, in an unstable region where change in key indicators – e.g., violence, economic growth – is historically common (a base rate of 80% or 90%), a 50% estimate of no change might well be a serious underestimate.i

Forecasters are at risk of over-predicting change (assigning too high probabilities to change and too low probabilities to no change) whenever

a. They get caught up in the moment and embrace dramatic-change scenarios. They will be at risk of not giving enough weight to a long track record of stability.

b. They subscribe to a strong theory about why things must change (or have a strong preference for things changing). Again, beware of the confirmation bias.

c. They slip into using 50% for two-outcome forecasts as a way of saying “I have no idea” without considering base-rates. For instance, if you’re working in a stable region where change is rare (a base rate of 10% or 20%), assigning a 50% estimate to “change” may well be a substantial overestimate.

Be on the lookout for both sets of mistakes in your reasoning and in that of others.

Step 3: Seek the best outside guidance and share that information with others

Forecasters often listen to too narrow a range of opinions about issues. They tune into the same news sources and talk to the same people – and more often than not, these sources of information have views strikingly similar to their own. To avoid the echo-chamber trap, ask yourself whether your sources of outside guidance often disagree with each other and point to different probability estimates for possible futures. For instance, if you are a liberal who reads only The New York Times, it is a good idea to read high quality conservative-leaning news outlets, such as the Wall Street Journal. And if you’re conservative, the same advice applies but in reverse.

Ask yourself whether you are sampling from a variety of viewpoints and make sure other forecasters are doing the same.

In addition to seeking diversity, look for expertise. It is common to mock the performance of political and economic experts, but the simple fact is that, unreliable though their forecasting performance often is, it is still usually better than that of non-experts.

Of course, it is often not easy to pin down experts. Let’s say you read an article in a high-quality source, such as The Economist, which predicts “growing turmoil in the Saudi kingdom in 2012. You now have the challenge of translating vague verbiage into a numerical probability. Be honest about the guesswork involved – and, if you accept their premise, you might reason along these lines: “Predicting the demise in one year of a well-entrenched regime is a long shot. But this magazine is a credible source. I’ll reduce my high base rate probability of autocratic-regime survival from 90% to 80% or even 70% – and then wait for breaking news.

But there are also external sources of guidance that provide more precise probability estimates. One example is prediction markets. One such market, Intrade, assigned a 58% chance to the reelection of President Obama in July 2011. Prediction markets are not perfect but many have good track records of assigning realistic odds to possible futures. Bring this information to the attention of your entire team – and provide updates (prediction markets update their probabilities continuously in response to news, and you should too – 58% may be old news).

There are many high-quality sources of information – so seek them out.

Step 4: Acknowledge uncertainties and don’t exaggerate your forecasts

Forecasters often have a hard time answering a surprisingly simple question: what would induce you to change your mind? This is an excellent question to pose to yourself as well as other forecasters. Here are signs that you may not be as open-minded as you need to be to perform well in the forecasting exercises:

a. You have difficulty identifying two or three conditions under which your prediction outcome would become much less likely.

b. You are unable to distinguish facts from the assumptions you have made about unknowns. The sharper the distinction in your mind between facts and assumptions, the easier it becomes to let go of old opinions.

c. You have difficulty revising your probability estimates in response to new information.

Suppose you are asked to predict the stability of the Saudi monarchy. To follow Step 4, you might begin by noting that the likelihood of Saudi stability would rise or fall with: 1) the price of oil, 2) the effectiveness of the state security apparatus in repressing or co-opting dissent, and 3) the health of the King. These factors highlight key unknowns. Also distinguish facts and assumptions. It is one thing to base a forecast on the opinion of a high-status source (such as, The Economist expects “growing turmoil”) and another thing to base it on fact (e.g., a near-miss assassination attempt – if one were to occur). Finally, be open to information that challenges prior beliefs and regularly update estimates in response to new developments.

Be aware of the potential fallibility of the assumptions underlying your probability estimates and be prepared to revise them in response to breaking developments.

Three Steps for Enhancing the Accuracy of Others

Step 1: Use clarity and specificity when explaining your own forecasts

In explaining your own forecasts, sketch – as precisely as possible – why you made the prediction. Instead of vague talk about how "a weak economy makes an incumbent politically vulnerable," be more specific: "weak economy" might mean "unemployment above 10%" or "GDP growth below 1%" and "politically vulnerable" might mean "approval ratings below 40%" or "cabinet-level resignations from the government." Be as specific as possible about what it would take to make you change your mind: "even an incumbent with weak approval ratings can beat a challenger with weaker ratings.”

We now offer some hypothetical examples:

Consider this question. How likely is dictator X in a particular sub-Saharan African country to remain in power by the end of 2012? A forecaster who knows very little should openly concede that:

Probability: 90%

Explanation: I know almost nothing. X has held power over five years, and dictators in power that long tend to stay in power. My best guess is 90%, but I’d change my mind if I saw breaking news about instability of the sort seen in the Middle East and North Africa.

A forecaster who knows more should reveal the more detailed bases for his or her opinion:

Probability: 90%

Explanation: X is well entrenched. He controls enough funds to support his patronage network and elite security forces that suppress dissent. In general, dictators in power that long are likely to survive another year. And this particular dictator is only 52.

Consider another question. How likely is it that Barack Obama, Mitt Romney, or someone else will win the Presidential election in 2012? A forecaster who knows very little might say:

Probability: 55% for Obama, 25% for Romney, and 20% for someone else

Explanation: I know almost nothing. But people whose judgment I respect tell me that prediction markets are one of the most accurate means of placing odds of futures and that Intrade is now selling Obama reelection futures priced at a probability of 52%. But this estimate doesn’t take into account how strong the GOP candidate will be. Knowing so little, I will take my cues from these trading prices.

A more knowledgeable forecaster might say:

Probability: 50% for Obama, 30% for Romney, and 20% for someone else

Explanation: Incumbent presidents have an advantage. But their prospects for reelection fall when unemployment is above 8% – and fall further when approval ratings go below 40%. Considering all that, I put Obama’s chances at 40-60. But he will be the better campaigner in the general election. Romney is still his likely challenger, but he has little oomph. And the alternatives to Romney would be even weaker in the general election. So, Obama still has a 50-50 chance.

Step 2: Offer constructive critiques of others’ explanations – clarity, specificity and diplomacy

In commenting on others’ forecasts, encourage each other to replace vague terms with more specific ones: "Please clarify what you mean by ‘rising Pakistani anger at US policies.’ Which US policies are provoking Pakistanis? Why? What evidence do you have for ‘Pakistani anger’ and your claim about the trend line?" Also encourage each other to specify the conditions under which you would change your forecasts (and, ideally, by how much).

When you do challenge someone's reasoning or assumptions, be sure to do so diplomatically. Avoid ad hominem attacks, such as “only an idiot would think…” or “this argument reveals no understanding of….” Encourage each other to be open-minded, not defensive.

Commentary on Forecaster’s Explanation of Answer to Question about Dictator Survival Prospects:

You may be right that the year-to-year survival rate for dictators in sub-Saharan Africa is as high as 90% (someone should check that), but the survival rate is lower if you narrow the comparison class to West African countries with bitter ethnic-religious-linguistic divisions. My estimate is 70% or lower. And it falls even further when you take into account the recent riots (chances fall to 50/50 at most).

Commentary on Forecaster’s Explanation of Answer to Question about Presidential Reelection Prospects in 2012:

All the factors you identify will influence the reelection probability for Obama. But in my opinion you underestimate the negative impact that high unemployment, slow growth and low approval ratings are having on his prospects and you overestimate his campaign skills relative to his challengers. Romney (if he is the candidate) has a better chance than Obama. Right now, I assign 35% to Obama, 50% to Romney, and 15% to someone else.

Example of a Forecaster’s Reply to Commentary on Forecaster’s Explanation of Answer:

Question 2 (Presidential Reelection Prospects in 2012)

OK – there’s a lot of pessimism about the economy now, but I think it will bounce back somewhat by fall 2012 (unemployment slightly below 8%) so I am not changing here. But you may be right that I am making too strong assumptions about the quality of the campaigners – so I nudge Obama’s chances down just a bit, say, 5%.

Step 3: Build an effective team

Effective teams will be built on the foundation of effective communication among team members. So, Steps 1 and 2 will be essential to Step 3. But effective teams are more than the sum of their individual parts. Teams need norms and procedures to facilitate their operations.

Your forecasting problems will cover a wide range of issues – and no one will be best equipped to handle all of them. After you have had time to get to "know" one another through the explanations and comments you will exchange in the first month of forecasting, we recommend that your team evolve a division of labor – and begin by identifying a team coordinator who facilitates conversations about team-organization issues. The ideal coordinator should be broadly knowledgeable, but should also know the limits of his or her knowledge – and be socially skilled.

Once you have a team coordinator, consider the value of assigning specific roles within the team. Encourage people to volunteer whenever they have special expertise for particular problems, ideally as soon as each problem is announced. As time passes, consider clustering into specialized subgroups. One division of labor might be geographic (China or European Union specialists); another might be functional (military or technology or macroeconomic or, electoral politics specialists).

Also, encourage people to volunteer for specialized information-seeking roles if they are skillful at ferreting out useful sources of information (e.g., high-quality news outlets, elite newspapers and magazines, prediction markets, financial markets, the blogosphere).

Do not interpret impossible-to-resolve disagreements in your team as a sign of bad process. Your official “team prediction” will simply be an average of individual member judgments – and diversity of viewpoints may well help your average accuracy. You are all in this together: sink or swim as one. Your value as a team member depends on your individual accuracy and your contributions to your team’s accuracy.

The first wave of questions will be posted on September 1, 2011. You will begin entering your individual forecasts immediately, but should also begin offering explanations and constructive critiques to help team members be more accurate. And, after October 1, 2011, take advantage of the tools we will offer you to begin more formal organization of your team.

Remember: The Seven Commandments of Effective Cyber-Team Management

1. Do not let attrition demoralize you. There will inevitably be some attrition in each team. People have busy lives – and this project does require some sustained attention and effort. There are advantages as well as disadvantages to having smaller teams.

2. Do not worry when – after candid-but-diplomatic exchanges of views – you are unable to resolve disagreements within your group. There is a good chance that teams with more internal disagreement will do better, on average, than teams with less disagreement.

3. Set aside time to think about how your team is thinking. There are different strategies for making probability estimates of complex real-world phenomena. It helps when people take the time to be explicit about how they are making their judgments.

4. Learn from experience about which members of your team are better at which tasks – and capitalize on the efficiency advantages of division of labor.

5. Do not be embarrassed about admitting ignorance (these forecasting problems will cover an extraordinary range of topics).

6. Treat each other respectfully (especially others with views that you find objectionable). Constructive conflict promotes accuracy. Destructive conflict just makes people angry – and defensive.

7. Expect mistakes (some are inevitable because some things just aren't all that predictable) – but still try to learn from your mistakes and those of others.

We hope these instructions are helpful. Good luck with your team’s forecasts and your own forecasts.

Measures of Close Calls

The closeness of forecasting problem resolutions was a combination of subjective and objective metrics. The subjective assessment consisted of ratings of two experts on two items:

Closeness Assessment: On a 0-10 scale, to what extent do you think the outcome of this IFP was “close”? Enter 0 if you cannot imagine it turning out the other way, and enter 10 if you feel an alternative outcome came as close to happening as is possible, without actually happening.

Surprise Assessment: Imagine an informed observer at the initial launch date of the forecasting question. Suppose someone came back from the future and told this forecaster exactly how the outcome did or didn't happen. How surprised would the forecaster be? Enter 0 if you think an informed observer with this perspective would not be surprised at all, and enter 10 if you think an informed observer with this perspective would be extremely surprised.

The inter-rater agreement was adequate (ICC=0.57), while the within-expert, between-item reliability was high by conventional standards (Cronbach’s α=0.90). The subjective closeness index for each forecasting problem was computed as the mean rating across items and raters.

The objective metrics of closeness focused on three aspects of resolutions: timing, vote and other quantitative metric. One or more of these aspects was relevant to each of the forecasting problems.

Timing: the large majority of forecasting problems asked if an event would occur by a pre-specified date. This measure was the number of days between resolution date and planned question closing date. A resolution that occurs only 3 days before the deadline would be considered extremely close, while a resolution occurring four months after the deadline would be

considered not at all close. In an alternative specification, the metric was adjusted for planned length of the problem; the number of days between start date and planned suspend date. The two specifications yielded very similar results. Timing ratings were converted to ordinal scale as follows: questions for which events occurred 14 days or less apart from the scheduled closing date (either before or after), received a rating 5, denoting an extremely close call. Resolutions between 15 and 29 days received a rating of 4; 40-59, a rating of 3, 50-89 days a 2 rating, 90-119, a 1 rating, and events that had not occurred for 120 days after the closing date, or occurred more than 120 before that date were received 0 ratings. Election: This variable was the margin of victory in percentage points. A candidate winning by a single percentage point over an opponent is considered a close call. A victory with a 20-percentage point margin is not. The ordered bins for votes were based on margin of victory, or distance in percentage points from a threshold value. Ordered from closest to most clear cut (5 to 0 ratings), the bins were as follows: 0% to 1.99% margin of victory, 2% to 4.99%, 5% to .99%, 10% to 19.99%, 20% to 35%, and more than 35%.

Other quantitative metrics: A few questions concerned quantitative variables that were unique: exchange rates, refugees, number of Bird flu cases, etc. In those cases, we used percent deviation from criterion (e.g. dollars, refugees, patients). An exchange rate that approaches the threshold value within 1% (e.g. $1.21 vs. $1.20 per Euro) would be considered extremely close. If actual values remain 20% away from the threshold, the resolution would be considered clear cut resolution. The deviation thresholds for classification in closeness bins were as follows: 0 to 1.9%, 2% to 4.9%, 5% to 12.49%, 12.5% to 25%, 25% to 40%, 40% or more. Questions first category received a rating of 5, while question in the latter category received a 0 rating.

Adjusting the bin thresholds for the any of the objective metrics did not materially affect the conclusions drawn from the resulting analysis. If more than one objective metric was relevant to a question, the objective index was the average between the two. After obtaining the subjective and objective metrics independently, each one was converted to a standard normal distribution with mean zero and standard deviation of 1. The standardized objective and subjective indices were averaged to produce the total closeness index.