Back in 2008, I criticized the book Predictocracy for proposing prediction markets whose contracts would be resolved without reference to ground truth.
Recently, Srinivasan, Karger, and Chen (SKC) published more scholarly paper titled Self-Resolving Prediction Markets for Unverifiable Outcomes.
Continue ReadingManifold Markets is a prediction market platform where I’ve been trading since September. This post will compare it to other prediction markets that I’ve used.
The most important fact about Manifold is that traders bet mana, which is for most purposes not real money. You can buy mana, and use mana to donate real money to charity. That’s not attractive enough for most of us to treat it as anything other than play money.
Play money has the important advantage of not being subject to CFTC regulation or gambling laws. That enables a good deal of innovation that is stifled in real-money platforms that are open to US residents.
Continue ReadingI participated last summer in Tetlock’s Existential Risk Persuasion Tournament (755(!) page paper here).
Superforecasters and “subject matter experts” engaged in a hybrid between a prediction market and debates, to predict catastrophic and existential risks this century.
Continue ReadingI’ve been brainstorming about what might happen with this year’s election. Here’s one of the more interesting (but not likely) scenarios that I’ve imagined:
Most parts of the US are experiencing their second or third wave of the pandemic. Two of the more heavily funded vaccine trials have just been declared to be failures. No US or European company expects to have a vaccine ready for FDA approval before December. Prediction markets say Trump’s chances of re-election have dropped to 20%.
China announces on October 27 that Sinopharm Group has a COVID-19 vaccine that meets China’s safety and effectiveness standards, and can deliver about 1 million doses by November 2.
China offers to allocate half of this initial batch of vaccines to the US, on the grounds that the US is a relatively needy country, and Donald Trump is currently a close friend of China.
Why, I hear you wonder, would China treat Trump as a friend?
China is strong enough that the government can afford to tolerate Trump’s annoying trade wars, and they maybe even see some benefit from reducing China’s somewhat excessive dependence on the US. It’s important to keep in mind that democracy is one of the larger threats to the Beijing regime (h/t scholars-stage). 2020 has proven to be a good year to mount a campaign to demonstrate that socialism with Chinese characteristics is superior to US-style democracy. I’ll leave as an exercise for my readers to figure out how many ways the idea of democracy could be discredited by a Trump re-election.
Why would a Chinese company be faster than US/European companies at producing a vaccine? My initial guess was differences in regulatory red tape would favor China, but my attempt to find evidence for such a pattern turned up nothing. My current guess is that differences between countries in who volunteers for a vaccine trial will have important effects on how quickly the control groups get infected. Trials in some countries may attract only people who are sufficiently risk-averse that the control groups are slower than expected at getting infected. Please don’t assume that I have any useful expertise here; I’m mostly just guessing.
I also wondered whether willingness to do human challenge trials would determine who verifies their vaccine first. I have vague intuitions that China is more likely than other countries to try that, but I haven’t found evidence to confirm those intuitions.
How would voters react to this scenario? My best guess is that the election would be surprisingly close, but Trump would still lose.
The stock market would rise due to the vaccine benefits. There would be no good way to infer the market’s opinion about the election until the results are announced. I’m still confused as to how this scenario should affect my investment strategies.
P.S. – China and Sinopharm aren’t willing to predict when they’ll able to submit the forms needed for FDA approval, and ask that the US consider the approval of China’s NMPA to be good enough evidence of the vaccine’s safety and effectiveness. How does the FDA react?
The stock market crash of the past two weeks looks like an over-reaction to COVID-19.
Is COVID-19 really the reason for the crash? I can’t find any other news that would explain the timing and which stocks were hit hardest.
Here’s a sample of some of the harder hit stocks, all travel-related (Friday’s close compared to the highest close in February):
It is, of course, possible that the market was in a mild bubble in early February, and the virus merely triggered a return to sanity. There were enough high-priced stocks that I’ll guess that’s explains a little of what happened. Hertz and Avis are maybe high-risk stocks due to the risks associated with the upcoming transition to robocars. But the others that I listed did not at all fit my stereotype of overpriced stocks.
And the stocks that I had been thinking were overpriced, in industries that don’t look to be especially hurt by the virus, declined roughly in line with the market.
Outside of travel-related stocks, it mostly looks like a general shift in preferences to more cash, and away from stock. I.e. a general increase in risk aversion.
The gold market is confused as to which direction a pandemic should move it. I agree. I’m confused as to how gold should react.
What scenario could explain the decline? Maybe a two month shutdown of 90+% of U.S. air travel? A multi-year reduction in travel of 10%? It would take something like that for the market reaction to make much sense. Yet I’d bet at roughly 10:1 odds against any one of those scenarios happening.
Metaculus is currently predicting 195k COVID-19 deaths this year.
Metaculus forecast trends ought to look a good deal like random walks, yet the charts I see there look more like exponential growth.
Metaculus is likely to be a more objective source of information than the news media storyteller industry or social media. But it’s likely more susceptible to selection effects and hype than are markets that have lots of money at stake. (Metaculus has token prizes, structured in a way that may encourage more extreme bets than a regular market would).
None of this implies much about where other reactions to the virus are sensible. There’s a much different asymmetry between getting sick versus being paranoid than there is between losing money due to a pandemic versus losing money due to selling on a false alarm.
I’ve got about a month’s supply of food, but that’s my normal preparation for a variety of disasters. I have no special insights about whether the current risks justify staying home.
P.S. Chinese stocks are supporting the view that the situation in China has improved over the past month.
No, this isn’t about cutlery.
I’m proposing to fork science in the sense that Bitcoin was forked, into an adversarial science and a crowdsourced science.
As with Bitcoin, I have no expectation that the two branches will be equal.
These ideas could apply to most fields of science, but some fields need change more than others. P-values and p-hacking controversy are signs that a field needs change. Fields that don’t care much about p-values don’t need as much change, e.g. physics and computer science. I’ll focus mainly on medicine and psychology, and leave aside the harder-to-improve social sciences.
The term “science” has a range of meanings.
One extreme focuses on “perform experiments in order to test hypotheses”, as in The Scientist In The Crib. I’ll call this the personal knowledge version of science.
A different extreme includes formal institutions such as peer review, RCTs, etc. I’ll call this the authoritative knowledge version of science.
Both of these meanings of the word science are floating around, with little effort to distinguish them [1]. I suspect that promotes confusion about what standards to apply to scientific claims. And I’m concerned that people will use the high status of authoritative science to encourage us to ignore knowledge that doesn’t fit within its paradigm.
Book review: Superforecasting: The Art and Science of Prediction, by Philip E. Tetlock and Dan Gardner.
This book reports on the Good Judgment Project (GJP).
Much of the book recycles old ideas: 40% of the book is a rerun of Thinking Fast and Slow, 15% of the book repeats Wisdom of Crowds, and 15% of the book rehashes How to Measure Anything. Those three books were good enough that it’s very hard to improve on them. Superforecasting nearly matches their quality, but most people ought to read those three books instead. (Anyone who still wants more after reading them will get decent value out of reading the last 4 or 5 chapters of Superforecasting).
The book’s style is very readable, using an almost Gladwell-like style (a large contrast to Tetlock’s previous, more scholarly book), at a moderate cost in substance. It contains memorable phrases, such as “a fox with the bulging eyes of a dragonfly” (to describe looking at the world through many perspectives).
The stock market reaction to the election was quite strange.
From the first debate through Tuesday, S&P 500 futures showed modest signs of believing that Trump was worse for the market than Clinton. This Wolfers and Zitzewitz study shows some of the relevant evidence.
On Tuesday evening, I followed the futures market and the prediction markets moderately closely, and it looked like there was a very clear correlation between those two markets, strongly suggesting the S&P 500 would be 6 to 8 percent lower under Trump than under Clinton. This correlation did not surprise me.
This morning, the S&P 500 prices said the market had been just kidding last night, and that Trump is neutral or slightly good for the market.
Part of this discrepancy is presumably due to the difference between regular trading hours and after hours trading. The clearest evidence for market dislike of Trump came from after hours trading, when the most sophisticated traders are off-duty. I’ve been vaguely aware that after hours markets are less efficiently priced. But this appears to involve at least a few hundred million dollars of potential profit, which somewhat stretches the limit of how inefficient the markets could plausibly be.
I see one report of Carl Icahn claiming
I thought it was absurd that the market, the S&P was down 100 points on Trump getting elected … but I couldn’t put more than about a billion dollars to work
I’m unclear what constrained him, but it sure looked like the market could have absorbed plenty more buying while I was watching (up to 10pm PST), so I’ll guess he was more constrained by something related to him being at a party.
But even if the best U.S. traders were too distracted to make the markets efficient, that leaves me puzzled about asian markets, which were down almost as much as the U.S. market during the middle of the asian day.
So it’s hard to avoid the conclusion that the market either made a big irrational move, or was reacting to news whose importance I can’t recognize.
I don’t have a strong opinion on which of the market reactions was correct. My intuition says that a market decline of anywhere from 1% to 5% would have been sensible, and I’ve made a few trades reflecting that opinion. I expect that market reactions to news tend to get more rational over time, so I’m now giving a fair amount of weight to the possibility that Trump won’t affect stocks much.
MIRI has produced a potentially important result (called Garrabrant induction) for dealing with uncertainty about logical facts.
The paper is somewhat hard for non-mathematicians to read. This video provides an easier overview, and more context.
It uses prediction markets! “It’s a financial solution to the computer science problem of metamathematics”.
It shows that we can evade disturbing conclusions such as Godel incompleteness and the paradox of the liar, by expecting to only be very confident about logically deducible facts (as opposed to being mathematically certain). That’s similar to the difference between treating beliefs about empirical facts as probabilities, as opposed to boolean values.
I’m somewhat skeptical that it will have an important effect on AI safety, but my intuition says it will produce enough benefits somewhere that it will become at least as famous as Pearl’s work on causality.
Automated market-making software agents have been used in many prediction markets to deal with problems of low liquidity.
The simplest versions provide a fixed amount of liquidity. This either causes excessive liquidity when trading starts, or too little later.
For instance, in the first year that I participated in the Good Judgment Project, the market maker provided enough liquidity that there was lots of money to be made pushing the market maker price from its initial setting in a somewhat obvious direction toward the market consensus. That meant much of the reward provided by the market maker went to low-value information.
The next year, the market maker provided less liquidity, so the prices moved more readily to a crude estimate of the traders’ beliefs. But then there wasn’t enough liquidity for traders to have an incentive to refine that estimate.
One suggested improvement is to have liquidity increase with increasing trading volume.
I present some sample Python code below (inspired by equation 18.44 in E.T. Jaynes’ Probability Theory) which uses the prices at which traders have traded against the market maker to generate probability-like estimates of how likely a price is to reflect the current consensus of traders.
This works more like human market makers, in that it provides the most liquidity near prices where there’s been the most trading. If the market settles near one price, liquidity rises. When the market is not trading near prices of prior trades (due to lack of trading or news that causes a significant price change), liquidity is low and prices can change more easily.
I assume that the possible prices a market maker can trade at are integers from 1 through 99 (percent).
When traders are pushing the price in one direction, this is taken as evidence that increases the weight assigned to the most recent price and all others farther in that direction. When traders reverse the direction, that is taken as evidence that increases the weight of the two most recent trade prices.
The resulting weights (p_px in the code) are fractions which should be multiplied by the maximum number of contracts the market maker is willing to offer when liquidity ought to be highest (one weight for each price at which the market maker might position itself (yes there will actually be two prices; maybe two weight ought to be averaged)).
There is still room for improvement in this approach, such as giving less weight to old trades after the market acts like it has responded to news. But implementers should test simple improvements before worrying about finding the optimal rules.
trades = [(1, 51), (1, 52), (1, 53), (-1, 52), (1, 53), (-1, 52), (1, 53), (-1, 52), (1, 53), (-1, 52),]
p_px = {}
num_agree = {}
probability_list = range(1, 100)
num_probabilities = len(probability_list)
for i in probability_list:
p_px[i] = 1.0/num_probabilities
num_agree[i] = 0
num_trades = 0
last_trade = 0
for (buy, price) in trades: # test on a set of made-up trades
num_trades += 1
for i in probability_list:
if last_trade * buy < 0: # change of direction
if buy < 0 and (i == price or i == price+1):
num_agree[i] += 1
if buy > 0 and (i == price or i == price-1):
num_agree[i] += 1
else:
if buy < 0 and i <= price:
num_agree[i] += 1
if buy > 0 and i >= price:
num_agree[i] += 1
p_px[i] = (num_agree[i] + 1.0)/(num_trades + num_probabilities)
last_trade = buy
for i in probability_list:
print i, num_agree[i], '%.3f' % p_px[i]