How to Play a Prediction Market

The prediction market is a way to try to assign probabilities to events. Bettors buy YES bets on things they think are likely to happen (relative to the market price) and NO bets on things they think are unlikely to happen (relative to the market price). Market dynamics lead the market price to settle on what is, across the bettors, the best subjective probability of the event. This is useful if you are trying to assign probabilities to one-off future events.

In this post, I'll teach you how to place bets to most effectively get the largest payout possible. In so doing, you'll do more to calibrate the market to your predictions.

Let's get ready to corner the replication market!

How does a prediction market work?

A prediction market allows people to bet YES or NO on some outcome. As people bet that the outcome will happen, the price of YES shares increases. As people bet that the outcome won't happen, the price of YES shares falls.

The market price for a YES share is p, the probability of the outcome. The market price for a NO share is (1-p). If the event happens, all the YES shares pay out $1 each and the NO shares become worthless. If the event does not happen, all the NO shares pay out $1 each and the YES shares become worthless.

The probability of rolling a six is 1/6, so we should be willing to pay up to $1/6 for YES or $5/6 for NO.

Imagine we are betting that a roll of a six-sided die will yield a six. The probability of this is 1/6, or about 17 percent. YES shares will cost 17 cents and NO shares will cost 83 cents. With five dollars, you could buy 30 YES shares or 6 NO shares.

Your expected payout is the number of shares times the probability. In the die example, since the market price is correct, your expected value is five dollars whether you buy YES or NO. For YES shares, 30 shares * (1/6 payout chance) = $5. For NO shares, 6 shares * (5/6 payout chance) = $5.

If the market price is wrong, you have a chance to make a profit. Suppose we are still betting on the die, but for some reason the market price is set at 10 cents for a YES share. We know that the probability of the die rolling six is greater than this, so with our five dollars we can buy 50 shares with an expected value of 50 shares * (1/6 payout chance) = $8.33. This is a profit of three dollars. Another way to look at this is that it's a profit of six cents per share, the difference between the wrong market price (.10) and the true probability (.16).

But if the market price is wrong, and we are wrong with it, we will lose money. Buying NO shares at this price will turn our five dollars into 5.55 shares * (5/6 payout chance) = $4.62, a long-run loss of 38 cents.

The Big Picture of the Big Short

Like we covered above, playing the prediction market isn't simply about buying YES on things you think will replicate and NO on things you don't replicate. Otherwise, we would just buy NO shares on the die rolling six because we know it's unlikely relative to the die not rolling six. It's about evaluating the probability of those replications. Your strategy in a betting market should be to look for those opportunities where there is a difference between the market price and the probability that you'd assign to that event.

If the market is completely correct, it shouldn't matter what you buy -- your 50 tokens will have an expected value of $50. In our die example above, when the market price was right, YES and NO shares had the same expected value. But if the market is wrong, you have a chance to beat the market, turning your 50 tokens into several times their value.

In order to beat the market, you have to find places where the market price is miscalibrated. Maybe something is trading at 40% when it only has a 20% chance to replicate, in your view. If you are right, each NO share you buy will cost 60 cents but have an expected value of 80 cents. But if you are wrong, you will pay more for the shares than they are truly worth, getting a poorer return on your 50 tokens than had you just spread them across the market.

Below is my four-step process for turning your predictions into the largest possible payoff.

1. Evaluate your prices.

Before the market started, I wrote down my estimates of what would or wouldn't replicate. I assigned probabilities to these studies indicating what chance I thought they had of replicating.

Coming up with these estimates is the basis of the replication market. I ended up focusing on the things I thought wouldn't replicate. Some studies were a priori deeply implausible, others had weak p-values, some had previous failures to replicate, and some had a combination of factors. These were studies I felt pretty confident wouldn't replicate, and so I priced them at about 10% (2.5% chance of Type I error + 7.5% chance of true effect).

A peek at my spreadsheet, comparing my subjective probabilities to the market prices.

Some other studies seemed more likely to replicate, so I was willing to price them in the 50-80% range. I was less certain about these, so I saw these as riskier purchases, and tended to invest less in them.

It's also useful to remember the context of the last prediction market. In that market, the prices were much too high. Nothing below 40% replicated, and the highest-priced study (88%) also failed to replicate. The lowest price on a successful replication was about 42%.

2. Buy and sell to your prices.

To make profit on the replication market, you have to spend your money where you think the market price is most miscalibrated. Something that the market thinks is a sure thing (95%) that you think will flop (5%) would be a massive 90-cent profit per share. Something that seems reasonable (50%) that the market is afraid won't replicate (15%) could be a nice little profit of 35 cents per share.

I made a spreadsheet of my prices and the market's prices. I added a column representing the difference between those prices. The largest absolute difference indicates where I would expect the greatest profit per share.

If the difference is negative, then buy NO shares. Suppose something is trading at 50%, but you think it has only a 15% chance of replicating. You can buy NO shares for 50 cents that you think are worth 85 cents -- a 35 cent profit per share.

If the difference is positive, then buy YES shares. If something is trading at 50%, and you think it has a 75% chance of replicating, then every YES share costs 50 cents but is worth 75 cents.

Overly optimistic market prices meant that I placed most of my bets on certain studies not replicating.

Again, you only profit when you are right and the market is wrong. Look for where there is juice!

3. Diversify your portfolio

If you want to ensure a decent payout, it may make sense to spread your money around. Suppose there is a study priced at 50% chance of replicating, but you know the true chance of replication is 80%. If you're right, putting all 50 tokens on this one study has an 80% chance of earning you $100, but a 20% chance of earning you $0. Your expected value is $80, a nice $30 profit, but there's a lot of variability.

Payout	$100	$0
Frequency	80%	20%
EV = $80; SD = $41

By diversifying your bets, you can reduce the variability at the cost of reducing your expected value slightly. Consider if we divide your bets across two options, one with a slightly worse profit margin. Let's say Study 1 is priced at 50% but is worth 80%, and Study 2 is priced at 65% but is worth 75%. By putting half our money into Study 2, we reduce our average profit, but we also reduce the likelihood of suffering a blowout.

Payout	$88	$50	$38	$0
Frequency	60%	20%	15%	5%
EV = $70; SD = $26

In the recent market, for example, Sparrow, Liu, and Wegner tended to trade at 55%, whereas I thought it was worth about 15%. Although this 40-cent gap would have been my biggest profit-per-dollar, I felt it was too risky to put everything on this study, so I balanced it against other studies with smaller profit margins.

4. Day trading

As other people show up to the market and start twiddling their bets around, the market prices will change. The market may move towards some of your predictions and away from other of your predictions. If you like to procrastinate by watching the market, you can leverage out your bets for a higher potential payout.

Figure 1. You hold NO shares of Studies 1 and 2, which the market has evaluated at 35% (bars) but you think have only a 10% chance of replicating (dashed line). Each share represents 25 cents of profit to you.

Lets say you think Study 1 and Study 2 each have a 10% chance of replicating. You bought 30 shares of Study1 NO and Study2 NO for 65 cents a share each (35% chance to replicate). You see each of these shares as representing a 25 cent profit (Figure 1).

Figure 2. The market has shifted such that your Study1 NO shares are worth more and your Study2 NO shares are worth less. If you are ready to be aggressive, you can sell your Study1 NO shares to take advantage of cheaper Study2 NO shares.

Some time passes, and now the market has agreed with you on Study1, dropping its probability to 20%, but it disagrees with you on Study2, raising the probability to 45% (Figure 2). The shares of Study 1 you're holding have already realized 15 cents per share of profit. The shares of Study 2 you're holding have lost 10 cents a share, but if you are right, then you can keep buying these shares at 55 cents when you think they are worth 90 cents.

Since the Study 1 shares have already realized their value, you can sell the Study1 NO shares to buy more cheap shares of Study2 NO. If the market fluctuates again, you can sell your expensive shares to pick up more cheap shares and so on and so on.

I watched the market and kept comparing the prices against my predictions. When one of my NO bets started to cap out (e.g., Gervais and Norenzayan reached 15%), I would sell my NO bets and reinvest them in another cheaper NO bet (e.g., buying NO on Kidd and Castano at 40%). Sometimes some poor credible soul (or somebody fumbling with the GUI) would buy a bunch of YES bets on Ackerman, driving the price way up (e.g., to 45%). When this would happen, I'd sell all my current bets to take advantage of the opportunity of cheap Ackerman NO bets.

It can be tempting to try to play the market, moving your tokens around to try to catch where other people will move tokens. I don't think there's much use in that. There aren't news events to influence the prediction market prices. Just buy your positions and hold them. If the market disagrees with you, you may consider doubling down on your bets now that they are cheaper. If the market agrees with you, you can release those options to invest in places where the market disagrees with you.

Summary

To make the biggest profits, you have to beat the market. To do this, you must: (1) Make good estimates of the probability to replicate. (2) Find the places where the market price is most divergent from what probability you would assign the study. (3) Spread your bets out across a number of studies to manage your risk. (4) Use day trading to take advantage of underpriced shares and increase your total leverage.