When your model diverges from reality, how do you decide which is wrong? The answer is usually obvious — until perhaps the object of study is humans, whereupon one feels the irresistible urge to declare that it is they who are mistaken. The reader of §2 is, I hope, now convinced that more people should be using prediction markets to satisfy their informational needs. And yet — Eppur si muove — they mostly do not. Why is this? We will try to avoid both the economist’s standard move of declaring them irrational, and the Hansonian move of re-describing them as rationally pursuing hidden motives. Instead, let us take their reluctance at face value, and ask what structural limitations of prediction markets themselves might explain their underuse1.
I.
I’ll begin by confessing to some equivocation. In my exposition and defence of prediction markets, I made an argument inspired by Hayek’s famous 1945 article, The Use of Knowledge in Society: that the strength of markets lies in their ability to incentivise the collection of information dispersed across individuals, and aggregate that information in price signals. Hayek was concerned with ordinary markets for goods and services, but my argument has been that prediction markets extend the same epistemic logic to beliefs about the future. But what kind of information would we need to aggregate, exactly, in order to usefully predict the future?
For Hayek, the price system allocates resources more effectively than any central planner could, by aggregating “the knowledge of the particular circumstances of time and place” — the local, often inarticulable, rapidly changing information about what people want (their subjective valuations); what resources and inputs are available where and in what condition; current production possibilities and bottlenecks; and the relevant opportunity costs. Prices transmit this diffused knowledge as signals of relative scarcity2, letting decentralised actors coordinate without anyone knowing the full underlying detail. Such tacit, practical knowledge is diffused across millions of individuals, and usually not written down — take the shopkeeper who knows what customers in their neighbourhood want this week — so a central planner cannot make use of it3, and thus cannot rival the adaptive efficiency of the market process.
Notice that the epistemic logic here depends on a specific ontology of knowledge — namely tacit, local, context-specific, and immediately actionable information about real resources and preferences. Crucially, it seems to presume that the relevant knowledge is already present, scattered across the minds of many individuals, waiting to be aggregated. The same logic underlies the Galtonian story from §1: the true weight of the ox had, in a sense, already leaked into the crowd — fragments of relevant knowledge were scattered among the villagers, converging to a remarkably accurate median estimate — and it simply needed to be extracted.
This invites the natural question: to what extent do anticipatory beliefs about the future have this structure? Or in other words, how leaky is the future to the present?
II.
Consider as a motivating example the case of Knightian uncertainty, a situation in which the relevant outcomes and their probabilities are not merely unknown but indeterminate, so that no coherent ex ante probability distribution can be assigned. Economist John Maynard Keynes has beautifully explained the concept4:
By “uncertain” knowledge, let me explain, I do not mean to distinguish what is known for certain from what is only probable. The game of roulette is not subject, in this sense, to uncertainty […] The sense in which I am using the term is that in which the prospect of a European war is uncertain, or the price of copper and the rate of interest twenty years hence, or the obsolescence of a new invention, or the position of private wealth-owners in the social system in 1970. About these matters there is no scientific basis on which to form any calculable probability whatever. We simply do not know. Nevertheless, the necessity for action and for decision compels us as practical men to do our best to overlook this awkward fact and to behave exactly as we should if we had behind us a good Benthamite calculation of a series of prospective advantages and disadvantages, each multiplied by its appropriate probability, waiting to be summed.5
Evidently, if the information we are seeking is like one of Keynes’ examples, then although the probability implied by a market price may be “better than nothing”, the error bars around it are enormous. In such cases, the marginal gain in precision from aggregating many uncertain guesses may not justify the cost of elicitation; there’s just too little dispersed knowledge that the market might usefully aggregate.
Even if one does not fully accept this notion of radical Knightian uncertainty6, with its unquantifiable probabilities, the argument nonetheless suggests an important lemma: it is often not that valuable to just aggregate all information readily available to market participants. If we want to estimate the likelihood that a radically new project succeeds, the manager’s best guess may be less accurate than the crowd’s; but if both arise from the same basic uncertainty, the expected informational surplus of pooling their decentralised judgments is unlikely to justify the coordination costs involved.
III.
I am not just making the boring and obvious Coasean argument that “[t]here is a cost of using the price mechanism” that might outweigh the informational benefits of prediction markets, leading organisations to prefer direct managerial judgement whenever it is cheaper7 — though this remains an important consideration, to which we will return. And I am certainly not making the boring and wrong argument that the future is so inscrutably uncertain that it’s pointless trying to reason probabilistically about it; that would ignore the clear forecasting success of financial markets.
The real point is subtler. Recall that our original question was the extent to which the future leaks clues about its own development to the present. What we have seen is that in many domains, it fails to supply such clues to anyone’s mind; there is little dispersed private knowledge for prices to aggregate. So if prediction market prices only aggregate what is already known, in the way Hayek (1945) describes, their usefulness will be constrained by the poverty of the underlying knowledge set.
But is that, in fact, all market prices do? This assumption neglects the crucial insight of a very different thinker8 to Hayek (1945). I refer of course to Hayek (1968), whose piece Competition as a Discovery Procedure reframes the market not as a device for transmitting a given stock of dispersed knowledge, but as a process that generates new knowledge through competitive experimentation. Whereas the 1945 essay shows how the price system enables coordination despite no individual possessing “the” data set, the latter work emphasises that the data set itself is not “given” in advance. Rather, market competition is an experimental process that creates knowledge: it directs attention, incentivises costly search, and forces entrepreneurs to form conjectures, then subjects those conjectures to an evolutionary process of trial and error, in which unsuccessful hypotheses are pruned through imitation and selection. Markets, in other words, are not only mechanisms for aggregating what people already know, but institutions that enable us to discover what nobody yet knows.
My claim will be that prediction markets are typically most valuable when they are not just aggregating dispersed knowledge, but when they actually succeed in financing and incentivising this process of knowledge discovery.
IV.
I am being ironic when I call this later Hayek a “very different thinker”. Although one could see this move as assigning markets two distinct functions — an aggregation function and a discovery function — it is more illuminating to understand the 1968 essay as a generalisation of the 1945 one. The earlier piece is about how markets aggregate information that exists in people’s minds, usually as tacit knowledge; the latter is about how markets aggregate information that is latent in the environment but perhaps not yet known, by providing incentives to unearth and incorporate it.
Having understood the 1968 move in this way, we can now return to our core framing, in which the central question is “how leaky is the future to the present, and where do the leaks show up?”. By sketching a taxonomy of “leakiness”, we will be in a better position to understand the domains in which prediction markets are likely to succeed.
At one extreme lies what we might call “mind-leaky” domains, situations in which relevant information leaks copiously and continuously into individual experience. For example, in supply-chain and retail contexts, dispersed actors observe shipment delays, changing input quality, or local demand fluctuations. Clearly markets excel in these domains, because they can efficiently pool what people know tacitly and observe locally into informative price signals.
At the opposite extreme lies the case of Knightian uncertainty discussed above. This is the limiting form of a broader class of “non-leaky”, or even “adversarially leaky”, domains. Sometimes9 the world does not contain stable, exploitable structure that maps present evidence to future outcomes; or the relevant causal pathways are so underdetermined that any model is mostly projection; or the information is strategically concealed and the observable signals are mostly noise or propaganda.
Prediction markets are ill-suited to both extremes. Most forecasting questions do not lie near the first extreme — except in limited cases, such as someone having insider information on a decision which is not yet public, future events are never directly observed or known in advance. And at the non-leaky end, market prices can always be formed — as one can always compress one’s uncertainty into a subjective probability — but as noted earlier, Coasean constraints quickly bind. While markets will provide a better forecast than alternative institutions10 (since they can incorporate their judgements), in non-leaky domains, the final probability estimate is too noisy11 for the marginal improvement to justify the costs of running and subsidising the market.
In between these poles lies a large class of domains which are “environment-leaky”, but mind-opaque. Here the future does leak information to the present, but not in the form of ready-made beliefs or local observations sitting in anyone’s head. The clues are instead “out there”, in the form of scattered data, weak correlations, and partial analogies that only become informative after someone does the work of extracting and synthesising them. In such settings, the market’s distinctive contribution is to create a competitive setting in which it is worth doing this hard epistemic work. This is the region in which prediction markets are most valuable, as they reward those who can translate these environmental traces into forecasts12 and make bad models costly to sustain, thereby endogenously selecting and amplifying methods that track reality.
V.
At this point, the reader may feel that the preceding taxonomy has been a digression. After all, if markets both aggregate existing knowledge and generate new knowledge, then why worry about whether the future is “mind-leaky” or not? Even if no individual already possesses the relevant information, can’t prediction markets simply lean on their capacity for discovery — rendering distinctions about leakiness largely irrelevant?
What this leaves out is how the process of market discovery actually works in practice. Although I have so far described it in abstract terms — conjectures, experimentation, selection — in real markets, this process is mediated by a very concrete institutional mechanism: specialisation and division of labour. This matters little if markets are only aggregating what is known, since anyone already possessing information has an incentive to trade on it. But discovery is not driven by diffuse marginal improvements; it depends on asymmetric investments by a subset of participants in model-building, data collection, hypothesis testing, and method development, and so forth. Prices then act as a selection mechanism over these heterogeneous epistemic strategies.
This dynamic is starkly visible in public financial markets, which we have repeatedly used to motivate the claim that prediction markets can attain informational efficiency. Because these markets are large, liquid, and continuously active, they can sustain an entire ecology of full-time discovery — analysts, quants, short-sellers, data-vendors — all competing to extract weak signals from the environment and turn them into tradable beliefs. Asset prices are the result of intense competition among specialised actors, including quantitative funds fitting statistical models, discretionary traders forming macro narratives, industry experts tracking regulatory minutiae, and so on.
This works better than simply commissioning some designated expert to produce a model, not only because of the familiar advantages of skin in the game, but because the market is itself a mechanism for recruiting, organising, and filtering expertise. By attaching rewards to predictive accuracy, it attracts effort from whomever is best positioned to extract signal, and places them in a competitive environment. In effect, it can assemble a small research community oriented around the question — an ad hoc academia whose membership and methods are determined by performance.
VI.
The metaphor of an “ad hoc academia” is, regrettably, not mine. It’s a phrase Arnold Brooks used on a Minds Almost Meeting podcast episode with Robin Hanson and Agnes Callard on polls versus prediction markets, to gloss a conceptual distinction Hanson drew there. He separates raw data, such as polls, from forecasts built from that data — like Nate Silver’s models, or an academic paper — and from the forum in which forecasts are disputed, like a prediction market or an academic field or journal.
But in that discussion, it comes out as13: “we can turn our managers into a little mini ad hoc academia that will aggregate this information.” The implied picture is of a fixed set of insiders, repurposed as forecasters, whose dispersed views are distilled into a price — so the market functions mainly as an internal communication system, only marginally superior to other communication systems, through which pre-existing knowledge is transmitted. In that frame, scale helps by adding noise traders, whose uninformed trades create incentives for the informed to reveal what they know14.
But as we have seen, the benefits of a marginally better communication system are often small. The deeper value of markets lies in discovery, and once that more general frame is adopted, the benefits of scale come with an extra requirement: that the market can select who does epistemic work. Different people are differently positioned to extract signal from the environment, and doing so is itself a specialised skill. If the “academics” are chosen in advance, the selection mechanism disappears, and with it, much of what prediction markets are supposed to be buying you.
Thus, the framework I have developed makes clear why the mechanism by which prediction markets deliver value collapses inside firms. Internal markets are typically closed to outsiders, and insiders are already specialised in their operational roles; investing in becoming good at internal prediction competes with the job they are hired, evaluated, and promoted to do. Lacking openness, liquidity, and a population for whom discovery itself is a viable specialisation, such markets cannot sustain the ecology of epistemic competition that makes large public markets informative.
VII.
We have now answered the motivating question. Prediction markets are not more widely used because in many settings, the expected benefits of sponsoring one are small relative to the costs. Aggregating what is already known can be done far more cheaply, with results that are not much worse — because when forecasting the future, there is little readily available information. And while specialisation in goods markets emerges almost automatically, limited scale in many prediction markets prevents the specialisation and division of epistemic labour needed to discover new information.
Seen in this light, at least some of the underuse of prediction markets does not require the kind of explanation Robin Hanson has offered, in which people profess to value accuracy and good decision-making while in fact pursuing hidden social or political motives15. Firms may not be perfectly rational, but they are profit-maximising entities; the reason they do not use prediction markets is that there is no competitive advantage to be gained from doing so. Indeed, Google has twice experimented with internal prediction markets16, and neither experiment ultimately endured, suggesting that the obstacle is not a reluctance to experiment, but a structural impediment.
If my diagnosis is correct, the practical path forward looks quite different from what Hanson’s account would suggest. Rather than fighting an uphill battle to overcome organisational and status quo bias inside firms, advocates should focus on domains where prediction markets have a comparative advantage: large-scale, high-stakes questions where substantial resources can be mobilised to support specialised epistemic labour. This points towards public rather than private applications — macroeconomic targets for central banks, fiscal projections by institutions like the CBO and OBR, or the ex ante evaluation of major policy interventions. In these cases, the Hansonian explanation regains some force: public actors like governments, central banks, and international organisations lack a profit motive, making it more plausible that they fail to adopt prediction markets despite their potentially large social value.
The discussion in this piece has deliberately remained abstract and theoretical, apart from these tentative gestures at what will follow. In the next section17, we will develop these suggestions in more detail by applying the preceding analysis to empirical case studies, spelling out the predictions the framework generates, and using them to derive recommendations for how advocates of prediction markets should proceed.
Title cribbed from Somewhat Contra Marcus On AI Scaling - by Scott Alexander.
As he writes in that essay:
The marvel is that in a case like that of a scarcity of one raw material, without an order being issued, without more than perhaps a handful of people knowing the cause, tens of thousands of people whose identity could not be ascertained by months of investigation, are made to use the material or its products more sparingly; that is, they move in the right direction.
Since the central planner collects and has access only to statistics and “data”, and is therefore unable to make use of this local knowledge, which is tacit and not recorded.
Alternatively, see Frank Knight’s own description, in Risk, Uncertainty, and Profit (1921):
It will appear that a measurable uncertainty, or “risk” proper, as we shall use the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all. We shall accordingly restrict the term “uncertainty” to cases of the non-quantitive [sic] type. It is this “true” uncertainty, and not risk, as has been argued, which forms the basis of a valid theory of profit and accounts for the divergence between actual and theoretical competition.
That is, I invoke Knightian uncertainty as an intuition pump, not as a premise on which my argument crucially depends. I later embed it in a more general taxonomy as the “limiting form” of cases where the future is opaque to us, but you need not think such a thing in fact exists to accept my lemma. Indeed, my friend Ben Shindel — self-described “forecasting hobbyist” and author of Manifold’s weekly newsletter — was having none of it:
I’ll quantify any uncertainty. […]
I don’t think there’s such a thing as truly “unquantifiable uncertainty” and I don’t think there’s a clear distinction between risk and uncertainty of the nature Knight proposes.
I agree with Ben that actually, some of Keynes’ examples are pretty bad; as he says:
[T]he prospect of a European war is not fundamentally different from a roulette wheel. It’s just far more complex. […] There are also tons of “scientific bases” upon which to forecast. Buildup of armaments. Population growth. Number of border incidents over time. Etc etc.
I still think that Knightian uncertainty is a real phenomenon, and that the distinction between risk and uncertainty is a legitimate one, for the following kind of reason:
What if I phrased it in terms of uncertainty re the space of outcomes to which we’re assigning probabilities? So obviously probabilities require some kind of event space, and you’re saying clearly there will always be better and worse probability assignments. […] But the point is that (say, scientific) experimentation can reveal outcomes (elements of the sample space) which we didn’t know existed. And so there’s a kind of irreducible uncertainty in that we can’t assign ex ante probabilities to *those* outcomes.
But like I say, nothing in this blog post turns on resolving this dispute. If you’re interested, you can see our full conversation here. You should also subscribe to Ben’s Substack.
From The Nature of the Firm (1937):
“The main reason why it is profitable to establish a firm would seem to be that there is a cost of using the price mechanism.”
— Section II
“The costs of carrying out a transaction by means of an exchange on the open market must be less than the costs of organizing the same transaction within the firm.”
— Section IV
“A firm will tend to expand until the costs of organizing an extra transaction within the firm become equal to the costs of carrying out the same transaction by means of an exchange on the open market.”
— Section IV
I understand that this is a Friday Night Dinner reference; thanks Izzy!
There may be an interesting way to formulate the ideas in this post in terms of Stephen Wolfram’s and Jonathan Gorard’s ideas about computational irreducibility. Our goal is to predict the future. But in some sense, the universe already does predict the future: it computes the transition from its state at the present time to all its future states, at the speed of time. Forecasters would like to “shortcut” this computation, that is, to find a procedure that reaches the answer with less computational work than letting the system unfold. Prediction markets work by enabling one to bid for some of the computational effort out there, by rewarding agents who invest resources in extracting structure from the present that bears on the future. But in certain domains, this effort is (nearly) futile. These correspond to cases of computational irreducibility, in which no systematic shortcut exists, because of the principle of computational equivalence — any process capable of producing accurate predictions must effectively mirror the underlying evolution it is trying to predict.
Of course, it is right that we compare markets to these existing institutions, rather than to an omniscient God. But that means comparing not only their informational gains, but also the costs. By “better”, I mean only “at least as good as, in terms of informational efficiency”.
Charles Dillon notes that my use of “noisy” here is slightly overloaded:
Do you mean "they are uncalibrated"? Or "they are no better than an estimate one could make without them"? I'm not sure what "noise" is supposed to represent in terms of the true underlying distribution.
I essentially mean high conditional variance around the (latent) true event probability. This needn’t imply miscalibration in the technical sense — for instance, forecasts labelled 30% can still occur about 30% of the time on average. Rather, it means the forecast is a very imprecise signal of the underlying probability at the case level. Only in the limiting case — where that conditional variance is so large that the signal contains essentially no incremental information — does the market become “no better than” baseline alternatives. As that limit is approached, the marginal advantage over substitutes goes to zero, and prediction markets correspondingly become less cost-effective.
By “discovered”, I don’t necessarily just mean finding facts, hidden away in a file drawer. Often the discovery is: which model class works? Which proxy variable tracks the underlying process? Which dataset is reliable? Which confounder matters? Which seemingly-plausible frame collapses under a backtest?
To clarify, it is Brooks who says this, but Hanson responds affirmatively — admittedly to a whole bundle of claims, but without correcting what I take to be a misleading implication of the metaphor.
Incidentally, Grossman & Stiglitz (1980) argue that markets cannot be perfectly informationally efficient, since if prices revealed all relevant information, then no trader has any incentive to incur the costs of producing that information in the first place, so full informational efficiency cannot be an equilibrium. But in my framing, no such paradox presents itself. The decision to acquire information depends on the leakiness/opacity of the domain — whether there exists exploitable structure in the environment not yet incorporated into prices. Yet this is not something agents can know ex ante: to discover whether discovery is worth doing, one must already do some discovery. This means we can relax some of the formal Grossman-Stiglitz assumptions — that information is fixed and well-defined in advance, that agents know its value before acquiring it, and that information acquisition is marginal. Consequently, it is conceptually possible for market prices in equilibrium to be perfectly informationally efficient: all discoverable information is incorporated into prices, but agents can’t know this and so keep investing in discovery. This can be understood as Hayek’s knowledge problem applied to the leakiness parameter itself.
Hanson on the same Minds Almost Meeting episode:
So, as you know, I’m co-author of this book, The Elephant in the Brain: Hidden Motives in Everyday Life, and this was a key piece of information to me in my life — to see that we had this plausible, easily understood, widely accepted rationale for why people should be interested in prediction markets, and then to see that they didn’t seem to be very interested. This was a key data point about the differences between what we say we want and what we actually want, which I explored in many other contexts in this book. This book doesn’t actually talk about prediction markets, but I’d say, yes, a key fact about the social world is that we often talk as if we had certain motives and certain priorities. And then our actions, if you look carefully, seem to belie that, that is, they don’t choose the things we would choose if we had the priorities we say we have.
With thanks to Anjali for drawing my attention to the linked piece! It catalysed much of the thinking elucidated in this post.
Which, if you’re reading this, doesn’t exist yet! Watch this space…
Returning to this post because, having gotten into betting on Manifold a lot recently, I've developed some Opinions. I don't think anything you say here is wrong per-se. And you are absolutely correct to suggest public over private applications, and to refer to the "expected benefits *relative to the costs*".
But I think the emphasis is a misplaced; the underuse of prediction markets isn't primarily due to the fact that they are better suited for public over private uses. My sense is that two factors are at play, which come from the inherent limitations of market construction. These are:
1. Constructing good questions is actually fairly difficult. (Partly this is just a public goods problem, but still, someone has to fund question creation. And maybe reward betting too, as with Metaculus tournaments.)
2. Even in so far as "good questions" can be constructed, the vast majority either do not give an edge over expert sources (such as for inflation rates or election results) *or* it is the case that "probable" is good enough irl and the market being 5% more accurate than a well considered individual opinion is trivial.
Combining 1&2, the work of market creation doesn't end after resolution. Since questions must be very tightly constructed to have a high likelihood of a fair resolution (preferably within a reasonable time frame), they are much more helpful when information from many markets is amalgamated together and when large, overarching-questions are broken down into sub-questions, sometimes ones with only minutely different resolution criteria.
This all being the case, I think something-something AI is the answer to using prediction markets better. Both for forecasting and for something like this: https://www.metaculus.com/notebooks/42293/map-the-future-before-you-build-it/