Collective intelligence has never been measured by any civilization, even as societies built precise yardsticks for armies, wealth, population, and productivity. For thousands of years humanity has counted what it can see while leaving the quality of its own collective judgment unexamined. This article investigates whether prediction markets, wisdom-of-crowds research, and superforecasters point toward a real scoreboard for judgment itself, and whether AllSportsMarket could become an early test of scoring civilization’s decisions. The central question is deceptively simple: could civilization get smarter if it finally learned to score its own predictions?

Why Civilization Has Never Scored Its Own Judgment

A defense ministry can tell you, to the tank and to the dollar, exactly what it owns. It can count warheads, sorties, personnel, and procurement contracts with ruthless precision. What it cannot tell you is how often its strategic calls were right. The same asymmetry runs through every modern institution. Nations maintain meticulous ledgers for what they possess and almost none for how well they think. We measure the inputs of power and the outputs of production, but we have never built a serious instrument for measuring the quality of collective intelligence itself.

That omission is stranger than it first appears. For thousands of years, societies have competed on metrics they could count. Empires tallied soldiers and tributaries. Mercantilist states tallied bullion and trade. Twentieth-century governments tallied GDP, industrial output, literacy rates, and life expectancy. Each metric was imperfect, and each was gamed, but each gave societies a feedback loop. When the numbers moved, policy moved with them. Now consider the decisions that matter most: whether to invade, whether to regulate, whether to lock down, whether to spend. On these questions, the metric is missing. We have no running score for whether the judgments behind those decisions were sound, and no systematic mechanism for learning from the ones that were not.

This is not a failure of data or computing power. It is a failure of design. A state that can model a supply chain in real time cannot say whether its foreign policy forecasts beat a coin flip. A central bank that publishes inflation targets to two decimal places has no comparable public record of the accuracy of its crisis calls. The measurement gap persists because counting things is administratively convenient while scoring judgment is politically uncomfortable. Possessions can be audited without embarrassment. Decisions invite accountability. So civilizations have spent centuries perfecting the first ledger and leaving the second almost blank.

The gap matters more now than at any point in history. The problems that define this century — pandemics, financial contagion, climate risk, the governance of powerful new technologies — are problems of judgment under uncertainty, not problems of production. They reward calibrated probabilistic reasoning and punish confident error. Yet our institutions are still structured to reward decisiveness in the moment and to defer any reckoning until the moment has passed, by which time attention has moved on. Worst of all, the record of past predictions is rarely preserved in a form anyone can check. Civilizations forget their bad calls faster than they forget their defeats.

Which raises the central question of this article: could civilization itself become smarter if it learned how to score its own decisions? The proposition is not mystical. If collective intelligence is a real, measurable property of groups, and if consistent scoring and feedback are what improve judgment at the individual and team level, then the same discipline applied at scale could turn collective judgment from a slogan into an asset that compounds. The rest of this article builds that case in four movements. First, what research actually says about collective intelligence and the wisdom of crowds. Second, the emerging machinery of judgment — prediction markets, forecasting tournaments, and the rise of superforecasters. Third, why sports improve while politics often does not, and what that comparison implies about feedback speed, including a look at whether ASM (AllSportsMarket) could become a first large-scale system for scoring judgment itself. Finally, a practical answer to how civilization might learn to score its decisions, with the open questions that remain.

---

• Inputs are easy to count: armies, GDP, population, and output all lend themselves to administration and audit.
• Judgment is harder to score: it requires recording forecasts before outcomes, then comparing prediction to result in public.
• Accountability is delayed: political and economic errors are often absorbed slowly, so feedback loops stay weak.
• The central question: if collective intelligence can be measured, can scoring it make civilization smarter?

What Collective Intelligence Actually Means in Theory and Research

Before asking whether civilization can be scored, we need a working definition. Collective intelligence is the capacity of a group to solve problems, make predictions, and reach judgments that are better than what any single member could produce alone. The phrase carries a technical meaning in research, not just a poetic one: it describes measurable aggregation effects, where combining many independent estimates yields accuracy that no individual contributor commands. That definition matters because it makes the concept falsifiable. If collective intelligence is a real property of groups, it should show up in data, and it does.

The founding anecdote of the field is Francis Galton’s 1907 ox-weighing contest at a livestock fair in Plymouth. Galton, a statistician skeptical that ordinary crowds could reason well, collected the guesses of roughly 800 fairgoers who tried to estimate the weight of a slaughtered ox. The individual guesses were wildly scattered. The median guess, however, landed within about one percent of the true dressed weight. Galton published the result in Nature under the title «Vox Populi,» and the finding became the touchstone for a century of research on the wisdom of crowds. The lesson was not that every guesser was smart. It was that errors, when sufficiently independent, cancel out during aggregation.

A parallel line of theory predates Galton. The Marquis de Condorcet’s jury theorem, formulated in 1785, shows mathematically that if each voter is more likely than not to choose the correct option, then adding voters increases the probability that the majority is right, approaching certainty as the group grows. The theorem comes with strict conditions: voters must be independent and individually better than chance. Violate those conditions and the arithmetic reverses. A crowd of dependent voters who all repeat the same error can be confidently wrong. This is why crowd forecasting research treats independence and diversity as engineering requirements rather than nice-to-haves.

The modern empirical literature has largely confirmed both the promise and the fragility. Scott Page’s work on diversity and the «diversity prediction theorem» shows that a group’s collective error can be decomposed into the average individual error minus the diversity of the group. More diversity, mathematically, reduces collective error even when average individual accuracy stays constant. Lu Hong and Page’s 2004 study in the Proceedings of the National Academy of Sciences demonstrated that diverse groups of heuristically reasoning agents can outperform groups of the individually best problem solvers. James Surowiecki’s 2004 book «The Wisdom of Crowds» popularized four conditions under which crowds excel: diversity of opinion, independence, decentralization, and aggregation. Remove any of the four and the crowd degrades.

Which brings us to the failure conditions. Groups do not automatically become smart when they get larger. Herding occurs when participants observe each other’s judgments and converge before information is fully revealed. Correlated errors arise when everyone draws on the same misleading data, the same media narrative, or the same institutional incentive. Social pressure, cascades, and information asymmetries can all turn a crowd into an amplifier of its own bias. The research consensus, in short, is conditional: groups beat experts when members contribute independent information, when that information is aggregated by a rule that rewards accuracy, and when feedback corrects the group over time. Prediction markets are one such mechanism, and they deserve attention precisely because they institutionalize those conditions rather than hoping for them.

Prediction Markets, Superforecasters, and the Machinery of Judgment

If collective intelligence is a capacity rather than a slogan, it needs instruments. The previous sections established that groups can, under specific conditions, aggregate information into judgments more accurate than those of most individual members. The obvious next question is mechanical: what does a working apparatus for scoring judgment actually look like? The most mature answers come from three overlapping traditions — prediction markets, forecasting tournaments, and the research program around superforecasters. Each attempts the same fundamental move: convert a belief into a number, attach a deadline and an outcome, and let the arithmetic of accuracy do the talking.

Prediction markets are the clearest of the three. Participants buy and sell contracts that pay out a fixed amount if a specified event occurs — an election result, a rate decision, a geopolitical rupture, a box-office threshold. The price of a contract trading at 63 cents is, roughly speaking, the market’s implied probability of that event: 63 percent. This design does two things that polling and commentary do not. First, it forces precision; there is no way to express «things look uncertain» in a market. Second, it rewards being right and penalizes being wrong, because the trader who holds a mispriced contract transfers money to the trader who spotted the error. The mechanism does not require participants to be disinterested or polite. It requires only that they have some information or reasoning advantage and a reason to act on it.

Election markets are the most visible example, and they are instructive precisely because they are contested. During major electoral cycles, prediction market prices have often tracked outcomes closely, and they have occasionally diverged sharply from punditry and even from conventional polling averages. But it is important to be honest about what the record shows. Market prices are informative aggregates, not oracles. They can be thin, they can be distorted by a small number of large traders, they can misprice low-probability tail events catastrophically, and they can be subject to regulatory constraints that limit who may participate. The claim worth defending is narrower and more durable than «markets are always right»: a well-designed market extracts and weights dispersed information faster than most alternative aggregation methods, and — crucially — it leaves a public, timestamped record that can later be checked against reality.

That last property is the one that matters for the larger argument of this article. A prediction market is not merely a forecasting tool; it is a memory device. It preserves what a population believed, at a specific moment, about a specific resolvable question. Without such a record, there is nothing to audit. With it, hindsight stops being a rhetorical weapon and becomes data.

Forecasting tournaments and the discipline of the resolvable question

Forecasting tournaments formalize the same instinct without necessarily using money. Participants register probabilistic forecasts on a battery of precisely worded questions, each with an explicit resolution criterion and a closing date. The precision is the point. «Will the economy improve?» is not a forecastable question; «Will the official unemployment rate be below 4.0 percent on the first Friday of next March?» is. A tournament is, in effect, a factory for producing well-posed questions and then grading the humans who answer them.

The most influential of these efforts was the Good Judgment Project, a multi-year research program conducted as part of a broader intelligence-community forecasting initiative. Its design was straightforward and rigorous: thousands of volunteers made probability estimates on geopolitical and economic questions over multiple seasons, and their accuracy was tracked continuously. Several findings from that program have become reference points in the field. First, forecasting skill is not evenly distributed; a small subset of participants consistently outperformed the rest by wide margins. Second, skill was not simply a proxy for domain expertise or credentials — some of the strongest performers were not professional analysts. Third, and most consequentially, the best performers improved over time in a way that looked like deliberate practice rather than innate talent.

The term «superforecasters» emerged from this work to describe that top tier. The label is sometimes treated as mystique, but the research portrait is more mundane and more useful. Superforecasters tended to think in explicit probabilities, update frequently in small increments as news arrived, break large questions into components, distinguish base rates from case-specific evidence, and — this is essential — keep score on themselves. They were not distinguished primarily by raw intellect. They were distinguished by habits built around feedback.

This is where forecasting tournaments connect back to the measurement problem. A tournament is a loop: forecast, resolve, score, adjust. The loop runs on a schedule. Participants cannot quietly forget a bad call, because the bad call is in the ledger, next to a number that says how bad it was. The tournament does not merely measure judgment; it manufactures the conditions under which judgment improves.

Calibration, resolution, and the Brier score in plain terms

To understand why scoring changes behavior, one has to understand what is being scored. Two properties matter, and they are often conflated.

• Calibration — the agreement between stated probabilities and observed frequencies. If you say «70 percent» a hundred times, roughly seventy of those events should occur. A forecaster who is well calibrated can be trusted at the level of confidence they express.
• Resolution — the ability to discriminate between events that will happen and events that will not. A forecaster who assigns 0.5 to everything may be perfectly calibrated in a trivial sense while being completely useless, because they never distinguish the likely from the unlikely. Good forecasting requires both calibration and resolution.

The most commonly used instrument for combining these properties is the Brier score, a proper scoring rule that penalizes the squared distance between a forecast probability and the actual outcome, coded as 1 or 0. If you forecast 0.9 and the event happens, your penalty is small: (0.9 − 1)² = 0.01. If you forecast 0.9 and it does not, the penalty is (0.9 − 0)² = 0.81. Lower is better. The rule is «proper» in a technical sense: it is mathematically designed so that the only way to minimize your expected penalty is to report your true belief. Hedging toward 0.5 to look safe does not help you. Overclaiming confidence to seem decisive does not help you. The scoring rule makes honesty the optimal strategy, which is an elegant property for any system intended to measure judgment.

Forecast	Outcome	Squared error (Brier contribution)
0.90	Event occurs (1)	0.01
0.90	Event does not occur (0)	0.81
0.50	Event occurs (1)	0.25
0.50	Event does not occur (0)	0.25
0.10	Event does not occur (0)	0.01

Two features of this arithmetic deserve emphasis. First, it rewards sharpness when sharpness is warranted — a confident correct forecast scores far better than a timid one — while punishing confident errors savagely. Second, it aggregates. Individual scores can be summed and averaged across a question set, a team, a season, or a population, producing a single number that describes how well a group’s judgment held up against reality. That is as close as the field currently comes to a score for collective intelligence on a defined question set.

It is worth being explicit about a subtlety: Brier scores are relative, not absolute. A score of 0.15 means something only in the context of the questions asked and the base rates involved. A set of questions where most events happen is easier than a set of near-coin-flips. Serious forecasting programs therefore compare scores against reference benchmarks — climatology-style base rates, always-0.5 forecasts, and sometimes market prices themselves — rather than treating the raw number as a universal grade.

The feedback loop is the active ingredient

Strip away the experimental apparatus and the finding that survives is almost embarrassingly simple: consistent, timely, unambiguous feedback improves judgment. This is not unique to forecasting. It is true of surgeons, chess players, weather forecasters, and athletes. What forecasting research adds is a demonstration that the principle applies to probabilistic reasoning about uncertain world events — a domain often assumed to be too messy for skill to accumulate.

The mechanism is not mysterious. Feedback does at least four things. It corrects overconfidence, because a forecaster who repeatedly assigns 0.95 and is wrong learns to shade toward 0.8. It surfaces base-rate neglect, because questions that keep resolving «yes» despite scary narratives teach the forecaster to weigh the reference class more heavily. It distinguishes luck from skill, because a single dramatic correct call cannot outweigh a season of scores. And it makes memory durable: the logged forecast, the date, and the resolution become an object that can be revisited rather than a hazy impression that can be revised.

The contrast with most public decision-making is stark. A pundit who predicts a recession that never arrives rarely faces a scored ledger; their reputation may even be enhanced by the memorable conviction of the claim. A policymaker whose forecast of a conflict’s duration proves badly wrong is typically evaluated on other dimensions — political loyalty, rhetorical skill, institutional survival — rather than on the accuracy of the forecast itself. The absence of a scoring rule does not merely fail to improve judgment; it actively selects for confident narrative over calibrated probability.

What the machinery cannot do

Honesty about limits is what separates a serious measurement proposal from a sales pitch, and the forecasting literature is reasonably candid here. Prediction markets and tournaments only score questions that can be resolved cleanly, which excludes many of the most consequential decisions in public life — decisions whose effects are diffuse, delayed, and entangled with other causes. Market prices can be gamed, thin markets can be noisy, and the regulatory environment for event contracts is unsettled in several jurisdictions. Forecasting skill measured on discrete questions does not automatically transfer to complex operational judgment, and the field has not established that a high Brier score on geopolitical questions predicts wisdom about institutional design.

There is also a selection problem. The populations that participate in forecasting tournaments and markets are not representative samples of humanity; they skew toward the analytically inclined, the numerate, and the online. A score computed on such a population measures that population’s judgment, not civilization’s. Any claim to have scored «civilization’s collective intelligence» would have to specify which population, which question set, and which benchmark — or it would be an overreach.

None of these limits is fatal. They are design constraints. They tell us that any credible system for scoring collective judgment must be explicit about resolvability, transparent about participation, and disciplined about comparing scores to benchmarks rather than celebrating raw numbers. They also point toward the deeper question taken up next: why some domains — sports, most obviously — have built dense, fast, unforgiving feedback loops around performance, while others have built almost nothing. The machinery of judgment exists and it works. The puzzle is where civilization chooses to install it.

Sports vs. Politics: Why Some Systems Learn and Others Do Not

Picture a professional sports league at the end of a season. Every match has a score, every score feeds a table, and the table decides who is champion and who is demoted. A striker who misses chance after chance sees their conversion rate printed in the program. A manager who keeps losing set pieces is eventually replaced. Fans argue furiously, but they argue within a system that produces an unambiguous verdict: the standings. The league is not necessarily smarter than the society it entertains, yet it has something societies rarely build — a tight, fast, public loop that connects a decision to its outcome and the outcome to a consequence.

That contrast is the heart of the case for scoring civilization’s judgment. The difference between a system that improves and one that stagnates is often not the intelligence of its participants. It is the speed, clarity, and consequence of the feedback they receive. When you look at how sports leagues learn and how public policy frequently fails to learn, the gap is structural. It comes down to how quickly the results of a decision come back, how legible those results are, and whether anyone is obliged to answer for them.

The Anatomy of a Learning Loop

Sports work as a learning environment because they satisfy four conditions at once. First, outcomes are measured precisely and consistently — goals, times, error rates, save percentages. Second, the measurement is public, so everyone works from the same numbers. Third, the cycle is short: a season lasts months, a match lasts ninety minutes, and post-match analysis happens within hours. Fourth, consequences attach to performance. Coaches are fired, players are traded, and ownership faces relegation. Each of these four elements — measurement, transparency, tempo, and consequence — is a feedback loop in decision making that turns experience into adjustment.

Sports leagues also institutionalize the comparison. League tables are a form of collective scorekeeping that lets thousands of separate contests be ranked on a common scale. This is precisely the property that is missing from most collective decision-making outside sport. Nations publish GDP, unemployment, and inflation, but they do not publish anything comparable for the accuracy of the judgments that produced those outcomes. A government can preside over a policy failure and still win the next election on unrelated issues. The feedback exists in principle — the world eventually tells you whether your forecast was right — but it arrives late, arrives tangled with other causes, and rarely attaches to the person or process that made the call.

Dimension	Sports league	Politics, economics, public policy
Measurement	Precise, standardized statistics	Contested, revised, often retrospective
Speed of feedback	Immediate to seasonal	Years to decades, sometimes never attributed
Transparency	Public and shared	Partial, partisan, or opaque
Consequence	Relegation, firing, trading	Diffuse; accountability in public policy is intermittent
Learning mechanism	Continuous adjustment against a common score	Blame cycles and narrative shifts

The table does not show that sports are more intelligent than politics. It shows that sports are better instrumented. And instrumentation, as the earlier evidence on forecasting and superforecasters suggests, is what allows judgment to improve. A politician facing a delayed and ambiguous scorecard has little opportunity to calibrate; a coach facing a live league table calibrates every week. This is why the question «Why doesn’t civilization learn from its prediction errors?» has a partial answer already: often it cannot see the errors in time, in a form it accepts, or with anyone assigned to correct them.

Why Policy Feedback Fails

Consider three mechanisms that routinely break the policy feedback loop. The first is lag. Infrastructure decisions, interest-rate changes, and education reforms produce their most important effects years later, by which point the political actors who chose them have moved on and the causal chain has been muddied by a hundred intervening events. Attribution becomes a rhetorical contest rather than a factual one.

The second is the absence of a control. In sports, the same team plays many games, so performance can be separated from luck across a large sample. In policy, a given country typically tries one version of a policy once. Without a comparison condition, and often without pre-registered forecasts of what should happen, it is nearly impossible to say whether the policy worked or the world simply moved. The movement toward structured forecasting in policy settings is an attempt to fix this missing baseline, and it is worth linking back to the prediction-market explainer for readers who want the underlying mechanics.

The third is incentive misalignment. In a league, everyone benefits when the scorekeeping is honest, because fans and sponsors need to trust the table. In politics, losing face can be fatal, so actors are rewarded for disputing the metric rather than absorbing the lesson. This is the core of the accountability problem in public policy: when the scorekeeper can be attacked, the score becomes negotiable. As the earlier sections of this article showed, calibration only improves when feedback is accepted as legitimate. A score nobody must respect cannot teach anyone anything.

AllSportsMarket as a Prompt for Investigation

This is where the AllSportsMarket (ASM) idea becomes interesting as an object of study. The proposition worth investigating is narrow and specific: could a large-scale, market-based scoring system built around a domain that already generates rich, immediate, unambiguous feedback — sports — become a working model for how judgment itself might be scored? Sports fixtures resolve cleanly, and they resolve quickly. That makes sports one of the few high-volume domains where a collective judgment can be recorded, resolved, and scored at scale without waiting years for the world to make up its mind.

The significance is not that a sports market is the same as a policy market. It is that it demonstrates the machinery. If thousands of participants can register forecasts, be scored against outcomes across many events, and accumulate a public track record that reflects judgment rather than luck, then the essential loop of a judgment-scoring civilization already exists in miniature. The open question is whether such an approach — examined as an investigation rather than endorsed as a solution — could be adapted to domains with slower and messier feedback. This article does not treat ASM as a proven remedy; the materials presented here support a careful, verification-oriented look at the mechanism, not sweeping claims about what it will accomplish. Readers and researchers should treat it as a hypothesis worth testing, with the design details verified as evidence emerges.

There is value even in treating sports as a proving ground. Every domain that wants to score collective judgment will face the same demands: a large population of forecasters, a common set of events with verifiable resolutions, a transparent scoring rule, and a way to show that a public track record correlates with genuine skill rather than noise. A sports-based market tests all of these on a fast cycle. If such systems cannot produce calibrated, trustworthy scores in the easiest domain available, there is little reason to expect them to work in harder ones. That makes them a useful diagnostic — a stress test of the whole idea of measurement before it is pointed at economics or public policy.

The Skeptic’s Objections

Any proposal to score collective judgment will meet three serious objections, and they deserve direct answers rather than deflection. The first is manipulation. If a score matters, someone will try to game it — by trading to distort market prices, by coordinating votes, or by exploiting gaps in the resolution rules. The mitigations are familiar from market design and forecasting research: deep liquidity, clear and pre-committed resolution criteria, detecting anomalous trading patterns, and treating the scoring rule as public infrastructure. The relevant factual question, which should be verified rather than assumed, is how robust any specific mechanism is against coordinated distortion.

The second objection is incentive distortion. Once a metric becomes a target, people optimize for the metric rather than the underlying reality. A scoring system for judgment could encourage forecasters to make safe, easy calls rather than useful, hard ones. This is a design problem, not a fatal flaw: scoring rules can reward difficult and informative forecasts, and track records can be weighted by how much a forecast moved collective belief. The system must score not just accuracy but informativeness.

The third objection is that prediction is not the same as policy. A market can tell you that a policy will fail without telling you how to make it succeed. This is fair, and it is a boundary the article should respect. Scoring judgment does not replace deliberation; it disciplines it. A public forecast of expected outcomes gives deliberation something to react against, and a public post-mortem gives it something to learn from. The claim is not that forecasting replaces politics. It is that politics currently lacks the scoreboard that would let it improve.

If these objections can be met in sports — the simplest domain — then the case for extending the approach becomes empirical rather than speculative. This is precisely why an investigation of AllSportsMarket and similar efforts matters. It is a test of whether a public, scalable scoring mechanism can survive the pressures that wreck less carefully built metrics. And it returns us to the central contrast: a league table works because the score is trusted, measured consistently, and attached to consequences. Any civilization that wants to become smarter must build the same four things — measurement, transparency, tempo, and consequence — and must first prove it can do so in a domain where the final whistle settles the argument.

The sports-versus-politics comparison therefore yields more than an analogy. It yields a checklist and a research agenda. The checklist tells us what any serious attempt at scoring civilization judgment must provide: fast and consistent measurement, public shared access, short feedback cycles, and real consequences for being wrong. The research agenda tells us what to test next: whether a large-scale market can generate calibrated, manipulation-resistant scores, and whether those scores track genuine skill. Sports already run the experiment every season. The question is whether civilization is willing to build a scoreboard of its own — and to read the table honestly when it does. For readers new to the underlying instruments, the earlier explainer on prediction markets provides the background mechanics this section builds on.

Could Civilization Learn to Score Its Own Decisions? A Practical Answer and FAQ

The answer to the central question is yes, but with a condition attached. Civilization can learn to score its own decisions only if it builds mechanisms that make judgment visible, comparable, and subject to feedback. The previous sections established that humans have never lacked intelligence so much as they have lacked instrumentation. We can measure the speed of light to nine significant figures, yet we have no accepted index for whether a parliament, a central bank, or a public health agency exercised good judgment last year. closing that gap is not a fantasy of technocrats. It is an engineering problem with known components: forecasts that can be registered in advance, outcomes that can be verified afterward, and scoring rules that reward calibration rather than confidence.

The practical answer therefore has three layers. First, a culture layer: institutions must treat calibration as a virtue, the way sports treats accuracy under pressure. A forecaster who says «70 percent likely» and is right seven times out of ten should be rewarded, not mocked for the three misses. Second, an infrastructure layer: public forecasting registries where officials, analysts, and citizens can record predictions with timestamps and confidence levels, so that hindsight cannot quietly rewrite the record. Third, a market layer: pilot prediction markets and scoring tournaments that aggregate dispersed knowledge into a single number, a collective intelligence score that can be tracked over time the way GDP or inflation is tracked now. Each layer is independently buildable. None requires a global government or a new superintelligence. They require only the decision to start keeping score.

Specific recommendations follow from the evidence reviewed earlier. Organizations should adopt feedback audits: after any major decision, compare the forecasts that were on record with what actually happened, publish the gap, and reward the analysts whose calibration held. Governments should run pilot markets on bounded, verifiable questions such as budget outturns, infrastructure completion dates, or disease trajectories, keeping stakes small and rules transparent so the exercise tests judgment rather than appetite for risk. Newsrooms and universities could host public forecasting registries, giving citizens a place to record and compare predictions on civic questions. Crucially, these pilots should be evaluated on calibration and Brier scores, not on whether the crowd happened to agree with any particular expert. A civilization that audits its forecasts will, over a decade, accumulate something no previous civilization possessed: a longitudinal record of how well it thinks.

The comparison with sports, developed earlier, points to the design principle that matters most: shorten the feedback loop. A league that waited twenty years to reveal the final score would never improve its players. Politics and public policy operate close to that absurd standard, with accountability arriving late, diluted, or never. Prediction markets and forecasting tournaments compress the loop. They force a claim into the open before the outcome is known, and they settle the claim automatically when reality arrives. That is the mechanism by which civilization prediction markets could function not as gambling parlors but as learning machines. The scoring is the point; the trading is merely the engine that produces the score.

None of this is guaranteed, and the honest version of the argument acknowledges the risks. Scoring systems can be gamed, captured, or ignored. Forecasters can be punished for unpopular accuracy. Markets can be thin and noisy. But the alternative, which is what civilization currently practices, is worse: an absence of any score at all, and therefore an absence of any reliable mechanism for learning from prediction errors. The investigation into ASM (AllSportsMarket) as a possible first large-scale system for scoring judgment should be read in that light. It is not a promise that one platform solves the problem. It is a test of whether the underlying architecture, immediate feedback plus public scoring, can be generalized from sports to the rest of human decision-making. The question worth carrying forward is not whether scoring civilization is difficult. It is whether we can afford another century without it.

Frequently Asked Questions

Does scoring collective intelligence require tracking individual people?

No. The most useful scores operate at the level of questions and institutions, not citizens. A public forecasting registry can record predictions under pseudonyms or organizational names, and market prices aggregate anonymous positions into a single number. Privacy concerns are real and should shape design, but they do not block the core mechanism, because the unit being scored is the quality of a forecast or decision, not the identity of the person who made it.

Can these systems be manipulated?

Any scoring system can be attacked, and prediction markets are no exception: participants can trade on noise, coordinate, or attempt to distort prices. The defenses are the same ones that work in sports and auditing: transparent rules, verifiable outcomes, volume limits on pilot programs, and evaluation on long-run calibration rather than single events. Manipulation is a design constraint to be managed, not a reason to abandon the attempt. A system that is honest about its vulnerabilities is more trustworthy than a political process that hides its own.

Is this expensive or utopian?

Pilot programs can start small. A forecasting tournament inside a single agency or university costs little beyond staff time, and public registries are largely software and process. The utopian charge usually assumes a single global scoring authority, which is not required and probably not desirable. The realistic path is plural and incremental: many registries, many markets, many audits, with comparison across them. What is genuinely ambitious is not the cost but the cultural change of treating calibration as a public virtue.

What would a collective intelligence score actually measure?

At its core, a collective intelligence score would summarize how well a group’s recorded forecasts match outcomes over time, adjusted for difficulty and expressed through calibration and proper scoring rules such as the Brier score. It would not measure wisdom in the abstract, nor intelligence in the IQ sense. It would measure something narrower and more useful: the reliability of a group’s judgment about the future. That narrower measure is precisely what allows comparison, accountability, and improvement.

How would we know whether it is working?

Success would look like this: forecasts on record before events, outcomes verified afterward, and scores that rise as participants learn. If a decade of pilot markets and public registries produces institutions whose calibration improves measurably, the experiment has worked, even if no single platform dominates. If scores stay flat while confidence rises, the experiment has failed, and we will have learned that too, which is itself the kind of lesson civilization currently lacks any machinery to collect.

Where can readers follow the underlying research?

The literature on forecasting tournaments, calibration, and proper scoring rules is publicly available through the research programs of Philip Tetlock and collaborators, including the Good Judgment Project, and through academic work on wisdom-of-crowds aggregation dating back to Galton and Condorcet. Readers interested in the broader debate should also follow reputable outlets that cover collective intelligence and prediction-market research; specific studies and links are best verified at the source before citation, since the field continues to evolve.