How it works
In short. Three AI models predict what the Supreme Court will do in real cases, before the Court acts. Each prediction is timestamped and saved to a public record, then scored against what the Court actually did. Because the answer doesn’t exist yet when the prediction is made, the models can’t have memorized it.
FedCourtsAI asks a simple question: when a frontier AI model reads a case that is actually pending before the Supreme Court, and commits to a prediction before the Court acts, how good is that prediction? Not on a benchmark the model may have seen in training, and not on cases whose outcomes are already written down somewhere, but on the live docket, where the answer key doesn’t exist yet. Everything below follows from that question.
The events
Each prediction is about one event: one question about one case, asked at one point in the case’s life. A case can pass through three stages, and each stage has its own question.
- Cert. Will the Court take the case? Asked when a petition is docketed, again when it is distributed for conference (and after each relist), and again if the Court asks for the Solicitor General’s views.
- Interim. Will the Court grant an emergency application for a stay or an injunction? Asked when the application arrives, after the Court calls for a response, and once the response is filed.
- Merits. Will the Court disturb the judgment below? Asked when the Court agrees to hear the case, and again once both sides’ merits briefs are in.
These moments are fixed in advance, in a closed table in the code. A case is predicted when it enters the pool, and predicted again only when something about it changes: a distribution, a relist, a call for the Solicitor General’s views, a new filing. Nothing runs on a fixed clock, and no one can pick a case by hand: each run works out what it owes from the committed record.
What a prediction contains
Each prediction is a small folder in the repository, written by one model about one event. At its heart is a JSON file with:
- a probability: of a grant at the cert and interim stages, or of the judgment below being disturbed at merits;
- the single outcome the model thinks most likely;
- at merits, a predicted judgment and vote.
Beside it are short prose files: why the number is what it is; what the model expects the Court to do, and on what grounds; and what the model consulted beyond the materials it was given. That last file is the model’s own account of its research. The system separately logs every tool call the model actually made.
Newer predictions also carry a prediction summary: one paragraph in plain words, which the board shows under each model’s figures.
The reasoning matters as much as the number. Predicting grant or deny on a cert petition is close to calling a coin with a known bias. The harder, more interesting question is whether a model can anticipate which justices will form the majority, who will write the opinion, and on what legal ground.
Pre-registration is about time, not evidence
A model isn’t restricted in what it may read. It can search the web, pull commentary, and consult whatever it likes, just as a forecaster in the real world would. The commitment is about timing: the prediction is committed to a public git branch, with a timestamp from the machine that ran it, before the Court releases its order list. That timestamp, together with the logged workflow run that produced the commit, is what shows the prediction came first.
This is deliberate. A forecaster allowed to see only a curated slice of the world is being tested on something narrower than forecasting. Instead, the design makes research visible. If a prediction leaned on something it shouldn’t have had, such as a leaked outcome, it can be spotted and excluded afterwards, without a gate that would also block legitimate research.
Who predicts, and who grades
Three model families compete: Claude, Codex, and Gemini, each running inside its own developer’s agentic coding tool. Each reads the same case materials and answers independently. A prediction is one model’s answer; the tournament is the comparison between them.
Once an event resolves, the same three models act as graders, and each grades every prediction on that event. Grading is blind: each prediction is shown under an anonymous label, so a model doesn’t know whose reasoning it is scoring, even when it is its own. The numerical score is plain arithmetic, computed by code from the outcome. The grader’s job is the qualitative part: judging the predicted reasoning against the opinion the Court actually issued, and checking the model’s research for leaks.
What counts
Before the first term’s predictions could be scored, the process was frozen. The prompts, model settings, and tool settings were fingerprinted and recorded in a tagged commit. A prediction counts toward a reported figure only if it was made after that moment and carries a stamp matching those fingerprints.
Predictions made while the pipeline was still being built stay in the repository with their timestamps, labeled as shakedown. They tested the machinery; nothing about them was pre-registered, and nothing is claimed from them.
When the process changes, its version number changes, and results from before and after are never pooled. Two processes working from two sets of information are two separate experiments.
How scoring works
Almost every cert petition is denied, so a model that answers “denied” every time will be right most of the time. Accuracy alone tells you nothing. Instead, each prediction is scored on skill against a base rate: the historical grant rate for similar petitions, grouped by how prominent a case is. Simply restating the base rate earns nothing; beating it earns credit. The probability is also scored for calibration, using the Brier score.
Two more things are pre-registered alongside each forecast. The big-case score is each model’s view of how much a case matters, explicitly not how likely it is to be granted. It is what the front page ranks by, and it is graded later by an independent evaluator rather than by the outcome. At the merits stage, the predicted vote is scored against the actual vote.
Which cases
The Court receives thousands of petitions a term. Predicting all of them would spend the whole budget on cases that turn out exactly as the base rate says. Instead, a fixed formula ranks the eligible petitions by how worthwhile each is to forecast, using signals a docket picks up as it moves, such as relists, a call for the Solicitor General’s views, or the court the case came from. The models predict the top-ranked cases, as many as the budget allows.
Some cases are always included, whatever their rank, and the ranking can’t push them out:
- a random sample of new filings, drawn from a seed fixed in advance;
- every petition the federal government files;
- every case where the Court has asked for the Solicitor General’s views.
Once a case is in, it stays in. A later budget decision never rewrites the record.
Some cases are collected but not predicted: petitions filed without a lawyer or without paying the filing fee (pro se and in forma pauperis), original-jurisdiction and miscellaneous matters, and dockets whose outcome can’t be read automatically. This is a budget line, not a judgment that those cases don’t matter.
Limitations
Some of these are temporary and some are built into the design. It’s worth being clear about which is which.
- The numbers are small. The first frozen set of cert predictions covers roughly a hundred events, all distributed for one conference. A hundred events can’t separate three models with any confidence, so figures from it are a first look, not a ranking.
- The first set isn’t a sample of the docket. It is what remained from earlier funded rounds, and it leans toward more prominent cases. Read it group by group, each against its own base rate; a single pooled number would flatter every model.
- The timestamps rest on GitHub, not on a notary. Git dates can be set to anything. What the design relies on is the clock of the machine that ran the model, which the model can’t change; the public push to a protected branch; and the logged workflow run that produced the commit. That’s strong enough for this purpose, but it isn’t cryptographic proof, and we don’t describe it as such.
- Research is self-reported and graded, not prevented. A model that read an outcome it shouldn’t have is caught by a grader reading its research log, not stopped by a wall. Leakage is grounds for exclusion; it isn’t made impossible.
- Three models, one setup each. A result about “Claude” here is a result about one particular model, running inside one particular coding tool, with particular prompts. Changing any of those changes the process version.
- Cert is the easy stage. The most interesting predictions, about reasoning and votes, can only be scored at merits, and merits cases resolve months after the forecast. Early results will be almost entirely grant-or-deny.
- Scope follows the budget. How many cases are predicted depends on capacity, and capacity depends on funding. A wider scope is a decision, not a discovery.
- None of this is legal advice. The predictions are experimental model outputs about how a court may act. They are not statements of fact about any judge, and not a basis for any decision. See the terms.
Case summaries
Most cases on the board come with a short plain-language summary: what happened, what the Court is being asked, and where the case stands. They’re there so a reader who doesn’t know the case can follow the predictions.
Each summary is written by Claude Sonnet 5, a model that is not one of the three being scored. It works only from the case’s docket and filings on the Supreme Court’s website as of the date shown. It has no web access and is told not to add anything it may know from elsewhere. It is instructed to describe both sides’ positions without predicting the outcome or judging how important the case is. A summary is rewritten when the case’s filings change, and each one is committed to the public repository along with the model and date that produced it.
The summaries are AI-generated and can contain mistakes. They aren’t predictions, and nothing is scored on them. For anything that matters, check the filings.
Prediction summaries
Under each model’s figures on the board is a short paragraph, written by that model in the same run as its prediction, that says the prediction in plain words: what it expects the Court to do, how likely it thinks that is, and the main reason. It restates the model’s numbers and must agree with them; it is not a second prediction. Nothing grades or scores it, and the graders never see it. Predictions made before these summaries were added show only the model’s reason for its big-case score.
Case topics
Cases on the board carry a topic tag, such as Criminal law or First Amendment, that says what the case is about. Each tag comes from a fixed list of fifteen subjects. It is assigned by a Claude model working as a labeler, separate from the three models being scored. For a petition, the labeler reads only its questions presented. An emergency application has no questions presented, so the labeler reads the application itself where the record holds it, along with the case name, the court below, and the docket entries filed before the Court acted. It labels the dispute behind the application, not the relief asked for. A few cases also carry a second topic.
Tags are added in batches, so newer petitions and emergency applications may not have one yet. Many applications have only a short docket to go on, and those the labeler can’t place get no tag. Before each batch is published, it is checked against a fixed reference set of cases labeled under the same written rules, and a batch that agrees with that set too rarely is not published. The reference labels were also made by AI models, so that figure measures consistency rather than accuracy. The lines between some subjects (civil procedure and constitutional rights, for instance) are genuinely blurry.
Topics are descriptive only: no prediction depends on them and nothing is scored on them. You can filter the board by topic. No topic collects the cases not yet labeled, plus the few the labeler couldn’t place in any subject. The rules and every label are in the public repository.
Updates
There are two optional weekly emails. The Supreme Court update says what the Court decided and did that week on the cases the project follows, with no forecast in it. The weekly forecasts compare what the Court resolved that week with what each model forecast, and list the forecasts newly committed to the ledger. Both are generated from the ledger, not written by hand, and say nothing the ledger doesn’t: a forecast is always shown as a forecast, never as a score. Each email starts as a draft, and a person reads and approves it before it goes out. Weeks with nothing to report are skipped. Every week’s forecasts are also kept, exactly as generated, on the Updates page.
Glossary
- Cert (certiorari)
- The Supreme Court chooses almost all of its cases. A party asks it to take a case by filing a petition for certiorari; the Court “grants cert” to hear the case or “denies cert” to leave the lower court’s decision standing. It denies the great majority.
- Docket
- The Court’s list of pending cases, and each case’s public record of filings and orders. “Docketed” means the case has been entered on it.
- Distribution and conference
- The justices decide which petitions to take at private meetings called conferences. A petition is “distributed” for a particular conference once the papers are complete.
- Relist
- When the Court considers a petition at conference but neither grants nor denies it, and instead carries it over to a later conference. Repeated relists often signal that the justices are still debating a case.
- CVSG (call for the views of the Solicitor General)
- An order asking the federal government’s top Supreme Court lawyer to say whether the Court should take a case in which the government is not a party. It raises the odds of a grant considerably.
- GVR (grant, vacate, and remand)
- A short order in which the Court takes a case, wipes out the lower court’s decision, and sends it back to be reconsidered, usually in light of another recent Supreme Court ruling. No briefing or argument takes place. The ledger counts a GVR as a grant.
- Summary reversal
- A decision reversing the lower court on the petition papers alone, without full briefing or oral argument. Also counted as a grant.
- Emergency application (the “interim” stage)
- A request to the Court to act quickly while a case is still moving through the lower courts, most often to pause (“stay”) a lower-court order or to block a government policy. These are sometimes called the “shadow docket”.
- Merits
- The stage after the Court has agreed to hear a case: full written briefs, oral argument, and a signed opinion deciding the legal question.
- Reverse, vacate, affirm
- What the Court can do with the lower court’s decision at the merits stage: reverse it (the other side wins), vacate it (set it aside and send the case back), or affirm it (leave it standing). The ledger’s merits question is whether the Court reverses or vacates.
- Petitioner and respondent
- The party asking the Court to take the case, and the party opposing. In a case name, the petitioner is listed first.
The full design, with the command behind every number that will appear on the results page, lives in the repository’s documentation.