The Limits of Judicial Analytics: What Published Judgments Don't Tell You

Judicial analytics can only see what the record shows, and the record is a narrow, self-selecting slice of what actually happens in court. Knowing precisely what it leaves out is the difference between a tool you can trust and one that quietly misleads you.

Any product built on published judgments makes an implicit promise: that a judge's decisions reveal how that judge thinks. The promise is real, and it is the foundation of a genuinely useful discipline. But it comes with a condition that the more excitable end of the legal technology market prefers to leave unsaid. A published judgment sits at the very end of a long funnel, and most disputes never travel far enough down it to produce one. The honest question is not what the data can tell you. It is what the data has never seen.

Naming that is not an admission of weakness. For an audience of litigators, trained to attack the reliability of evidence for a living, it is the only thing that makes the rest of the analysis credible. A tool that knows its own blind spots is worth more than one that claims to have none, and the reasons are worth setting out in full.

The funnel that produces a judgment

Consider what has to happen before a dispute becomes a data point in any analytics system.

First, it has to not settle. The overwhelming majority of civil disputes do. The Ministry of Justice's own figures show that only a small fraction of claims are contested all the way to trial: in the quarter to September 2024, somewhere in the order of twelve thousand claims reached trial, against a backdrop of hundreds of thousands issued, and widely cited estimates put the settlement rate for civil disputes well above ninety per cent. The cases that settle, which are arguably the ones most shaped by how sharply each side read the odds, leave no reasoned judgment behind. They vanish from the record entirely.

Second, if a case is decided, the decision has to be written down. A great many first-instance judgments are delivered orally, read into the record at the end of the hearing, and never transcribed unless someone asks and pays for it or the judge specifically directs it. An oral judgment that is never transcribed is, for analytical purposes, as if it never happened.

Third, even among written judgments, only a fraction are formally reported. By long-standing estimate, only around two and a half thousand judgments a year are reported in the law reports series, which is well under two per cent of the judgments handed down. A decision earns a report only if it lays down, changes or clarifies a principle of law. The ordinary, competent application of settled law to particular facts, which is most of what courts do, is not reportable, and mostly goes unrecorded in any processed form.

Fourth, even routine publication has gaps. When The National Archives launched its Find Case Law service in 2022, monitoring by the Incorporated Council of Law Reporting found that in its first full year around one in five judgments that might have been expected to appear were not there. County court judgments, which account for an enormous share of first-instance decisions, are not normally published at all.

Stack those filters and the shape of the problem becomes clear. By the time you arrive at a clean, searchable set of published judgments, you are not looking at the population of what courts decide. You are looking at the survivors: the cases that did not settle, were decided in writing, were considered worth recording, and actually made it into a database. That set is real and it is valuable. It is not representative.

The selection bias hiding in plain sight

This is where the honest caveat has to move from the footnote to the front page, because the survivors are not a random sample. They are systematically different from the cases that fell out of the funnel.

The cases that are fought to a reasoned judgment, and then judged worth reporting, tend to be the harder ones: more contested, higher in value, more novel on their facts or their law. That is frequently why they were not settled and why they were reported. The everyday matter that resolves on well-worn principles rarely generates a published decision. So a judge's published record over-represents the difficult and the unusual, and under-represents the routine work that fills most of their days.

The consequence for analytics is precise and important. A statement like "this judge tends to decide this kind of issue in this way" is, strictly, a statement about how the judge behaved in the atypical cases that survived to a published judgment. It is not a reliable statement about how they handle the ordinary run of matters, because those matters left no trace to analyse. This does not make the signal worthless. It makes it a signal that has to be read with its selection clearly in mind, which is a very different thing from a verdict to be taken at face value.

What a judgment leaves out even when you have it

Set the funnel aside and suppose you have the full text of a reserved, reported judgment in front of you. It is still a curated artefact, and it still withholds a great deal.

A judgment is the reasoned justification for a decision, composed after the fact. It is not a recording of what actually moved the judge. It does not show the argument that nearly succeeded, the concession in oral submissions that quietly reshaped the outcome, or the exchange across the courtroom that turned the case. It does not show the commercial pressure that had both parties eyeing a settlement until the last moment, nor the points the judge considered so obvious that they were never written down. It records the destination and says almost nothing about the route.

Analytics can read patterns across a great many destinations, and that is a legitimately powerful thing to be able to do. What it cannot do is watch the journey, and any tool that talks as though it can is selling a confidence the source material does not support.

Why naming the limits makes the tool better

Here is the turn that the market tends to get backwards. The instinct in a young category is to project omniscience, to hand the user a clean number and imply that the machine has seen everything. For this profession, that instinct is not just ethically shaky. It is a design mistake.

A tool that says, in effect, "here is what the record shows, here is what it structurally cannot show, and here is the primary source so you can check" is more useful than one that hands over an unqualified score. It is more useful because the professional can calibrate how far to rely on it. A number you cannot calibrate is a number you cannot safely lean on, however precise it looks. Stated limits are not disclaimers bolted on so that lawyers feel comfortable. They are the instructions for use, and without them the instrument is harder to use well, not easier.

This is also why the honest posture and the compliant posture turn out to be the same posture. The case for presenting organised evidence that a professional weighs, rather than a conclusion they are invited to accept, is usually made in regulatory terms. But it follows just as directly from the shape of the data. A record this selective can support preparation. It cannot support prediction dressed up as certainty, and a well-built tool does not pretend otherwise.

Designing for the record you actually have

All of this is why Probatur is built the way it is, around the blind spots rather than in denial of them.

It treats published judgments as precisely what they are: a rich but partial account of how a judge has reasoned in the cases that reached, and survived to, a written decision. It surfaces the patterns in that account as material for preparation, keeps the primary source a click away so that nothing has to be taken on trust, is candid about the coverage it does and does not have, and leaves the conclusion where it belongs, with the professional. It is designed to make the record's limits visible rather than to paper over them.

That is not a smaller promise than the one the overclaiming tools make. It is a more honest one, and for an audience whose job is to test the strength of evidence, honesty is the more persuasive pitch. A tool that starts by telling you what it cannot see is, paradoxically, the one you can most safely rely on for what it can.

The published record is among the most valuable open datasets in law, and among the most misunderstood. Its worth and its limits are the same fact viewed from two sides. It is a careful account of the decisions that mattered enough to be written down and preserved, which is exactly the reason it can never be a complete account of what judges actually do. The analytics worth trusting are the ones that understand that difference, and say so plainly.