Why we built this benchmark
Thousands of legal teams do their most important work on Legora every day. That trust comes with a responsibility to put the best models to work, and to focus our efforts on making the Legora product measurably better over time. To do this successfully, we need to measure ourselves in a repeatable, reliable way. The Legora BAR is how we do this.
To improve the quality of our product, we have to measure it the way our customers actually use it. Many legal benchmarks are designed as technical evaluations that run in synthetic environments with simplified harnesses built specifically for benchmarking. Those environments are functional but don't fully reflect what a legal team experiences in practice. Additionally, open-sourced cases enable labs to train their models on them, which can create artificially high scores on public benchmarks.
We're taking a different approach. The quality of a legal AI product is about more than the underlying model. It’s the combination of the model, the harness, the tools it has access to, and the system it operates within. Our benchmark evaluates models on real legal cases across practice areas, provided by our law firm partners, inside the Legora harness and the Legora aOS™. This is the same harness and system our customers use every day. The environment is real, so the results represent what our customers actually experience. This is the standard we are committed to measuring.
Our intention with the Legora BAR is not just to measure progress, but to accelerate it. We run the benchmark continuously, and every evaluation helps our team identify where to improve next, and each improvement ships directly into the Legora platform. This creates a constant loop of evaluation, identification, and improvement that results in a measurably better product and, ultimately, a higher quality experience for our customers over time.
Defining quality
When legal teams evaluate AI, they care about one thing: the quality of the work it produces. But quality does not come from the foundation model alone, it comes from the whole system working together.
The input prompt is the task you give Legora. Goal-oriented, well-scoped, and written with the variability of a real lawyer.
The case environment is the matter the agent works in. It holds the documents, playbooks, precedents, templates and other materials lawyers rely on.
The model is the reasoning engine. It supplies the core AI capability, and we run the best models from the leading labs.
The Legora harness equips the model with the tools, skills, legal sources, and workflows needed to perform legal work.
None of these work without the others. This is why we benchmark against the full setup, not the model on its own. It’s the only way to measure what a legal team actually experiences.
The Legora BAR
We maintain an evaluation corpus of 5,161 cases, each reviewed by a legal professional. The Legora BAR is a carefully selected subset, drafted by our law firm partners and in-house legal engineers. It reflects the same mix of practice areas, task types, and difficulty levels as the full suite, so the benchmark gives a representative view of Legora's performance across real legal work.
Practice Areas
28
Every major area, from high-volume review to end-to-end drafting and advisory work.
Cases
5,161
Source Documents
11,075
The files that make up the cases, from source material to precedents and prior drafts.
Each evaluation case, or eval for short, consists of four elements that mirror how legal work is created and executed:
Case description — The input prompt: an end-to-end workflow you would actually run in Legora. Each is classified as short, medium, long (see below).
The matter file room — The folder structure, and corresponding documents, that holds all documents relevant to the case. This includes, but is not limited to, firm-specific templates, playbooks, precedents, and case-specific documents.
The Legora answer — The output from Legora. This can be a single document or a coordinated set of PDFs, Word documents, spreadsheets, redlines, or an in-chat answer.
Rubrics — The criteria against which we compare the Legora answer. They are expert-written, binary, individually verifiable criteria covering facts, analysis, citations, and recommendations. They represent the gold standard output.
Each case is classified based on several categories, including its difficulty level. Legora needs to perform well on both simple and more complex prompts. Therefore, we decided to add the difficulty dimension to our benchmark, where its definition is rooted in something very familiar to legal professionals: how much time the work could take a legal expert. It’s the unit firms scope and bill by, and it translates across practice areas. Examples of corresponding prompts can be found below.
Level
What it looks like
Short
A few hours, up to ~half a day
A bounded, well-specified piece of work: a discrete question, a short clause, a single document checked against one point.
Medium
One to three days
A complete work product an associate would be assigned: a memo, a redline against a playbook, a pleaded section, a disclosure schedule.
Long
A week or more, up to hundreds of hours
An end-to-end delivery that moves the whole matter forward: a full position paper, a complete pleading, a deal's document set. Sustained senior judgment across a large, messy record.
Why the cases stay private
The benchmark runs on real client cases, so we do not open-source them. That constraint is also a strength. When benchmark cases are made public, they end up in the training data of the next model generation. This means a high score reflects familiarity with the test as much as capability. Our cases stay private. Every score reflects the agent seeing the work for the first time, the way a lawyer does.
For transparency, we have open-sourced one of the cases, co-created with AfterQuery. It shows exactly how we build and score a case, and AfterQuery reviewed it to confirm our data-curation methodology meets industry standards.
How we score
Scoring is simple. Every task comes with a checklist of the items a great answer must get right, written by a lawyer. We score the work by how much of that checklist it actually gets right. That percentage is the Quality Rubric Score.
01
A lawyer writes the checklist
The must-haves for a great answer: the facts, the analysis, the conclusions, and the sources.
02
The system does the task
It works through the matter and produces the deliverable, the same way a lawyer would be asked to.
03
We tick what it got right
The share of the checklist it satisfies is the score. Important items count for more than minor ones.
This answer got the high-importance points right and missed only a minor one, so it scores well. Miss something marked high and the score falls much further than missing a low item.
Some agent benchmarks grade all-or-nothing: a deliverable either satisfies every criterion or fails outright. All-or-nothing grading collapses every result into a single pass/fail and hides where systems actually differ, on the hard work where the differences matter most.
We weight our grading instead. Weighted scoring mirrors how a partner reviews work: a missed secondary filing deadline and a missed material change-of-control provision are both failures, and no partner treats them the same. Because high-importance misses are penalized heavily, a materially incomplete deliverable still scores like one.
We report this one number so there is a single, clear measure of legal quality, the kind a partner could sign off on.
Key findings
Model performance
The headline comparison is the models themselves, each run through the Legora harness on the same cases and tasks. They are closely matched on easy work and pull apart as it gets harder, where sustaining judgment across a full matter is what counts.
Quality by difficulty
Foundation models
The black line is the average across all models and all difficulty levels. Each bar shows how far a model sits above or below it within that difficulty (colors identify the labs, see the key above). The models cluster on easy work and spread out on the hard tier, where sustaining judgment across a full matter matters most.
Quality and latency
The strongest models lead on quality and run slower; the lighter models are fastest and trail. No model yet owns the fast, high-quality corner.
Quality vs median time per case
The strongest models lead on quality but run slower; the lighter models are fastest but trail. No model sits alone in the fast and high-quality corner. Across all difficulty levels.
Is the better model worth its price?
Right now, yes. At today's prices the cost of running a model is a rounding error next to the value of the work and the senior time a good draft saves. A few cents more on a task a partner would otherwise spend an hour reviewing is an easy trade, so the more capable model is almost always worth it. That calculus will tighten as volume grows and more work becomes routine, but for the matter-moving work this benchmark targets, quality is what to optimise for, and it is worth paying for.
Quality vs cost per case
Each dot is one model. Up is better quality, right is more expensive. The top-left corner is the place to be, stronger and cheaper. The frontier models trade quality for cost, so the real question is how much the extra quality is worth, and for this work it is worth a lot.
The Legora harness improves the performance of every model
We ran the same frontier models through the harness twice, two months apart: in May and again in July, calling the same model versions in both runs. Model providers do ship minor updates continuously, so no two-month window is perfectly controlled — but incremental model changes are small and uneven across providers. The uplift we measured is large, consistent, and holds across every model we tested. This pattern points to the one variable every run shares: the Legora harness.
Each line represents one model, evaluated unchanged in May and again in July; the bold green line is the average across all models. Every model improves over the period, and because the models themselves did not change, the gains reflect advances in the Legora harness rather than in any individual model. The chart shows the direction and relative scale of the improvement rather than absolute scores.
Benchmark improvements lead to real customer value
This is where our technical investment concentrates and compounds. We evaluate against these cases continuously, alongside the firms who help build them, and every failed rubric criterion shows us where the agent can be sharper. That feedback informs the improvements we make, ships into the product, and is measured again on the next run. As both the models and our harness advance, the combination gets better at real legal work, and every gain reaches the customers who rely on it.
Transactional · SHORT · a few hours
We received the counterparty's draft NDA for Project Falcon (in the matter folder). Redline it against our NDA playbook, with particular attention to the term, the definition of Confidential Information, and the non-solicit. Mark anything that has to stay out of policy, and give me a short cover note on the points that need a decision from us.
Expected output
A clean redline applying the playbook, with the three priority points addressed and any out-of-policy positions clearly marked. A short cover note listing only the points that need a decision.
Litigation · Medium · one to three days
Draft the Statement of Claim in the ICC arbitration against Meridian Components under the 2019 Supply Agreement. Work from the matter documents: the agreement, the termination correspondence, and the delivery and payment records. Plead our primary case on wrongful termination and run the unpaid-invoices claim in the alternative, keeping the pleading to what the record supports.
Expected output
A pleaded Statement of Claim in which every allegation is grounded in the matter's evidence, with the primary and alternative cases properly structured. Claims the record cannot support are left out rather than pleaded and hedged.
Tax · LONG · weeks of senior-associate work
Using the master file template provided in the matter, draft a transfer pricing Master File for the Freshworks Group for the financial year ended 31 December 2023, based on the documents in the matter folder.
Expected output
A complete Master File following the template, with the entity characterisations and TP methods grounded in the audited accounts, ledger and benchmark studies which are all part of the matter. Known inconsistencies in the record are flagged and routed for confirmation rather than silently resolved.
This case and environment is open sourced and can be downloaded here.
Help us shape it
We are co-creating the benchmark with a small number of leading law firms and model labs. The suite keeps growing, and the hard tier — full end-to-end matters — is growing fastest, because that is where agents are now capable enough for the measurement to mean something. If you are a firm or a lab that wants to help define how agentic legal work is measured, we would like to hear from you.
