The Legal AI Clause Playbook Consistency Report 2026: How Much Do AI-Generated First Drafts Actually Vary Across Major Legaltech Platforms for the Same Contract Type — and What Does That Variance Cost in Negotiation Time

Research briefing — a review of public sources. The Legal Stack did not conduct a survey, benchmark or interviews for this piece.

Updated October 2026. An earlier version described a hypothetical six-platform test and cited survey and cost figures that we could not verify, including an attributed CLOC statistic and a per-document rework cost. They have been removed.

The question, and why it has no published answer yet

How much do AI-generated first drafts vary between platforms for the same contract type? How much does that variance, especially inconsistency between clauses within a single draft, cost in negotiation time?

As of October 2026, we found no independent, published benchmark that measures this. The public benchmarks for legal AI test research, extraction, summarization and redlining. They do not test the clause-level coherence of drafts generated from the same prompt across vendors. Vendor marketing about "playbook-aligned" drafting is not independent evidence either.

So this briefing does three things:

It summarizes what the nearest independent evidence shows.
It explains the documented technical reasons drafts vary, even on the same platform.
It sets out a testing protocol legal ops teams can run themselves.

What independent benchmarks do show

Vals Legal AI Report (VLAIR), February 2025. The first VLAIR study tested Harvey, Thomson Reuters' CoCounsel, vLex's Vincent AI and Vecflow's Oliver on seven tasks:

document extraction
document Q&A
summarization
redlining
transcript analysis
chronology generation
EDGAR research

Drafting was not among them. Redlining is the nearest task to drafting, and there the lawyer baseline of 79.7% outperformed the AI tools. Lawyers also beat the tools on EDGAR research, with 70.1%. The tools collectively outperformed lawyers on document analysis, information retrieval and data extraction. LexisNexis withdrew from most tasks after the report was written.

VLAIR legal research, October 2025. The follow-up study reported AI tools scoring 79–81% on legal research against a lawyer baseline of 71%. All systems struggled with multi-jurisdictional questions: performance fell by an average of 11 points compared with single-state questions. Thomson Reuters, LexisNexis and vLex did not take part.

Stanford RegLab and HAI, May 2024. Researchers found hallucination rates above 17% for Lexis+ AI and Ask Practical Law AI, and above 34% for Westlaw AI-Assisted Research, on more than 200 legal queries.

Analysis. None of these studies measures drafting consistency. Together, though, they suggest two things. First, tools do best on extraction and question-answering over a given document. Second, tasks that require producing or changing legal text, or reconciling several legal regimes, are where human lawyers still hold an edge. A first draft of a cross-border agreement involves both.

Why drafts vary, even on one platform

Variance does not only come from differences between vendors. Large language models are hard to make fully reproducible.

In September 2025, Thinking Machines Lab published Defeating Nondeterminism in LLM Inference, as summarized by Stefano Quintarelli. It observed that LLM APIs remain nondeterministic in practice even at temperature 0, the setting usually assumed to give identical outputs. It traced the cause to floating-point non-associativity: the order of arithmetic operations changes the result.

The practical result is that the same prompt, sent to the same tool twice, can return a different draft. Any evaluation that runs each prompt only once will understate how much drafts vary.

Several other sources of variance are product-level. They are documented in vendor materials rather than independent studies, and include:

the underlying model, which vendors may change without notice
retrieval from clause libraries
playbook configuration
how much context the user provides

Where inconsistency is likely to bite: a worked example

Here is one documented rule set an evaluator can test against.

Incoterms. The ICC's Incoterms 2020 rules took effect on 1 January 2020, according to Addleshaw Goddard. The changes from 2010 include:

DAT was renamed DPU (Delivered at Place Unloaded).
CIP now requires the seller to obtain broader "all risks" cover under Institute Cargo Clauses (A).
FCA now allows on-board bills of lading where the parties agree.

The firm also recommends that contracts state the Incoterms version relied on. A draft supply agreement that says "DAT" or leaves out the version is checkably out of date. So is a CIP clause whose insurance provisions assume (C)-level cover.

For illustration, consider a draft in which the delivery term says CIP but the separate insurance clause allocates cargo insurance to the buyer. Each clause may read well on its own, but together they contradict each other. That cross-clause contradiction is the "internal consistency" problem this briefing is about. It can be tested without any vendor's cooperation.

Other pairs where clauses often interact, and where an evaluator should check that they agree:

the limitation-of-liability cap and the indemnity carve-outs
IP ownership and the licence-back or feedback clauses
the data processing clause and any data processing addendum (DPA) incorporation trigger
termination for convenience and fee-on-termination and survival
governing law and the dispute-resolution forum

Estimating rework cost without invented numbers

We do not publish a rework cost estimate, because no public data measures the extra review time an inconsistent AI draft causes. Teams can build their own using a simple model. This is analysis, not data:

Rework cost per year = (number of AI-drafted agreements) × (share with at least one cross-clause conflict) × (extra review hours per conflicted draft) × (blended reviewer rate)

The share of drafts with conflicts and the extra review hours should come from your own pilot testing, as described below. The rate should come from your own billing or outside-counsel data. None of these inputs should come from a vendor deck.

A testing protocol legal ops teams can run

These recommendations are analysis.

Build a prompt library. Write three to five prompts that reflect your highest-volume or highest-risk contract types. Use your own playbook positions, not generic ones.
Run each prompt several times on each finalist platform. Because outputs vary even at fixed settings, three to five runs per prompt shows how much a single platform varies, as well as how platforms differ from each other.
Score for internal consistency, not just quality. Use a checklist of clause pairs like the ones above. Count contradictions per draft, separately from drafting quality.
Check against objective rule sets. Include at least one prompt that turns on a documented external standard: an Incoterms 2020 term, a specific governing law, or a statutory requirement. This tests whether drafts are current and correct, not just fluent.
Time the review. Have reviewers record how long it takes to bring each draft to a sendable state. That gives you the inputs for the rework model.
Use two reviewers who don't know which platform produced each draft. Hide the platform names. Even two reviewers will surface material differences, and checking where they disagree catches scoring drift.
Re-test after model changes. Ask vendors to notify you when the underlying model changes, and re-run the library when they do.
Ask vendors for independent evidence. Ask whether the vendor took part in independent benchmarks such as VLAIR, and on which tasks. Ask for any drafting-specific evaluation methodology they can share.

Bottom line

There is no public, independent measure of how consistent AI first drafts are across legal platforms. Anyone quoting a precise variance figure or dollar cost is relying on private or invented data. What is documented is enough to act on:

Independent benchmarks show lawyers still outperform AI on redlining.
Performance drops on multi-jurisdictional work.
LLM outputs vary even at fixed settings.

A short, repeatable internal test with a consistency checklist is the most reliable evidence a legal ops team can get before signing.

Sources

LawSites, "Legal AI tools show promise in first-of-its-kind benchmark study, with Harvey and CoCounsel leading the pack" (Feb 2025): https://www.lawnext.com/2025/02/legal-ai-tools-show-promise-in-first-of-its-kind-benchmark-study-with-harvey-and-cocounsel-leading-the-pack.html
Vals AI, "Vals Legal AI Report" (2025): https://vals.ai/vlair
LawSites, "Vals AI's latest benchmark finds legal and general AI now outperform lawyers in legal research accuracy" (23 Oct 2025): https://www.lawnext.com/2025/10/vals-ais-latest-benchmark-finds-legal-and-general-ai-now-outperform-lawyers-in-legal-research-accuracy.html
Stanford HAI, "AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries" (23 May 2024, updated 30 May 2024): https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries
Quinta's weblog (Stefano Quintarelli), "Defeating Nondeterminism in LLM Inference – Thinking Machines Lab" (15 Sep 2025): https://blog.quintarelli.it/?p=38315
Addleshaw Goddard, "Incoterms 2020: what's changed" (2019): https://www.addleshawgoddard.com/en/insights/insights-briefings/2019/commercial-services/incoterms-2020-whats-changed/