Table of contents

What the output does not show.

Many people assume that if a model has read a document, it has considered every part of it and used the words as they are written. That is not always what happens, and the output does not show the difference.

AI models are very good at legal work, and we use them every day at Definely. Over the past two years our research and writing has kept returning to two themes that anyone relying on these tools should know about. The first is a set of blind spots that come from the way the models work. The second is what the research shows about how working with AI changes the way people make decisions.

How AI models work

01 Long documents. A model gives less attention to the middle of a long text than to the start and the end, and its accuracy falls as the amount of text grows. Legal documents are long, and the provisions that matter are often in the middle.

02 Retrieval. Before a model responds, a search step usually decides which parts of the document it sees. Search finds text that looks similar to the question, which is not always the text that is relevant, and it can treat “shall” and “shall not”, for example, as nearly the same.

03 Extraction. A clause goes in as written and comes out in the model’s own words. In one Definely test, eight of ten summaries changed or dropped a qualifier, such as “on or prior to” becoming “before”, and none of them looked wrong.

04 Jurisdictional rooting. A model’s sense of a legal term comes from its training data, much of it from the United States, which may not match the law that governs the document. This is our observation from practice.

05 Compounding error. In a task with several steps, an early slip becomes the starting point for every later step.

06 Shown reasoning. The reasoning a model displays is not necessarily how it reached its answer, which matters when a lawyer has to explain that answer to a client.

07 Hidden instructions. Before a lawyer’s prompt reaches the model, it is usually preceded by instructions from the vendor that the user never sees. Research from Anthropic suggests that an instruction meant to reduce flattery can produce a different bias, so a carefully written prompt is not proof of an objective answer.

How people work with them

Patterns that research has found in people who work with AI.

08 Comprehension debt. A term from software engineering for producing work beyond what you fully understand. In law it means relying on analysis you could not reconstruct yourself.

09 Cognitive surrender. Researchers use this for the point at which a person stops working out an answer and accepts one that arrived already formed.

10 Resilience erosion. In trials with more than a thousand people, those who had worked with AI did worse once it was taken away than those who never had it, and they gave up more often.

11 Judgment drift. In a study by Anthropic of its own model, experienced analysts went with the AI’s assigned position 86 percent of the time, and only 15 percent mentioned the AI when explaining their decisions. Those who followed it every time were as confident as those who did not.

What ties them together

In each case the output looks the same whether it is right or wrong, and the person using it may not notice that anything has changed.

A model does not give an answer it regards as wrong. It gives the most likely answer based on its training and what it was shown, and it has no reliable way of telling from the inside that it missed a passage or changed a word. Asking an AI whether it is sure of itself is not a check. The same blind spots that produced the error will be there when it looks again.

The two feed each other. The blind spots produce errors that look right, and the more people rely on the tools, the less likely they are to catch them. Judgment needs a moment in which slower, more deliberate thinking can engage, which psychologists call System 2, and a tool designed to be as seamless as possible does not leave that moment. We should still use AI tools, but users should understand where they are weak.

How Definely approaches this challenge

We believe every contract deserves rigorous, independent review, whether a person or AI drafted it. Our approach is to leave as little judgment to the model as possible. We use rules wherever a point can be tested with certainty, such as defined terms and cross-references, and we use models where rules are weak or the language is ambiguous. Definely reviews contracts against precedent and standard forms and puts the issues it flags in front of the lawyer, who is always the judge.

See Definely in action: book a demo

Read more