Understanding Legalese with LLMs
A practical reference that connects collaborative experimentation and peer-reviewed research to real-world uses.
Summary
At its core, AI education isn't about memorizing prompt templates. It's about experimenting, observing, and learning in context. This first experiment in our Mellonhead Labs series, revealed that when it comes to legal documents, AI performs best when given specific goals, structured formatting, and personal context, and that tools behave very differently unless guided carefully. While different models offer unique user experiences and stylistic nuances, the core outputs are often similar. The biggest factor in output quality isn’t the Generative AI model you choose, but it’s how you prompt it that makes all the difference when dealing with long, dense documents.
Click here to review the original experiment brief and research
Read on to see what our community revealed about their approaches, their prompting techniques and the models they used.
The Biggest Takeaways From Our Community
Choosing the AI model
Pick the AI model that suits you best. They all work, but some, like NotebookLM, automatically show citations, while others, like ChatGPT, generate outputs that are easier to digest.
Give Context & Categories
Tell the AI exactly what you're looking for, e.g. "special conditions for minors" or "data sharing." Without specific context your results will be generic.
Focus on the format
Clear titles, layout, and bullet points make complex legal text easy to understand. Guidance like "a format that's easy to understand" or a more specific "create a comparison table" works wonders.
Most Valuable Tactic: Prompt Chaining
The order and the clarity of your prompts shape the flow and efficiency of the conversation, which is critical with large amounts of data for the model to distill and analyze.
Sample 5-Step Prompt Chain for Legal Document Analysis
Prompt 1: Setup
Assign a role, provide context, legal text, and instruct the AI to read the text. Prompt 1: You are a contract lawyer tasked with identifying any risk to my personal data. I have attached 2 terms and condition documents. Please read each one 5 times and then provided a detailed summary of what you understand about each document. Why This Works Asking the AI to read documents multiple times ensures a thorough and robust initial comprehension. These initial summaries serve as a foundational understanding, creating a well-established context that is crucial for building effective and accurate prompt chains in subsequent steps.
Prompt 2: Annotate and extract
Prompt 2 :Next go back and read each document one section at a time. Extract quotes and citations from the document for these these 3 categories: 1. My data privacy 2. my legal rights 3. how this company will be using my data. At the end, present a detailed summary of what you found, organized by document and category. Why This Works By maintaining the same categories across documents, you create a consistent framework for comparison while building on the AI's understanding from the initial setup prompt. Without them, the model has to decide what’s important, which often leads to vague or inconsistent summaries. The "extract" keyword will increase faithfulness to the original language, rather than summary which will be reworded.
Prompt 3: Get the answers you want
Prompt 4: Answer the following questions for each document based on the summary completed. Organize you answers with a headline that is the title of Document, followed by the numbered question, the answer and citation (page number and extracted quote). Here are the questions: 1. How long does this company retain my data? 2. What personal data does this company retain? 3. Will my data or any of my content be used in their marketing? 4. What are my rights if I feel my data has been breached or misused without my consent? Why This Works Specific questions transform AI from a summarizer into an analyst, making the review faster, sharper, and more relevant.
Prompt 4: Validate
Always validate AI interpretations of legal documents against the original text! Prompt 5: Validate that the results to questions and the previous summary are indeed noted in the initial documents. Flag any answers or details that lack nuance, cannot be tied to the original text, or are otherwise incorrect. Include corrected information. This critical final step ensures: Grounded Analysis Ensures the AI's analysis is firmly rooted in the actual text of the documents. Reduces Hallucinations The risk of incorrect information is never zero, but validation will add nuance and will catch some errors.
Prompt 5: Follow Up Questions
Prompt 5: I have additional questions I want you to answer based only on the material you have read and analyzed so far. What do the terms say about damages if I feel my data is misused? Why This Works You can use this to go deeper based on areas you have ore detailed questions. A final round of questioning is valuable because it pushes the AI beyond a first-pass summary and helps ensure nothing important was missed.
Hits and Misses
When working with long legal documents some techniques worked more than others.
What Worked
Provide Context
Guide the model by providing examples or background information to focus its output.
Re-reading
Instruct the model to "reread" documents multiple times improved comprehension.
Precise Language
Use clear, specific verbs like "extract" or "summarize" to define the desired action and output format.
Validation Steps
Implement steps like asking the model to check its own work or cross-reference information.
What Didn't Work
Vague Prompts
Using generic prompts with undefined superlatives (e.g. "the best privacy policy" was unhelpful.
Unspecified Roles
Assigning a role without clear details or context on our goals led to less relevant responses.
Introducing Bias
When asking questions about a topic the LLMs implicitly decided that was something more important.
Get Started
If you're looking to use Generative AI to understand legal documents try the following approach:
Choose a model
- ChatGPT for easy-to-digest outputs
- NotebookLM for direct citations to the source document to make validation easier
- Gemini and Claude for documents with more than 100 pages
Upload your documents and have the model read it multiple times
Remember to provide context and role that it will use for framing, as it reads.
Organize, annotate and extract details
Word choice is key. "Extract" quotes directly, whereas "summarize" will rewrite which can cause future answers to lose nuance.
Ask for bullet points or a comparison table to answer specific questions
Review and validate. LLM responses may have errors.
Make it a conversation.
Compare across models, if you're unsure about the results.
Why Join?
- We focus on teaching people how to think with AI, not just use it
- Every technique is backed by peer-reviewed research
- We connect individual insights into shared community intelligence
- Drop-in flexibility—participate when experiments match your needs
Who is it for?
Professionals who want research-backed AI skills they'll actually use. Participation is free and open to all.
Mellonhead Labs Learning Resource