Disclaimer: Do not include confidential, personal data, proprietary, or privileged information in prompts for generative AI products. I am using ChatGPT4.
KEY TAKEAWAYS:
- Document comparison is historically a frustrating and time-consuming process as contracts professionals struggle to find a tool that can automate this comparison process with accuracy and ease.
- There are a variety of file types that can be uploaded to ChatGPT-4, including Microsoft Office file types like .docx.
- Use the WARP prompt drafting message to get ChatGPT to run a specific document comparison analysis for you.
Contracts professionals often spend time comparing two versions of a contract during the redlining and negotiation process. This usually comes up when one party has shared contract redlines with the other, or when the parties are working on finalizing the contract for execution.
Many use Microsoft Word’s Compare feature, while others use tools like Clause or native CLM features. All claiming to do the same thing–compare two versions of a contract and highlight the differences between the two.
But historically, this has been a frustrating and time-consuming process as contracts professionals struggle to find a tool that can automate this comparison process with accuracy and ease. Depending on the settings, the number of changes, the length of the document, the number of contributors, and the counterparty’s practices this can range from easy to work with to impossibly difficult. We are still looking for the perfect document comparison solution…
As an in-house attorney who uses ChatGPT every day at work, I wondered whether ChatGPT-4 could help us with this problem. So I tested out my theory that ChatGPT could be a useful and reliable document comparison tool and will share my findings with you in this article.
Learn More: How Can ChatGPT Help Us Negotiate Contracts More Efficiently? by Laura J. Greenberg
Understanding ChatGPT’s Features
A lot has changed since ChatGPT was released on November 30, 2022. Users of ChatGPT-4 now have access to image generation through DALL-E, can browse the web with Bing, and can upload documents with analysis. All within the same chat. However, the usage limit is 40 messages every 3 hours.
Working with Different File Types in ChatGPT-4
There are a variety of file types that can be uploaded to ChatGPT-4. For contract professionals, the following file types are relevant:
- Microsoft Office (DOCX, XLS, PPT)
- Portable Document Format (PDF)
- Text (TXT)
- Images (JPG, PNG, WEBP)
- ZIP (ZIP): You can upload ZIP files containing any of the above types of files. The system can extract and interact with the contents of the ZIP file.
For Microsoft Office file types, ChatGPT-4 can:
- Summarize the content into main points or themes.
- Answer questions based on the content of the text.
- Analyze the text for style, tone, grammar or thematic elements.
- Suggest edits or corrections.
- Perform data analysis.
- Create charts.
- Answer data-specific questions for Excel files.
- Provide feedback on the structure, design, and content of PowerPoint presentations.
- Generate the files.
- Run a comparison of two different versions of the same document–the topic of this article!
For example, you can upload an Excel sheet (e.g., containing contracting data) and ask ChatGPT-4 to generate some visualizations of the data for your PowerPoint presentation. In addition, ChatGPT can generate for download all of the above types of files.
But, for Microsoft Office file types, ChatGPT-4 cannot:
- Directly edit uploaded files. This means ChatGPT cannot redline a Word document for you.
- Accurately read redlined Word or PDF files (Claude is better at this).
- Produce lengthy responses (because there is a token or character limitation).
Free Download: How to Redline Contracts Using Microsoft Word by Nada Alnajafi
Creating a Custom Prompt for Comparing Two Document Versions
You can upload two documents into ChatGPT-4 and request a comparison. However, with a very generic prompt, you will get back very generic information. To get to a helpful end result, it is important to clearly articulate the following in your prompt: (a) what type of output you want (i.e., format) and (b) what information you want ChatGPT to analyze or focus on.
Define the Output Format
The default output format of a document comparison exercise in ChatGPT-4 is a bullet-pointed list. However, you can also request that the output generated by ChatGPT-4 be in the following formats:
- Table
- Microsoft Office files (Word, Excel, PowerPoint)
- Plain Text (with markdown or other simple formatting)
- HTML
I’ve found that when ChatGPT cannot generate a detailed comparison in the chat due to token or character limitations, it can accomplish the same task if it generates the output in another format like a Plain Text or Word document.
Describe the Type of Analysis Desired
ChatGPT can run various types of analyses on your data set so it is important to tell it exactly what you want it to pay attention to. I’ve learned that the best way to learn what works for you and your work is to experiment. Here are some of the different types of analysis that ChatGPT can run for your document comparison prompt:
- Highlight Differences:
- Ask ChatGPT-4 to identify and highlight any textual changes between the two versions such as added, deleted, or altered words and sentences. Request the changes be highlighted in bold and that every change be noted.
- Summarize Key Changes:
- Request a summary of the most significant changes, such as alterations in specific terms, conditions, clauses, or overall structure.
- Examine Specific Sections:
- Instruct ChatGPT to focus on specific sections or clauses of the documents and compare those in detail.
- Check Consistency:
- Instruct ChatGPT to verify if certain standard phrases, terminologies, or legal requirements remain consistent between the versions.
- Impact Analysis:
- Request an overview of how the changes might impact the agreement, relationship, revenue, etc.
- Version Suitability:
- Request an opinion on which version might be more suitable for a specific purpose, considering the changes made.
Using WRAPS Method to Refine Prompt
In speaking with in-house teams and law firms, I’ve noticed a consistent theme. Many lawyers don’t know how to get started writing a prompt. So, I’ve developed WARPS to help you get started constructing prompts.
WARPS Method for Prompts
W – What: What is the end product or format you want ChatGPT-4 to produce?
A – Audience: Who is the audience of the output?
R – Role: What is your role in this process?
P – Purpose: What is the purpose, goal, or desired result of the output?
S – Specifics: What other information and context is needed to help generate a helpful output?
Bonus: Tell ChatGPT what you want it to do and then ask it, “What information do you need to complete this task?”
Note that the order is irrelevant. This is a framework to help ensure that you’ve gotten the key elements of constructing a successful prompt that, hopefully, will yield helpful results that you can use when reviewing and negotiating agreements. And the more information or context you provide ChatGPT, the better and more tailored the response will be to your specific situation.
On-Demand Webinar: Using ChatGPT to Draft & Negotiate Contracts with Laura J. Greenberg, Foster Sayers, and Nada Alnajafi
Real-Life Example and Results: Comparing Two Versions of a Non-Disclosure Agreement
Let’s apply the WARPS method to a real-life document comparison example. In this example, I used two clean versions (aka no visible markups or redlines) of a non-disclosure agreement (NDA). Here’s the example prompt I used:
[Audience] I want to create this comparison table of these two agreements, for myself. [What] I want an output that enables me to see a detailed comparison outlining the changes between two versions of a NDA. [Purpose] I want a detailed overview outlining the differences between the two documents and accompanied by the impact of that change so I can see the holistic picture when negotiating this agreement. [Role] I am an in-house counsel for a company. I am pressed for time but need to be detail-oriented and exact. [Specifics] Due to length constraints, tell me what other information you need to complete this task. I am open to having the table as a .txt or .pdf file.
This is how it looks in ChatGPT-4:
Because ChatGPT is generative, it is unlikely that the same output will be generated for the same request each time it is asked. So, I used the above prompt with the same two NDAs several times to see the different responses ChatGPT-4 generated.
Attempt 1
The first attempt was disappointing because ChatGPT-4 seemed unable to produce a thorough and complete overview of changes. It told me it was processing the information and it would contact me once the request was complete.
If you run into a similar result, don’t wait for ChatGPT to “process information”! ChatGPT is not processing the information–it has, for lack of a better word, hallucinated that it is working. This response means it’s time to alter your approach or open up a new chat window and start again.
Attempt 2
This time, ChatGPT-4, responded with a request for additional information to complete the request.
I responded simply with: “I want to know where there have been substantive changes that impact the meaning of the contract. Can you highlight the changes in bold?”
ChatGPT-4 did the work this time and created a bullet-pointed list of changes. However, it did not outline the impact of those changes. So I followed up with: “What is the impact of each of these changes?”
In response, ChatGPT-4 provided a bullet point list of 11 changes with their impact and an overall summary of the impact of the changes. Here are a few samples of the rather long response.
If you are a ChatGPT Plus subscriber, you can see the whole conversation here.
Attempt 3
The third time, ChatGPT-4 responded with an overview of its approach and a request for additional information.
I responded “Yes” so that ChatGPT could review the documents. It guided me through the process more and produced the following result.
Then, I requested a PDF file for the output but that did not come out formatted properly:
I prompted: “That doesn’t work. Can you provide the same table in this chat or as an Excel file?”
ChatGPT-4 provided an Excel file, which was formatted properly, but did not provide any more detailed information than what was previously provided in the chat.
So I prompted: “I thought you said you would provide more detailed information?” In response, it provided a more thorough analysis as it related to the changes it identified in the Excel format.
Then I prompted: “Can you reproduce the Excel chart and make it a table in this chat window?” Here was the result.
As you can see, this attempt generated a less complete list of the changes but focused more on the analysis of the impact of the changes.
If you are a ChatGPT Plus subscriber, you can see the whole conversation here.
In Conclusion
In general, I’ve had the best success with ChatGPT document compare when working with small sections of text (as opposed to entire contracts). This approach has enabled me to get the detail and precision required for legal professionals.
Specifically, I like to ask how the change impacts that section (and occasionally, the entirety of the agreement). But when I ask for ChatGPT-4 to review larger sections of text or whole documents, I’ve found that the responses vary, and at most, it is best at providing high-level summaries rather than detailed, granular information. That is why I chose an NDA, a short agreement, to use as an example for this article.
For a detailed comparison between two versions of a contract, I would stick with Microsoft Word’s Compare feature over ChatGPT. But ChatGPT-4 is useful to: (a) compare changes within limited sections or clauses, (b) to produce a high-level summary of the changes and identify themes, and (c) to analyze how the changes impact the clause or agreement. In conclusion, while Microsoft Word is great for editing contracts and tracking revisions, ChatGPT can potentially offer more in terms of understanding the implications of those changes.
Happy prompting!
For more expert tips about using ChatGPT with contracts:
- Read more articles in the author’s guest column Chatting Contracts with GPT
- Follow Laura on LinkedIn and reach out to her if you want customized trainings for your legal team
- Subscribe to the Contract Nerds newsletter for weekly tips about contracts straight to your inbox ????????
