GPT-3 was surprisingly capable at legal reasoning, going largely unnoticed and unused despite its potential.
Winston Weinberg discovered that GPT-3 could produce lawyer-quality legal answers when tested on landlord-tenant questions, with 86% of answers deemed acceptable by attorneys — a finding that even surprised OpenAI's leadership.
transcript
Winston Weinberg: And what had happened was he showed me GPT-3, which at the time was public and I was, first of all, just incredibly surprised that no one was talking about GPT-3 and no one was using it in any way, shape, or form. And he showed me that and I showed him kind of my legal workflows. And we started, the kind of aha moment was we went on r/legaladvice, which is basically a subreddit where people ask a bunch of legal questions and almost every single answer is 'So who do I sue?' Almost every single time. And we took about 100 landlord-tenant questions and we came up with kind of some chain of thought prompts. And this is before anyone was talking about chain of thought or anything like that. And we applied it to those landlord-tenant questions and we gave it to three landlord-tenant attorneys. And we just said nothing about AI. We just said, here's a question that a potential client asked and here's an answer. Would you send this answer without any edits to that client? Would you be fine with that? Is it ethical? Is it a good enough answer to send? And 86 out of 100 was yes.