GPT stands for Generative Pre-trained Transformer (GPT), a type of language model that uses deep learning to generate human-like, conversational text. GPT-4 is the newest version of OpenAI’s language model systems. Its previous version, GPT 3.5, powered the company’s wildly popular ChatGPT chatbot when it launched in November of 2022.
GPT-4, the new multimodal deep learning model from OpenAI, has passed the Uniform Bar Exam, demonstrating an enormous leap for machine learning and proving that an artificial intelligence program can perform complex legal tasks on par with or better than humans, according to a new paper co-authored by Daniel Martin Katz, professor of law at Illinois Institute of Technology’s Chicago-Kent College of Law.
The Uniform Bar Examination (UBE) is coordinated by the National Conference of Bat Examiners, and is composed of the Multistate Essay Examination (MEE), two Multistate Performance Test (MPT) tasks, and the Multistate Bar Examination (MBE). It is uniformly administered, graded, and scored and results in a portable score that can be transferred to other UBE jurisdictions.
In a period of roughly four years, the leading large language model family (GPT) has progressed from zero percent on the Multistate Bar Exam for GPT-2 to nearly 76 percent on the Multistate Bar Exam in GPT-4. (The Multistate Bar Exam is the multiple-choice section of the Uniform Bar Exam.) Overall, the authors report a 297 Uniform Bar Exam score for GPT-4, which reflects passing the bar by a fairly comfortable margin. The highest threshold in the country is Arizona at 273 (Illinois is 266).
“GPT-4 represents a new frontier in AI’s role in the legal profession and society at large,” says Katz, who collaborated with the legal AI company Casetext and fellow researcher Michael Bommarito. “The bar exam is an interesting test for AI to pass because it highlights the prospect of such technology becoming a ‘force multiplier’ that expands access to legal services to all members of society, including those who couldn’t previously afford to hire a lawyer.”
Katz did predict that within a few years, perhaps even by the end of this year, large corporations like Microsoft and Google may start experimenting with offering AI legal services. Nothing that could replace an attorney in a courtroom – yet – but work that is typically performed by law researchers and paralegals, like compiling data or answering clients’ questions about the law.
GPT-4 scored a 75 percent on the bar exam, higher than the 68 percent average and good enough to place in the 90th percentile. In a previous paper that Katz co-wrote, GPT-3.5 scored a 50 percent and passed only two multiple choice portions of the bar exam, placing it in the 10th percentile.
In this test, GPT-4 not only took the multiple choice sections, but also the essays (worth 30 percent) and performance test (worth 20 percent). Although many have been skeptical about AI’s ability to pass sections that require generating language, GPT-4 did so by a significant margin, giving responses that were generally on par with the “representative good answers” provided by many state bars.
The latest GPT model also shows fewer “hallucinations,” in which an AI language model confidently asserts wrong answers that have no basis in reality.
Passing the bar exam requires the command of not just ordinary English, but of complex “legalese,” which is difficult even for humans. GPT’s rapid advancement in this field is sure to have wide implications for the legal profession.
“Lawyers need to figure out how to really use these tools. And those that do, it’ll be a very positive thing for them. We’re sitting on the dawn of a major increase in potential capacity,” says Katz. “These are tools that allow you to more effectively do your work, so you need to learn how to use them to maximum efficacy.”
“I’m generally very optimistic about it,” said Matthew Shepard, a public defender and board member of the Chicago chapter of the National Lawyers Guild, which often provides pro bono counsel to arrestees and activists. “I think it will be helpful in assisting attorneys with filing motions and going through large quantities of discovery”
“Sounds like it could help out a lot of pro se litigants,” agreed Zane Thompson, a workers’ compensation attorney with the Chicago law firm Ganan & Shapiro.