The California Bar Exam and the Future of AI in Grading:
AI Offers Opportunities and Challenges
In October 2024, the California Supreme Court issued a groundbreaking ruling that could reshape the future of bar exam testing in the state. The Court recommended that the State Bar of California explore the potential use of emerging technologies, particularly artificial intelligence (AI), in assessing bar exam essays and performance exams. This directive aligns with a growing interest in AI within the legal field and opens the door for the integration of technology into one of the most challenging exams in the United States.
The Court acknowledged the need for innovation in legal testing, particularly with respect to assessing seven key practical skills, such as client interviewing and negotiation. The justices specifically urged the State Bar to investigate how AI could “innovate and improve upon the reliability and cost-effectiveness of such testing.” This suggests that the Court sees AI as a potential tool to enhance the bar exam process, not just for grading, but for improving the broader testing experience.
The Court also referenced the 1980 Assessment Center, an early attempt to introduce a more holistic approach to bar exams by testing practical skills. It appears that the justices are looking at AI as a way to improve upon such initiatives, potentially allowing for a more nuanced and sophisticated evaluation of crucial legal skills, such as communication, strategy, and problem-solving.
The introduction of AI into bar exam grading is not without its risks. One of the main concerns is that AI, in its current form, may lack the nuance and understanding that a human grader brings to the table. Legal writing often requires a deep understanding of context, judgment, and reasoning that is challenging for AI to replicate. While AI can certainly detect certain patterns and keywords, it may miss the more subtle elements of a student’s response that a human grader would notice.
This leads to a potentially reductive approach to grading: AI could focus on matching keywords or phrases in an essay response to a predetermined set of “correct” answers. If a student’s response doesn’t align with the exact terminology or structure that the AI expects, the student could be penalized unfairly—even if their analysis is sound. For example, an applicant who demonstrates a thorough understanding of the law but uses different phrasing than what the AI is programmed to recognize could risk receiving a lower score than deserved.
Human grading, especially in the context of complex legal exams like the bar, provides a level of flexibility and discernment that AI currently cannot replicate. A skilled human grader can understand the context of an answer and assess whether the applicant is displaying critical legal thinking, even if their answer deviates from a specific pattern. Furthermore, human graders can account for subjectivity in legal analysis—what might be an acceptable interpretation of the law for one issue may differ from another, depending on the examiner’s judgment.
Human graders are also more adept at providing individualized feedback—something that is critical for bar exam preparation. For example, Executive Bar Review tutorial courses offer personalized feedback based on a student’s individual strengths and weaknesses. These courses provide targeted advice that goes beyond simple answer keys, offering insights into how an applicant can refine their approach to legal reasoning, argumentation, and writing. In contrast, AI, while useful for quick feedback, lacks the personalization that many students find invaluable in their preparation.
AI should not be viewed as a replacement for human grading, but rather as a complementary tool. While AI has the potential to enhance the grading process by providing more consistent, standardized assessments, human expertise remains essential for nuanced evaluations of legal reasoning. Moreover, AI-generated feedback should never be relied upon as the sole means of improving a student’s legal skills. In this regard, AI could be used to supplement traditional preparation methods, such as bar review courses, practice exams, and individualized tutoring. For example, AI could help students pinpoint common mistakes, suggest areas for improvement, and provide practice questions. However, students still need the human element—whether it’s a professor, tutor, or bar review instructor—who can explain complex legal concepts, provide strategic advice, and help students refine their overall approach to legal analysis.
