Evaluating Large Language Models as Judicial Decision-Makers – Oren Gazal-Ayal (University of Haifa)

PLAMADISO Talk by Oren Gazal-Ayal (University of Haifa) – 26 May 2026 || Organized by the Research Group "Digital Economy, Internet Ecosystem & Internet Policy" @Weizenbaum Institute for the Networked Society Title "Evaluating Large Language Models as Judicial Decision-Makers" Abstract Large Language Models (LLMs) are increasingly shaping various domains, yet their ability to align with human judgment remains a critical challenge. This study explores the extent to which LLMs can serve as judicial decision-makers by comparing their sentencing decisions to those of 123 retired judges on two fictional cases involving rape and violence. We evaluate GPT, Gemini, and Claude using zero-shot, few-shot, and chain-of-thought prompts. LLMs showed greater consistency, producing significantly lower sentence disparity than judges. To assess accuracy, we treated the judges' average sentence as a conservative benchmark — acknowledging that the "correct" sentence is unknown. If models outperform even this minimal standard, they are closer to any plausible ground truth. Remarkably, all LLMs deviated less from the judges' mean than the judges themselves, suggesting that when properly prompted, LLMs can deliver more accurate sentencing decisions than human judges. Link to paper: Gazal Ayal, O., Elyoseph, Z., & Solomon, A. (2026). Evaluating Large Language Models as Judicial Decision-Makers. Justice Quarterly, 1–36. https://doi.org/10.1080/07418825.2026... Short Bio Prof. Oren Gazal-Ayal is the Vice President and Dean for Research and Development at the University of Haifa. Previously, he served as the Dean of the Faculty of Law and the director of the Center for the Study of Crime, Law and Society. He is an expert in criminal law and procedure, sentencing law, and law and economics at the University of Haifa. He regularly publishes in leading journals, including Duke Law Journal, The Journal of Law & Economics, The Journal of Empirical Legal Studies, Law & Social Inquiry, and all leading Israeli legal journals. His papers are frequently cited in Israeli Supreme Court decisions. More about Oren Gazal-Ayal Google Scholar: https://scholar.google.com/citations?... Stay tuned for future events: https://plamadiso.weizenbaum-institut.de Follow us on LinkedIn:   / plamadiso   Find all PLAMADISO Talks in our playlist:    • Plamadiso Talks   The Weizenbaum Institute for the Networked Society: https://weizenbaum-institut.de/