The LLM as Expert Conflict Coder: A Structured Adjudication Methodology and Its Limits

Human-coded ordinal indicators remain central to peace and conflict measurement, but human coding is slow, costly, and difficult to scale. Large language models offer an obvious substitute. This talk presents IEP's structured adjudication methodology for using LLMs as expert coders: defined ordinal scales, detailed coding rubrics, worked country-year anchors, and curated text inputs that together produce scored assessments with justifications. We discuss how the approach is being applied to conflict measurement, and report honestly on what we have learned about its limits. A workable LLM-coder pipeline has to confront several frictions that are easy to overlook. Models do not natively respect the ordinal structure analysts impose on them. Prompt design choices that look harmless can quietly collapse the output distribution and create false precision. And there are real tradeoffs between reproducibility and useful uncertainty estimation The aim is to offer a useful map of where LLM adjudication can credibly supplement expert coding, and where it cannot. Thomas Morgan, Chief Research Officer, Institute for Economics and Peace