AIn't Nothing But a Survey? Using Large Language Models for Coding Open-Ended Survey Responses

2026-04-22 | Input Talk | Leah von der Heyde Abstract Large language models (LLMs) have the potential to make survey research more efficient, including the classification of open-ended survey responses. However, as most existing research on this topic has focused on English-language text or single LLMs, it is unclear whether their applicability generalizes and how the quality of classifications compares to established methods. In this talk, I will demonstrate how LLMs can be used for coding open-ended responses using different access options and prompting and fine-tuning techniques. I will present a study testing these approaches on a dataset of German open-ended survey responses, comparing several LLMs to human coders and other automated methods. Finally, I will discuss the implications of the study findings for practitioners, including the many trade-offs researchers need to consider. Presenter(s) Leah von der Heyde is a computational social scientist and survey methodologist. Her research focuses on the potential and pitfalls of new data sources, such as large language models, for improving the measurement and representation of public opinion. Substantively, she is particularly interested in political attitudes and voting behavior. Leah received her PhD in Social Data Science and Research Methodology from the University of Mannheim. She has a background in political science from LMU Munich, the University of Mannheim, and Georgetown University. Previously, Leah was part of the Social Data Science and AI Lab at LMU Munich, worked for the European Social Survey at GESIS - Leibniz Institute for the Social Sciences, the European Parliamentary Research Service, and several market and public opinion research institutes in Germany and Sweden. At GESIS, she is part of KODAQS, researching and educating social scientists on AI applications in survey research and their implications for data quality.