SICSS Istanbul 2026 | Reproducible AI: Replication, Prompt Stability in Social Science (ChrisBarrie)

Can research built on Large Language Models be replicated? As LLMs increasingly replace human annotators, coders, and research assistants, ensuring that AI-assisted research remains transparent, reproducible, and scientifically reliable has become one of the central methodological challenges in computational social science. In this lecture, Dr. Chris Barrie (New York University) examines how generative AI changes the meaning of replication and reproducibility in social science. Drawing on recent research published in leading political science journals, he demonstrates why LLM outputs may vary across prompts, model versions, parameter settings, and even repeated executions of the same pipeline. The lecture introduces new methods for evaluating prompt robustness, annotation reliability, and the stability of AI-assisted coding workflows before discussing their implications for scientific inference. The second half of the session presents practical tools for auditing LLM-based research pipelines, including Prompt Stability Scores (PSS), adaptations of Krippendorff's Alpha, and empirical comparisons between human annotators and modern language models. Through hands-on exercises in Python, participants learn how to diagnose fragile prompts, evaluate annotation consistency, and design reproducible AI workflows that meet the standards of contemporary computational social science. Topics covered Large Language Models (LLMs) Replication and reproducibility AI-assisted research methods Prompt engineering Prompt Stability Score (PSS) Krippendorff's Alpha Annotation reliability Human versus LLM annotation Prompt sensitivity Research transparency Scientific reproducibility Python workflows for LLM evaluation Open-source and proprietary language models Benchmarking LLM performance Validation of AI-generated annotations Computational social science methodology Best practices for reproducible AI research This lecture was delivered during the Summer Institute in Computational Social Science (SICSS Istanbul 2026) for graduate students and researchers interested in computational social science, artificial intelligence, political methodology, and the development of transparent, reproducible, and trustworthy AI-assisted research pipelines.