Naoki Egami: Conformal Policy Learning with Distribution-Free Safety Guarantees

Subscribe to the channel to get notified when we release a new video. Like the video to tell YouTube that you want more content like this on your feed. See our website for future seminars: https://sites.google.com/view/ocis/home Tuesday, May 19, 2026: Speaker: Naoki Egami (Massachusetts Institute of Technology) Details: Zoom link, Meeting ID: 968 8371 7451, Passcode: 414559 Title: Conformal Policy Learning with Distribution-Free Safety Guarantees: Application to AI-Powered Interventions Abstract: Generative AI is emerging as a new class of intervention in the social sciences, with applications designed to change attitudes and behaviors through scalable, personalized interactions. For example, conversational agents have been used to reduce political polarization and improve workplace productivity. At the same time, recent empirical studies highlight an important risk: while such interventions may benefit many individuals and tasks, they may also harm others. How, then, can AI interventions be deployed safely? In this paper, we develop a new statistical framework, conformal policy learning, to deliver pre-specified safety guarantees when deciding whether individuals should receive a new intervention or the status quo. For instance, a researcher may require that the probability that an individual is harmed by the chosen intervention is below 1%. Using tailored conformal hypothesis testing, our method provides finite-sample safety guarantees under the standard exchangeability assumption, without relying on any modeling assumptions. It also achieves asymptotically optimal power or welfare maximization when the conditional expectation functions of outcomes are correctly specified. Thus, our treatment assignment rule is guaranteed to be safe in finite samples while attaining optimality under standard modeling assumptions. In practice, our framework enables researchers to deploy AI safely by assigning AI interventions only to people and tasks that satisfy user-specified safety requirements, and by reverting to the status quo otherwise. This offers a middle ground between two undesirable extremes: unfiltered deployment that ignores AI risks and total avoidance due to safety concerns. We illustrate the method through extensive simulations and an experiment in which randomly assigned AI chatbots are used to reduce conspiracy beliefs. This is joint work with Ying Jin. Discussant: Eli Ben-Michael (Carnegie Mellon University)