1W-MINDS, April 9: Flavio du Pin Calmon (Harvard University), Inference-Time Information Theory

Inference-Time Information Theory In this talk, we argue that "inference-time" Large Language Model (LLM) operation, where we interact with these models post-training without modifying their weights, is fertile ground for information-theoretic methods. We focus on one challenge in particular: watermarking LLM-generated text. Watermarks enable authentication of text provenance and help curb misuse of machine-generated content. We present recent results establishing a close connection between LLM watermarking and coding theory, showing that classical tools such as the Plotkin bound yield fundamental limits on watermark performance. This perspective also informs the design of two practical watermarks: SimplexWater and HeavyWater. We show that these watermarks achieve high detection accuracy with minimal impact on text quality, even in low-entropy tasks such as code generation. We also briefly survey other inference-time challenges that can be addressed with information theory, such as inference-time alignment. These results illustrate a broader opportunity: as LLMs increasingly serve as black-box components of more complex systems, information and coding theory offer a principled toolkit for shaping, verifying, and controlling their outputs.