Arjun Guha: How Language Models Model Programming Languages & How Programmers Model Language Models

In this Tech Talk, Arjun Guha takes a deeper look at how LLMs internally reason about programming concepts, and how programmers reason about the programming abilities of LLMs. We first examine how language models perform at the task of predicting type annotations. Although language models can reason about complex type structures, it is also obvious that they often rely on lexical heuristics such as "n is an int" and "s is a string". Can we disentangle genuine reasoning from heuristics at all? Towards this goal, we present evidence that models do in fact learn a robust internal mechanism for type prediction in their hidden layers. We show that this mechanism often fails to activate on out-of-distribution programs, but when we force it to turn on, we can correct many type prediction failures. We also show that this mechanism is shared across programming languages. We then examine how people perform at the task of steering language models on programming tasks. Many programmers, and new programmers in particular, have skill issues when it comes to writing prompts. But, what does it mean to be good at writing programming prompts? When we interview students struggling to prompt models and instructors trying to teach prompting, a common refrain is that new programmers lack the technical vocabulary needed to effectively describe programming tasks. With a causal intervention experiment on a dataset of prompts, we show that this is not the case. Instead, we build an abstraction to study prompt editing at scale. Our work shows that prompts go wrong when they lack details that are essential for the task at hand and have implications for how language models fit into computing education.