#156 Bayesian Experimental Design & Active Learning, with Adam Foster

Support & Resources → Support the show on Patreon:   / learnbayesstats   → Bayesian Modeling Course (first 2 lessons free): https://topmate.io/alex_andorra/1011122 Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work at https://bababrinkman.com/ ! Takeaways: Q: What is Bayesian experimental design and what problem does it solve? A: It's the practice of using a Bayesian model to decide how to collect data before you collect it. Most statistical thinking starts with a fixed dataset. Bayesian experimental design sits upstream -- you have control over experimental parameters (which questions to ask, which reagents to mix, which conditions to test) and you want to choose them optimally. The Bayesian angle is to ask: what new data would most reduce my current uncertainty? Q: When should you actually use Bayesian experimental design? A: When two conditions hold: you have active control over how data is collected (not just passive observation), and you have a Bayesian model whose prior predictive distribution gives a reasonable picture of what typical data might look like. It's especially valuable when data collection is expensive or irreversible -- when the "committal step" of running an experiment has real cost, it's worth doing the analysis first. Q: What is expected information gain (EIG) and why is it central to Bayesian experimental design? A: EIG is the score you assign to a candidate experimental design -- the amount of information you expect to gain about your model parameters by running an experiment with that design. You compute it by simulating datasets from your prior predictive, doing Bayesian inference on each, and averaging how much the uncertainty decreased. What's remarkable is that you can derive the same quantity from two completely different starting points -- reducing parameter uncertainty, or maximizing outcome uncertainty while correcting for noise -- and arrive at the same formula. That convergence is why EIG keeps being re-discovered independently across fields. Q: Why is computing EIG so hard, and what is "double intractability"? A: Bayesian inference is already intractable in general because it requires integrating over the full prior space. Computing EIG compounds this: you have to synthesize many datasets, run Bayesian inference on each one, and average the results -- so you're doing something hard in a loop. This is what Adam and Tom Rainforth call "double intractability," and it's the main computational bottleneck for continuous-outcome models. Q: How does variational inference help overcome double intractability in experimental design? A: By training an amortized variational inference network that approximates the posterior as a neural network function of the data, you avoid having to solve a full inference problem from scratch for every simulated dataset. Instead, you feed data into the network and get an immediate posterior approximation -- making the EIG computation loop tractable. Full takeaways at: https://learnbayesstats.com/episode/1... Chapters: 00:00:00 What is Bayesian experimental design and why does it matter? 00:06:02 What problem does Bayesian experimental design actually solve? 00:08:54 When should practitioners use Bayesian experimental design? 00:12:00 Is Bayesian experimental design changing how scientists work in practice? 00:15:04 What are the limitations of Bayesian experimental design? 00:17:55 What is expected information gain (EIG) and how does it work? 00:21:05 How do you compute expected information gain in practice? 00:23:48 What is active learning and how does it connect to Bayesian experimental design? 00:41:02 What is active learning by disagreement? 00:48:57 What is deep adaptive design and when should you use it? 00:56:02 How is Bayesian experimental design applied in protein dynamics and quantum chemistry? 01:01:58 What does a practical Bayesian experimental design workflow look like? 01:06:50 What are the future directions for Bayesian experimental design research? Thank you to my Patrons (https://learnbayesstats.com/#patrons) for making this episode possible! Links from the show at https://learnbayesstats.com/episode/1...