#158 Bayesian Workflows & Foundation Models, with Stefan Radev

Support & Resources → Support the show on Patreon:   / learnbayesstats   → Bayesian Modeling Course (first 2 lessons free): https://topmate.io/alex_andorra/1011122 Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work at https://bababrinkman.com/ ! Takeaways: Q: Why are prior predictive checks so underused in practice, and how do simulations help? A: They're underused because researchers don't always think to run them before seeing data -- but also because doing them rigorously (in the style Michael Betancourt advocates, with prior push-forward checks on interpretable summaries) takes effort. Simulations make it cheap to generate thousands of “what-if world” datasets from your model and check whether they look plausible, catching bad priors before you ever touch real data. Q: How can generative AI help with prior elicitation? A: Rather than forcing a domain expert to choose a distributional family and parameterize it, you can use a generative model to translate their qualitative knowledge directly into a prior. The expert describes what realistic data should look like; the generative model produces synthetic datasets matching that description; those datasets are used to fit a prior distribution. It removes the assumption that experts can think in terms of parameters and replaces it with the more natural question: does this look like your data? Q: What would a foundation model for Bayesian inference actually look like? A: Stefan's bet is that it won't be a fine-tuned general LLM. The right analogy is chess: you don't fine-tune GPT to play chess, you teach it when to call Stockfish. For Bayesian inference, you'd want a semantic layer – an LLM that understands the analysis goal – calling specialized numerical engines (MCMC samplers, amortized inference networks) that do the actual computation. Agent skills are already a step in this direction; the longer-term vision is engines that have been trained from scratch to generalize across large families of models and priors. Q: What is self-consistency in amortized inference and why does it matter? A: Self-consistency is the idea that when you train a network on simulations, you can use the network's own outputs as additional training signal – essentially creating a feedback loop where the network teaches itself to be more internally consistent. This can significantly improve data efficiency, letting you get better posteriors from the same number of simulations. It's particularly useful when running the simulator is expensive. Q: Why is sensitivity analysis so important yet so underused in Bayesian practice? A: Every Bayesian analysis rests on a set of implicit choices – the prior, the likelihood, the data preprocessing steps, the specific model architecture. Change any of those and you might get a different answer. A striking example from the 2020 COVID pandemic: nine analysis teams asked to estimate the same reproduction number R came up with estimates so different that the between-team uncertainty dwarfed the within-team uncertainty. Every team was underestimating total epistemic uncertainty by treating their analytical choices as fixed. Sensitivity analysis makes this uncertainty explicit – but classically it's so expensive that it only gets done when reviewers force it. Full takeaways at https://learnbayesstats.com/episode/b... Chapters: 00:00 How does amortized inference fit into modern Bayesian workflows? 06:01 What role do simulations play across the full Bayesian workflow? 12:12 How do you elicit priors from a domain expert who doesn't think in distributions? 19:01 What would a foundation model for Bayesian inference actually look like? 35:32 What is self-consistency in amortized inference and why does it matter? 39:22 How does semi-supervised learning improve simulation-based inference? 43:16 Why is sensitivity analysis so important yet so underused in Bayesian practice? 47:40 What is multiverse analysis and how does it change how we report Bayesian results? 51:32 How does amortized inference make sensitivity and multiverse analysis affordable? 01:02:47 How do amortized inference and classical MCMC complement each other? 01:10:08 What are the next major directions for BayesFlow and amortized inference research? Thank you to my Patrons (https://learnbayesstats.com/#patrons) for making this episode possible! Links from the show at https://learnbayesstats.com/episode/b...