Making Small Models Cool Again | Context Window w/ Ed Anuff + Anant Jhingran

Are bigger AI models always better? Or is it time to rethink scale? In this episode of Context Window, hosts Ed Anuff and Anant Jhingran sit down with Kate Soule (IBM, Granite) to explore the rise of smaller, fit-for-purpose models. They dig into the tradeoffs of large models (cost, latency, customization), the advantages of small models (efficiency, flexibility, faster fine-tuning), and what this shift means for enterprises building real-world AI systems. 👉 Tune in for insights on model size, agent design, and why Kate says: “It’s time to make small models cool again.” ____________________ Chapters: 00:00:00 Intro & Today’s Theme – Small Models 00:01:15 Headlines: AI Agents Interviewing Humans 00:04:00 Is Silicon Valley Still the Tech Capital? 00:06:30 Optimists vs. Doomers: The AI Debate 00:11:52 Guest Intro: Kate Soule, IBM Granite 00:14:00 What Are Small Models? (Granite family, 1–30B params) 00:17:20 Small vs. Quantized Models – Trade-offs & Use Cases 00:19:20 The Ensemble Approach: No One Model to Rule Them All 00:22:20 Where Small Models Shine (RAG, grounding, tool use) 00:26:00 Small Models & Agents – NVIDIA’s Bet 00:29:30 Designing Agents Differently for Small Models 00:32:00 Do Small Models Need Fine-Tuning? (Adapters, LoRA, alternatives) 00:36:00 Specialization Beyond Fine-Tuning (prompts, memory, KV stores) 00:38:30 Are Models Becoming Commodities? Future Directions 00:41:10 Wrap-Up with Kate – Key Takeaways 00:42:35 Host Debrief: Fine-Tuning, Agents, & Architecture Lessons 00:47:30 Databases vs. Models: Lessons in Boundaries 00:49:20 Teaser for Next Episode with David Cox 00:50:55 Closing Banter & AI Doomsday Clock