The simple test every AI feature needs to pass | The Assessment Engineers

Building a slick AI demo? Easy. But shipping something that works every time, at scale, for real users? That's the hard part. In the first episode of The Assessment Engineers, host Kevin Holloway (VP of K-12 Business Development at Learnosity) sits down with Kate Hake (VP of Product) to talk about what it actually takes to build AI assessment products that work in production. They chat about the simple test every AI feature should pass before you start building it, the hidden complexity of authoring, why your AI grader can't fix a rubric that was never fully written down, and more. Got questions? Email [email protected] for answers. 0:00 Intro — the hard part of shipping AI 1:20 Kate's background & Learnosity's AI team 3:27 From demo to production: the real challenges 4:32 Hidden difficulties when building assessment products 6:12 Why authoring tools are more complex than you think 7:53 What actually makes AI assessment & feedback better for learners 10:07 The gap between AI hype and what works in production 13:33 What authors and publishers actually need from AI authoring tools 15:38 AI content generation & item bank health 15:45 Guardrails for AI grading and feedback 17:14 Identity masking and student data safety 18:09 Accessibility: why you can't "vibe code" a product 20:25 What assessment will look like in 2 years 24:25 Advice for PMs with AI features on the roadmap 26:06 What benefit AI has for builders 27:52 Wrapping up