We're Overestimating Medical AI — and Underestimating the Harm (Jessica Morley, Yale)

AI ethicist Jess Morley: these chatbots are giving medical advice — so regulate them as medical devices. Part of The Agentic Patient, a Faces of Digital Health series on how patients actually use AI — which tools, which prompts, which safeguards. In this episode, host Tjaša Zajc sits down with Dr Jess Morley, Associate Research Scientist at the Yale Digital Ethics Center and a former AI subject-matter expert at the UK Department of Health and Social Care, for a clear-eyed account of where health AI is going wrong — and how to use it well anyway. Morley argues we systematically overestimate what these tools can do and underestimate the harm. She makes the case for "skeptical optimism," explains why bioethics principles built for one-to-one care break down against many-to-many AI harms, and reframes ambient scribes as inference engines rather than transcription services — with real consequences for coding, billing and patient records. Then she gets practical: the guardrails, prompts and habits patients (and clinicians) can use today. Guest: Dr Jessica Morley — Associate Research Scientist, Yale Digital Ethics Center; formerly UK Department of Health and Social Care and the Bennett Institute, University of Oxford. What the conversation covers: Why "skeptically optimistic" is the honest position on health AI AI adoption as "a hammer looking for nails" — and what needs-led design would look like instead OpenEvidence, EU rules and the question of regulatory capture The DeepMind–Royal Free case and why law alone isn't enough Beneficence, non-maleficence, autonomy, justice — and where they fail for AI Ambient AI scribes, miscoding, billing inflation and phantom tests Paid vs free models and the widening access gap The "ask why" rule and knowing when to walk away from a chatbot Red-teaming your own assumptions and playing models off each other Building a personal "harness" with skills so AI works from your history The last-mile problem and the case for regulating LLMs as medical devices Whether AI is narrowing how clinicians think Chapters: 00:00 — Intro: The Agentic Patient and the case for skeptical optimism 02:52 — "A hammer looking for nails": adoption pressure without a plan 04:25 — OpenEvidence, EU rules and regulatory capture 06:42 — The DeepMind–Royal Free lesson: why law needs ethics 11:29 — The bioethics principles and what they were built to do 16:40 — Autonomy, consent and the ambient-scribe problem 19:49 — Scribes as inference engines: miscoding, fraud and phantom tests 26:06 — Paid vs free models and the access gap 30:25 — Using AI safely: the "ask why" rule 38:38 — Knowing when to walk away: engagement design and degradation 41:58 — Red-teaming and playing models off each other 46:00 — Harnesses and skills: making the model work for you 48:38 — The last-mile problem and regulating AI as a medical device 55:00 — Does AI narrow the clinician's mind? The Agentic Patient series: https://www.facesofdigitalhealth.com/... Website: https://www.facesofdigitalhealth.com Newsletter: https://fodh.substack.com LinkedIn:   / faces-of-digital-health