Designing Better Formatives: Simulated Use, Realism, and the Risks of Playing It Safe

Formative usability studies are where device teams either uncover real-world use errors early—or get blindsided later during summative validation (when changes are expensive and timelines are tight). In this Friday In-Focus session, we break down how to design better formative studies by increasing realism in the right ways (without over-building prototypes), choosing the right level of fidelity, and avoiding the common trap of “playing it safe” in a way that hides real risks. Key themes include the difference between formative vs. summative evaluation, how to set the right fidelity across five dimensions (physical, functional, environmental, procedural, participant), how to handle small sample sizes and outliers, and why startups need a focused-but-defensible approach to formative testing to avoid costly post-market consequences. You’ll learn: • Formative vs. summative: what each is for—and why confusing them creates regulatory risk • The five dimensions of study fidelity and how to choose what matters most for your device • Environmental realism: why use location, noise, lighting, and workflow assumptions can change risk • Participant fidelity: selecting representative users (and why “engineers as users” often fails) • Handling small-N studies and outliers using targeted follow-up tests + root cause analysis • Testing sensitive products responsibly while maintaining realism and participant comfort • How to evaluate the full “user interface” (hardware + software + labeling + IFU + training) together Chapters: 00:00 – Welcome & session overview 01:45 – Formative vs. summative: purpose, timing, and why mixing them causes problems 07:47 – Why formative is the best investment (and why skipping it makes summative risky) 14:49 – Cost, timing, and regulatory consequences of “playing it safe” 16:28 – The 5 dimensions of study fidelity (physical, functional, environmental, procedural, participant) 19:59 – Environmental fidelity: how real-world context changes use risk 24:27 – Participant fidelity: selecting representative users (and avoiding biased samples) 27:14 – Small sample sizes: interpreting outliers and deciding what to investigate 30:43 – Foreseeable vs. non-foreseeable risks: how to treat outliers 32:10 – Targeted follow-up testing and adaptive recruitment strategies 35:24 – Startups vs. large companies: right-sizing formative approaches 37:09 – Why startups can’t afford to miss use-related risks 40:29 – When usability impacts outcomes even if “safety risk” seems low (e.g., SaMD) 45:11 – The full UI scope: hardware + software + labeling/IFU + training must be evaluated together 46:35 – Sensitive products: realism vs participant comfort + ethical considerations 49:48 – Upcoming webinars: legacy devices (HF) and connected device cybersecurity 51:54 – Next session preview: regulatory intelligence + AI in post-market surveillance