I Tested Open Evidence on 5 Oncology Trials. It Was Missing the Document Every Time.

A clinical AI tool used by 40% of US physicians answered clinical questions confidently from trials it had only read in part. I tested Open Evidence on five recent oncology trials. In every one, the AI was missing the document the question depended on. The five trials: TROPION-Breast02, DESTINY-Breast05, OptiTROP-Lung04, SERENA-6, and INAVO120. Across all five, Open Evidence did not have access to the supplements, protocols, statistical analysis plans, or disclosure forms that the workflow of evidence-based oncology depends on. The architectural fact held in every test. Behavior varied — sometimes the AI proceeded as if it had retrieved the source, sometimes it acknowledged the gap on direct question. The underlying gap was the same. This video walks through each test in detail, with verbatim Open Evidence responses on screen. ── Companion Substack write-up (verbatim prompts, full transcripts, reproducibility instructions): https://allenlimd.substack.com/p/open... ── Chapters: 0:00 — The primer - Bishal Gyawali's "How I Read a Clinical Trial Report?" 1:06 — OpenEvidence's marketing and the unexpected test results 2:03 — Test 1: TROPION-Breast02 5:07 — Test 2: DESTINY-Breast05 7:05 — Test 3: OptiTROP-Lung04 8:05 — Test 4: SERENA-6 9:12 — Test 5: INAVO120 (the disclosure layer) 11:07 — "No access" across five trials 12:01 — What this leaves you with ── A note on retrieval scope: Open Evidence markets full-text content partnerships with NEJM, JAMA, NCCN, Cochrane, and Wiley. One of the five trials tested (TROPION-Breast02) was published in Annals of Oncology, which is not on Open Evidence's named partnership list. It is possible that this is part of why retrieval was limited to the abstract for that trial. This does not change the architectural finding. Open Evidence produced a confident clinical recommendation citing a trial whose retrieval was abstract-only, without flagging the limitation until I asked directly. A clinician relying on the recommendation would not know what Open Evidence had and had not seen. The other four trials were published in NEJM, which is on the partnership list. Across all four NEJM trials, the supplements, protocols, statistical analysis plans, and disclosure forms remained outside what Open Evidence could retrieve. ── Anchoring reference: Bishal Gyawali, MD, PhD. "How I Read a Clinical Trial Report? A Primer for Busy Clinicians." JCO Oncology Practice, 2026. ── Methodology: All five trials were independently appraised first using documented Source Report methodology. Open Evidence was then tested using prompts derived from the data points each appraisal had already identified as critical. Test sessions were conducted between April 25–27, 2026. Verbatim transcripts of all five sessions are linked in the Substack write-up. Quotes from Open Evidence shown in this video are reproduced from the test session transcripts. Slide cards display the verbatim text; spoken delivery may condense quotes for natural pacing. The text on screen is the verbatim record. Reproduced under fair use for criticism and journalism. ── Disclosures: Allen Li, MD is a board-certified hematologist-oncologist in community practice. The Oncology AI Lab and The Source Report are independent editorial projects. The views expressed are my own and do not represent my employer or any institution I am affiliated with. This video is journalism and analysis of clinical AI tools — not medical advice and not a clinical guideline. No financial relationship with Open Evidence, with any of the trial sponsors mentioned. ── #OncTwitter #AIinMedicine #EvidenceBasedMedicine #openevidence #oncologyai