What Makes Science True? | NOVA

What makes science reliable? The ability to reproduce the results of an experiment, known as reproducibility, is one of the hallmarks of a valid scientific finding. But science is facing what many consider a reproducibility crisis, and the stakes are high. Many scientific claims cannot be replicated, and many clinical trials fail as a result. This 15-minute video examines the reproducibility crisis and reports on the outcome of five experiments designed to test the reproducibility of cancer biology studies. While NOVA had no editorial involvement in this video, we believe it is a thoughtful and well-researched exploration of a complex topic, and we are pleased to share it with you. The video was produced by Dakin Henderson and Kelly Thomson, and was funded entirely by the Laura and John Arnold Foundation (LJAF). In the interest of full transparency: LJAF has an interest in this subject matter, is one of a handful of funders that supports the cost of conducting reproducibility experiments, and provided funding to the organizations featured in this video in order to carry out the reproducibility studies covered here. We have been assured that LJAF had no editorial involvement in the video. You can also learn more about reproducibility by reviewing the following publications: Many analysts, one dataset: Making transparent how variations in analytical choices affect results https://osf.io/j5v8f/ False-Positive Psychology : Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant http://journals.sagepub.com/doi/pdf/1... The Statistical Crisis in Science http://www.stat.columbia.edu/~gelman/... Why Most Published Research Findings Are False http://robotics.cs.tamu.edu/RSS2015Ne... Journals unite for reproducibility http://science.sciencemag.org/content... Promoting an open research culture http://karlan.yale.edu/sites/default/... Estimating the reproducibility of psychological science http://www.psykologforbundet.se/Docum... Design, power, and interpretation of studies in the standard murine model of ALS https://www.researchgate.net/publicat... Raise standards for preclinical cancer research http://www.nature.com/nature/journal/... Believe it or not: how much can we rely on published data on potential drug targets? http://www.nature.com/nrd/journal/v10... A Survey on Data Reproducibility in Cancer Research Provides Insights into Our Limited Ability to Translate Findings from the Laboratory to the Clinic http://journals.plos.org/plosone/arti... ASCB Member Survey on Reproducibility http://www.ascb.org/wp-content/upload... 1,500 scientists lift the lid on reproducibility http://www.nature.com/news/1-500-scie... Interpretations of the results of the Reproducibility Project Psychology and the National Institute for Neurological Disorders and Stroke replications vary depending on how "reproducibility" is defined. Original sources for each reproducibility study are: Amgen: http://www.nature.com/nature/journal/... Bayer: http://www.nature.com/nrd/journal/v10... NINDS: https://www.ncbi.nlm.nih.gov/pmc/arti... ALS TDI: http://www.nature.com/news/preclinica... RP:P: https://osf.io/ezcuj/wiki/home/ RP:CB: https://osf.io/e81xl/wiki/home/ We hope that this information on reproducibility will start a conversation. We encourage you to share this video and the publications above, and if you have feedback, let us know in the comments. We’d like to hear from you. Credits: Produced and Directed by Dakin Henderson & Kelly Thomson Animation by Kent Leon Welling Original Music by Louis Weeks Editing, Cinematography and Narration by Dakin Henderson Funding provided by the Laura and John Arnold Foundation Featuring Trevor Butterworth John Ioannidis Elizabeth Iorns Brian Nosek Additional Music “6 Ghosts I” and “18 Ghosts II” by Nine Inch Nails Thank You Jay Bradner Peter Gotzsche Marcia McNutt Ben Pender Cudlip Nicole Perfito Lawrence Rajendran Jackson Solway Deborah Zarin Archival Material abc7 News Atlantic Monthly Group, Inc. Conus Archive The Economist Group Getty Images KARGER The Lancet/Elsevier NASA Nature Publishing Group NBC Bay Area News PLOS One Pond5 Salon Media Group, Inc. Science/AAAS Scientific American Shutterstock The Week Publications, Inc. Wiley *** NOVA Facebook:   / novaonline   NOVA Twitter:   / novapbs   NOVA Instagram:   / novapbs