Logical Experimental Design and Execution in the Biomedical Sciences

Daniel J. Holder1, Michael J. Marino2

1 Biometrics Research, Merck Research Laboratories, West Point, Pennsylvania, 2 Neuroscience, Merck Research Laboratories, West Point, Pennsylvania
Publication Name:  Current Protocols in Pharmacology
Unit Number:  Appendix 3G
DOI:  10.1002/cpph.20
Online Posting Date:  March, 2017
GO TO THE FULL TEXT: PDF or HTML at Wiley Online Library


Lack of reproducibility has been highlighted as a significant problem in biomedical research. The present unit is devoted to describing ways to help ensure that research findings can be replicated by others, with a focus on the design and execution of laboratory experiments. Essential components for this include clearly defining the question being asked, using available information or information from pilot studies to aid in the design the experiment, and choosing manipulations under a logical framework based on Mill's “methods of knowing” to build confidence in putative causal links. Final experimental design requires systematic attention to detail, including the choice of controls, sample selection, blinding to avoid bias, and the use of power analysis to determine the sample size. Execution of the experiment is done with care to ensure that the independent variables are controlled and the measurements of the dependent variables are accurate. While there are always differences among laboratories with respect to technical expertise, equipment, and suppliers, execution of the steps itemized in this unit will ensure well‐designed and well‐executed experiments to answer any question in biomedical research. © 2017 by John Wiley & Sons, Inc.

Keywords: causality; experimental controls; experimental design; experimental execution; power analysis

PDF or HTML at Wiley Online Library

Table of Contents

  • Introduction
  • Experimental Design
  • Proper Execution of Experiments
  • Conclusion
  • Literature Cited
  • Figures
  • Tables
PDF or HTML at Wiley Online Library


PDF or HTML at Wiley Online Library



Literature Cited

Literature Cited
   Almeida, J. L. , Cole, K. D. , & Plant, A. L. (2016). Standards for cell line authentication and beyond. PLoS Biology, 14, e1002476. doi: 10.1371/journal.pbio.1002476.
   Armitage, P. , McPherson, C. K. , & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society. Series A, 132, 235–244. doi: 10.2307/2343787.
   Baker, M. (2015). Reproducibility crisis: Blame it on the antibodies. Nature, 521, 274–276. doi: 10.1038/521274a.
   Baker, M. (2016). How quality control could save your science. Nature, 529, 456–458. doi: 10.1038/529456a.
   Begley, C. G. , & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483, 531–533. doi: 10.1038/483531a.
   Begley, C. G. , & Ioannidis, J. P. (2015). Reproducibility in science: Improving the standard for basic and preclinical research. Circulation Research, 116, 116–126. doi: 10.1161/CIRCRESAHA.114.303819.
   Bordeaux, J. , Welsh, A. , Agarwal, S. , Killiam, E. , Baquero, M. , Hanna, J. , Anagnostou, V. , & Rimm, D. (2010). Antibody validation. Biotechniques, 48, 197–209. doi: 10.2144/000113382.
   Bradbury, A. & Pluckthun, A. (2015). Reproducibility: Standardize antibodies used in research. Nature, 518, 27–29. doi: 10.1038/518027a.
   Collins, F. S. , & Tabak, L. A. (2014). Policy: NIH plans to enhance reproducibility, Nature, 505, 612‐613, doi: 10.1038/505612a.
   Colquhoun, D. (2006). The quantitative analysis of drug‐receptor interactions: A short history. Trends in Pharmacological Sciences, 27, 149–157. doi: 10.1016/j.tips.2006.01.008.
   Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of p‐values. Royal Society Open Science, 1, 140216. doi: 10.1098/rsos.140216.
   Conover, E. (2015, April 7). Why did you become a scientist? Researchers speak out with #IAmAScientistBecause. Retrieved from http://News.Sciencemag.Org/Scientific‐Community/2015/04/Why‐Did‐You‐Become‐Scientist‐Researchers‐Speak‐Out‐Iamascientistbecause.
   DeLoach, A. , Cozart, M. , Kiaei, A. , & Kiaei, M. (2015). A retrospective review of the progress in amyotrophic lateral sclerosis drug discovery over the last decade and a look at the latest strategies. Expert Opinion on Drug Discovery, 1‐20. doi: 10.1517/17460441.2015.1067197.
   Entzeroth, M. , Flotow, H. , & Condron, P. (2009). Overview of high‐throughput screening. Current Protocols in Pharmacology, 44, 9.4.1‐9.4.27. doi: 10.1002/0471141755.ph0904s44.
   Finkbeiner, S. , Frumkin, M. , & Kassner, P. D. (2015). Cell‐based screening: Extracting meaning from complex data. Neuron, 86, 160–174. doi: 10.1016/j.neuron.2015.02.023.
   Freedman, L. P. , Cockburn, I. M. , & Simcoe, T. S. (2015). The economics of reproducibility in preclinical research. PLoS Biology, 13, e1002165 doi: 10.1371/journal.pbio.1002165.
   Geraghty, R. J. , Capes‐Davis, A. , Davis, J. M. , Downward, J. , Freshney, R. I. , Knezevic, I. , … Cancer Research UK (2014). Guidelines for the use of cell lines in biomedical research. British Journal of Cancer, 111, 1021‐1046. doi: 10.1038/bjc.2014.166.
   Green, A. R. , Cross, A. J. , & Goodwin, G. M. (1995). Review of the pharmacology and clinical pharmacology of 3,4‐methylenedioxymethamphetamine (MDMA or “Ecstasy”). Psychopharmacology (Berl), 119, 247–260. doi: 10.1007/BF02246288.
   Gusella, J. F. , & MacDonald, M. E. (2006). Huntington's disease: Seeing the pathogenic process through a genetic lens. Trends in Biochemical Sciences, 31, 533–540. doi: 10.1016/j.tibs.2006.06.009.
   Halsey, L. G. , Curran‐Everett, D. , Vowler, S. L. , & Drummond, G. B. (2015). The fickle P value generates irreproducible results. Nature Methods, 12, 179–185. doi: 10.1038/nmeth.3288.
   Hill, A. V. (1909). The mode of action of nicotine and curari, determined by the form of the contraction curve and the method of temperature coefficients. Journal of Physiology, 39, 361–373. doi: 10.1113/jphysiol.1909.sp001344.
   Howson, C. (2000). Hume's problem: induction and the justification of belief. Oxford: Claredon Press.
   Hume, D. (1739). A treatise of human nature. Oxford: Claredon Press.
   Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2, e124. doi: 10.1371/journal.pmed.0020124.
   Ioannidis, J. P. , Greenland, S. , Hlatky, M. A. , Khoury, M. J. , Macleod, M. R. , Moher, D. , … Tibshirani, R. (2014). Increasing value and reducing waste in research design, conduct, and analysis. Lancet, 383, 166–175. doi: 10.1016/S0140‐6736(13)62227‐8.
   ISO (1994). Accuracy (trueness and precision) of measurement methods and results – Part 1: General principles and definitions. International Organization for Standardization ISO, 5725‐1, 1994.
   Jarvis, M. F. , & Williams, M. (2016). Irreproducibility in preclinical biomedical research: Perceptions, uncertainties, and knowledge gaps. Trends in Pharmacological Sciences, 37, 290–302. doi: 10.1016/j.tips.2015.12.001.
   Jennison, C. , & Turnbul, B. W. (2000). Group sequential methods with applications to clinical trials. Boca Raton, FL: Chapman Hall.
   Kelder, T. , Conklin, B. R. , Evelo, C. T. , & Pico, A. R. (2010). Finding the right questions: Exploratory pathway analysis to enhance biological discovery in large datasets. PLoS Biology, 8, e1000472. doi: 10.1371/journal.pbio.1000472.
   Kenakin, T. , & Christopoulos, A. (2011). Analytical pharmacology: The impact of numbers on pharmacology. Trends in Pharmacological Sciences, 32, 189–196. doi: 10.1016/j.tips.2011.01.002.
   Kimmelman, J. , Mogil, J. S. , & Dirnagl, U. (2014). Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLoS Biology, 12, e1001863. doi: 10.1371/journal.pbio.1001863.
   Krstic, D. , & Knuesel, I. (2013). The airbag problem‐a potential culprit for bench‐to‐bedside translational efforts: Relevance for Alzheimer's disease. Acta Neuropathologica Communications, 1, 62. doi: 10.1186/2051‐5960‐1‐62.
   Langmuir, I. (1918). The adsorption of gases on plane surfaces of glass, mica and platinum. Journal of the American Chemical Society, 40, 1361–1403. doi: 10.1021/ja02242a004.
   Larsen, K. E. , Fon, E. A. , Hastings, T. G. , Edwards, R. H. , & Sulzer, D. (2002). Methamphetamine‐induced degeneration of dopaminergic neurons involves autophagy and upregulation of dopamine synthesis. The Journal of Neuroscience, 22, 8951–8960.
   Levine, M. , & Ensom, M. H. (2001). Post hoc power analysis: An idea whose time has passed? Pharmacotherapy, 21, 405–409. doi: 10.1592/phco.21.5.405.34503.
   Limbird, L. E. (2005). Cell surface receptors: A short course on theory and methods (3rd ed.). New York: Springer.
   Macarron, R. , Banks, M. N. , Bojanic, D. , Burns, D. J. , Cirovic, D. A. , Garyantes, T. , … Sittampalam, G. S. (2011). Impact of high‐throughput screening in biomedical research. Nature reviews. Drug Discovery, 10, 188–195. doi: 10.1038/nrd3368.
   Marino, M. J. (2014). The use and misuse of statistical methodologies in pharmacology research. Biochemical Pharmacology, 87, 78–92. doi: 10.1016/j.bcp.2013.05.017.
   McNutt, M. (2014). Journals unite for reproducibility. Science, 346, 679. doi: 10.1126/science.aaa1724.
   Mills, J. S. (1843). A system of logic: Ratiocinative and inductive. London: John W. Parker.
   Motulsky, H. J. (2014). Common misconceptions about data analysis and statistics. The Journal of Pharmacology and Experimental Therapeutics, 351, 200–205. doi: 10.1124/jpet.114.219170.
   Neimark, J. (2015). Line of attack. Science, 347, 938‐940. doi: 10.1126/science.347.6225.938.
   Nuzzo, R. (2014). Scientific method: Statistical errors. Nature, 506, 150–152. doi: 10.1038/506150a.
   Perrin, S. (2014). Preclinical research: Make mouse studies work. Nature, 507, 423–425. doi: 10.1038/507423a.
   Popper, K. R. (1963). Conjectures and refutations: The growth of scientific knowledge. Abingdon, United Kingdom: Routledge.
   Prinz, F. , Schlange, T. , & Asadullah, K. (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews. Drug Discovery, 10, 712. doi: 10.1038/nrd3439‐c1.
   Ricaurte, G. A. , Yuan, J. , & McCann, U. D. (2000). (+/‐)3,4‐Methylenedioxymethamphetamine (‘Ecstasy’)‐induced serotonin neurotoxicity: Studies in animals. Neuropsychobiology, 42, 5–10. doi: 10.1159/000026664.
   Ricaurte, G. A. , Yuan, J. , Hatzidimitriou, G. , Cord, B. J. , & McCann, U. D. (2002). Severe dopaminergic neurotoxicity in primates after a common recreational dose regimen of MDMA (“ecstasy”). Science, 297, 2260–2263. doi: 10.1126/science.1074501.
   Ricaurte, G. A. , Yuan, J. , Hatzidimitriou, G. , Cord, B. J. , & McCann, U. D. (2003). Retraction of Severe dopaminergic neurotoxicity in primates after a common recreational dose regimen of MDMA (“ecstasy”). Science, 301, 1479. doi: 10.1126/science.301.5639.1479b.
   Rosenthal, R. , & Fode, K. L. (1963). The effect of experimenter bias on the performance of the albino rat. Behavioral Science, 8, 183–189. doi: 10.1002/bs.3830080302.
   Scott, S. , Kranz, J. E. , Cole, J. , Lincecum, J. M. , Thompson, K. , Kelly, N. , … Heywood, J. A. (2008). Design, power, and interpretation of studies in the standard murine model of ALS. Amyotrophic Lateral Sclerosis, 9, 4–15. doi: 10.1080/17482960701856300.
   Tukey, J. W. (1980). We need both exploratory and confirmatory. American Statistician, 34, 23–25. doi: 10.2307/2682991.
   Uhlen, M. , Bandrowski, A. , Carr, S. , Edwards, A. , Ellenberg, J. , Lundberg, E. , Rimm, D. L. , Rodriguez, H. , Hiltke, T. , Snyder, M. , & Yamamoto, T. (2016). A proposal for validation of antibodies. Nature Methods, 13, 823–827. doi: 10.1038/nmeth.3995.
   Wadhawan, S. , Runz, H. , & Burchard, J. (2015). The genome as pharmacopeia: Association of genetic dose with phenotypic response. Biochemical Pharmacology, 94, 229–240. doi: 10.1016/j.bcp.2015.02.005.
PDF or HTML at Wiley Online Library