Open-Source MUltiple Tests Corrections and FOrmatted Tables Software (MUFOS)

Nikolay Petrov; Vasil Atanasov; Trevor Thompson

doi:10.5334/jors.350

Open-Source MUltiple Tests Corrections and FOrmatted Tables Software (MUFOS)

Journal of Open Research Software

Volume 10 (2022): Issue 1

By: Nikolay Petrov , Vasil Atanasov and Trevor Thompson

Open Access

|Mar 2022

Abstract

The p value statistic remains a ubiquitous indicator of the verisimilitude of experimental hypotheses. However, testing multiple hypotheses poses a problem as Type I error rate is inflated. Despite known solutions, this problem remains largely neglected for two reasons: 1) most data analysis tools offer limited multiple tests correction options; 2) the learning curve of existing tools requires hefty time investment. To address these concerns, we present a free, easy-to-use and convenient Software, built around established python libraries, that allows users to apply a MUltiple tests correction and get the results in a readily-understood, FOrmatted table (MUFOS) – https://github.com/nikbpetrov/mufos.

References

Šidák Z. Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American Statistical Association. 1967; 62(318): 626–633. DOI: 10.1080/01621459.1967.10482935
Open DOI Search in Google Scholar Back to article
Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika. 1988; 75(4): 800–802. DOI: 10.1093/biomet/75.4.800
Open DOI Search in Google Scholar Back to article
Hommel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika. 1988; 75(2): 383–386. DOI: 10.1093/biomet/75.2.383
Open DOI Search in Google Scholar Back to article
Holm S. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics. 1979; 65–70.
Search in Google Scholar Back to article
Benjamini Y, Hochberg Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological). 1995; 57(1): 289–300. DOI: 10.1111/j.2517-6161.1995.tb02031.x
Open DOI Search in Google Scholar Back to article
Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics. 2001; 29(4): 1165–1188. DOI: 10.1214/aos/1013699998
Open DOI Search in Google Scholar Back to article
Benjamini Y, Krieger AM, Yekutieli D. Adaptive linear step-up procedures that control the false discovery rate. Biometrika. 2006; 93(3): 491–507. DOI: 10.1093/biomet/93.3.491
Open DOI Search in Google Scholar Back to article
Benjamini Y, Hochberg Y. On the adaptive control of the false discovery rate in multiple testing with independent statistics. Journal of educational and Behavioral Statistics. 2000; 25(1): 60–83. DOI: 10.3102/10769986025001060
Open DOI Search in Google Scholar Back to article
Bretz F, Hothorn T, Westfall P. Multiple Comparisons Using R. Chapman and Hall/CRC; 2016. DOI: 10.1201/9781420010909
Open DOI Search in Google Scholar Back to article
Farcomeni A. A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Statistical Methods in Medical Research. 2008; 17(4): 347–388. DOI: 10.1177/0962280206079046
Open DOI Search in Google Scholar Back to article
Lee S, Lee DK. What is the proper way to apply the multiple comparison test? Korean Journal of Anesthesiology. 2018; 71(5): 353–360. DOI: 10.4097/kja.d.18.00242
Open DOI Search in Google Scholar Back to article
Tamhane AC, Gou J. Advances in p-Value Based Multiple Test Procedures. Journal of Biopharmaceutical Statistics. 2018; 28(1): 10–27. DOI: 10.1080/10543406.2017.1378666
Open DOI Search in Google Scholar Back to article
Wang D, Li Y, Wang X, Liu X, Fu B, Lin Y, Larsen L, Offen W. Overview of multiple testing methodology and recent development in clinical trials. Contemporary Clinical Trials. 2015; 45: 13–20. DOI: 10.1016/j.cct.2015.07.014
Open DOI Search in Google Scholar Back to article
Armstrong RA. When to use the Bonferroni correction. Ophthalmic and Physiological Optics. 2014; 34(5): 502–508. DOI: 10.1111/opo.12131
Open DOI Search in Google Scholar Back to article
Gyorffy B, Gyorffy A, Tulassay Z. The problem of multiple testing and its solutions for genom-wide studies. 2005; 146(12): 559–563.
Search in Google Scholar Back to article
Lakens D. The 20% Statistician: Error Control in Exploratory ANOVA’s: The How and the Why; 2016. URL http://daniellakens.blogspot.com/2016/01/error-control-in-exploratory-anovas-how.html. [Online; accessed 17-July-2020].
Search in Google Scholar Back to article
Lakens D. The 20% Statistician: Why you don’t need to adjust your alpha level for all tests you’ll do in your lifetime; 2016. URL http://daniellakens.blogspot.com/2016/02/why-you-dont-need-to-adjust-you-alpha.html. [Online; accessed 17-July-2020].
Search in Google Scholar Back to article
Noble WS. How does multiple testing correction work? Nature biotechnology. 2009; 27(12): 1135–1137. DOI: 10.1038/nbt1209-1135
Open DOI Search in Google Scholar Back to article
Streiner DL. Best (but oft-forgotten) practices: the multiple problems of multiplicity—whether and how to correct for many statistical tests. The American Journal of Clinical Nutrition. 2015; 102(4): 721–728. DOI: 10.3945/ajcn.115.113548
Open DOI Search in Google Scholar Back to article
Bennett CM, Miller, MB, Wolford GL. Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction. Neuroimage. 2009; 47(Suppl 1): S125. DOI: 10.1016/S1053-8119(09)71202-9
Open DOI Search in Google Scholar Back to article
Austin PC, Mamdani MM, Juurlink DN, Hux JE. Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. Journal of clinical epidemiology. 2006; 59(9): 964–969. DOI: 10.1016/j.jclinepi.2006.01.012
Open DOI Search in Google Scholar Back to article
Lesack K, Naugler C. An open-source software program for performing Bonferroni and related corrections for multiple comparisons. Journal of pathology informatics. 2011; 2. DOI: 10.4103/2153-3539.91130
Open DOI Search in Google Scholar Back to article
MATLAB. 9.7.0.1190202 (R2019b). The MathWorks Inc.; 2018.
Search in Google Scholar Back to article
R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2020. URL https://www.R-project.org/.
Search in Google Scholar Back to article
Van Rossum G, Drake FL. Python 3 Reference Manual. CreateSpace; 2009.
Search in Google Scholar Back to article
IBM Corp. IBM SPSS Statistics for Windows; 2017. URL https://hadoop.apache.org.
Search in Google Scholar Back to article
Microsoft Corporation. Microsoft Excel; 2018. URL https://office.microsoft.com/excel.
Search in Google Scholar Back to article
Baker M. Scientific computing: Code alert. Nature. 2017; 541(7638): 563–565. DOI: 10.1038/nj7638-563a
Open DOI Search in Google Scholar Back to article
JASP Team. JASP (Version 0.13.1)[Computer software]; 2020. URL https://jasp-stats.org/.
Search in Google Scholar Back to article
American Psychological Association. Publication manual of the American Psychological Association. American Psychological Association; 2020. DOI: 10.1037/0000165-000
Open DOI Search in Google Scholar Back to article
McKinney W, others. Data structures for statistical computing in python. In: Proceedings of the 9th Python in Science Conference. Austin, TX; 2010. DOI: 10.25080/Majora-92bf1922-00a
Open DOI Search in Google Scholar Back to article
Oliphant TE. A guide to NumPy. Trelgol Publishing USA; 2006.
Search in Google Scholar Back to article
Seabold S, Perktold J. Statsmodels: Econometric and statistical modeling with python. In: 9th Python in Science Conference; 2010. DOI: 10.25080/Majora-92bf1922-011
Open DOI Search in Google Scholar Back to article
Van Rossum G. The Python Library Reference, release 3.8.2. Python Software Foundation; 2020.
Search in Google Scholar Back to article
n.d. Python-docx. URL https://python-docx.readthedocs.io/en/latest/#. [Online; accessed 13-August-2020].
Search in Google Scholar Back to article
Gazoni E, Clark. openpyxl – A Python library to read/write Excel 2010 xlsx/xlsm files, version 3.0.4; 2020. URL https://openpyxl.readthedocs.io/en/stable/index.html. [Online; accessed 13-August-2020].
Search in Google Scholar Back to article
Reitz K. requests: Python HTTP for Humans; 2020. URL http://python-requests.org/.
Search in Google Scholar Back to article
Cortesi D. PyInstaller Manual, Release 4.0; 2020. URL https://pyinstaller.readthedocs.io/en/stable/.
Search in Google Scholar Back to article
NSIS Team. Nullsoft Scriptable Install System (NSIS), version 3.06.1; 2020. URL http://nsis.sourceforge.net/.
Search in Google Scholar Back to article

Articles in this issue

DOI: https://doi.org/10.5334/jors.350 | Journal eISSN: 2049-9647

Journal RSS Feed

Language: English

Submitted on: Sep 29, 2020

Accepted on: Mar 3, 2022

Published on: Mar 17, 2022

Published by: Ubiquity Press

In partnership with: Paradigm Publishing Services

Publication frequency: 1 issue per year

Keywords:

Multiple tests adjustment,

p value,

© 2022 Nikolay Petrov, Vasil Atanasov, Trevor Thompson, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 10 (2022): Issue 1