Undermind - AI-Powered Scientific Research Assistant

Research topic

I want to find empirical research quantifying the prevalence of the statistical error where people infer significance from a nonsignificant result in comparison to a significant one, often termed as the "difference between significant and nonsignificant is not itself significant" error, across various scientific disciplines or within specific types of studies.

Report

Empirical studies quantifying the prevalence of the error where significance is inferred from a nonsignificant result alongside a significant one are scarce, but some studies highlight similar statistical misinterpretations in specific disciplines such as animal cognition [1], urology [2], and psychology [3].

Detailed Summary:

Empirical Quantification of the Error:
- The specific error of interpreting the difference between significant and nonsignificant results as itself significant is not directly quantified across multiple disciplines in the studies retrieved. However, related statistical misinterpretations are analyzed within specific fields.

Field-Specific Studies:
- Animal Cognition: Research [1] quantifies how often non-significant results are reported as no effect, highlighting a lack of proper interpretation of statistical results.
- Urology: Study [2] examines how often abstracts misuse the term “trend toward significance” as a misinterpretation near p-value thresholds.
- Psychology: Literature [3, 6] reports frequent misinterpretations of nonsignificant results as accepting the null hypothesis and inadequate evidential support for such claims through Bayes factors.
General Discussions and Concerns:
- Articles such as [4] and [10] discuss the broader implications of statistical misinterpretations concerning significance, stressing the need for statistical literacy and correct method application.
- General calls for re-evaluating the use of p-values and significance thresholds due to their contribution to misinterpretations and replication issues in research [7, 9].

Overall, while studies directly quantifying the prevalence of the specific error across disciplines are lacking, similar issues with statistical significance are documented in specialized analyses within individual fields.

Empirical Quantification of the Specific Statistical Error

Papers that explicitly empirically quantify the prevalence of the statistical error or closely related errors, such as misinterpretation of nonsignificant results as significant.
References: [1], [3], [5], [6], [9]
Details:
- [1] examines misinterpretation of nonsignificant p-values in animal cognition studies.
- [3] investigates misinterpretation of nonsignificant results in Chinese psychological research.
- [5] looks at terms like "marginally significant" in interpretation of results in personality and social psychology studies.
- [6] analyzes nonsignificant results' evidential impact in psychology.
- [9] investigates statistical significance misuses in public health research.

General Discussion on the Statistical Error

Papers that discuss the theoretical aspects of the statistical error without necessarily providing empirical quantification data.
References: [4], [10]
Details:
- [4] discusses the conceptual problem of inferring significance from nonsignificant results.
- [10] critiques the overreliance on 'statistical significance' as a decision criterion.

Statistical Misinterpretations Related to P-Values

Papers focusing broadly on common statistical misinterpretations related to p-values, including descriptions of “trends toward significance.”
References: [2], [7]
Details:
- [2] analyzes use of "trend toward significance" in urology.
- [7] reviews the impact of dichotomizing p-values into significance categories on research reproducibility.

Methodological or Policy-Oriented Suggestions

Papers proposing methodological or policy changes to address broader issues in statistical interpretation.
References: [8], [10]
Details:
- [8] compares correct vs. incorrect methods for interpreting statistical effects.
- [10] proposes retirement of the statistical significance threshold.

This categorization highlights papers of highest relevance to the researcher's interest in empirical quantification of a specific statistical error, with additional context on theoretical discussions and broader contexts or solutions offered.

References

Last 5 years

Last 2 years

> 1 citation per year

> 5 citations per year

Topic Match

Cit./Year

Year

Paper

Paper Relevance Summary

98.7%

2.1

2022

[1] Reporting and interpreting non-significant results in animal cognition research B. Farrar, ..., and L. Ostojić PeerJ 2022 - 6 citations - Show abstract - Cite 98.7% topic match

98.4%

0.0

2023

[2] kJkIPEWqNQYsAohDAF Xu AWuuUpnItJId SiWcDlM seKPqXh wbmw zPD gJhjLZP AcoddjDOh Rr jVc hbStcYLA YkeYsWdqet sHxQgq XqWkqn LvDC kxr DgPCzCV LPJaEPpr Cureus 2023 - 0 citations - Show abstract - Cite 98.4% topic match

Unlock with Undermind Pro

90.8%

0.0

2021

[3] Interpreting nonsignificant results: A quantitative investigation based on 500 Chinese psychological research Jun Wang, ..., and Chuan-Peng Hu Advances in Psychological Science 2021 - 0 citations - Show abstract - Cite - PDF 90.8% topic match

86.1%

49.7

2006

[4] HiQ UAZZNMWMme oQruXrR pQBeIEUWbVCqj oIO FcDS uWyKcQpktaLu MP ief yaxwLh xcleCmhPpTNqg VBNZbFOwPzh Hb TBYrVL Cvo NQ ktiCK The American Statistician 2006 - 901 citations - Show abstract - Cite 86.1% topic match

Unlock with Undermind Pro

83.2%

None

[5] Digital Commons @ East Tennessee State University Digital Commons @ East Tennessee State University Investigating the Misrepresentation of Statistical Significance in Investigating the Misrepresentation of Statistical Significance in Empirical Articles Empirical Articles No author found Journal Not Provided None - 0 citations - Show abstract - Cite 83.2% topic match

75.6%

12.3

2018

[6] Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation B. Aczel, ..., and E. Wagenmakers Advances in Methods and Practices in Psychological Science 2018 - 79 citations - Show abstract - Cite - PDF 75.6% topic match

75.0%

34.5

2017

[7] The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research Valentin Amrhein, ..., and Tobias Roth PeerJ 2017 - 256 citations - Show abstract - Cite - PDF Abstract: The widespread use of ‘statistical significance’ as a license for making a claim of a scientific finding leads to considerable distortion of the scientific process (according to the American Statistical Association). We review why degrading p-values into ‘significant’ and ‘nonsignificant’ contributes to making studies irreproducible, or to making them seem irreproducible. A major problem is that we tend to take small p-values at face value, but mistrust results with larger p-values. In either case, p-values tell little about reliability of research, because they are hardly replicable even if an alternative hypothesis is true. Also significance (p ≤ 0.05) is hardly replicable: at a good statistical power of 80%, two studies will be ‘conflicting’, meaning that one is significant and the other is not, in one third of the cases if there is a true effect. A replication can therefore not be interpreted as having failed only because it is nonsignificant. Many apparent replication failures may thus reflect faulty judgment based on significance thresholds rather than a crisis of unreplicable research. Reliable conclusions on replicability and practical importance of a finding can only be drawn using cumulative evidence from multiple independent studies. However, applying significance thresholds makes cumulative knowledge unreliable. One reason is that with anything but ideal statistical power, significant effect sizes will be biased upwards. Interpreting inflated significant results while ignoring nonsignificant results will thus lead to wrong conclusions. But current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings. Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that ‘there is no effect’. Information on possible true effect sizes that are compatible with the data must be obtained from the point estimate, e.g., from a sample average, and from the interval estimate, such as a confidence interval. We review how confusion about interpretation of larger p-values can be traced back to historical disputes among the founders of modern statistics. We further discuss potential arguments against removing significance thresholds, for example that decision rules should rather be more stringent, that sample sizes could decrease, or that p-values should better be completely abandoned. We conclude that whatever method of statistical inference we use, dichotomous threshold thinking must give way to non-automated informed judgment. 75.0% topic match

63.0%

0.0

2016

[8] Supplemental Text S 1 M. Siwiak, ..., and P. Zielenkiewicz Journal Not Provided 2016 - 0 citations - Show abstract - Cite 63.0% topic match

62.1%

0.0

2023

[9] Statistical significance abuses in public health research: a retrospective investigation of recent trends and possible solutions A. Rovetta medRxiv 2023 - 0 citations - Show abstract - Cite - PDF 62.1% topic match

51.4%

44.6

2019

[10] MPazSE lJCqgAzryqV plTPBpTKRRYw PhIUmnfo fBpkhkhh aRyE HLC hG AjbCWVQ Journal Not Provided 2019 - 265 citations - Show abstract - Cite 51.4% topic match

Unlock with Undermind Pro

50.7%

0.2

1983

[11] Statistics for diagnostic procedures. II. The significance of "no significance": what a negative statistical test really means. C. Phillips, ..., and G. Blasczcynski AJR. American journal of roentgenology 1983 - 10 citations - Show abstract - Cite 50.7% topic match

49.4%

0.3

2018

[12] On Statistical Non-Significance Alberto Abadie arXiv: Other Statistics 2018 - 2 citations - Show abstract - Cite - PDF 49.4% topic match

46.5%

7.4

2006

[13] Impact of Criticism of Null‐Hypothesis Significance Testing on Statistical Reporting Practices in Conservation Biology F. Fidler, ..., and N. Thomason Conservation Biology 2006 - 135 citations - Show abstract - Cite 46.5% topic match

45.2%

0.3

1987

[14] 'No statistically significant difference'. So what? W. Bourne Archives of ophthalmology 1987 - 12 citations - Show abstract - Cite 45.2% topic match

38.7%

0.0

2016

[15] Are controversies in science driven by randomness and misbeliefs M. Siwiak, ..., and P. Zielenkiewicz https://doi.org/10.7287/PEERJ.PREPRINTS.2249V1 2016 - 0 citations - Show abstract - Cite 38.7% topic match

32.8%

0.0

2017

[16] Interpreting nonsignificant findings in psychological research B. Pálfi, ..., and E. Wagenmakers Cognitive Science 2017 - 0 citations - Show abstract - Cite 32.8% topic match

32.2%

0.4

2019

[17] Reducing our dependence on null hypothesis testing: A key to enhance the reproducibility and credibility of our science K. Murphy SA Journal of Industrial Psychology 2019 - 2 citations - Show abstract - Cite - PDF 32.2% topic match

27.8%

0.0

2005

[18] Special Section NEEDED : A BAN ON THE SIGNIFICANCE TEST J. Hunter Journal Not Provided 2005 - 0 citations - Show abstract - Cite Abstract: The significance test as currently used is a disaster. Whereas most researchers falsely believe thai the significance test has an error rale of 5%, empirical studies show the average error rale across psychology is 60%—)2 times higher than researchers think it to he. The error rate for inference using the significance test is greater than the error rate using a coin toss to replace the empirical study. The significance test has devastated the research review process. Comprehensive reviews cite conflicting results on almost every issue. Yet quantitatively accurate review of the same results shows that the apparent conflicts stem almost entirely from the high error rate for the significance test. If 60% of studies falsely interpret their primary results, then reviewers who base their reviews on the interpreted study ' findings'' will have a 100% error rate in concluding that there is conflict between study results. Consider a parable; A new young science has addressed significant problems and has attracted a large number of bright, tireless, empirical scientists. There is only one problem. This new science uses a defective decision-making technique that has been shown to have a 60% error rale. Worse yet, people in that science falsely believe that the error rate is 5%. Thus, people take the results of the decision-making device to be almost always right even though it is actually wrong tnore often than it is right. The question is this; Do you expect a high or a low rate of progress in this new science? Before you answer, consider this ugly fact. Suppose two researchers in this new science are testing a new hypothesis. One does an empirical study and the other flips a coin. The person who does the study has a 60% chance of error and thus has a 40% probability of being right. The person who flips a coin has a 50% probability of being right. That is, the person who flips a coin will be right more often than the person who does a study. My personal prediction is that progress in this new science will come at a glacial rate. 27.8% topic match

21.4%

0.0

2018

[19] Science, Medicine, and the Anesthesiologist N. Engl Anesthesiology 2018 - 0 citations - Show abstract - Cite - PDF 21.4% topic match

19.7%

3.6

2011

[20] How to Statistically Show the Absence of an Effect E. Quertemont Psychologica Belgica 2011 - 48 citations - Show abstract - Cite - PDF 19.7% topic match

19.2%

0.2

1979

[21] The importance of significance and the significance of importance. R. Riegelman Postgraduate medicine 1979 - 8 citations - Show abstract - Cite 19.2% topic match

17.1%

11.1

1997

[22] Needed: A Ban on the Significance Test J. Hunter Psychological Science 1997 - 310 citations - Show abstract - Cite 17.1% topic match

12.7%

0.2

2018

[23] Concise, Simple, and Not Wrong: In Search of a Short-Hand Interpretation of Statistical Significance J. Spence and David Stanley Frontiers in Psychology 2018 - 1 citations - Show abstract - Cite 12.7% topic match

10.7%

0.0

2005

[24] Frequent mistakes in the statistical inference of biomedical data. M. Martínez‐Sellés, ..., and Inmaculada Herranz Italian heart journal : official journal of the Italian Federation of Cardiology 2005 - 0 citations - Show abstract - Cite 10.7% topic match

9.9%

21.1

2016

[25] Marginally Significant Effects as Evidence for Hypotheses Laura Pritschet, ..., and Z. Horne Psychological Science 2016 - 181 citations - Show abstract - Cite 9.9% topic match

8.1%

0.0

2018

[26] Effect of loading dose of atorvastatin prior to planned percutaneous coronary inter- vention on major adverse cardiovascular events in acute coronary syndrome: The SECURE-PCI randomized clinical trial. JAMA 2018; 319:1331–40 No author found Journal Not Provided 2018 - 0 citations - Show abstract - Cite 8.1% topic match

6.8%

11.6

2017

[27] Some Natural Solutions to the p-Value Communication Problem—and Why They Won’t Work A. Gelman and J. Carlin Journal of the American Statistical Association 2017 - 86 citations - Show abstract - Cite Abstract: It is well known that even experienced scientists routinely misinterpret p-values in all sorts of ways, including confusion of statistical and practical significance, treating nonrejection as acceptance of the null hypothesis, and interpreting the p-value as some sort of replication probability or as the posterior probability that the null hypothesis is true. A common conceptual error is that researchers take the rejection of a straw-man null as evidence in favor of their preferred alternative (Gelman 2014). A standard mode of operation goes like this: p < 0.05 is taken as strong evidence against the null hypothesis, p > 0.15 is taken as evidence in favor of the null, and p near 0.10 is taken either as weak evidence for an effect or as evidence of a weak effect. Unfortunately, none of those inferences is generally appropriate: a low p-value is not necessarily strong evidence against the null (see, e.g., Morris 1987; Gelman and Carlin 2014), a high p-value does not necessarily favor the null (the strength and even the direction of the evidence depends on the alternative hypotheses), and p-values are in general notmeasures of the size of any underlying effect. But these errors persist, reflecting (a) inherent difficulties in the mathematics and logic of p-values, and (b) the desire of researchers to draw strong conclusions from their data. Continued evidence of these and other misconceptions and their dire consequences for science (the “replication crisis” in psychology, biology, and other applied fields), especially in light of new understanding of how common it is that abundant “researcher degrees of freedom” (Simmons, Nelson, and Simonsohn 2011) and “gardens of forking paths” (Gelman and Loken 2014) allow researchers to routinely obtain statistically significant and publishable results from noise, motivated the American Statistical Association to release a Statement on Statistical Significance and p-values in an attempt to highlight the magnitude and importance of problems with current standard practice (Wasserstein and Lazar 2016). At this point, it would be natural for statisticians to think that this is a problem of education and communication. If we could just add a few more paragraphs to the relevant sections of our textbooks, and persuade applied practitioners to consult more with statisticians, then all would be well, or so goes this logic. 6.8% topic match

6.7%

0.0

2024

[28] Statistical inference on partially shape-constrained function-on-scalar linear regression models Kyunghee Han, ..., and Soo-Young Kim Journal Not Provided 2024 - 0 citations - Show abstract - Cite - PDF 6.7% topic match

5.5%

5.4

2019

[29] Appreciating the Significance of Non-significant Findings in Psychology D. Mehler, ..., and K. Matić Journal of European Psychology Students 2019 - 29 citations - Show abstract - Cite - PDF 5.5% topic match

4.7%

0.0

1974

[30] Casework as a Causative Agent in Client Deterioration: A Research Note on the Fischer Assessment W. Hudson Social Service Review 1974 - 1 citations - Show abstract - Cite 4.7% topic match

4.2%

10.7

1993

[31] The Case Against Statistical Significance Testing, Revisited R. P. Carver Journal of Experimental Education 1993 - 338 citations - Show abstract - Cite 4.2% topic match

3.8%

15.9

1959

[32] Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa T. Sterling Journal of the American Statistical Association 1959 - 1044 citations - Show abstract - Cite 3.8% topic match

2.6%

1.5

1994

[33] The concept of statistical significance testing B. Thompson https://doi.org/10.7275/3GSD-TK11 1994 - 45 citations - Show abstract - Cite 2.6% topic match

2.3%

0.2

2018

[34] [The uncertainties of statistical "significance"]. Raúl J Domenech Revista medica de Chile 2018 - 1 citations - Show abstract - Cite - PDF 2.3% topic match

2.2%

1.6

2018

[35] Pitfalls of significance testing and $p$-value variability: An econometrics perspective N. Hirschauer, ..., and C. Becker Statistics Surveys 2018 - 11 citations - Show abstract - Cite 2.2% topic match

1.5%

0.5

1993

[36] The Role of Statistics in Research W. Asher Journal of Experimental Education 1993 - 16 citations - Show abstract - Cite 1.5% topic match

1.4%

5.2

2006

[37] Probability as certainty: Dichotomous thinking and the misuse ofp values R. Hoekstra, ..., and Addie Johnson Psychonomic Bulletin & Review 2006 - 93 citations - Show abstract - Cite - PDF 1.4% topic match

1.1%

0.2

2001

[38] Does statistical language constitute a "significant" roadblock to readers' interpretations of research results? D. Robinson, ..., and Duane Halbur-Ramseyer Journal of Educational Psychology 2001 - 4 citations - Show abstract - Cite 1.1% topic match

1.0%

3.6

2017

[39] Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence Jae H. Kim, ..., and Kamran Ahmed Financial Accounting eJournal 2017 - 26 citations - Show abstract - Cite - PDF 1.0% topic match

1.0%

0.0

2021

[40] Detecting faking responses during empirical research: a study in a developing country environment G. Tetteh, ..., and A. Kwarteng International Journal of Lean Six Sigma 2021 - 0 citations - Show abstract - Cite Abstract: Purpose Several research studies on Lean Six Sigma (LSS) have been done using the survey methodology. However, the use of surveys often relies on the measurement of variables, which cannot be directly observed, with attendant measurement errors. The purpose of this study is to develop a methodological framework consisting of a combination of four tools for identifying and assessing measurement error during survey research. Design/methodology/approach This paper evaluated the viability of the framework through an experimental study on the assessment of project management success in a developing country environment. The research design combined a control group, pretest and post-test measurements with structural equation modeling that enabled the assessment of differences between honest and fake survey responses. This paper tested for common method variance (CMV) using the chi-square test for the difference between unconstrained and fully constrained models. Findings The CMV results confirmed that there was significant shared variance among the different measures allowing us to distinguish between trait and faking responses and ascertain how much of the observed process measurement is because of measurement system variation as opposed to variation arising from the study’s constructs. Research limitations/implications The study was conducted in one country, and hence, the results may not be generalizable. Originality/value Measurement error during survey research, if not properly addressed, can lead to incorrect conclusions that can harm theory development. It can also lead to inappropriate recommendations for practicing managers. This study provides findings from a framework developed and assessed in a LSS project environment for identifying faking responses. This paper provides a robust framework consisting of four tools that provide guidelines on distinguishing between fake and trait responses. This tool should be of great value to researchers. 1.0% topic match

0.7%

63.6

2013

[41] Revised standards for statistical evidence V. Johnson Proceedings of the National Academy of Sciences 2013 - 705 citations - Show abstract - Cite - PDF 0.7% topic match

0.7%

29.3

2016

[42] When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment Dénes Szűcs and J. Ioannidis Frontiers in Human Neuroscience 2016 - 234 citations - Show abstract - Cite - PDF 0.7% topic match

0.0%

0.2

2019

[43] Use of significance test logic by scientists in a novel reasoning task R. Morey and R. Hoekstra https://doi.org/10.31234/osf.io/sxj3r 2019 - 1 citations - Show abstract - Cite 0.0% topic match

0.0%

2.8

2010

[44] All evidence is equal: the flaw in statistical reasoning S. Gorard Oxford Review of Education 2010 - 42 citations - Show abstract - Cite - PDF 0.0% topic match

Research topic

Report

References

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations