Conflating relevance with practical significance and other issues: Commentary on Sen,Smith,and Van Note’s “statistical significance versus practical importance in information systems research”

Abstract

Expanding on the current debate on the issues of statistical and practical significance in information systems research, where the article by Sen, Smith, and Van Note is a recent contribution, this commentary cautions against conflating relevance with practical significance. We emphasize that relevance is 1) about the real-world usefulness of research findings rather than their impressiveness for the researcher audience, 2) an essential quality of research spanning beyond its findings and not merely limited to statistical studies, and 3) determined by nonacademics rather than academics. We also comment on other aspects of the article by Sen et al., such as the term “practical importance,” the treatment of effect size measures, and the presentation of “marginal effects.”

Keywords

Information systems research statistical significance practical significance relevance effect size theory testing

Sen, Smith, and Van Note (2021) (henceforth, SSVN) pursue the admirable goal of communicating the message to information systems (IS) researchers that practical significance (which SSVN call practical importance) deserves recognition apart from statistical significance and is important in its own right. This same message was earlier communicated to the audience of IS researchers by Mohajeri, Mesgari, and Lee (2020) (henceforth, MML) and mentioned by Mertens and Recker (2020) (henceforth, MR). Scholars across numerous areas of scholarship have formed a broad intellectual movement focused on questioning the value of statistical significance in the way that this concept has traditionally been (mis)used. We welcome SSVN to this movement and we laud their efforts to further communicate the message to the particular audience of IS researchers.

We wish to comment on some issues raised by SSVN’s treatment of practical significance. First is the issue of distinguishing relevance and practical significance; second, there is the issue of what the term, “practical importance,” as used by SSVN, means; third, we address SSVN’s treatment of “practical importance” and effect size measures; fourth, we comment on how SSVN present “marginal effects”; fifth, there is the matter of differences between MML and SSVN in how they judge the coverage of practical significance by certain MIS Quarterly articles; and finally, sixth, we correct SSVN’s misstatement about MML’s hypothetical example pertaining to relevance.

Distinguishing relevance from practical significance

As important as practical significance is, it is different from relevance. MML take the position that relevance and practical significance have been conflated and need to be distinguished from each other, and that relevance is a research quality not only separate from, but also equal in importance to practical significance. Indeed, one of the major points in MML’s article is that practical significance cannot be properly understood unless it is distinguished from relevance. SSVN, however, do not acknowledge any distinction between relevance and practical significance. MML’s treatment of practical significance may not be properly understood or discussed unless it is distinguished from relevance.

In this section, we first review how MML characterize and distinguish the concepts of practical significance and relevance. After that, we comment on how SSVN proceed without acknowledging any distinction between the two concepts.

Practical significance

SSVN (p. 5) correctly attribute to MML the words “research impressiveness of statistical results” in referring to what MML mean by “practical significance,” but in MML’s usage, these words make up but a parenthetical phrase, not a definition as SSVN claim. In attempting to convey what MML mean by “practical significance,” SSVN omit the rich context that MML provide for the term “research impressiveness” (p. 528):

Historically speaking, along with placing emphasis on the statistical discernibility [significance] of results, statisticians were also aware of another demanding issue of a different nature: the research impressiveness of results. In other words, researchers conducting inferential statistical analysis also had to make sure that the magnitude of their reported results are sufficiently impressive so that they could consider their hypothesized effects/relationships to actually matter in the research area they are concerned about. Kirk (1996) explains that Pearson (1900) and then Fisher (1925) were among the first to react to the issue of practical significance. In particular, Fisher (1925) proposed that researchers supplement the [statistical] significance test in analysis of variance with the correlation ratio or eta, which measures the strength of the association between the independent and dependent variables (Kirk, 1996: 748).

Later reactions to this issue can also be identified in the behavioral and educational literatures (e.g., Cohen, 1977; Daniel, 1977; Kerlinger and Pedhazur, 1973), and the sociological literature (e.g., Selvin, 1957), among others.

Moreover, regarding the mention of the term “research impressiveness” in the above quotation, MML attach a footnote, the text of which is (p. 558):

We acknowledge the fact that “in some cases effect sizes are most impressive by being as small as possible” (Kelley and Preacher, 2012: 148). Therefore, we prefer to use the term impressiveness (as suggested by Kelley and Preacher 2012) in lieu of the term largeness, when referring to the concept of practical significance.

The foregoing quotations provide the rich context for MML’s characterization of “practical significance” as involving the “research impressiveness of statistical results.” Without this context, MML’s characterization may not be properly understood or conveyed.

Relevance

SSVN state (p. 6), “we do not believe that all research must be relevant in the narrowly defined sense used by MML.” As for SSVN’s understanding of what MML mean by relevance, SSVN offer this description: “MML describe their third criterion, relevance, as follows: ‘we consider the relevance of an independent/mediating variable to be concerned with the understandability, and when appropriate, actionability, of the variable in the eyes of nonacademic stakeholders’ (p. 535).” This description, however, omits the context with which MML themselves describe relevance.

Contrary to the impression given by SSVN’s quotation of MML, relevance is not just another statistical or mathematical quality like practical significance or statistical significance, but rather, a quality of research in general, whether statistical/mathematical or not. In MML’s Table 1 (p. 530, which we reproduce below), MML define and characterize research which has relevance as having the quality of “real-world usefulness”—a term absent in SSVN’s article. Table 1 also makes explicit the difference that relevance is determined in the eyes of practitioners or the nonacademic audience while practical significance is determined in the eyes of researchers or the academic audience. MML’s Venn diagram (see MML’s Figure 1, p. 530) indicates that not all practically significant research bears relevance, and vice versa. For example, MML provide an illustration in which a research finding, in the eyes of the researchers, bears statistical significance as well as practical significance but, in the eyes of the practitioners in a firm, lacks relevance (p. 532). MML also point out that “relevance,” as a quality of research in general and not just statistical research, is accompanied by a substantial literature discussing it; as MML indicate, this literature can be found both inside and outside the IS field, where MML offer the following as examples (pp. 528–529): AACSB International, 2008; Dennis, 2001; Desouza et al., 2006; Hambrick, 1994; Hassan, 2014; Koontz, 1961; Robey and Markus, 1998; Rosemann and Vessey, 2008; and all the 1999 MIS Quarterly essays on “rigor versus relevance.”¹ This overall context, including the literature, is missing in SSVN’s discussion.

Table 1.

The key distinctions among statistical significance, practical significance and relevance. (from Mohajeri, Mesgari, and Lee, 2020, Table 1, p. 530).

	Corresponding Quality	Applicability Scope	Primary Adjudicator
Statistical Significance	Statistical Discernibility	Only Quantitative Findings	Researchers/Academic Audience
Practical Significance	Research Impressiveness	Only Quantitative Findings	Researchers/Academic Audience
Relevance	Real-World Usefulness	Research Findings plus Research Models/Constructs/Variables	Practitioners/Nonacademic Audience

Not distinguishing relevance and practical significance

Regarding their study examining 306 papers published in MIS Quarterly, SSVN state (p. 10): “We do not assess relevance.” Documentation of the extent to which relevance, not just practical significance, is or is not recognized and treated in statistically conducted research in 306 statistically conducted MIS Quarterly articles would have been a major contribution. How might the failure to distinguish relevance and practical significance be explained? This result need not be surprising, given that MML anticipated this (p. 529):

[Regarding] the distinction between practical significance and relevance, … there exists a tendency in the literature (e.g., Kelley and Preacher, 2012; Kirk, 1996) to equate the research impressiveness of quantitative results with the real-world usefulness of research or its results, and hence not to make a full distinction between practical significance and relevance. One might speculate this occurs, in part, due to the fact that the relevance of the conceptual aspects (e.g., research models, hypotheses, variables, etc.) of statistically conducted research is often a taken-for-granted quality among statisticians. On this basis, statisticians mostly presume that the only remaining condition to achieve relevance for the entire research is to obtain impressive results.

The text in this quotation of MML is but a way of reminding us of the lessons that research with practical significance can still lack relevance, that research with relevance can lack practical significance and that, therefore, relevance is a research quality that deserves recognition in its own right.

SSVN’s term, “practical importance”

Rather than use the term “practical significance,” SSVN prefer the term “practical importance,” but what SSVN mean by the latter term is not entirely clear. SSVN do state (p. 5): “Economists have been sharply and repeatedly criticized by McCloskey (1985, 1999, 2002) for confusing statistical significance with practical importance (what she calls ‘oomph’).” However, SSVN do not define “oomph,” a term we find problematic. Suppose there is research which the researchers themselves consider to have practical significance, but which practitioners consider to lack any relevance; may this research be considered to have “oomph”? Likewise, suppose there is research which practitioners consider to be relevant, but which the researchers themselves consider to lack practical significance; may this research be considered to have “oomph”? These two questions suggest that who is making the judgment is a determinative factor, where explicitly acknowledging this would lead to MML’s distinction between their concept of relevance (which would be determined in the eyes of the practitioners) and their concept of practical significance (which would be determined in the eyes of the researchers). SSVN, however, do not make this distinction.

Potentially helping to clarify what SSVN mean by “oomph” or “practical importance” is that SSVN put these words in the same company as two other terms (p. 5): “[MML’s] definition of practical significance, however, is very different from what McCloskey calls oomph, MR call the ‘magnitude of an effect,’ the ASA statement calls ‘the size of an effect,’ and we call practical importance.” However, MR and the ASA statement (Wasserstein and Lazar, 2016) neither define nor describe what they mean by “magnitude of an effect” and “the size of an effect.” The term “practical significance” does not even appear in the ASA statement. Moreover, we agree with Grissom and Kim (2005) who have noted that “effect size … is not synonymous with practical significance” (p. 4). Thus, what SSVN mean by “oomph” or “practical importance” remains unclear.

SSVN’s “practical importance” and effect size measures

SSVN’s presentations of effect size and its association with the concept of practical significance (which SSVN call practical importance) are confusing or misleading. We refer to the statements listed in Table 2.

Table 2.

Quotations from SSVN on “practical importance” and effect size.

Quotation No.	Page No.	Quotation Text
I	4	Subjective opinions—not statistical measures like p-values, correlations, or effect sizes—are needed to judge practical importance.
II	4	Although the label effect size suggests a measure of the magnitude of an effect, Cohen did not intend these indexes to gauge whether the explanatory variables have substantial effects on the dependent variable…
III	4	Cohen’s effect size is a measure of a model’s goodness-of-fit, and not a measure of the practical importance of any of the model’s explanatory variables.
IV	4	The effect size does not depend on the units the variables are measured in and tells us nothing about the practical importance of the model’s coefficients.
V	5	As discussed above, Cohen’s effect sizes and other statistics related to goodness-of-fit do not measure practical importance.

Quotation I is confusing. How is it possible to judge “practical importance” without needing to know the research findings reported in the form of statistical measures? Perhaps, SSVN are trying to convey a different meaning here. A possibility is that SSVN are implying what MML (p. 533) already stated by quoting Grissom and Kim (2005: 4): “knowledge of a result’s effect size can [only] inform a judgment about practical significance.” In other words, knowing the magnitude of statistical findings is necessary but not sufficient to make judgments about their practical significance. MML (p. 533) provide this account on the matter:

As for practical significance, interpretation of obtained effect sizes is a matter of human judgment and also relies on convention. “Researchers study varied phenomena using a plethora of paradigms and methodologies” (Cortina and Landis, 2009, p. 306). Consequently, depending on the context and research domain/design, there can certainly be situations where a small effect size can denote a large practical impact and vice versa (e.g., see Hsieh et al., 2012; Lewandowsky and Maybery, 1998; Prentice and Miller, 1992; Rosenthal, 1990; Rosenthal and Rubin, 1979). On this basis, factors such as population and sample size, the particular effect size measures involved, and the nature of the phenomenon in question, are in fact determinants of how the magnitude of effect sizes obtained in a given research study should be interpreted.

In Quotation II, the treatment of the term “effect” is also confusing. SSVN appear to imply that the term “effect” is only valid for referring to the effect of independent variables on dependent variables. The term “effect” can indeed refer to the effect of independent variables on dependent variables, but that is only one form of effect in statistically conducted research. In the literature on effect size, the term “effect” has a more general meaning than what SSVN indicate; the more general meaning is, “a quantitative reflection of a phenomenon” (Kelley and Preacher, 2012: 140). We also agree with Cohen (1988), who broadly describes effect size as meaning “the degree to which the phenomenon is present in the population” or “the degree to which the null hypothesis is false” (pp. 9–10).

Quotations III, IV, and V are also confusing. In those statements, SSVN appear to imply that “practical importance” is a matter definable or assessable only in relation to statistical measures generally known as path/regression coefficients where a model of explanatory variables is involved. Again, we believe that this represents but one possible view of “effect.” Practical significance is a pertinent issue whenever and wherever any measure of effect size (or magnitude), including but not limited to path/regression coefficients, is reported. MML and many other works in the literature (e.g., Grissom and Kim, 2005; Kelley and Preacher, 2012; Kirk, 1996; Vacha-Haase and Thompson, 2004) have already established this point.

SSVN’s presentation of “marginal effects”

SSVN provide an excellent explanation of “marginal effects.” For instance, they follow their statement, “A straightforward way of helping readers assess the practical importance of a model’s coefficients is to report the predicted marginal effects of ceteris paribus changes in the explanatory variables on the model’s dependent variable” (pp. 4–5), with additional discussion and a table showing that the marginal effect for a regression model can be derived by taking the derivative of the equation for the regression model.

We note that the concept of marginal analysis as related to practical significance has been previously covered. MML mention and discuss the utility of marginal analysis throughout their paper. For instance, MML state (p. 543):

With regard to advice to facilitate the task of offering interpretations of magnitude, we wish to draw attention to two particular tactics here. The first tactic deals with utilizing a type of analysis known as “marginal analysis,” whenever possible. … As stated earlier in this essay, through marginal analysis, the sensitivity of a dependent variable to changes in an independent variable may be measured and reported by researchers (see Lee and Mohajeri, 2012; Lin et al., 2013; Vittinghoff et al., 2005). Indeed, in the case of nonlinear models, which are quite common in IS research, marginal analysis is a more robust way—and sometimes the only way—to interpret effect size, compared to looking at the p-value or magnitude of the coefficient (Lin et al., 2013: 909).

Marginal analysis can therefore be instrumental to not only deliver an interpretation of magnitude … , but also to articulate and present a sound rationale … for such an interpretation.

Differences in judging practical significance in MIS Quarterly articles

SSVN compare MML’s analysis of 27 MIS Quarterly articles published in 2015 with their (SSVN’s) own assessment of the same articles. SSVN draw a parallel between MML’s criterion of “Does the paper offer any judgment, with supporting rationale, indicating whether the RES [reported effect sizes] are practically significant?” and their own criterion of “Does the paper discuss whether the signs and magnitudes of the estimated coefficients are plausible?” SSVN disagree with MML’s assessment that two papers (Kankanhalli et al., 2015; Tian and Xu, 2015) partially communicate judgments on practical significance. SSVN state (p. 10):

Tian and Xu (2015) reported Cohen’s F² values, and MML judged this a satisfactory measure of practical significance, but we did not consider F² values to be a measure of practical importance, for reasons explained earlier. MML also gave a yes assessment for Kankanhalli et al. (2015) because the authors gauged measurement invariance for two models by reporting that the difference between the comparative fit index (CFI) was less than 0.01. We gave a no assessment because the authors did not discuss the estimated magnitudes of the effects of the explanatory variables on their dependent variables.

Regarding Tian and Xu (2015) and Kankanhalli et al. (2015), SSVN repeat their position that they do not consider practical significance (which SSVN call practical importance) a matter pertinent to the statistical measures Cohen’s f ² and CFI (and ΔCFI).² As we mentioned previously, SSVN appear to confine the matter of practical significance to explanatory variable coefficients (or what we call path/regression coefficients). That is an incomplete conceptualization of practical significance. On the one hand, we emphasize again that practical significance is concerned with interpretations of magnitude for all statistical measures known as measures of effect size (or magnitude) (see Kirk, 1996; Thompson, 2002; Vacha-Haase and Thompson, 2004). On the other hand, as MML (p. 535) indicate, the literature (e.g., Breaugh, 2003; Kelley and Preacher, 2012; Kirk, 1996) already recognizes that there is “a broad range of statistical measures [known] as effect size measures. This includes path/regression coefficient, correlation coefficient, fit indices, measures of variance explained and so forth.”

Unlike MML’s assessments, SSVN offer positive assessments for four other papers. For the paper, Wang et al. (2015), SSVN provide the following quotation as a basis to justify their positive assessment:

Our results indicated that the business value of an application (BVM) increased the risk of an application experiencing unauthorized attempts ( $\hat{β}$ ₁ = 0.22), supporting H1. One level increase in BVM increased the instantaneous probability (or the hazard rate) of an application experiencing unauthorized attempts by 24 percent on average with everything else being equal. (p. 104)

Our question is, considering SSVN’s own criterion (“Does the paper discuss whether the signs and magnitudes of the estimated coefficients are plausible?”), where does the quotation of Wang et al. (2015) include any discussion of whether the magnitude of the 24 percent increase is plausible? SSVN also do not provide any explanation of how Wang et al. (2015) satisfy their criterion by discussing “the size of the coefficients in terms that can be assessed by readers” (SSVN, p. 10). The same type of argument and question apply to the three other articles (Faraj et al., 2015; Rai et al., 2015; Ramasubbu et al., 2015), for each of which SSVN only provide a single quotation with no further explanation.

SSVN on MML’s hypothetical example of relevance

SSVN take issue with MML’s interpretation of relevance in a hypothetical example that MML provide. SSVN state (p. 5):

They give a hypothetical example involving a multiple regression equation in which the dependent variable is an individual’s proficiency in the use of a firm’s business intelligence (BI) program and the explanatory variables are the person’s age and gender. They argue that the coefficients of these two explanatory variables are not relevant because, the firm does not consider the factors, indicated by the independent variables, to be actionable. Specifically, the firm does not consider it feasible to change an employee’s age or gender, or consider it ethical to select an employee based on age or gender. (p. 532)

SSVN then argue with this hypothetical example by offering ways in which the firm can nonetheless act on age and gender. However, a complete reading of MML reveals that MML already anticipated and accounted for this; they state in footnote 10 (p. 532):

It is possible that a different firm could interpret the results differently. For instance, a different firm might use the results to institute a training program, targeting participants by age and gender, with the goal of enhancing their individual BI usage proficiency. Of course, the point remains that relevance, however construed, is a different matter from practical significance.

Conclusion

In concluding this commentary, we make three points.

First, we emphasize that we commend SSVN on their efforts to further communicate the message—that practical significance is important—to the particular audience of IS researchers.

Second, we sound a note of caution with regard to the mathematical rigor with which SSVN pursue their conception of practical importance. As IS researchers have long been aware, the pursuit of increased rigor can result (even if unwittingly) in decreased relevance. Of course, this is not a reason to refrain from pursuing increased rigor in practical significance (e.g., in the form of using derivatives in efforts to measure marginal effects and elasticities), but rather a reason to, in addition, pursue relevance by documenting it alongside practical significance and statistical significance. For a discipline such as IS which is not only academically oriented but also practice oriented, it is not enough for research to achieve rigor in its methods (such as the rigor associated with statistical significance and practical significance); the research needs no less to achieve relevance, or “real-world usefulness.”

Third, even if a statistical analysis is completed so that all the concerns of MML and SSVN are addressed, there remains the matter of theory testing. For instance, consider a theory operationalized as a statistical equation in a regression analysis or PLS study. Lee and Hubona (2009) and Lee, Mohajeri, and Hubona (2017) explain how the statistical equation can be used to make predictions which are judged to either succeed or fail, where the overall record of successes or failures can then be used to confirm or refute the theory. Until this testing is done, the statistical analysis just engages in theory fitting, that is, the matter of how well the theory fits the data, rather than the matter of whether the data confirm or refute the theory. Where the validity of the theory itself remains to be determined, any differences between MML and SSVN amount to nothing more than a tempest in a teapot.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Kaveh Mohajeri

Allen S Lee

Notes

References

AACSB International (2008) Impact of Research. Tampa, FL: The Association to Advance Collegiate Schools of Business.

Breaugh

(2003) Effect size estimation: factors to consider and mistakes to avoid. Journal of Management 29(1): 79–97. DOI:10.1016/S0149-2063(02)00221-0.

Cheung

Rensvold

(2002) Evaluating goodness-of-fit indexes for testing measurement invariance. Structural equation modeling: a multidisciplinary journal 9(2): 233–255.

Cohen

(1977) Statistical Power Analysis for the Behavioral Sciences. Revised, ed. New York, NY: Academic Press.

Cohen

(1988) Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NY: Lawrence Erlbaum Associates.

Cortina

Landis

(2009) When Small Effect Sizes Tell a Big Story, and When Large Effect Sizes Don't, in Statistical and Methodological Myths and Urban Legends: Doctrine, Verity and Fable in the Organizational and Social Sciences. CE Lance and RJ Vandenberg (eds.). New York: Routledge, pp. 287–308.

Daniel

(1977) Statistical significance versus practical significance. Science Education 61(3): 423–427.

Dennis

(2001) Relevance in information systems research. Communications of the Association for Information Systems 6. Article 10.

Desouza

Sawy

Galliers

, et al. (2006) Beyond rigor and relevance towards responsibility and reverberation: information systems research that really matters. Communications of the Association for Information Systems 17. Article 16.

10.

Faraj

Kudaravalli

Wasko

(2015) Leading collaboration in online communities. Management Information Systems Quarterly 39(2): 393–412.

11.

Fisher

(1925) Statistical Methods for Research Workers. London: Oliver & Boyd.

12.

Grissom

Kim

(2005) Effect Sizes for Research: A Broad Practical Approach. Mahwah, NJ: Erlbaum.

13.

Hambrick

(1994) What if the academy actually mattered? Academy of Management Review 19(1): 11–16.

14.

Hassan

(2014) Value of IS research: is there a crisis?. Communications of the Association for Information Systems 34. Article 41.

15.

Hsieh

JP-A

Rai

Petter

, et al. (2012) Impact of user satisfaction with mandated CRM use on employee service quality. Management Information Systems Quarterly 36(4): 1065–1080.

16.

Kankanhalli

Teo

(2015) Comparing potential and actual innovators: an empirical study of mobile data services innovation. Management Information Systems Quarterly 39(3): 667–682.

17.

Kelley

Preacher

(2012) On effect size. Psychological Methods 17(2): 137–152. DOI:10.1037/a0028086.

18.

Kerlinger

Pedhazur

(1973) Multiple Regression in Behavioral Research. New York: Holt, Rinehart and Winston.

19.

Kirk

(1996) Practical significance: a concept whose time has come. Educational and Psychological Measurement 56(5): 746–759. DOI:10.1177/0013164496056005002.

20.

Koontz

(1961) The management theory jungle. Academy of Management Journal 4(3): 174–188.

21.

Lee

Mohajeri

Hubona

(2017) Three roles for statistical significance and the validity frontier in theory testing. In: The 50th Hawaii International Conference on System Sciences, pp. 5737–5746.

22.

Lee

Hubona

(2009) A scientific basis for rigor in information systems research. Management Information Systems Quarterly 33(2): 237–262.

23.

Lee

Mohajeri

(2012) Linking relevance to practical significance. In: The 45th Hawaii International Conference on System Sciences, pp. 5234–5240. DOI: 10.1109/HICSS.2012.416.

24.

Lewandowsky

Maybery

(1998) The critics rebutted: a pyrrhic victory. Behavioral and Brain Sciences 21: 210–211.

25.

Lin

Lucas

Jr Shmueli

(2013) Research commentary-too big to fail: large samples and thep-value problem. Information Systems Research 24(4): 906–917.

26.

McCloskey

(2002) The Secret Sins of Economics. Chicago, IL: Prickly Paradigm Pamphlets.

27.

McCloskey

(1985) The Rhetoric of Economics. Madison, WI: University of Wisconsin Press.

28.

McCloskey

(1999) Cassandra’s open letter to her economist colleagues. Eastern Economic Journal 25(3): 357–363.

29.

Mertens

Recker

(2020) New guidelines for null hypothesis significance testing in hypothetico-deductive IS research. Journal of the Association for Information Systems 21(4): 1072–1102. DOI: 10.17705/1jais.00629.

30.

Mohajeri

Mesgari

Lee

(2020) When statistical significance is not enough: investigating relevance, practical significance, and statistical significance. Management Information Systems Quarterly 44(2): 525–559. DOI: 10.25300/MISQ/2020/13932.

31.

Pearson

(1900) On the correlation of characters not quantitatively measurable. Philosophical Transactions of the Royal Society of London, Series A, Containing Papers of a Mathematical or Physical Character 195. The Royal Society London: 1–47.

32.

Prentice

Miller

(1992) When small effects are impressive. Psychological Bulletin 112(1): 160–164.

33.

Rai

Arikan

Pye

, et al. (2015) Fit and misfit of plural sourcing strategies and IT-enabled process integration capabilities: consequences of firm performance in the U.S. electric utility industry. Management Information Systems Quarterly 39(4): 865–885.

34.

Ramasubbu

Bharadwaj

Tayi

(2015) Software process diversity: conceptualization, measurement, and analysis of impact on project performance. Management Information Systems Quarterly 39(4): 787–807.

35.

Robey

Markus

(1998) Beyond rigor and relevance. Information Resources Management Journal 11(1): 7–16.

36.

Rosemann

Vessey

(2008) Toward improving the relevance of information systems research to practice: The role of applicability checks. Management Information Systems Quarterly. 32(1):1-22.

37.

Rosenthal

(1990) How are we doing in soft psychology? American Psychologist 45(6): 775–777.

38.

Rosenthal

Rubin

(1979) A note on percent variance explained as a measure of the importance of effects. Journal of Applied Social Psychology 9(5): 395–396.

39.

Selvin

(1957) A critique of tests of significance in survey research. American Sociological Review 22(5): 519–527.

40.

Sen

Smith

Van Note

(2021) Statistical significance versus practical importance in information systems research. Journal of Information Technology 1: 026839622110622. DOI: 10.1177/02683962211062236.

41.

Thompson

(2002) “Statistical,” “practical,” and “clinical”: how many kinds of significance do counselors need to consider? Journal of Counseling & Development 80(1): 64–71.

42.

Tian

(2015) How do enterprise resource planning systems affect firm risk? Post-implementation impact. Management Information Systems Quarterly 39(1): 39–60.

43.

Vacha-Haase

Thompson

(2004) How to estimate and interpret various effect sizes. Journal of Counseling Psychology 51(4): 473–481.

44.

Vittinghoff

Glidden

Shiboski

, et al. (2005) Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models. New York, NY: Springer.

45.

Wang

Gupta

Rao

2015) Insider threats in a financial institution: analysis of attack-proneness of information systems applications. Management Information Systems Quarterly 39(1): 91–112.

46.

Wasserstein

Lazar

(2016) The ASA statement on p-values: context, process, and purpose. The American Statistician 70(2): 129–133. DOI: 10.1080/00031305.2016.1154108.