How Can We Move Advanced Methodology into Practice More Effectively?

Abstract

Methodology serves an essential role in advancing psychological science. However, meta-science research points to a leaky translational pipeline in which substantive research often fails to utilize recommended methodological practices. Various explanations for this problem include valuing the development of methods over methodology (making tools over using tools), incentives for methodological research, incentives for promoting pedagogy in methodology, an insufficient number of quantitative methodologists in the discipline, and scarcity of resources for substantive researchers seeking more advanced methodological training. Policy makers might consider several recommendations that could mitigate extant leaks in the translational pipeline.

Keywords

quantitative methodology methods training pedagogy

Key Points

Aptly employing quantitative methodology in substantive research potentially advances scientific knowledge and its application.

Meta-science research points to a leaky translational pipeline of moving methodology into substantive research.

The leaky translational pipeline can be attributed to valuing methods (i.e., tools) over methodology (i.e., how best to use tools), low incentives for pedagogy and dissemination, a declining number of quantitative methodologists to provide for the training needs of the field, and over-reliance of substantive researchers on informal training due to inaccessibility of more formal training resources.

Funding agencies, research institutions, and professional societies should provide incentives to increase the number of quantitative methodologists, promote synergistic research among quantitative methodologists and substantive (non-methods) researchers, encourage continuing training for substantive researchers, and encourage pedagogy and dissemination efforts on the part of quantitative methodologists.

To move quantitative methodology into research practice, we must incentivize methodology (i.e., how best to use statistical models) over methods (i.e., statistical procedures), encourage pedagogy and training opportunities, and increase the number of quantitative methodologists.

Introduction

Quantitative methodology (QM) provides mathematical and statistical underpinnings to psychological research and is therefore central to advancing scientific knowledge. For example, test theory and psychometric models developed to measure individual differences are often applied in standardized tests in education (Jones & Thissen, 2007). Integrative data analysis allows researchers to pool multiple data sets to answer broad questions, including the generalizability of findings, while attending to superficial differences in measurement across studies (Curran & Hussong, 2009). Recently, network models have emerged as an alternative foundation for conceptualizing constructs, with applications to further understanding mental disorders (Fried et al., 2017). While methodologists develop approaches to analyze quantitative data, the uptake of QM by substantive (non-methods) researchers is often limited. The persistence of less optimal QM approaches is a clear challenge to advancing the field. Meta-science research on statistical mediation, construct validity, and power analysis provides examples of how current practice fails to incorporate QM recommendations, impeding progress. Solutions require a structural change to incentivize best practices.

Statistical Mediation

Statistical mediation aims to infer the mechanisms (or process) through which a predictor transmits an effect to an outcome. To conclude that a putative mediating variable (e.g., attitude change) conveys a causal effect (e.g., effect of treatment on smoking behavior), the “gold standard” design is an experiment (e.g., random assignment to treatment vs. control) with longitudinal observation of the process as it unfolds (e.g., how changes in attitudes over time change smoking behavior). However, most studies that examine mediation are neither experiments (18% in Flake et al., 2020; 24% in Gelfand et al., 2009) nor longitudinal (26% in Flake et al., 2020; 21% in Gelfand et al., 2009; 10% in Maxwell & Cole, 2007). Thus, most studies purporting to assess mediation do not have optimal design conditions to infer causal effects, much less the processes through which these effects occur, impeding our understanding of how treatments affect relevant psychological and health outcomes.

Construct Validity

Much of psychological science focuses on latent constructs that cannot be explicitly observed (e.g., personality, attitudes). Typically, constructs are measured indirectly through responses on a measurement instrument (e.g., a battery of items). These responses are thought to reflect levels of the construct of interest (e.g., ratings of loneliness and hopelessness both reflect underlying depression); however, this presumption should be empirically verified (e.g., see Benson, 1998). Without construct validity evidence, one cannot infer that calculated scores represent the construct of interest. Among 301 multiple-item measures in social and personality psychology, 23% had no construct validity information (Flake et al., 2017). Subsequently, 20% of multiple-item measures used in the Reproducibility Project: Psychology (Open Science Collaboration, 2015; currently the largest replication study) had no construct validity information (Flake et al., 2022). Taken together, a sizeable portion of studies lack construct validity evidence. Without construct validity, one cannot be sure that a given study is measuring the construct of interest or that any two studies purporting to measure the same construct are doing so, rendering the cumulation of knowledge a quixotic pursuit.

Statistical Power

In the study design, statistical power is the likelihood of identifying an effect that truly exists across study samples of a given size. For instance, a smoking cessation treatment might truly produce a meaningful reduction in cravings for nicotine. Yet a study design with modest number of participants might only have a 30% chance of detecting this effect. More broadly, power analyses allow researchers to consider “what if” scenarios that are useful for planning research studies (Kraemer & Blasey, 2015), framing the planned study within a context of prior research and theory (Wilkinson & The TFSI, 1999), and determining the sample size of a study (Cohen, 1988). Although power is strictly a pre-study concept (i.e., only relevant before a study is implemented; Greenland, 2012), power has been incorrectly applied in the post-study context to “evaluate” results (e.g., attribute statistically non-significant findings to an “underpowered study”; see Hoenig & Heisey, 2001). Despite increased emphasis in the methodology literature, by granting agencies, and in journal submission guidelines (e.g., Psychological Science), the proportion of studies in psychology that use power analysis to determine sample size remained constant at about 40% from 2017 to 2021 (Hoisington-Shaw et al., under review). Alarmingly, this stable rate in power analysis was accompanied by an increase in post-study power analysis (from n = 2 in 2017 to n = 21 in 2021). Power analysis thus seems to have increased in its incorrect rather than correct application. Failing to conduct pre-study power analysis risks poorly designed studies that waste resources and misapplying post-study power analysis risks ascribing false credibility to results of completed studies (Pek, 2022).

Meta-Science Observations

As these observations indicate, researchers readily apply methods that could increase the risk of errors and provide a poor guide for policy. What explains this persistence of poor QM practice, despite current recommendations? In part, it reflects standards of training in QM. Surveys have shown an improvement in the breadth and depth of QM courses offered in masters and doctoral training programs from the early 1990s to 2008 (Aiken et al., 1990, 2008). Many training programs, however, still do not offer or even require rigorous and comprehensive training in QM, in part reflecting the scarcity of quantitative methodologists entering the field who can provide this training. When averaged across all areas and programs of psychology, the median number of years of training in QM was only 1.2. That is, most graduates undergo only a first-year statistics sequence, which seldom covers the more complex models typical of most research in the field. Because QM continues to undergo refinement and innovation, career-long continuing education is necessary to keep up with new developments and recommendations. In a survey conducted in 2016, Flake et al. (2020) asked respondents where they learned about theoretical and statistical concepts related to mediation; the most frequent responses were: on their own by reading books, articles, or visiting webpages. This finding corroborates the view that poor QM practices in psychological research persist in part because researchers lack comprehensive training in QM. A crucial question for the field is how to support and incentivize the translation of recommended QM into research practice.

Policy Implications

Quantitative methodologists play a crucial role in bridging the gap between methodological developments and substantive application. They develop new methodologies, refine the application of existing methodologies, facilitate concordance with best practices in research design and analyses, and play a critical role in disseminating methodological recommendations to researchers. Currently, methodologists are not always successful in communicating new developments to substantive researchers. Some do not even try. Compounding the problem, substantive researchers’ methodological needs are not always recognized by quantitative researchers. Even when they are, substantive researchers may not have sufficient training and resources to access these developments. The next section explores what can strengthen the bridge between methodologists and substantive researchers, mostly considering incentives for methodologists in the domains of research and teaching. For substantive researchers, changing some aspects of training might better move methodology into practice. Finally, meta-science research would provide a feedback loop required to monitor and improve how QM moves into substantive research.

Research

Research in QM is evaluated using the traditional criteria of novelty, creativity, and impact. Rightly or (as we would argue) wrongly, these criteria have led methodologists to value and produce highly technical work and to undervalue dissemination, accessibility, and application by substantive researchers. Consider Psychometrika, the official journal of the Psychometric Society, which was founded to develop psychology as a quantitative and rational science (Thurstone, 1937). While prestigious among quantitative methodologists, this flagship journal currently does not reach the larger audience of psychological researchers. A Web of Science search, conducted in October 2022, indicated that the most highly cited Psychometrika article was by Cronbach (1951) on coefficient alpha (cited 22,411 times) followed by Kaiser (1974) on factor solutions (cited 6,067 times). The next 50 most-cited Psychometrika articles (with > 600 citations) were mostly published before 2000. In contrast, the most highly cited QM paper, Baron and Kenny (1986) on mediation and moderation (cited 49,012 times), was published in the substantive outlet Journal of Personality and Social Psychology (JPSP). The most-cited papers in Psychological Bulletin (a high-impact journal that emphasizes integrative reviews of psychological research), focus on methodology instead of a substantive topic (e.g., Cohen, 1992 cited 26,027 times; Anderson & Gerbing, 1988 cited 22,665 times, and Shrout & Fleiss, 1979 cited 16,191 times). Similarly, in Annual Review of Psychology, several of the most-cited papers also focus on methodology (e.g., Podsakoff et al., 2012 cited 6,426 times; MacKinnon et al., 2007 cited 3,821 times, and Graham, 2009 cited 3,771 times). In 1996, the methodology section of Psychological Bulletin moved to create the stand-alone journal Psychological Methods (PM). Since this move, however, no paper in PM has been cited more than 8,000 times, suggesting that methodologists are now largely writing for other methodologists rather than for the broader field. Even accounting for the cumulative nature of citation counts, QM papers appear to have lost much of their salience for substantive researchers. Hence, the limited uptake of QM developments.

Despite these overall trends, accessible QM papers can be highly influential and generate exceptional citation counts. Reflecting on this observation, several societies encourage publishing pedagogically oriented methodological review papers. For example, the American Psychological Association (APA) has a “Tutorials” category for papers on Psychological Methods. Similarly, the Psychometric Society has papers on “Application Reviews and Case Studies” in Psychometrika. In 2018, the Association for Psychological Science (APS) created a new journal, Advances in Methods and Practices in Psychological Science, to feature methodology papers relevant to substantive researchers. Despite these calls, quantitative methodologists often regard papers developing and communicating best practices as less prestigious than technical papers that feature novel technical contributions (even if these contributions are narrow, inaccessible, and fail to address the needs of substantive researchers).

What to Do About QM Research?

What stands out from a qualitative review of the most highly cited quantitative papers is their content. While these papers are mathematically elegant, they focus on methodological concepts and theory (e.g., reliability, dimensionality of a construct, modeling process), more so than technical prowess (e.g., deriving second-order derivatives for a standard error, e.g., Pek et al., 2011). To increase impact, journal editors and reviewers should evaluate contributions in terms of how developments contribute to methodology beyond method. Past President of the APA division of Quantitative and Qualitative Methods Beaujean (2021) pointed out that a method is defined as a particular tool or technique that guides our observations (e.g., the PROCESS macro; Hayes, 2022), whereas methodology combines method with logos (i.e., discourse). Thus, methodology is “knowledge about, or a reasoned argument for, doing something a certain way” (e.g., test theory by Lord & Novick, 1968). We emphasize incentivizing methodology over methods to improve practice.

Method development has historically had strong support from quantitative journals. We do not seek to diminish these activities, which are important for putting tools in the hands of substantive researchers (e.g., the mirt R package for fitting item response models; Chalmers, 2012). What is lacking, however, are similar incentives for methodology development. Though journals often call for such papers, they are not as highly valued by the institutional culture. This value system is at odds with the actual impact of methods versus methodology papers, the latter tending to have more influence on subsequent quantitative research and applications. For instance, generalizability theory (Cronbach et al., 1963) led to considering different types of reliability (Shrout & Fleiss, 1979, one of the most highly cited papers noted above), and an alternative perspective to replication (Shrout & Rodgers, 2018). Thus, editors, reviewers, professional societies, and institutions should place higher status on methodology papers.

Methods papers tend to be written by quantitative methodologists for other quantitative methodologists. In contrast, methodology tends to be developed within the context of a substantive psychological question. For instance, factor analysis (Spearman, 1904; Thurstone, 1934) and test theory (Lord & Novick, 1968) were motivated by questions about the structure of intelligence. Moderated nonlinear factor analysis (Bauer, 2017) addressed measurement differences when integrating data across multiple studies of the development of adolescent substance use (Bauer & Hussong, 2009).

To encourage the development of QM within the context of substantive research, funding agencies and universities should encourage team science that includes methodologists at the core. Funding should prioritize research projects that include a mix of substantive and methodological research aims, encouraging QM developments to attain substantive research goals. Both substantive and methodological outputs should count as independent scholarly contributions of the methodologist, equally valued by departmental tenure and promotion committees. The current lack of dual recognition tends to encourage methodologists to develop methods instead of methodology. To counter this tendency, substantive researchers should come to recognize that the quantitative methodologist is not simply a service technician to whom to contract out a research design, data analysis, and result reporting, absent from an appreciation of the substantive context (i.e., a statistical consultant), but an equal partner in generating knowledge about the substantive research question.

From the standpoint of scholarly publications, emphasizing methodology accompanies writing for the broader audience beyond quantitative methodologists. Thus, journal editors should encourage the publication of review articles centered on methodology, while featuring different methods (e.g., see Bakk & Kuha, 2021). Such papers serve an important role in consolidating methods under a more general methodological frame. Review papers tend to be written in a more accessible style (e.g., see Graham, 2009), serving a pedagogical role in bridging methodological developments with substantive research. Among quantitative methodologists, acknowledging the value of reviews on methodology should naturally extend to pedagogical publications because they move methodologies and methods into practice. Furthermore, non-methods journals (e.g., JPSP, Journal of Clinical Psychology) within each subdiscipline of psychology (e.g., social, clinical) should encourage submission of methodological papers relevant to improving their science (e.g., special issue on Mediation and Moderation in Health Psychology in 2008). When it comes to conducting research, encouraging an equal partnership of quantitative methodologists with substantive researchers will encourage methodology beyond methods and highlight the importance of pedagogy.

Training

Quantitative methodologists have a unique role in that many of us train students across subdisciplines within psychology as well as mentor future quantitative methodologists. How we teach is central to prioritizing methodology over methods. Training resources vary along a continuum from formal (classroom instruction) to informal (online forums). More formal options are typically characterized by greater scope of coverage and depth of training but also usually require a greater investment of resources in time and finances. Discussion begins with more formal training, progressing toward greater informality.

Courses

Much training in methodology and methods occurs in the classroom, from undergraduate through graduate training. Yet this training often does not go beyond a basic two-semester sequence and seldom touches upon complex QM, despite those approaches being prevalent in research applications. Further, methods and methodology are constantly evolving, so what one learns formally in the classroom has a half-life. In one example, Experimental Design and ANOVA as a course have slowly been replaced by Multiple Linear Regression, leaving a generational gap in communicating ideas framed within the context of the General Linear Model. Unfortunately, opportunities and resources are rare for post-PhD substantive researchers to take formal methodology coursework. Too often, senior investigators must rely on their students for up-to-date QM knowledge and skills, undermining meaningful supervision of this aspect of the research.

A related issue is who is teaching these courses, using what materials, and aiming at whom. Ideally, quantitative psychologists train both undergraduates and graduates in the classroom, using materials developed by experts in methods and methodology. But a shortage of quantitative psychologists continues (Clay, 2005), and most undergraduate and first-year graduate courses in methodology are taught by psychologists with substantive and not methodological expertise (Aiken et al., 1990, 2008). Likewise, textbooks at the undergraduate level tend to be predominantly written by substantive researchers. In our review of 143 undergraduate textbooks published from 2004 to 2021, only 20% of first authors identify as quantitative methodologists. Reflecting the small output of quantitative methodologists in producing textbooks, graduate-level texts are limited and in much need of updating (e.g., Cohen et al., 2003; Raudenbush & Bryk, 2002). Additionally, with a mean of 3.69 (mdn = 3.31, SD = 1.54) of full-time equivalent faculty members per quantitative program (APA Task Force for Increasing the Number of Quantitative Psychologists [TFINQP], 2009), faculty within a department offer only a constrained selection of courses beyond the first-year graduate sequence (see also Aiken et al., 1990, 2008). Worse, many departments do not have QM programs, and many have no quantitative methodologists on their faculty. In sum, few departments afford substantive researchers a full spectrum of QM training. Remedying this problem requires a substantial investment of resources into training and hiring new QM faculty and incentivizing them to focus on pedagogy.

Workshops and Short Courses

In part due to the absence of expertise and formal training opportunities within their own departments, many substantive researchers turn to external training options. Workshops are often developed by quantitative methodologists and typically offered in the summer (e.g., ICPSR summer program https://www.icpsr.umich.edu/web/pages/sumprog/; CenterStat https://centerstat.org/) or as pre-conference activities organized by societies (e.g., APS Convention, Annual Meeting of the Psychometric Society). Workshops vary greatly in scope, with some representing primers or introductions and others approaching equivalence with formal training (e.g., comparability to a semester-length class but without an evaluation component). Most, though not all, require a financial investment on the part of the attendee, whether through direct payment, membership fees for a professional society, or fees to attend a conference. These costs can be a significant barrier for many researchers. Providing funding opportunities for individuals to attend QM workshops would be a high-yield, low-cost way to strengthen psychological research across many areas.

Online Resources

The most informal, but also perhaps most used, resources are websites, blogs, videos, online forums, and podcasts. These resources are typically easy to access and free but are highly variable in terms of developers, topics, and target audiences. For example, the University of California Los Angeles (UCLA) Office of Advanced Research Computing unit in Statistical Methods and Data Analytics maintains an excellent repository of statistical software demonstrations for textbook examples (https://stats.oarc.ucla.edu/other/examples/). Examples of popular blogs include Statistical Modeling, Causal Inference, and Social Science (https://statmodeling.stat.columbia.edu/), Data Colada (http://datacolada.org/), and Statistical Horizons (https://statisticalhorizons.com/blog/). Videos are often distributed over YouTube channels (e.g., CenterStat “Office Hours” https://www.youtube.com/@CenterStat). Online forums include StackExchange and listservs dedicated to specific analytic techniques (e.g., the multilevel modeling list at https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=multilevel). A popular podcast is Quantitude (https://quantitudepod.org/). All of these represent important informal training resources for substantive researchers; however, none would purport to replace more formal training in quantitative methodology. Rather, websites, blogs, videos, forums, and podcasts typically serve to supplement and clarify training received through other venues.

What to Do About Training?

Increased training holds the potential to strengthen the bridge between quantitative methodologists and substantive researchers, facilitating movement of advanced QM into practice. To shift emphasis from methods to methodology, training providers should encourage statistical thinking. Statistical thinking, as supported by the Guidelines for Assessment and Instruction in Statistics Education College Report (GAISE College Report American Statistical Association Revision Committee, 2016) encourages modeling data (Rodgers, 2010; Tukey, 1969) over applying the “correct” procedure to data (Wild & Pfannkuch, 1999). Statistical thinking is consistent with valuing methodology over method and sets the stage for future researchers to regard quantitative methodologists as partners instead of contractors in research. The next sections consider how professional societies, institutions, and funding agencies can encourage statistical thinking by emphasizing methodology over methods through incentives for training.

Professional societies. Courses taught by quantitative methodologists, which are ideally guided by how textbooks are written, have the most potential to emphasize methodology over methods (e.g., see Abelson, 1995; Maxwell et al., 2017). Yet, most undergraduate courses and texts tend to focus on methods by promoting a cookbook style of analyzing data by following “recipes” (Schoenfeld, 1998; e.g., emphasizing steps in significance testing instead of the logic behind the test). Because of a general lack of quantitative methodologists to fulfill all formal training needs, professional societies should seek to (a) increase the number of quantitative psychologists by increasing efforts to connect with potential students (outreach; e.g., see APA TFINQP, 2009), (b) encourage conversations about teaching statistical thinking among substantive researchers, quantitative methodologists, and statisticians (e.g., holding co-sponsored events at conferences between APA Division 2 [Teaching of Psychology] and Division 5 [Quantitative and Qualitative Psychology]), (c) consider providing continuing education credits for attendance in workshops on QM, and (d) support the development of teaching (digital) material among substantive researchers, quantitative methodologists, and statisticians. Societies could also incentivize the development of training materials on methodology that can serve dual purposes of pedagogy and outreach. For example, a living, online, and open textbook with chapter contributions by methodological experts could be developed with sufficient funding sources (e.g., https://www.learn-mlms.com/ by Shaw & Flake, 2022).

Universities. Institutions play an important role in incentivizing training in methodology over methods. For instance, to encourage the development of resources focused on methodology, evaluation criteria on scholarly contributions by quantitative methodologists should weight graduate textbooks more highly because these are particularly impactful in the social sciences (e.g., Cohen et al., 2003 was cited 131,033 times in Google Scholar; Green, 2016). Currently, textbooks are often seen as secondary to articles, and early-career faculty are often advised against writing textbooks for this reason. More broadly, the emphasis on novel contributions that is typical for tenure-track faculty should be paired with a corresponding emphasis specifically for quantitative methodologists on dissemination, outreach, and application.

Departments could also consider creating tenure-track teaching positions for the purpose of coordinating undergraduate and graduate training in methodology among faculty from substantive and methodological backgrounds. By considering the methodological training of undergraduates and graduates as a single arc and coordinating input from faculty teaching different courses along the sequence, statistical thinking and methodology can emphasize identifying and addressing gaps in teaching. For example, a free web application and active learning exercises were developed in conversation with faculty teaching undergraduates the abstract concept of a sampling distribution (e.g., see Hoisington-Shaw & Pek, 2021).

Institutions may also consider a different model for graduate training in methodology by sharing expertise across institutions in a training consortium. In response to COVID-19, hybrid courses (i.e., students attending the course both in person and remotely over zoom) opened the possibility for students to access courses unavailable in their home institution. For example, a course on missing data at UCLA could be attended remotely by students from the Ohio State University (OSU) in exchange for UCLA students remotely attending a course on SEM offered by OSU. Inter-institutional agreements could also allow graduate students or postdoctoral fellows to take courses from a consortium of departments with graduate programs in quantitative methodology. Alternatively, public–private partnerships could be developed, wherein workshops on QM are provided by external organizations for use in courses led by faculty to scaffold a department's curriculum. The provision of such materials could enable fewer faculty to supervise courses on a greater breadth of QM topics, filling the current gap due to the scarcity of QM faculty.

Funding agencies. Funding opportunities could also strategically incentivize the application of QM in substantive research via pedagogy. Funding agencies should offer training grants to individuals and institutions seeking to increase breadth and depth of expertise in QM among substantive research trainees and for more seasoned researchers who need to refresh or extend their knowledge. Many of these grants could be relatively low budget (e.g., a typical course-equivalent workshop costs less than $1000) yet have a large payoff in enhancing methodological practices. Currently, investigators must write this funding into larger research or fellowship grants (e.g., R-, F-, or K-award NIH proposals); few stand-alone mechanisms exist for obtaining funding to attend workshops or conferences on QM.

Funding meta-science research . Meta-science research indicates that recommended methodology has not moved into substantive research although researchers continue to value it, as evidenced in surveys (e.g., Collins & Watt, 2021) and the high citations of methodological papers. These reviews also identify misconceptions and misapplications of methodology that can threaten the validity of results, pointing to the importance of forming a stronger bridge between methodology and substantive research. Surveys (conducted by Aiken et al., 1990, 2008) also provide valuable information on the health of QM as a discipline, describing changes in what is being taught and raising the alarm on the shortage of quantitative methodologists. Because meta-science research provides invaluable insights into current practices, funding agencies should support this research, particularly when it aims to develop strategies to improve methodological training and application. Beyond supporting reviews to identify burgeoning methodologies that need further development and training, research focused on identifying misconceptions and misapplications to motivate clarifications and corrections will advance psychological science. Surveys on the state of QM training (cf. Aiken et al., 1990, 2008) should, on a regular basis, monitor progress in meeting the methodological needs of substantive researchers.

Summary and Discussion

Methodology remains essential to moving psychological science forward. However, recent findings in meta-science research reveal that recommended QM is not moving into substantive research. This leaky translational pipeline reflects an emphasis on methods over methodology within the QM discipline, coupled with valuing innovation over pedagogy and a backdrop of a declining number of quantitative methodologists in the field. Moving methodology into substantive research more effectively will require changing incentives for both QM and substantive researchers. Policymakers should (a) develop incentives to promote methodological over method development; (b) emphasize funding team research that integrates methodological aims with substantive aims and within which methodologists and substantive researchers serve as equal contributors; and (c) value pedagogy in methodology equally as QM research by encouraging the publication of textbooks and review articles especially in substantive outlets. In terms of training needs, policymakers can (a) encourage the teaching of statistical thinking in collaborations among substantive researchers, quantitative methodologists, and statisticians; (b) incentivize and fund continuing education in QM; (c) work toward increasing the number of quantitative methodologists in the field (e.g., APA TFINQP, 2009); and (d) create a stronger presence with informal training resources to supplement formal training resources. Finally, policymakers should fund meta-science research on methodological practice and surveys on the health of QM as a discipline. This research can provide invaluable insights on how to develop and evaluate actionable policies that can fix the leaky translational pipeline moving methodological recommendations into practice.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Abelson

R. P.

(1995). Statistics as principled argument. Taylor & Francis.

Aiken

L. S.

West

S. G.

Millsap

R. E.

(2008). Doctoral training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno’s (1990) survey of PhD programs in North America. American Psychologist, 63(1), 32–50. https://doi.org/10.1037/0003-066X.63.1.32

Aiken

L. S.

West

S. G.

Sechrest

Reno

R. R.

Roediger

H. L.

III Scarr

Kazdin

A. E.

Sherman

S. J.

(1990). Graduate training in statistics, methodology, and measurement in psychology: A survey of PhD programs in North America. American Psychologist, 45(6), 721–734. https://doi.org/10.1037/0003-066X.45.6.721

American Psychological Association (2009). Report of the task force for increasing the number of quantitative psychologists. https://www.apa.org/science/leadership/bsa/quantitative

Anderson

J. C.

Gerbing

D. W.

(1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411–423. https://doi.org/10.1037/0033-2909.103.3.411

Bakk

Kuha

(2021). Relating latent class membership to external variables: An overview. British Journal of Mathematical and Statistical Psychology, 74(2), 340–362. https://doi.org/10.1111/bmsp.12227

Baron

R. M.

Kenny

D. A.

(1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182. https://doi.org/10.1037/0022-3514.51.6.1173

Bauer

D. J.

(2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507–526. https://doi.org/10.1037/met0000077

Bauer

D. J.

Hussong

A. M.

(2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14(2), 101–125. https://doi.org/10.1037/a0015583

10.

Beaujean

A. A.

(2021, October). Message from the new president. The Score, the Division 5 (Quantitative and Qualitative Methods) Newsletter of the American Psychological Association. https://www.apadivisions.org/division-5/publications/score/2021/10/message-president

11.

Benson

(1998). Developing a strong program of construct validation: A test anxiety example. Educational Measurement: Issues and Practice, 17(1), 10–17. https://doi.org/10.1111/j.1745-3992.1998.tb00616.x

12.

Chalmers

R. P.

(2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

13.

Clay

(2005). Too few in quantitative psychology. Monitor on Psychology, 36(8), 26–28.

14.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Academic Press.

15.

Cohen

(1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98–101. https://doi.org/10.1111/1467-8721.ep10768783

16.

Cohen

West

S. G.

Aiken

L. S.

(2003). Applied multiple regression/correlation analysis for the behavioral sciences. Lawrence Erlbaum.

17.

Collins

Watt

(2021). Using and understanding power in psychological research: A survey study. Collabra: Psychology, 7(1), Article 1. https://doi.org/10.1525/collabra.28250

18.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555

19.

Cronbach

L. J.

Rajaratnam

Gleser

G. C.

(1963). Theory of generalizability: A liberation of reliability theory. British Journal of Statistical Psychology, 16(2), 137–163. https://doi.org/10.1111/j.2044-8317.1963.tb00206.x

20.

Curran

P. J.

Hussong

A. M.

(2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14(2), 81–100. https://doi.org/10.1037/a0015914

21.

Flake

J. K.

Davidson,

I. J.

Pek

(2020). Is methods moving into practice? The critical role of quantitative training. In Rodgers

J. E.

(Ed.), Teaching statistics and quantitative methods into the 21st century (pp. 70–86). Taylor & Francis.

22.

Flake

J. K.

Davidson

I. J.

Wong

Pek

(2022). Construct validity and the validity of replication studies: A systematic review. American Psychologist, 77, 576–588. https://doi.org/10.1037/amp0001006

23.

Flake

J. K.

Pek

Hehman

(2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378. https://doi.org/10.1177/1948550617693063

24.

Fried

E. I.

van Borkulo

C. D.

Cramer

A. O.

Boschloo

Schoevers

R. A.

Borsboom

(2017). Mental disorders as networks of problems: A review of recent insights. Social Psychiatry and Psychiatric Epidemiology, 52(1), 1–10. https://doi.org/10.1007/s00127-016-1319-z

25.

GAISE College Report American Statistical Association Revision Committee (2016). Guidelines for assessment and instruction in statistics education college report 2016. http://www.amstat.org/education/gaise

26.

Gelfand

L. A.

Mensinger

J. L.

Tenhave

(2009). Mediation analysis: A retrospective snapshot of practice and more recent directions. The Journal of General Psychology, 136(2), 153–178. https://doi.org/10.3200/GENP.136.2.153-178

27.

Graham

J. W.

(2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60(1), 549–576. https://doi.org/10.1146/annurev.psych.58.110405.085530

28.

Green

(2016, May 12). What are the most-cited publications in the social sciences (according to Google Scholar)? LSE Impact Blog. https://blogs.lse.ac.uk/impactofsocialsciences/2016/05/12/what-are-the-most-cited-publications-in-the-social-sciences-according-to-google-scholar/#author

29.

Greenland

(2012). Nonsignificance plus high power does not imply support for the null over the alternative. Annals of Epidemiology, 22(5), 364–368. https://doi.org/10.1016/j.annepidem.2012.02.007

30.

Hayes

A. F.

(2022). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach (2nd ed.). Guilford Press.

31.

Hoenig

J. M.

Heisey

D. M.

(2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897

32.

Hoisington-Shaw

K. J.

Pek

(2021). Using dynamic graphics to teach the sampling distribution with active learning. The Quantitative Methods for Psychology, 17(2), v1–v9. https://doi.org/10.20982/tqmp.17.2.v001

33.

Hoisington-Shaw

K. J.

Wong

Matthews

Wegener

D. T.

Pek

(under review). The practice of power analysis and its implications: A meta-science review .

34.

Jones

L. V.

Thissen

D. T.

(2007). A history and overview of psychometrics. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Psychometrics (vol. 26, pp. 1–27). Elsevier.

35.

Kaiser

H. F.

(1974). An index of factorial simplicity. Psychometrika, 39(1), 31–36. https://doi.org/10.1007/BF02291575

36.

Kraemer

H. C.

Blasey

(2015). How many subjects? Statistical power analysis in research (2nd ed.). Sage Publications.

37.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Information Age Publishing.

38.

MacKinnon

D. P.

Fairchild

A. J.

Fritz

M. S.

(2007). Mediation analysis. Annual Review of Psychology, 58(1), 593–614. https://doi.org/10.1146/annurev.psych.58.110405.085542

39.

Maxwell

S. E.

Cole

D. A.

(2007). Bias in cross-sectional analyses of longitudinal mediation. Psychological Methods, 12(1), 23–44. https://doi.org/10.1037/1082-989X.12.1.23

40.

Maxwell

S. E.

Delaney

H. D.

Kelley

(2017). Designing experiments and analyzing data: A model comparison perspective. Routledge.

41.

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science (New York, N.Y.), 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

42.

Pek

Losardo

Bauer

D. J.

(2011). Confidence intervals for a semiparametric approach to modeling nonlinear relations among latent variables. Structural Equation Modeling: A Multidisciplinary Journal, 18(4), 537–553. https://doi.org/10.1080/10705511.2011.607072

43.

Pek

(2022, October). Uncertainty quantification in power analysis. The Score, the Division 5 (Quantitative and Qualitative Methods) Newsletter of the American Psychological Association, 44(4). https://www.apadivisions.org/division-5/publications/score/2022/10/power-analysis

44.

Podsakoff

P. M.

MacKenzie

S. B.

Podsakoff

N. P.

(2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63(1), 539–569. https://doi.org/10.1146/annurev-psych-120710-100452

45.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods. Sage.

46.

Rodgers

J. L.

(2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65, 1–12. https://doi.org/10.1037/a0018326

47.

Schoenfeld

A. H.

(1998). Making mathematics and making pasta: From cookbook procedures to really cooking. In Greeno

J. G.

Goldman

S. V.

(Eds.), Thinking practices in mathematics and science learning (pp. 299–319). Lawrence Erlbaum.

48.

Shaw

Flake

J. K.

(2022). Introduction to multilevel modeling. https://www.learn-mlms.com/

49.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420

50.

Shrout

P. E.

Rodgers

J. L.

(2018). Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual Review of Psychology, 69(1), 487–510. https://doi.org/10.1146/annurev-psych-122216-011845

51.

Spearman

(1904). 'General intelligence,' objectively determined and measured. The American Journal of Psychology, 15(2), 201–293. https://doi.org/10.2307/1412107

52.

Thurstone

L. L.

(1934). The vectors of mind. Psychological Review, 41(1), 1–32. https://doi.org/10.1037/h0075959

53.

Thurstone

L. L.

(1937). Psychology as a quantitative rational science. Science, 85, 227–232. https://doi.org/10.1126/science.85.2201.227

54.

Tukey

J. W.

(1969). Analyzing data: Sanctification or detective work? American Psychologist, 24(2), 83–91. https://doi.org/10.1037/h0027108

55.

Wild

C. J.

Pfannkuch

(1999). Statistical thinking in empirical enquiry. International Statistical Review, 67(3), 223–248. https://doi.org/10.1111/j.1751-5823.1999.tb00442.x

56.

Wilkinson

, & The Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594–604. https://doi.org/10.1037/0003-066X.54.8.594

How Can We Move Advanced Methodology into Practice More Effectively?

Abstract

Keywords

Key Points

Tweet

Introduction

Statistical Mediation

Construct Validity

Statistical Power

Meta-Science Observations

Policy Implications

Research

What to Do About QM Research?

Training

Courses

Workshops and Short Courses

Online Resources

What to Do About Training?

Summary and Discussion

Footnotes

Declaration of Conflicting Interests

Funding

References