Abstract
In this issue's Statistical Sidebar, I would like to continue the theme from the sidebar included in the last issue (“Statistics, broadly speaking,” March–April 2024) and look not at a specific statistical test but rather a larger analytical approach: meta-analysis. It is enticing to take the results from a single study and hold those results as answering a question or proving what the state of and issue is. However, our discussions of statistics have shown that every study has its limitations and all results, to some degree, leave open the possibility that the findings are due to chance and do not actually reflect the true state of affairs. Having results replicated by other researchers adds to the weight of a finding and makes it more unlikely that the first set of results are spurious. Beyond simply repeating what others have done, another way to leverage the work of past researchers is to conduct what is known as a meta-analysis.
In a meta-analysis, a research question is addressed not by collecting new data or by replicating a previous study but by systematically combining all previous results that fit a strict criteria. By gathering together all results from previous studies that are able to be linked together due to similar designs, participants, and measures, the power of each individual study's findings is enhanced by adding them together to the findings of related studies. Instead of trying to collect data from many participants, a large number of participants is obtained by combining previous related studies.
Now, there are plenty of pitfalls in this approach. Any bias or low quality data in a study will skew the overall findings. In a meta-analysis, like in the article “Gait characteristics and development in pediatric populations with visual disorders: Where do we stand and where are we going?” by Montagnani, Bradley, and Smith, as many studies as possible that are related to the research topic are reviewed and evaluated for the quality of their methods, measures, and statistical analysis. Ideally, acceptable studies from a large pool of candidates are those that used a randomized controlled trial approach. However, depending on the field, randomized controlled trial studies may not be common. In these cases, studies with lower levels of evidence must be considered. It is here that a researcher may need to decide whether a meta-analysis is possible or whether a systematic review must be conducted.
The difference between a meta-analysis and a systematic review is that the meta-analysis combines data from different studies in order to obtain a larger sample size than any of the component studies has and then conduct analyses on the larger dataset. A systematic review is no less stringent on evaluating the quality of studies, but focuses more on reviewing and analyzing the results of a set of studies that meet a given quality standard. In the article by Montagnani, Bradley, and Smith, the level of quality required to be included in the research led to a reduction from 1,531 papers down to 8 acceptable articles. However, the designs and measures in the 8 studies did not allow for a combining of the data across studies for new analyses. Therefore, the authors were forced to conduct a systematic review and focus on the results instead.
If data are similar enough across studies so that they can be combined, a weighted pooled estimate of an effect or intervention can be calculated. This is where the results from each study are assigned a weight that is determined by comparing samples sizes, measures, validity, risk of bias, and how much information the study contributes to the meta-analysis. If outcome variables of interest across several studies are all continuous variables (define), the meta-analysis can calculate a mean difference or standardized mean difference for the combined data. If the data are dichotomous, odds ratio can be calculated. Essentially, if it is determined that data across studies is similar enough that they can be combined, the characteristics of those data determined what statistical approach can be used to combine and report on the combined dataset. However, due to the many pitfalls possible when combining data across studies, this should be done by a person well versed in a range of statistical approaches. For a deeper, more thorough treatment of elements to be considered when conducting a systematic review, consider perusing the Cochrane Handbook for Systematic Reviews of Interventions at https://training.cochrane.org/handbook.