Meta-analysis done right: Steps and tips for reliable and impactful results
Meta-analyses are crucial to research—they provide a comprehensive picture of the overall evidence for a specific topic or field. But they have to be done properly to be useful. In this article, we’ll cover what meta-analyses are and their advantages, explore how to conduct them, and give some practical tips to ensure your next meta-analysis gives both meaningful and reliable results.
What is meta-analysis?
Definition
Meta-analysis is a statistical technique that combines results from multiple independent studies addressing the same research question. Meta-analyses provide an estimate, or overall effect size, which measures the strength or magnitude of a relationship, difference, or effect of the included studies.
Imagine you want to know how effective weight training is for improving bone density in the elderly. A meta-analysis will combine the individual data from multiple studies with such interventions and synthesize, or pool, them together. This will give you an overall effect estimate of the intervention.
The typical effect sizes used in meta-analyses are Cohen’s d or Hedges’ g, which represent the standardized mean difference between two groups. Effects close to 0 indicate negligible effects, effects of 0.2 to 0.5 suggest small effects, 0.5 to 0.8 indicate medium effects, and effects above 0.8 represent large effects.
Effect sizes can also be represented as a risk or odd’s ratio, which are used when you have binary outcomes, comparing the likelihood of an event occurring in two groups; by Pearson’s r when you want to assess the strength and direction of linear relationships between variables; or by a risk difference, which measures the absolute difference in event rates between groups.
When to use a meta-analysis
Meta-analyses are used when you want to get a more precise and reliable estimate of an effect. You’ll want to opt for a meta-analysis when:
There are many individual studies, or studies with varying or conflicting results
You want to get an overall effect rather than summarize findings qualitatively
Understanding trends or resolving inconsistencies across studies is important
You need to answer questions not posed by the individual studies
You need to make important evidence-based decisions, such as in healthcare policies or educational practices
Meta-analyses are especially useful in fields like medicine, psychology, and education, where individual studies often give varying results because of differences in methods, populations, or sample sizes; and where it’s crucial to understand the overall effects of a treatment, intervention, or phenomenon. Understanding these overall effects helps guide evidence-based decision-making.
Going back to our weight training in the elderly example, the results of the meta-analysis can help doctors have solid evidence to recommend weight training as a treatment for improving bone health in older adults. Similarly, for policymakers, meta-analyses can help inform public health initiatives and shape policies promoting weight-training programs for the aging population.
Meta-analysis vs. other reviews
Meta-analyses are quantitative, meaning they use statistical techniques to arrive at conclusions. They should not be confused with other, qualitative types of reviews, such as systematic or narrative reviews.
Systematic reviews summarize and evaluate the findings of multiple studies in a structured and comprehensive way using clear search parameters, but they do not necessarily use statistical methods to combine results. They focus on summarizing trends and conclusions, rather than quantifying the overall effect.
Narrative reviews provide a broad, non-statistical overview of a topic, offering insights from various studies without using structured criteria or statistical analysis. Scoping reviews are used when you want to get an overview of the existing literature on a broad topic, highlighting evidence and identifying research gaps. Unlike systematic reviews and meta-analyses, they don’t aim to answer specific research questions but rather explore a wider subject area.
The type of review you’ll need to do will depend on your research question. But, if you want to ensure a precise, quantitative estimate of an effect or relationship by combining data from multiple studies, a meta-analysis is your best bet.
Types of meta-analyses
There are several types of meta-analysis and the type you go for will depend on your research goals. Understanding the different types helps researchers choose the best approach for synthesizing data and addressing their specific research questions.
A network meta-analysis allows researchers to compare multiple interventions for a specific condition by combining both direct and indirect evidence from various studies. For example, if studies directly compare treatment A to B, and treatment B to C, network meta-analyses can estimate the relative effectiveness of A to C, even if there is no direct study comparing these two. This method helps provide a comprehensive view of all available treatment options, especially when multiple competing interventions exist for the same condition.
A Bayesian meta-analysis incorporates Bayesian statistical methods, which use prior knowledge or expert opinion alongside data from studies. By combining both prior distributions and observed data, Bayesian meta-analysis provides updated probability estimates for treatment effects. It can be especially useful when data is sparse or when researchers wish to incorporate prior research into the analysis.
A multilevel, or hierarchical, meta-analysis is a type of meta-analysis that accounts for the nested structure of data within studies. It is particularly useful when the studies being analyzed have multiple levels of data, such as when studies contain multiple outcome measures, multiple treatment groups, or data from different time points. In a multilevel meta-analysis, the data are grouped at different levels (e.g., study-level, participant-level, or measurement-level), and the analysis accounts for the potential correlation of outcomes within these groups.
How to conduct a meta-analysis
Meta-analyses follow a rigorous process that must be followed to a tee if you want to ensure your meta-analysis gives the best results. In this section, I will briefly describe the main steps of a meta-analysis.
Research question
The first step in any research project is to come up with a clear, novel and specific question you want your research to answer. Having a clear and specific research question will ensure you find the most papers you can on your topic of interest.
Good research questions should conform to the PICO criteria, which is framework for formulating clear and focused research questions in meta-analyses. It outlines the Population being studied, the Intervention or exposure under investigation, the Comparison group, and the specific Outcome being measured.
For example, a good research question for our meta-analysis on the elderly and bone density following the PICO criteria would be: Among elderly adults (Population), how does weight training (Intervention) compare to no exercise or alternative forms of exercise (Comparison) in improving bone density (Outcome)?
Selection criteria
The next step is to establish a strict set of selection criteria that will help you determine which articles to include in your meta-analysis. These criteria are based on factors like the population studied, the type of intervention or exposure, the comparison group, the outcomes measured, and the study design (e.g., randomized controlled trials). Establishing clear inclusion and exclusion criteria ensures consistency and helps focus your analysis on studies that are directly relevant to your research question.
For example, for the effects of weight training on bone density in the elderly example, you will want to include studies that feature:
Participants aged 60 and older (or another age criteria)
An intervention where weight-training or resistance exercises are the focus
A measure of bone density using specific metrics like bone mineral density or MRI
A comparison group to the intervention, such as those receiving no exercise or alternative forms of exercise (e.g., aerobic or flexibility exercises)
Randomized controlled trials or longitudinal observational studies
Systematic literature search
The main part of a meta-analysis is the literature search. It is called systematic because you need to systematically look for articles that examine your research question. This is where you conduct a comprehensive search for ALL relevant studies on your topic using keyword queries on multiple databases (e.g., PubMed, Scopus, etc) and keep a clear record of how many records, or “hits,” each database retrieves.
“Grey,” or unpublished, literature can also be searched here to ensure you don’t miss any relevant studies. This includes thesis, pre-prints or other unpublished works.
A good search strategy is comprehensive, uses clear search terms, and is carefully documented so others can reproduce it. You may also need to try out different boolean operators as you refine the search. A search for our weight training interventions in the elderly example might look like this: ("weight training" OR "resistance exercise") AND ("bone density" OR "bone health") AND ("elderly" OR "older adults"), where tweaking some of the syntax of the query might be required for each database.
Study selection
After doing the comprehensive literature search, the next step is to screen and evaluate the studies for inclusion based on your selection criteria. For instance, you will likely exclude studies that look at the effects of weightlifting on health outcomes in high school students or studies looking at outcomes other than bone health and only include studies where older adults were included and bone density was measured as the main outcome.
A good meta-analysis will use tools like the PRISMA flowchart to document the process of selecting and excluding studies, ensuring transparency. This flowchart will then be displayed in your manuscript so others can easily follow and replicate the same process.
Data extraction
Once you’ve selected all the relevant studies for your meta-analysis, you will then extract and code study information such as effect sizes, sample sizes, and study characteristics. This is the information you’ll need to conduct the actual analysis part of the meta-analysis. Depending on the size of your meta-analysis, this step might be the most time-consuming as at least two researchers, or raters, are required to minimize bias and errors in the coding process.
If you are unable to extract certain data from a study, such as key statistics needed for effect size calculation, that study should be excluded from the meta-analysis.
Assessing quality and risk of bias
Meta-analyses are only as good as the studies they include, so evaluating the quality and risk of bias in the included studies is crucial. This is because low-quality or biased studies can distort the overall findings and lead to the wrong conclusions. Risk of bias is rated based on how well each paper follows a set of criteria or judgements using the Risk of Bias tool (see here for more information on the tool).
The quality of studies is done using the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) approach. GRADE is a systematic method for assessing the quality (certainty) of evidence and the strength of recommendations in healthcare. It helps organizations and researchers make clear, transparent, and reliable decisions by evaluating factors like study design, consistency of results, and potential biases. GRADE is widely used in developing clinical guidelines to ensure recommendations are based on the best available evidence (see here for more information on how to assess quality in meta-analysis).
If you’re conducting a systematic literature review on a topic, the steps up till here are generally the same. Meta-analysis then goes a step further and quantifies the extracted data using a model.
Analysis
In this step, the extracted data from the individual studies (e.g., their effect sizes) are pooled together to calculate the overall effect size using statistical methods. This will give you the overall effect estimate. The results are typically shown in a "forest plot,” which displays each study's individual results and the overall pooled effect (Fig 1.). The forest plot helps you easily compare different treatment effects. Each study's result is represented by a square, with the overall pooled estimate shown as a diamond.
Fig 1.
Forest plot illustration (data is simulated). Boxes represent effect sizes of each study, with varying sizes based on weights. Green diamond represents the overall effect size and the blue line below that is the prediction interval.
As a side note here: only studies that can be compared should be. If the methodologies or outcomes are too different amongst your included studies, it might be better to present the results in a systematic review instead of meta-analysis.
Quantifying problems
Once the meta-analysis has been conducted, differences between the studies may still exist. These differences are known as heterogeneity, which is the variation in results across studies due to differences in methods, populations, or contexts; and publication bias, which occurs when studies with significant or positive results are more likely to be published than those with null or negative findings.
Cochran's Q test and the I² statistic are used to assess the degree of heterogeneity. If significant heterogeneity is found (e.g., p < .05), the decision must be made whether combining studies is valid. For instance, if a large study with notable clinical differences stands out, excluding it may resolve heterogeneity. To resolve or better understand heterogeneity, researchers sometimes conduct meta-regression which looks at how study-level factors like study size or country contribute to the overall effects.
More complex statistical tests like PET-PEESE are sometimes also used that take the study’s precision into account. While there is no direct way to “solve” publication bias, one way it can be handled is to look at multiple measures of publication bias to get a more accurate estimate and quantification of it’s effects.
Interpreting
After conducting the analysis of the meta-analysis, it’s time to interpret what it all means. A positive overall effect size indicates a positive relationship or impact, while a negative effect size suggests a negative relationship or impact.
For example, if our meta-analysis on the impact of weight-training in the elderly found an overall effect of g = 0.65, 95% CI [0.53-1.78), that’s a good indicator that weight-training is moderately effective at improving bone density in the elderly.
It’s also important to look at the confidence interval (CI) here, as that will tell you the range within which the true effect likely lies, reflecting both the precision of the estimate and the consistency of the included studies. In this case, the CI tells us that the true effect of this intervention is estimated to fall somewhere between 0.53 and 1.78. Because this range is pretty narrow and does not include 0 (which would indicate no effect), this means we can be relatively certain about the intervention’s effectiveness.
Tips for researchers
Before you take on a meta-analysis, consider the following tips:
Meta-analyses are not for the “lone wolf” researcher. Research can often be lonely and isolating, but not meta-analyses. You’ll need a great team around you to assist in literature search, study selection and data extraction. This is done to ensure consistency and reliability.
It’s not a quick turnaround. Like all research, meta-analyses are time-consuming and require a high degree of diligence, attention to detail, and deliberation. Sometimes, the literature search alone can take a few months depending on the research topic, so be prepared to be in it for the long haul if you undertake a meta-analysis.
Be transparent and follow best practices. Meta-analyses have the potential for far-reaching impact in society, so it’s crucial they are conducted in the best way possible. Clearly report methods, criteria, and data sources so others can replicate your work and follow best practice guidelines (for a review, see Maassen et al., 2020).
If you’re not doing a meta-analysis currently or in the near future, you can still contribute to meta-analyses. Good meta-analyses start with good research, so you can ensure your own papers are conducted well and are “meta-analysis ready.” This means you included all raw data, reported all outcomes, and included the means, standard deviations and effect sizes in your results. This will make it easier for a future researchers to include your paper in a meta-analysis and for that meta-analysis to produce reliable findings based on solid research.
Takeaways
Meta-analyses are extremely useful and rewarding research projects, but they need to be executed correctly to be informative and beneficial to both science and society. A clear research question, strict inclusion criteria, thorough literature search, and careful data extraction are essential for producing reliable results. Researchers should also assess study quality using frameworks like GRADE and follow PRISMA guidelines for transparent reporting. When done properly, meta-analyses help resolve conflicting evidence, identify trends, and support evidence-based decision-making in healthcare and beyond.
References
Harrer, M., Cuijpers, P., Furukawa, T.A., & Ebert, D.D. (2021). Doing Meta-Analysis with R: A Hands-On Guide. Boca Raton, FL and London: Chapman & Hall/CRC Press. ISBN 978-0-367-61007-4.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5 (updated August 2024). Cochrane, 2024. Available from www.training.cochrane.org/handbook.
