Systematic Reviews

studies

What are systematic reviews?

A systematic review is a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research and to extract and analyse data from the studies that are included in the review. In other words, systematic reviews use explicit and rigorous methods to identify, critically appraise, include and synthesise relevant research studies. Systematic reviews differ from narrative reviews. In narrative reviews, work is described but not systematically identified, assessed for quality and synthesised. Statistical methods (meta-analysis) may be used to combine results of the studies into a summary measure. Commonly, systematic reviews are used as part of developing clincial guidelines and so you may find that some systematic reviews include recommendations that have Roman Numerals attached, indicating what level of evidence the recommendation is based on.

Systematic reviews do not equate meta-analysis. Quite often, a subject area is new and developing or lacks funding and so the evidence does not contain experimental trials which makes a meta-analysis difficult to conduct. Or there can be a large heterogenetity of study results which means that a meta-analysis is not possible. Results of systematic reviews can be reported qualitatively. The key is that the review is conducted systematically and therefore is replicable (i.e. if someone else conducted the review using your protocol, they could come up with the same results as you).

Aims of Systematic Reviews

1. They seek to minimise bias by using a replicable, scientific and transparent approach.

2. They seek to summarise results of otherwise unmanageable quantities of research. They also combine studies which gives more statistical power.

3. They do not reflect the views of 'experts' as they generate balanced inferences based on a collation and analysis of the available evidence.

4. They establish whether scientific findings are consistent and can be generalised across populations or whether findings vary significantly by particular subsets.

Stages of a Systematic Review

1. Identify the need for the review

2. Develop a protcol & formulate review questions

3. Conduct searches

4. Select studies according to selection criteria

5. Assess study quality and bias

6. Extract data & conduct data synthesis

7. Write report and disseminate findings

Appraising a Systematic Review

1. Is the topic well defined?

2. Was the search for papers thorough?

3. Were the criteria for inclusion of studies clearly described and fairly applied?

4. Was study quality assessed by blinded or independent reviewers?

5. Was missing information sought from the original study investigators?

6. Do the included studies seem to indicate similar effects?

7. Were the overall findings assessed for their robustness? (Think bias, chance, confounding, real effects)

8. Are the recommendations based firmly on the quality of evidence presented?

Types of Systematic Reviews

Systematic reviews of randomised controlled trials

These reviews seek to evaluate effectiveness.

A drawback is that there may be a lack of power of individual studies.

Systematic reviews of analytical (observational) studies

These reviews seek to evaluate risks (causes)

A common problem encountered when conducting this type of review is bias.

Meta-Analysis

A meta analysis is the analytical or statistical part of a systematic review and is used to combine results. Data combination increases power which enables small differences to be seen and makes a more precise estimate of a treatment effect. The precision with which the size of any effect can be estimated depends upon the number of patients studied. So combining trials leads to (1) more patients and more power to detect small but clincially significant effects and (2) more precise estimates of the size of effects. Meta-analysis differs from data-pooling as each trial is weighted before combining is done. The types of studies used in meta-analysis are randomised controlled trials, analytical (observational) studies or a single multi-centre observational study. Estimates, confidence intervals and/or standard errors are usually reported.

Forrest Plot

This is a visual representation of contribution of each study displaying estimates (symbols) and confidence intervals (displayed by length). There is also a visual representation of combined estimate and hetrogeneity of studies.

Heterogeneity

Heterogeneity is variation between the estimates over and above the natural sampling variation. Test for heterogeneity is performed at the same time as the estimates are combined (usually given as a chi-squared statistic). Overall, estimate and confidence interval can take heterogeneity into account - random effects meta-analysis. A random effects model assumes that in addition to the presence of random error, differences between studies can also result from real differences between the study populations and procedures. Fixed effects meta-analysis, on the other hand, assumes that all studies are a certain random sample of one large common study and that differences between study outcomes only result from random error. Pooling is then simple as it consists of calculating a weighted average of individual study results. It is worth noting that there is no agreement amongst statisticians on whether fixed or random effects meta-analysis is better. Reasons for heterogeneity should be investigated.

When different studies have largely different results, this can be random error or heterogeneity. To test homogeneity, use chi-square or Fisher's exact test for small studies. Power of test tends to be low but offers guidance. Basic but informative method is to produce a graph in which individual outcomes are plotted together with 95% CI.

Publication Bias

Research with statistically significant results is potentially more likely to be submitted, published or published more rapidly than work with non-significant results. False positive results are more likely to be published than false negative ones. Meta-analysis could, therefore, lead to incorrect conclusion. You will need to use a funnel plot to check publication bias.

Funnel Plot

This plot displays effect size by sample size. The plot will be funnel shaped if all studies that estimate the same quantity have been identified. It will be asymmetrical if trials are missing - usually smaller studies showing no effect. You can then estimate the number of missing studies that would change conclusion. Funnel plots can also show publication bias (if asymmetrical). Asymmetry can also be due to tendency for smaller studies to show larger treatment effects (tendency to have less rigorous methodology). Relative risks and odds ratio are plotted on a logarithmatic scale so that effects of same magnitude but in opposite directions are equidistant from 1.0 (e.g. 2 and 0.5). They are plotted against precision: 1/standard error. This emphasizes differences between larger studies.

Sensitivity Analysis

This looks at how the study quality is affected by bias, error, reporting and power. It examines results in relation to decisions made in systematic reviews, e.g. inclusion/exclusion criteria for studies, impact of each study and random and fixed effects.

Note on Hierarchy of Evidence

While randomised controlled trials top the ranks of evidence, it may not always be appropriate to restrict systematic reviews to just those of RCTs. For example, for studying risk factors, use of cohort studies rather than RCTs are more applicable. Some interventions, such as defibrillation for ventricular fibrillation, have an impact so large that observational data are sufficient to show it. In relation to rare or infrequent adverse outcomes, these would only be detected by RCTs so large that they are rarely conducted. Thus, observational methods such as postmarking surveillance of medicines are the only alternative. Sometimes, observational data provide a realistic means of assessing the long-term outcome of interventions beyond the timescale of trials.

Tips on Doing a Systematic Review

1. Don't underestimate how long a systematic review will take. Typically, a review will take between 3 and 9 months (average 6 months), depending on how easy it is to get hold of the articles for quality assessment.

2. Don't restrict your searches to just Medline, EMBASE and the other 'big name' databases. A 1997 study found that Medline and EMBASE only covered 6,000 of 20,000 journals. Check what databases are available to you at your library and use them all. You should also include hand searches, snowballing (i.e. using the list of references at the end of an article to generate more literature) and talk to people in the field as this will help you to locate grey literature.

3. Be thorough! This is a systematic review of all the evidence on a certain topic. You need to take great care in finding all relevant studies (published and unpublished), assess each study, synthesise the findings from individual studies in an unbiased way and present a balanced and impartial summary.

4. Spend time refining your review questions. These are what will drive your quality assessment and help shape your data synthesis. They will also help flag up gaps in the literature.

5. Make sure you establish a clear need for the review. When preparing a protocol, undertake a preliminary assessment of the literature. Have any reviews been done on this topic before? If so, see if you can build upon these reviews, for example, if a review was done 5 years ago, you could argue that it needs to be updated.