Systemic reviews are generally considered to be at the top of the evidence pyramid, providing one of the best sources of information. But just like any other type of research, a systemic review is only as good as the work and data that goes into it. A systemic review carefully looks at all of the evidence using a rigorous, predefined system of methods, and draws conclusions based on the information gathered. That rigorous, predefined system of methods is critical to a good systemic review. The researchers go through several steps:

• Select a specific, well-defined question
• Lay out their criteria for searching and selecting the evidence
• Conduct a very thorough search of all the available literature
• Evaluate the studies, rejecting the studies that are of poor quality
• Review the studies that make the cut
• Make a recommendation for practice

A systemic review differs from a general review of the literature in the methodology used. A systemic review starts by formulating specific criteria that will be used to judge which studies will be included and which will be excluded. The criteria are set before any of the studies are reviewed; ideally this will prevent bias and make for a stronger, more valid result.

Some systemic reviews include a meta-analysis, where statistical techniques are used to combine the results of the included studies and use the larger sample size to draw a stronger conclusion. But don't assume that all meta-analyses use the systemic review process! It's entirely possible to conduct a meta-analysis of a group of studies chosen in an incomplete or biased manner. The questions below can help you identify which meta-analyses use the systemic review process.

When reading a systemic review or meta-analysis, here are some questions to consider:

1. How well is the question defined? There should be a clear statement of what the review would like to show. Then double check the results to make sure that question actually got answered.

2. Is it the right question? If two things are being compared, is the comparison appropriate? Do all the studies included use the same comparison/control?

3. Do the authors describe their search? Was it thorough? Authors should discuss how they went about searching for the articles they evaluated. A thorough researcher will look at multiple databases, use variations of the key words, and include studies published in other languages. Limiting to studies published in English is convenient, but you may miss valuable research. Unpublished studies are also sometimes included, as it can be difficult to get a study published if your results showed no dramatic differences. This can help avoid publication bias, but unpublished studies should still be thoroughly checked for quality. The search and selection methods should be so clearly outlined that someone could duplicate them.

The Cochrane Collaboration is best known for conducting systemic reviews. In the Cochrane organization, reviewers publish their protocols before conducting the review. (If you are a Lamaze member, you can access these protocols on the Cochrane site by logging in through the Lamaze Member Center.) This is not generally done elsewhere, but in the published paper, the reviewers should explain their protocol. Any potential conflicts of interest should also be addressed.

4. Did the authors evaluate the quality of the studies reviewed? Not all studies are of equal quality, though sometimes studies with quality issues can still provide useful information. One example of this is the Cochrane Review on skin-to-skin, which included some studies that did not have completely random groupings if it appeared that the groups were otherwise equal. For this reason, many systemic reviews will rate the quality, size and applicability of the studies as they evaluate them, and assign them a weight so that the most appropriate studies are more heavily represented in the results.

5. Are there any biases in the inclusion/exclusion criteria? Read through them very carefully and evaluate this aspect. Easier when you're already familiar with the studies out there, or if the excluded studies are listed for your viewing. Overly restrictive criteria lead to smaller sample sizes and less reliable results.

6. Were the outcome measures clearly defined? What are the benefits or risks the researchers were looking at? Are outcomes lumped into groupings of debatable usefulness? Do they matter? As Amy Romano recently pointed out sometimes the outcomes have little or no real life importance.

7. Are there biases in how subgroups are analyzed? In one interesting study researchers created random data with the roll of the dice. The studies used dice that were identical other than color, and each time a 6 was rolled, it was counted as a patient death. The researchers then manipulated subgroup analysis to show that red dice had significantly higher death rates than other colors!

Don't assume that the general inclusion/exclusion criteria were applied to each subgroup. As we learned with the Wax meta-analysis on home birth, selective subgroup analysis does happen.

Bias in subgroup analysis is less likely if the subgroups were defined before the reviewers selected the included studies. Cochrane systematic reviews generally are conducted this way.

8. How old is it? As new research is done, even systematic reviews become outdated. It's hard to give a specific age that is 'too old', though, since some topics will have many new studies each year, and might be outdated at 18 months old while others rarely get anything new, and might still be relevant three years after publication.

9. How can I apply this? Again, this is always your last, but very important, consideration. Is this information that is applicable to your practice? To the population you serve? How can you best use this information?

Systemic reviews are among the best types of evidence out there, but only if they are done well. Read carefully, and consider the quality of the analysis as well as the quality of the included studies.

Next up: A roundup of other types of articles you may encounter in journals.

Published: November 13, 2010

Tags