Systematic Reviews Project Assignment

BOX 29.1: Guidelines for Critiquing Systematic Reviews

THE PROBLEM

·  Did the report clearly state the research problem and/or research questions? Is the scope of the project appropriate?

·  Is the topic of the review important for nursing?

·  Were concepts, variables, or phenomena adequately defined?

·  Was the integration approach adequately described, and was the approach appropriate?

SEARCH STRATEGY

·  Did the report clearly describe criteria for selecting primary studies, and are those criteria reasonable?

·  Were the bibliographic databases used by the reviewers identified, and are they appropriate and comprehensive? Were key words identified, and are they exhaustive?

·  Did the reviewers use adequate supplementary efforts to identify relevant studies?

·  Was a PRISMA-type flowchart included to summarize the search strategy and results?

THE SAMPLE

·  Were inclusion and exclusion criteria clearly articulated, and were they defensible?

·  Did the search strategy yield a strong and comprehensive sample of studies? Were strengths and limitations of the sample identified?

·  If an original report was lacking key information, did reviewers attempt to contact the original researchers for additional information—or did the study have to be excluded?

·  If studies were excluded for reasons other than insufficient information, did the reviewers provide a rationale for the decision?

QUALITY APPRAISAL

·  Did the reviewers appraise the quality of the primary studies? Did they use a defensible and well-defined set of criteria, or a respected quality appraisal scale?

·  Did two or more people do the appraisals, and was interrater agreement reported?

·  Was the appraisal information used in a well-defined and defensible manner in the selection of studies, or in the analysis of results?

DATA EXTRACTION

·  Was adequate information extracted about methodologic and administrative aspects of the study? Was adequate information about sample characteristics extracted?

·  Was sufficient information extracted about study findings?

·  Were steps taken to enhance the integrity of the data set (e.g., were two or more people used to extract and record information for analysis)?

DATA ANALYSIS—GENERAL

·  Did the reviewers explain their method of pooling and integrating the data?

·  Was the analysis of data thorough and credible?

·  Were tables, figures, and text used effectively to summarize findings?

DATA ANALYSIS—QUANTITATIVE

·  If a meta-analysis was not performed, was there adequate justification for using a narrative integration method? If a meta-analysis was performed, was this justifiable?

·  For meta-analyses, were appropriate procedures followed for computing effect size estimates for all relevant outcomes?

·  Was heterogeneity of effects adequately dealt with? Was the decision to use a random effects model or a fixed effects model sound? Were appropriate subgroup analyses undertaken—or was the absence of subgroup analyses justified?

·  Was the issue of publication bias adequately addressed?

DATA ANALYSIS—QUALITATIVE

·  In a metasynthesis, did the reviewers describe the techniques they used to compare the findings of each study, and did they explain their method of interpreting their data?

·  If a metasummary was undertaken, did the abstracted findings seem appropriate and convincing? Were appropriate methods used to compute effect sizes? Was information presented effectively?

·  In a metasynthesis, did the synthesis achieve a fuller understanding of the phenomenon to advance knowledge? Do the interpretations seem well grounded? Was there a sufficient amount of data included to support the interpretations?

CONCLUSIONS

·  Did the reviewers draw reasonable conclusions about the quality, quantity, and consistency of evidence relating to the research question?

·  Were limitations of the review/synthesis noted?

·  Were implications for nursing practice and further research clearly stated?

All systematic reviews

Systematic reviews of quantitative studies

Metasyntheses

In drawing conclusions about a research synthesis, a major issue concerns the nature of the decisions the researcher made. Sampling decisions, approaches to handling quality of the primary studies, and analytic approaches should be carefully evaluated to assess the soundness of the reviewers’ conclusions. Another aspect, however, is drawing inferences about how you might use the evidence in clinical practice. It is not the reviewers’ job, for example, to consider such issues as barriers to making use of the evidence, acceptability of an innovation, costs and benefits of change in various settings, and so on. These are issues for practicing nurses seeking to maximize the effectiveness of their actions and decisions. Systematic Reviews Project Assignment

Figure 2.1  shows that  systematic reviews  are at the pinnacle of the hierarchy (Level I), regardless of the type of question, because the strongest evidence comes from careful syntheses of multiple studies. The next highest level (Level II) depends on the nature of inquiry. For Therapy questions regarding the efficacy of an intervention (What works best for improving health outcomes?), individual RCTs constitute Level II evidence (systematic reviews of multiple RCTs are Level I). Going down the “rungs” of the evidence hierarchy for Therapy questions results in less reliable evidence—for example, Level III evidence comes from a type of study called quasi-experimental. In-depth qualitative studies are near the bottom, in terms of evidence regarding intervention effectiveness. (Terms in  Figure 2.1  will be discussed in later chapters.)

FIGURE 2.1

Evidence hierarchy: levels of evidence.

For a Prognosis question, by contrast, Level II evidence comes from a single prospective cohort study, and Level III is from a type of study called case control (Level I evidence is from a systematic review of cohort studies). Thus, contrary to what is often implied in discussions of evidence hierarchies, there really are multiple hierarchies. If one is interested in best evidence for questions about Meaning, an RCT would be a poor source of evidence, for example. We have tried to portray the notion of multiple hierarchies in  Figure 2.1 , with information on the right indicating the type of individual study that would offer the best evidence (Level II) for different questions. In all cases, appropriate systematic reviews are at the pinnacle. Information about different hierarchies for different types of cause-probing questions is addressed in  Chapter 9 .

Of course, within any level in an evidence hierarchy, evidence quality can vary considerably. For example, an individual RCT could be well designed, yielding strong Level II evidence for Therapy questions, or it could be so flawed that the evidence would be weak.

Thus, in nursing, best evidence refers to research findings that are methodologically appropriate, rigorous, and clinically relevant for answering persistent questions—questions not only about the efficacy, safety, and cost-effectiveness of nursing interventions but also about the reliability of nursing assessment tests, the causes and consequences of health problems, and the meaning and nature of patients’ experiences. Confidence in the evidence is enhanced when the research methods are compelling, when there have been multiple confirmatory studies, and when the evidence has been carefully evaluated and synthesized.

Of course, there continue to be clinical practice questions for which there is relatively little research evidence. In such situations, nursing practice must rely on other sources—for example, pathophysiologic data, chart review, quality improvement data, and clinical expertise. As Sackett and colleagues (2000) have noted, one benefit of the EBP movement is that a new research agenda can emerge when clinical questions arise for which there is no satisfactory evidence. Systematic Reviews Project Assignment