Free
Special Communication  |   August 2007
Evidence-Based Medicine, Part 4. An Introduction to Critical Appraisal of Articles on Harm
Roberto Cardarelli, DO, MPH; Margaret M. Seater, OMS IV
Author Notes
  • From the Department of Family Medicine at the University of North Texas Health Science Center—Texas College of Osteopathic Medicine in Fort Worth. 
  • Address correspondence to Roberto Cardarelli, DO, MPH, Department of Family Medicine, Texas College of Osteopathic Medicine, University of North Texas Health Science Center, 855 Montgomery St, Patient Care Center, 2nd Fl, Fort Worth, TX 76107-2553.E-mail: rcardare@hsc.unt.edu 
Article Information
Evidence-Based Medicine
Special Communication   |   August 2007
Evidence-Based Medicine, Part 4. An Introduction to Critical Appraisal of Articles on Harm
The Journal of the American Osteopathic Association, August 2007, Vol. 107, 310-314. doi:https://doi.org/10.7556/jaoa.2007.107.8.310
The Journal of the American Osteopathic Association, August 2007, Vol. 107, 310-314. doi:https://doi.org/10.7556/jaoa.2007.107.8.310
Abstract

This article provides an introductory step-by-step process to appraise an article on harm. The authors introduce these principles using a systematic approach and case-based format. The process of assessing the validity of an article on harm, determining its importance, and applying it to an individual patient is reviewed. The concepts of study population homogeneity, equal treatment, sufficient follow-up periods, and completeness are discussed to help physicians determine an article's validity. Instruction on calculating odds ratios, relative risk, absolute risk increase, and the number needed to harm is provided and applied to the clinical scenario. Finally, information that is learned from the previous two steps is applied to patient care. Study generalizability and the role of patient values, patient expectations, and patient concerns are also addressed. The skills learned from appraising an article on harm in the manner outlined provides a solid basis for lifelong learning and improved patient care.

Evidence-based medicine (EBM) is the practice of using the most current research information to help guide clinical decisions while also fully taking into account patient values and circumstances.1-3 This method of inquiry demands skill-development in appraising the validity, importance, and applicability of new clinical evidence to individual patients— and it begins when a physician formulates a clinical question during daily practice. 
As noted, in this article, we introduce a strategy for busy physicians, physician residents, and medical students to critically assess the medical literature on harm. In-depth details of research methods are beyond the scope of this introductory series on EBM. Readers are encouraged to seek further training on these topics with supplemental learning opportunities and continuing medical education. Finally, the clinical scenario described has been simplified to provide readers with an illustrative example for the general concepts introduced. 
Searching the Evidence
As with other types of clinical questions, the most authoritative evidence for treatment decisions comes from systematic reviews.1 This standard is especially true for clinical questions that include harm because individual randomized controlled trials are seldom sufficiently powered to allow researchers to conduct thorough assessments for potential harm to patients. Unfortunately, well-designed systematic reviews are uncommon and individualized studies frequently must be used in EBM. For this reason, the present article will focus on how to use such studies in lieu of preferred systematic reviews. 
Validity of Articles on Harm
A valid study aims to discover objective truth while also attempting to exclude competing explanations.1,2 Only a study that is deemed valid should be further reviewed. The following set of questions will help a clinician elucidate the validity of a study on harm: 
  • Are the defined groups of patients similar—other than through their exposure status to the treatment under study? When assessing the effectiveness or harm of an exposure or intervention, factors that may influence the outcome of interest must be balanced or accounted for in all study groups to avoid confounding results and conclusions. Treatment and comparison groups should be clearly defined, and crossover between allocated groups should be minimized.
    Both groups should share similar baseline characteristics, especially characteristics that may confound the outcome of interest. In a study examining a potentially harmful treatment, healthier subjects should not be overrepresented in the treatment group because such a group allocation may confound the study's outcome (ie, healthier people may be able to tolerate the treatment better). Information on baseline characteristics can usually be ascertained quickly from a study population characteristics table. In most research articles, subject characteristics are placed in the first table.
    In addition, patients assigned to receive active treatment should not also receive the control drug.
  • Were the measures of interest ascertained in the same way for both groups?
    To properly compare study results among study groups, outcome assessments should be objective. Such clarity can be achieved only through the use of clearly stated case definitions.
    For example, in a study investigating cardiovascular outcome, the investigators decide that a myocardial infarction (MI) must meet specific criteria such as EKG changes or positive troponins. The definition of an MI cannot be left open for interpretation.
  • Were participants and researchers blinded to the measures of interest?
    Assessment should also be blinded to prevent observer bias. Observer bias occurs when the researcher unconsciously (or consciously) looks harder for outcomes in the treatment group than in the control group. Blinding means that the researcher is unaware of the participants' group assignment. Double-blinding means that study participants are also masked to group allocation.
  • Was the follow-up period sufficiently long and complete? Follow-up in a valid study should be sufficiently long and complete. Short follow-up periods may allow too little time for the disease under investigation to manifest. The appropriate length of the study is dependent on the study question, the intervention used, the outcomes of interest, and special circumstances (eg, funding).
    A clinical investigation is considered complete when all study participants are accounted for at the study's completion. Patients who dropped out of the study early because of death or adverse effects should be included and analyzed with their original group assignment. This is called an intention-to-treat analysis.
    A failure to perform an intention-to-treat analysis may direct researchers to providing readers with misleading results. The “5-and-20 rule” can be used by a critical reader to evaluate a study's completeness. If less than 5% of the study population is lost to follow-up, one can be assured that the loss minimally impacted the results. If, however, more than 20% of the study population is lost to follow-up, caution is advised when making clinical decisions based on study findings. An attrition rate of 5% to 20%—and its impact on the researchers' results—must be determined by the reader based on other specifics of the study.
  • Do the results of the study satisfy some of the criteria for causation?
    While most studies provide some insight as to whether an exposure is associated with an outcome, that contact may not determine causality (ie, the exposure causes the outcome). The following five questions will help a cautious reviewer determine if the criteria are met to presume causality:
     
    • Does the exposure precede the outcome?
      To establish causation, the exposure must take place before the outcome.
    • Is a dose-response phenomenon present?
      The more one is exposed (eg, dosage), the greater the effect.
    • Is there evidence in the change of the outcome when the exposure is removed and reintroduced?
      Causation is suggested when the health status or clinical outcome improves (or deteriorates) after exposure is discontinued. In addition, the outcome recurs if the exposure is reintroduced.
    • Do other studies find consistent results?
      The assumption of causation is strengthened when more than one group of researchers reports similar findings.
    • Does the association make biological sense?
      It is important for the relationship between the exposure and the study outcome to be a logical one from the perspective of basic biology.
Study Results
Having ascertained a study's validity, it is necessary to determine if the study's findings are important.1,2 In the medical literature, importance refers to the study's results and is measured in terms of the magnitude and precision of the association. 
  • What is the magnitude of the association between the exposure and the outcome?
    The magnitude of association quantifies the benefit or risk of an intervention when the results seen in the treatment group are compared with that of the control group. In retrospective studies, this data is reported as an odds ratio (hazards ratio). An odds ratio of less than 1 indicates that the intervention confers a protective effect. An odds ratio greater than 1 indicates that the intervention confers a detrimental effect. An odds ratio equal to 1 indicates that there is no difference in the outcome between the treatment and control groups. In the everyday practice of EBM, correctly interpreting the odds ratio is more important than knowing how to calculate it (Figure 1).
    Relative risk, which is used in prospective studies (ie, clinical trials and cohort studies), is another way to report the magnitude of an association. Relative risk is defined as “the ratio of the risk in the treated group (experimental event rate) to the risk in the control group (control event rate).”1,4 Interpretation of the relative risk is the same as the odds ratio (Figure 1).
    For patients and physicians alike, the odds ratio and relative risk are difficult to conceptualize. These measures are more digestible when reported as the number needed to harm (NNH) when investigating adverse outcomes (Figure 1). The NNH is defined as “the number of patients who need to be exposed to the causative agent to produce one additional harmful event.”1,2 The following methods can only be used for clinical trials and cohort studies.
    The NNH can also be calculated from case-control studies. Calculations are not shown because they are complicated and unrealistic to perform in real practice. However, free EBM calculators are widely available online (http://www.cebm.utoronto.ca/). Such calculators can easily convert an odds ratio into a NNH.
  • What is the precision of the relationship between the exposure and the outcome?
    Measures of the magnitude of this association (eg, odds ratio, relative risk, and number needed to harm) are estimates of some unknown “true” value. If one were to repeat the experiment on different samples, they would yield similar, but not identical, results.1 The results of repeat experiments are dispersed above and below the “true” value. This sampling variation is referred to as precision. The principle measure of precision is the 95% confidence interval (CI). The 95% CI quantifies the uncertainty of a measurement by reporting a range of values within which there is a 95% certainty that the true value lies for the entire population.1,2,4 With respect to an odds ratio or relative risk, results are said to be significant if the 95% CI does not include 1.0. If the 95% CI includes 1.0, there is a 95% chance that there is no difference in the outcome between the comparison groups.
Figure 1.
Simple calculations used to analyze the results of clinical trials and cohort studies for risk to patients (ie, patient harm). *The absolute risk increase is the experimental event rate minus the control event rate. *The number needed to harm is 1 divided by the absolute risk increase.
Simple calculations used to analyze the results of clinical trials and cohort studies for risk to patients (ie, patient harm). *The absolute risk increase is the experimental event rate minus the control event rate. *The number needed to harm is 1 divided by the absolute risk increase.
View Original  |  Slide (.ppt)
Figure 1.
Simple calculations used to analyze the results of clinical trials and cohort studies for risk to patients (ie, patient harm). *The absolute risk increase is the experimental event rate minus the control event rate. *The number needed to harm is 1 divided by the absolute risk increase.
Simple calculations used to analyze the results of clinical trials and cohort studies for risk to patients (ie, patient harm). *The absolute risk increase is the experimental event rate minus the control event rate. *The number needed to harm is 1 divided by the absolute risk increase.
View Original  |  Slide (.ppt)
Figure 2.
Clinical scenario.
Clinical scenario.
View Original  |  Slide (.ppt)
Figure 2.
Clinical scenario.
Clinical scenario.
View Original  |  Slide (.ppt)
Figure 3.
Clinical scenario (continued).
Clinical scenario (continued).
View Original  |  Slide (.ppt)
Figure 3.
Clinical scenario (continued).
Clinical scenario (continued).
View Original  |  Slide (.ppt)
Practical Use
Having verified the validity and importance of a study on harm, a physician's ultimate consideration of the research would then become how its results and the authors' conclusions can be applied in practice.1,2 Making an evidence-based decision requires skills that integrate the patient's unique values and circumstances with current and valid evidence. One of the goals of EBM is to assist physicians in developing a system for backing up clinical decisions with sound evidence. Those who criticize EBM presumably focus on the first goal of treating patients according to the evidence. However, the goal of EBM is twofold. The second, and most important, goal of EBM is to shift physicians' focus from the disease to the patient, restoring the patient to the center of the clinical decision-making process—and better aligning the healthcare professions with long-standing osteopathic principles and practice. 
  • Are the study subjects different in any significant way from your specific patient?
    The findings of a valid, important study are considered applicable and relevant if the patient to be treated is similar to those described in the study. As mentioned earlier, such information can be accessed by reviewing the study population characteristics.
  • Based on the study's results, what is your patient's individualized potential for risk and benefit from treatment?
    Physicians can estimate the potential for risk from treatment (ie, adverse effect) for an individual patient versus a study population (Figure 2). For example, if a patient has a medical history that includes one previous MI, the physician may determine that this particular individual is five times more likely than study participants to have a cardiovascular event if prescribed the nonsteroidal anti-inflammatory drug celecoxib— especially once it is known that a previous MI was in the study's exclusion criteria. Or the opposite may be true when the patient's general physical health is much better than study participants (eg, comorbidities), reducing his or her overall risk of cardiovascular events by half (Figure 3). This personalized rate of risk is called the F statistic, and it allows clinicians to personalize a study's NNH for individual patients easily, by dividing the NNH by the F statistic: NNH/f.
    If a study's NNH is 42 (Figure 4), for example, a personalized NNH for the patient could be determined by dividing 42 by 5 for an answer of 8.4. Therefore, according to this theoretical study (N=42), 8 people would need to take celecoxib for 2.8 years to cause one adverse event or complication.
  • What are the patient's concerns and expectations from the treatment?
    When a study is declared a landmark by journalists, its results are irrelevant if incorporating them in practice would violate a patient's preferences, concerns, or expectations. For example, any “wonder drug” would probably be unacceptable to a patient who is completely opposed to the idea of taking medication.
  • Are other alternatives available?
    The physician and the patient, together, should explore the relative safety of alternative treatment options (Figure 4). Alternative choices may include lifestyle modifications and other nonpharmacologic treatment modalities.
Figure 4.
Clinical scenario (continued).
Clinical scenario (continued).
View Original  |  Slide (.ppt)
Figure 4.
Clinical scenario (continued).
Clinical scenario (continued).
View Original  |  Slide (.ppt)
Conclusion
Although most clinicians are already incorporating EBM principles in their practices, often instinctively, some physicians may require a more organized approach to integrating this relatively new model of self-education. Improved comfort levels and true expertise in the practice of EBM are the result of additional education, repetition, and self-assessment. The principles of EBM allow physicians to stay informed while also improving the quality of the information communicated to patients during patient encounters. The systematic approach that is used to appraise an article on harm is but one step in practicing EBM. Remember, the goal is always to provide the best care possible to patients—using one's clinical expertise to address patient values and expectations for treatment. 
  [Editor's note: This article is part 2 of a six-article series intended to introduce the principles of evidence-based medicine (EBM) to busy clinicians, physician residents, and medical students. Because the application of EBM is a career-long process, further training is needed beyond the information provided within this article and series. A foundation of knowledge about research methods is critical in understanding EBM; however, such details, though introduced, are beyond the scope of this series.]
 
Straus SE, Richardson WS, Glasziou P, Haynes RB. Evidence-Based Medicine: How to Practice and Teach EBM. 3rd ed. London: BMJ Books; 2005.
Levine M, Walter S, Lee H, Haines T, Holbrook A, Moyer V; Evidence-Based Medicine Working Group. Users' guides to the medical literature. IV. How to use an article about harm. JAMA. 1994;271:1615-1619.
Greenhalgh T. How to Read a Paper: The Basics of Evidence-Based Medicine. 3rd ed. Malden, Mass: Blackwell Publishing Limited; 2006.
Rosner B. Fundamentals of Biostatistics. 6th ed. Belmont, Calif: Duxbury Press; 2005.
Figure 1.
Simple calculations used to analyze the results of clinical trials and cohort studies for risk to patients (ie, patient harm). *The absolute risk increase is the experimental event rate minus the control event rate. *The number needed to harm is 1 divided by the absolute risk increase.
Simple calculations used to analyze the results of clinical trials and cohort studies for risk to patients (ie, patient harm). *The absolute risk increase is the experimental event rate minus the control event rate. *The number needed to harm is 1 divided by the absolute risk increase.
View Original  |  Slide (.ppt)
Figure 1.
Simple calculations used to analyze the results of clinical trials and cohort studies for risk to patients (ie, patient harm). *The absolute risk increase is the experimental event rate minus the control event rate. *The number needed to harm is 1 divided by the absolute risk increase.
Simple calculations used to analyze the results of clinical trials and cohort studies for risk to patients (ie, patient harm). *The absolute risk increase is the experimental event rate minus the control event rate. *The number needed to harm is 1 divided by the absolute risk increase.
View Original  |  Slide (.ppt)
Figure 2.
Clinical scenario.
Clinical scenario.
View Original  |  Slide (.ppt)
Figure 2.
Clinical scenario.
Clinical scenario.
View Original  |  Slide (.ppt)
Figure 3.
Clinical scenario (continued).
Clinical scenario (continued).
View Original  |  Slide (.ppt)
Figure 3.
Clinical scenario (continued).
Clinical scenario (continued).
View Original  |  Slide (.ppt)
Figure 4.
Clinical scenario (continued).
Clinical scenario (continued).
View Original  |  Slide (.ppt)
Figure 4.
Clinical scenario (continued).
Clinical scenario (continued).
View Original  |  Slide (.ppt)