Free
Special Communication  |   August 2007
Evidence-Based Medicine, Part 5. An Introduction to Critical Appraisal of Articles on Prognosis
Author Notes
  • From the Department of Family Medicine at the University of North Texas Health Science Center—Texas College of Osteopathic Medicine in Fort Worth. 
  • Address correspondence to Roberto Cardarelli, DO, MPH, Department of Family Medicine, Texas College of Osteopathic Medicine, University of North Texas Health Science Center, 855 Montgomery St, Patient Care Center, 2nd Fl, Fort Worth, TX 76107-2553.E-mail: rcardare@hsc.unt.edu 
Article Information
Evidence-Based Medicine
Special Communication   |   August 2007
Evidence-Based Medicine, Part 5. An Introduction to Critical Appraisal of Articles on Prognosis
The Journal of the American Osteopathic Association, August 2007, Vol. 107, 315-319. doi:10.7556/jaoa.2007.107.8.315
The Journal of the American Osteopathic Association, August 2007, Vol. 107, 315-319. doi:10.7556/jaoa.2007.107.8.315
Abstract

This article provides an introductory step-by-step process to appraise an article on prognosis. The authors introduce these principles using a systematic approach and case-based format. The process of assessing the validity of an article on prognosis, determining its importance, and applying it to an individual patient is reviewed. The concepts of study population homogeneity, sufficient follow-up periods, and completeness are discussed to help physicians determine an article's validity. Instruction on how to evaluate confidence intervals is provided. Finally, information that is learned from the previously mentioned steps is applied to patient care. Study generalizability and the role of patient values, expectations, and concerns are also addressed. The skills learned from appraising an article on prognosis in the manner outlined provides a solid basis for life-long learning and improved patient care.

After patients receive a diagnosis, a common question they ask their physicians is, “What is my prognosis?” Clear prognoses are important for patient care and counseling, but they also guide the physician and patient in selecting the most effective treatment. This complex process requires physicians to connect certain clinical predictors of disease and associated comorbidities to the temporal outcome of that disease.1 
However, not every physician can confidently provide appropriate prognoses in terms of the evidence available in the most current medical literature. For most physicians, prognostic skills develop with practice and experience and alongside clinical research. A framework exists in which any physician can master the art of making a prognosis and helping patients make informed decisions on the basis of available evidence provided by the current literature, the use of one's clinical expertise, and the unique circumstances relevant to a patient. This approach to medicine, called evidence-based medicine (EBM), is a way for physicians to accelerate their effectiveness in practicing various clinical skills, such as prognosis.2,3 Applying EBM to prognoses requires that physicians thoroughly and critically review the current pertinent literature to verify the accuracy and validity of the evidence and its applicability to a specific clinical case. 
In this article, we introduce a strategy for busy physicians, physician residents, and medical students to critically assess the medical literature on prognosis. In-depth details of research methods are beyond the scope of this introductory series on EBM. Readers are encouraged to seek further training on these topics with supplemental learning opportunities and continuing medical education. Finally, the clinical scenario described has been simplified to provide readers with an illustrative example for the general concepts introduced. 
Searching the Evidence
Choosing a high quality article to evaluate is a critical first step in practicing EBM. The preferred articles to support prognoses are systematic evidence-based reviews of studies of the specific disease or disorder in peer-reviewed journals. Unfortunately, systematic reviews on prognosis are sometimes hard to find. In such instances, individual research articles can be used—with caution. 
Often the best study designs for prognostic articles are cohort studies (Figure 1). In such studies, investigators identify a group of individuals with a previously specified characteristic of interest and follow up with them over a predetermined period of time. The results of the group with the disease of interest are then compared with normal subjects who do not have the disease.2-4 Other study designs, such as randomized controlled trials (RCTs) and case-control studies, have important roles in research and are introduced elsewhere in this series.5-8 
Figure 1.
Clinical scenario.
Figure 1.
Clinical scenario.
Validity of Articles on Prognosis
The process of critically appraising an article on prognosis begins with evaluating the quality of information provided in the selected article. To ascertain the validity of an article, physicians need to determine not only if the methods used to arrive at the conclusions were free of error and bias but also if the study's results and conclusions were accurately deduced. Physicians may use the following questions2 to help them determine the validity of an article on prognosis: 
  • Are the study subjects truly representative of the population of interest?
    A study of an entire population with a specific disease or disorder would be impractical. Instead, studies analyze a representative sample of a patient population and generalize the results to a larger population. The representative sample used in a study can have a direct impact on the usefulness of the results. For example, if the study's population has substantially different characteristics (eg, age, race or ethnicity, comorbidity) compared with a clinician's patient, the applicability of the results in this particular instance becomes questionable. Therefore, a study population may not be completely representative of the general population or a particular clinician's patients.
  • Were the study subjects chosen at a common point in the disease course?
    It is important to determine the stage of the disease that researchers began to evaluate the study's participants because the stage of the disease can impact whether the results are valid and applicable to a specific patient. For example, if the subjects in a study are all in an advanced stage of the disease (eg, New York Heart Association [NYHA] class III or IV for congestive heart failure [CHF]9), a physician whose patient is in an earlier stage of the disease (eg, NYHA class I or II for CHF) might incorrectly deduce that his or her patient has a small amount of time before the patient will experience an expected endpoint. To assess the applicability of the article regarding the disease stage of recruited study participants, physicians should look at the article's abstract and its “Methods” section (Figure 2).
  • Was the follow-up period sufficiently long and complete? Follow-up in a valid study should be sufficiently long and complete. Short follow-up periods may allow too little time for an outcome of interest to occur. The appropriate length of the study is dependent on the study question, the intervention used, the outcomes of interest, and any special circumstances (eg, funding).
    A clinical investigation is considered complete when all study participants are accounted for at the study's completion. Subjects who dropped out of the study early because of adverse effects or death should be included and analyzed with their original group assignment. This is called an intention-to-treat analysis.
    Failure to perform an intention-to-treat analysis may direct researchers to provide readers with misleading results. The “5-and-20 rule” can be used by a critical reader to evaluate a study's completeness. If less than 5% of the study population is lost to follow-up, one can be assured that the loss minimally impacted the results. If, however, more than 20% of the study population is lost to follow-up, caution is advised when making clinical decisions based on study findings. An attrition rate of 5% to 20%—and its impact on the researchers' results— must be determined by the reader based on other specifics of the study (Figure 3). For example, a study design that is complex with high participant burden or a long follow-up period can contribute to a large number of patients lost to follow-up.
  • Were the study subjects and investigators blinded to the measures of interest?
    Assessment should also be blinded to prevent observer bias, which occurs when the researcher unconsciously (or consciously) looks harder for outcomes in the study group than in the control group. Blinding means that the researcher is unaware of the participants' group assignment. Double-blinding means that study participants are also masked to group allocation. In prognosis articles, which are often presented as cohort studies, clinicians are usually single-blinded. In other words, clinicians are blinded to the method by which the outcome is measured, but not to the characteristics of the patients involved in the study (Figure 4).
  • Were adjustments made for important prognostic factors? Studies on prognosis often stratify the study group into cohorts based on comorbid conditions that influence prognostic outcomes.10 These factors can include the stage of disease or other diseases, such as coronary artery disease, that may affect the prognosis. In addition, factors that may confound the prognostic outcomes, such as age, sex, and environmental risk factors (eg, living conditions, pollution) must be accounted for. It is important to make sure that the authors of any study have identified factors applicable to their study's participants and determined if any of these factors influenced the predictive value of their conclusions (Figure 5).
Figure 2.
Clinical scenario (continued).
Figure 2.
Clinical scenario (continued).
Figure 3.
Clinical scenario (continued).
Figure 3.
Clinical scenario (continued).
Study Results
After the quality of a study on prognosis has been ascertained, physicians must determine what the results are. Evaluating the endpoints and projected outcomes can help physicians analyze the results and determine their importance. 
  • How likely are the projected outcomes over time?
    This question deals with the common question most physicians receive from their patients: “What are my chances of having another event?” Valid and accurate studies allow physicians to respond to this question quickly and within a reasonable amount of confidence. Often these findings can be located in the “Results” section of an article. One caveat that must be considered is the study endpoint. As presented in Figure 6, the endpoint can be defined as (1) an emergency department visit, (2) hospitalization as a result of CHF, or (3) death.
    Many times, studies provide a more stratified endpoint, such as only death caused by CHF, which may make the prognosis results more clinically applicable by allowing the clinician to provide information that is specific to the patient's concern. Moreover, composite endpoints vary in the severity of the outcome, making the results less clinically applicable. For example, an emergency department visit is substantially different from death.
     
    • What is the precision of the relationship between the end-point and the outcome?
      Odds ratios and relative risks are estimates of some unknown “true” value. If one were to repeat the study with different subjects, it would yield similar, but not identical, results.2 The results of repeat studies are dispersed above and below the “true” value. This sampling variation is referred to as precision. The principle measure of precision is the 95% confidence interval (CI). The 95% CI quantifies the uncertainty of a measurement by reporting a range of values within which we can be 95% certain that the true value lies for the entire population.2,11 Results are said to be significant if the 95% CI does not include 1.0. If the 95% CI includes 1.0, there is a 95% chance that there is no difference in the outcome between the comparison groups (Figure 7).
Figure 4.
Clinical scenario (continued).
Figure 4.
Clinical scenario (continued).
Figure 5.
Clinical scenario (continued).
Figure 5.
Clinical scenario (continued).
Figure 6.
Clinical scenario (continued).
Figure 6.
Clinical scenario (continued).
Practical Use
Having verified the validity and importance of a study on prognosis, a physician's ultimate consideration of the research would then become how the study's results and the authors' conclusions can be applied in practice.2 Making an evidence-based decision requires skills that integrate the patient's unique values and circumstances with current and valid evidence. One of the goals of EBM is to assist physicians in developing a system for backing up clinical decisions with sound evidence. Those who criticize EBM presumably focus on the first goal of treating patients according to the evidence. However, the goal of EBM is twofold. The second, and most important, goal of EBM is to shift physicians' focus from the disease to the patient, restoring the patient to the center of the clinical decision-making process—and better aligning the healthcare professions with longstanding osteopathic principles and practice. 
  • Are the study subjects substantially different from your specific patient?
    The findings of a valid, important study are considered applicable if the patient to be treated is similar to those described in the study. As mentioned earlier, such information can be accessed by reviewing the study population characteristics.
  • How important is the evidence for your specific patient?
    If the evidence from the study shows that the prognosis for the patient is disastrous without treatment, then this information should be presented to and discussed with the patient (Figure 8). The risks and benefits must be weighed while taking into account the patient's special circumstances (eg, financial concerns, health insurance coverage, and personal wishes). Any evidence becomes moot if the patient is not informed or willing to undergo further diagnostic tests or treatment.
Figure 7.
Clinical scenario (continued).
Figure 7.
Clinical scenario (continued).
Figure 8.
Clinical scenario (continued).
Figure 8.
Clinical scenario (continued).
Conclusion
Although most clinicians are already incorporating EBM principles in their practices, often instinctively, some physicians may require a more organized approach to integrating this relatively new model of self-education. Improved comfort levels and true expertise in the practice of EBM are the result of additional education, repetition, and self-assessment. The principles of EBM allow physicians to stay informed while also improving the quality of the information communicated to patients during patient encounters. The systematic approach that is used to appraise an article on prognosis is but one step in practicing EBM. Remember, the goal is always to provide the best care possible to patients—using one's clinical expertise to address patient values and expectations for treatment. 
  [Editor's note: This article is part 5 of a six-article series intended to introduce the principles of evidence-based medicine (EBM) to busy clinicians, physician residents, and medical students. Because the application of EBM is a career-long process, further training is needed beyond the information provided within this article and series. A foundation of knowledge about research methods is critical in understanding EBM; however, such details, though introduced, are beyond the scope of this series.]
 
Hagland M. Medicare & Medicaid. A case against cookie-cutter care. Hosp Health Netw. . (1997). ;71: 78 ,80.
Straus SE, Richardson WS, Glasziou P, Haynes RB.Evidence-Based Medicine: How to Practice and Teach EBM. . 3rd ed. London, England: Churchill Livingstone; (2005). .
Laupacis A, Wells G, Richardson WS, Tugwell P. Users' guides to the medical literature. V. How to use an article about prognosis. Evidence-Based Medicine Working Group. JAMA. . (1994). ;272: 234-237.
Rothman KJ, Greenland S. Causation and causal inference. In: Rothman KJ, Greenland S, eds. Modern Epidemiology. . 2nd ed. Boston, Mass: Lippincott Williams & Willkins; (1998). : 7-28.
Cardarelli R, Virgilio RF, Taylor L. Evidence-based medicine, part 2. An introduction to critical appraisal of articles on therapy. J Am Osteopath Assoc. 2007;107:299-303. Available at: http://www.jaoa.org/cgi/content/full/107/8/299.
Schranz DA, Dunn MA. Evidence-based medicine, part 3. An introduction to critical appraisal of articles on diagnosis. J Am Osteopath Assoc. 2007; 107:304-309. Available at: http://www.jaoa.org/cgi/content/full/107/8/304.
Cardarelli R, Seater MM. Evidence-based medicine, part 4. An introduction to critical appraisal of articles on harm. J Am Osteopath Assoc. 2007;107:310-314. Available at: http://www.jaoa.org/cgi/content/full/107/8/310
Sanderlin BW, AbdulRahim N. Evidence-based medicine, part 6. An introduction to critical appraisal of clinical practice guidelines. J Am Osteopath Assoc. 2007;107:321-324. Available at: http://www.jaoa.org/cgi/content/full/107/8/321.
The Stages of Heart Failure—NYHA Classification. Heart Failure Society of America Web site. 2002. Available at: http://www.abouthf.org/questions_stages.htm. Accessed August 8, 2007.
Hemingway H. Prognosis research: why is Dr Lydgate still waiting? J Clin Epidemiol. 2006;59:1229-1238.
Rosner B. Fundamentals of Biostatistics.6th ed. Belmont, Calif: Duxbury Press; 2006.
Figure 1.
Clinical scenario.
Figure 1.
Clinical scenario.
Figure 2.
Clinical scenario (continued).
Figure 2.
Clinical scenario (continued).
Figure 3.
Clinical scenario (continued).
Figure 3.
Clinical scenario (continued).
Figure 4.
Clinical scenario (continued).
Figure 4.
Clinical scenario (continued).
Figure 5.
Clinical scenario (continued).
Figure 5.
Clinical scenario (continued).
Figure 6.
Clinical scenario (continued).
Figure 6.
Clinical scenario (continued).
Figure 7.
Clinical scenario (continued).
Figure 7.
Clinical scenario (continued).
Figure 8.
Clinical scenario (continued).
Figure 8.
Clinical scenario (continued).