Were subjects randomly assigned to the treatment group?

Randomization is a pivotal step in determining the validity of a study.³ It ensures that each subject has the same probability of being selected for active treatment protocols rather than for a control treatment or placebo. It also allows study results to be generalized to a larger population of interest. Randomization can be as simple as “flipping a coin” or using a random number generator.

The strengths and weaknesses of each article must be evaluated independently. If clinical practice recommendations based on RCTs are not available, physicians may choose to look at the results of nonrandomized studies. Causation (“X causes Y”) cannot be established using observational studies (eg, case control study, cohort study), however. In most instances, only an association between a therapeutic intervention and desired outcome can be interpreted from the results of observational studies.⁴

Were all the subjects accounted for and attributed at the end of the study?

All enrolled subjects must be accounted for at the end of the investigation. A large “lost to follow-up” group may lead researchers to present biased study results. For example, if very ill subjects do not complete a study and are not ultimately accounted for, the study's outcome may appear favorable when a more thorough analysis of the data gathered may have led researchers to very different conclusions.

All study participants should be analyzed in the groups to which they were originally assigned. This principle is called “intention-to-treat.”⁵ This research model allows group randomization to be preserved, and the known and unknown factors affecting patient prognosis have an equal probability of impacting subjects assigned to each study group. Researchers who conduct their analysis excluding very ill subjects would obtain results that suggest the therapy was efficacious—though those study results would be compromised through bias.

Was the study “blinded”?

Study participants, clinicians, and investigators should not know which subjects are assigned to the intervention or control group. Participants who are aware of their group assignments may bias study results by behaving and/or responding differently. Clinicians who are aware of subject assignments may treat individuals in the intervention group differently than those in the control group, unconsciously (or consciously) manipulating the study design or analysis.

Double-blinding refers to the processes of keeping group assignments concealed from study subjects and investigators. Sometimes, however, a double-blinded research protocol is simply not possible given the intervention used. For example, a study that investigates the efficacy of surgical vs nonsurgical procedures is unable to conceal group assignments from the subjects or surgeons. When possible, however, all measures of progress and improvement for such studies should be concealed from primary investigators through the use of independent evaluators.

Were the study groups similar at the start of the investigation?

The demographics and description of study participants are usually found in the first table of an article. There are always some established risk factors that may affect the study outcome. Therefore, it is important to determine whether these factors are equally balanced between the intervention and control groups. If one wanted to determine the overall benefit of a surgical procedure, it would be important to know if other comorbid conditions are balanced between the two groups (eg, coronary heart disease, diabetes mellitus). Randomization does not always guarantee an equal balance of demographic factors and medical history between groups. If the difference of a variable (eg, age) is large, it may bias study results. It is important for clinical investigations to have a sufficient number of subjects (sample size) so that the results would be able to find a desired difference in the outcome (sufficient power).⁶ Small studies have a greater probability of having an unequal distribution of baseline subject characteristics. At times, however, a deficiency of this kind may be overcome using appropriate statistical tools (eg, regression analyses).

Were the study groups treated equally?

As one might imagine, if subjects in the intervention and control groups were treated differently, maintaining blinding protocols would be difficult. For example, if subjects in the intervention group had more frequent follow-up visits than other study participants, they may inadvertently be “unmasked” to group assignments—suddenly aware that they are receiving the intervention. Data gathered from a study in which the intervention group receives more frequent follow-up visits may also skew data by virtue of the fact that subjects in such a group would have more opportunities to report adverse events (ie, significant between-groups differences). It is important that study groups are treated equally in all aspects of the study (Figure 2).

How big was the treatment effect?

Once it has been determined that a particular article on therapy is valid, the physician should evaluate the magnitude of the treatment effect and its precision. Only basic mathematical and statistical skills are required for this kind of postpublication review and analysis of the medical literature.

Most journal articles report outcomes in a dichotomous fashion. For example, one may chose to evaluate whether or not daily use of aspirin prolongs life 2 years after an initial myocardial infarction. Therefore, a researcher might then compare an event of interest (eg, mortality) among those who received aspirin and those who received nothing (or placebo). The proportion of those who died in the placebo group determine what is called the control event rate (CER). The CER is considered the baseline risk for patients who meet study inclusion and exclusion criteria. The proportion of study subjects in the intervention group who died determines the experimental event rate (EER). The truncated table shown in Figure 3 provides a hypothetical example of just one possible study outcome (ie, mortality ≤2 y postinfarction).

Although the difference as reported in the table between the EER and CER may appear to be statistically significant, the data presented does not provide us with any clinically useful information.

Numerical terms that can be applied to our patients and allow us to explain potential outcomes are needed. It is for this reason that many articles report the relative risk reduction (RRR). When the EER is subtracted from the CER and that total is then divided by the CER, the result is the RRR.⁷ To elaborate using the example provided by the data in the table (Figure 3), the RRR=[0.15-0.05]/0.15=0.67 or 67%. A physician reading this number in a medical journal can then safely say that aspirin, relative to placebo, decreased patients' risk of death in the 2 years after an initial myocardial infarction.

Although this finding appears to be impressive since it confers a large treatment effect, it still conveys incomplete information to the reader because it does not attempt to evaluate patients' baseline risk (ie, CER) of death during the 24 months postinfarction. One cannot discriminate large treatment effects from small ones. Therefore, for example, with a postinfarction CER of 0.00015% and an EER of 0.00005%, the RRR will still be 67%. Because the baseline risk (ie, CER) is small, a further decrease in risk will have only minimal clinical impact. It is for this reason that the RRR is not the best calculation to use in clinical practice.

The absolute risk reduction (ARR), which is calculated by subtracting the EER from the CER, takes the baseline risk into account.⁷ Using the same example (Figure 3), the ARR would be calculated by subtracting the 0.05 EER from the 0.15 CER for an ARR of 0.10 or 10%. This analysis would lead physicians practicing EBM to reach a very different conclusion from physicians who consider only the RRR of 67%.

At times, it is difficult to recall an ARR value. In addition, physicians want to convey the information available to their patients in a manner that is easy for them to understand. For these purposes, the number needed to treat (NNT) can be calculated. The NNT is computed by taking the reciprocal of the AAR, or dividing 1 by the ARR.¹ To calculate the NNT for the study reported in the table (Figure 3), one would divide 1 by 0.10 for an NNT total of 10. In other words, 10 patients need to take 81 g of aspirin daily for 2 years postinfarction to prevent one mortality.

Physicians must then decide whether an NNT of 10 is clinically significant or remarkable. This determination can be made by comparing the number to other NNTs for interventions with a similar therapy duration. The disease itself and the severity of the outcome must also be taken into consideration. For example, one may be willing to administer a particular therapy when an outcome is severe (eg, NNT of 50 at 1 year of treatment for cancer). Yet, with the same NNT, one may be reluctant to prescribe an antibiotic to manage a mild upper respiratory infection when it is known that the medication would shorten the symptomatic phase of the illness by only 1 or 2 days (Figure 4).

Can I apply these results to my patient?

Physicians practicing EBM can answer this question by determining if his or her patient meets inclusion and exclusion criteria of the study and if the patient has any comorbidities that would bias or contraindicate the desired outcomes of the investigation. One should be able to generalize the results of a study to a particular patient most times—but this should never be taken for granted. The key is not to be overly stringent with study inclusion or exclusion criteria, but rather, to determine if there are any compelling reasons that one should not generalize the results to the patient under consideration.

Were “disease-oriented” or “patient-oriented” outcomes considered?

It is important to assess the types of outcomes studied. Many times researchers strive to improve markers of disease (eg, systolic blood pressure, pulse rate) rather than patient-oriented outcomes (eg, mortality rates, pain resolution). In our example with aspirin therapy, the reduction in deaths is an important clinical outcome. If the outcome was related only to platelet counts (a disease-oriented outcome), researchers would not have informed their readers about any significant clinical outcomes.

What are the benefits versus costs of this treatment?

Patients may procrastinate in starting a new treatment due to concerns about potential adverse events and financial costs. Deleterious effects of therapies must be considered in every clinical decision, and they should be addressed directly and collaboratively with patients. For example, although taking aspirin may decrease overall mortality rates postinfarction, the potential for an increase in gastrointestinal or cerebral bleeding rates must also be considered, especially in conjunction with patient comorbidities.