Online First Open Access
Public Health and Primary Care  |   January 2021
An analysis of research quality underlying IDSA clinical practice guidelines: a cross-sectional study
Author Affiliations & Notes
  • Benjamin Greiner, DO, MPH
    Department of Internal Medicine, University of Texas Medical Branch, Galveston, TX, USA
  • Ryan Essex, OMS III, BS
    Oklahoma State University Center for Health Sciences, College of Osteopathic Medicine, Tulsa, OK, USA
  • Denna Wheeler, PhD
    Oklahoma State University Center for Health Sciences, College of Osteopathic Medicine, Tulsa, OK, USA
  • Corresponding author: Benjamin Greiner, DO, MPH, Department of Internal Medicine, University of Texas Medical Branch, 400 Harborside Drive, Galveston, TX, 77550, USA, E-mail: ben.greiner10@gmail.com  
Article Information
Public Health and Primary Care   |   January 2021
An analysis of research quality underlying IDSA clinical practice guidelines: a cross-sectional study
The Journal of the American Osteopathic Association Published Online First on January 18, 2021. doi:https://doi.org/10.1515/jom-2020-0081
The Journal of the American Osteopathic Association Published Online First on January 18, 2021. doi:https://doi.org/10.1515/jom-2020-0081
Abstract

Context: As a result of new developments in medicine, the need for evidence-based clinical practice guidelines (CPG) is of utmost importance. However, studies have shown that many medical societies are using low quality research to develop CPGs.

Objectives: To evaluate the quality of research underlying the CPGs issued by the Infectious Diseases Society of America (IDSA).

Methods: We examined 29 CPGs issued between January 1, 2012 and December 31, 2019 and classified each by research quality according to levels reported by the CPG authors and previously specified by the IDSA: Levels I through III, corresponding to high, moderate, and low quality of evidence, respectively. Each ranking was cross-checked with a second researcher to improve inter-rater reliability. To analyze evolution of research quality over time, three updated CPGs were randomly selected and compared to their original versions. Chi-square analysis was then performed to determine statistical significance.

Results: We evaluated the quality of research for 2,920 recommendations within the 29 CPGs that met our criteria and found that 418 (14%) were developed using high-quality (Level I) research from randomized, controlled trials. Of the remaining recommendations, 928 (32%) were based on moderate quality research (observational studies) and 1574 (54%) on low quality research (expert opinion). A Pearson chi-squared analysis indicated no-statistically significant difference between original guidelines or their subsequent updates for Clostridium difficile2=0.323; n=85; degrees of freedom [df]=2; p=0.851), candidiasis (χ2=4.133; n=195; df=2; p=0.127), or coccidiomycosis (χ2=0.531; n=95; df=1; p=0.466).

Conclusions: The proportion of high-quality research underlying guideline recommendations is remarkably low, indicating that moderate and low quality evidence is still influencing infectious disease guidelines despite IDSA standards. Moreover, the quality of research has not significantly changed over time. IDSA CPGs are a formidable source of information for clinicians, but an increased number of quality studies should be utilized to further guide CPG development.

Clinical practice guidelines (CPG) function to relay rapidly expanding medical research information to clinicians and aid in clinical decision making by summarizing key scientific evidence.1,2,3 CPGs can help minimize adverse patient outcomes, decrease the use of unnecessary and expensive diagnostic tests and procedures, and increase consistency of care between different populations of patients.4,5,6 In response to new developments in medicine and the need for evidence-based clinical guidelines to assist physicians, the Infectious Diseases Society of America (IDSA) partnered with the Institute of Medicine in 2011 to develop a handbook focused on more critical appraisal of literature used in CPG development.7,8 The new standards outlined in the handbook mandated that CPGs be developed using the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach, the most widely used technique for developing trustworthy CPGs.1,7,8 Interest in the evaluation of guidelines has grown in recent years.9,10,11 One study12 conducted prior to publication of the IDSA handbook found that only 14% of CPGs published before 2011 were developed using Level I quality evidence.12 In this study, we aimed to evaluate the quality of research supporting recommendations from the IDSA clinical guidelines, since the improved guideline development standards have been in place for nearly a decade. Our primary outcome was to determine the proportion of CPGs (and recommendations given within them) developed using Level I research. Our secondary outcome was to analyze the evolution of research quality underlying CPGs over time. 
Methods
This study was conducted in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines.13 Recommendations within CPGs were analyzed for strength and quality of research, as reported by the CPG authors, following GRADE criteria.8 Strength of recommendations, research quality, and publication year were tracked using an Excel spreadsheet (Microsoft, Inc.). CPGs met inclusion criteria for this study if they were documented as “current,” developed between January 1, 2012 and December 31, 2019, and reported on the IDSA website. CPGs were excluded if they did not pertain specifically to infectious disease principles. The strength of each recommendation within each CPG was classified by a process previously utilized for IDSA guidelines, in coordination with the GRADE approach: Grade Levels A, B, and C corresponded to strong, moderate, and weak recommendation strength.14 The quality of research supporting each recommendation was also specified in previous IDSA guidelines with GRADE Level I, II, and III rankings, corresponding with high, moderate, and low quality of evidence. Each recommendation was first extracted by one researcher (B.G.) and then cross-checked by a second researcher (R.E.) to improve inter-rater reliability. The proportions of high-, moderate-, and low-quality evidence as well as strength of recommendations were then analyzed for each CPG. The evolution of research quality underlying CPGs was then assessed by tabulating the proportion of recommendations guided by Level I, II, and III research in three of the most updated CPGs on Clostridium difficile, candidiasis, and coccidiomycosis. These were randomly selected using Microsoft Excel randomization. A Pearson chi-squared analysis was then constructed to compare the quality of research underlying original vs. updated recommendations. Statistical analysis was performed using SPSS (IBM Corp.) on April 8, 2020. 
Results
Of 31 total CPGs, 29 CPGs involving 2,920 recommendations met inclusion criteria. Figure 1 compares the quality of recommendations for all guidelines. There was 98.4% inter-rater reliability with 100% agreement on all evidence rankings after consensus meetings were conducted. Four hundred and eighteen (14.3%) of the recommendations were developed using Level I quality data (Table 1). Of the remaining recommendations, 928 (31.8%) were based on Level II data and 1,574 (53.9%) on Level III data. Next, we compared the proportion of evidence quality in the high strength recommendation category (Level A). Three hundred and fifty two (84.2%) recommendations of Level I quality were associated with Level A strength, while 620 (35.9%) and 756 (43.8%) of Level II and III quality research, respectively, were associated with Level A strength recommendations. 
Figure 1:
Comparison of guidelines by quality of evidence. Comparison of 29 guidelines using percentage distribution of quality of evidence underlying individual recommendations. ASP, antibiotic stewardship program; HIV, human immunodeficiency virus; OI, opportunistic infection.
Figure 1:
Comparison of guidelines by quality of evidence. Comparison of 29 guidelines using percentage distribution of quality of evidence underlying individual recommendations. ASP, antibiotic stewardship program; HIV, human immunodeficiency virus; OI, opportunistic infection.
Table 1:
Strength and quality of research underlying 29 IDSA clinical practice guidelines.
Evidence quality level I Evidence quality level II Evidence quality level III Total
Strength level A 352 620 756 1,728
Strength level B 52 211 357 620
Strength level C 14 97 461 572
Total 418 928 1,574 2,920
Table 1:
Strength and quality of research underlying 29 IDSA clinical practice guidelines.
Evidence quality level I Evidence quality level II Evidence quality level III Total
Strength level A 352 620 756 1,728
Strength level B 52 211 357 620
Strength level C 14 97 461 572
Total 418 928 1,574 2,920
×
Finally, we considered the evolution of three randomly selected guidelines to assess the changes in proportion of Levels I, II, and III quality research over time. All three updated guidelines had either no change or a decrease in Level I research over time (Figure 2). A Pearson chi-squared analysis indicated no statistically significant difference between original or updates guidelines for C. difficile2=0.323; n=85; degree of freedom [df=2]; p=0.851), candidiasis2=4.133; n=195; df=2; p=0.127), or coccidiomycosis2=0.531; n=95; df=1; p=0.466). The total number of recommendations in the CPGs increased for all guidelines over time. Candidiasis recommendations increased from 54 to 141 (261%) while recommendations in the coccidiomycosis and C. difficile CPGs increased from 26 to 69 (265%) and 40 to 45 (113%), respectively. 
Figure 2:
Evolution of guidelines and the underlying quality of evidence. Comparison of three current guidelines with respective previous versions. The total number of recommendations is depicted within each subdivision.
Figure 2:
Evolution of guidelines and the underlying quality of evidence. Comparison of three current guidelines with respective previous versions. The total number of recommendations is depicted within each subdivision.
Discussion
This study showed that 14% of recommendations in IDSA CPGs were based on Level I research, which is unchanged from the 14% reported in 2011.12 The proportion of guidelines developed using Level II research increased since that time (928 [32%], up from 31%), while the proportion of Level III research based guidelines decreased (1,574 [54%] down from 55%). Thus, the quality of research underlying current CPG recommendations has not changed since the IDSA adopted the GRADE approach to evaluating research. Additionally, we found that Level I quality evidence more strongly influenced Level A strength recommendations than lower quality evidence. We found that 84% of Level I quality recommendations provided Level A recommendation strength, while 36% of Level II and 44% of Level III quality recommendations resulted in Level A strength. Considering this, we concluded that randomized controlled trials had a greater impact on the strength of recommendations being offered. 
In assessing the evolution of guidelines over time, we found no statistically significant difference between quality of research underlying recommendations in any of the original or updated guidelines, while the total number of individual recommendations increased between 113 and 265%. Thus, although updated CPGs now carry a higher total number of recommendations, the quality of research underlying those recommendations is not improving. The minimal increase in randomized controlled trials since 2011 may be partially attributed to physician and patient barriers to trial recruitment. Multiple studies have found that recruitment to RCTs is increasingly difficult due to patient concerns for complexity and stringency of protocols, discomfort with randomization, and potential side effects.15,16 Physician barriers to recruitment were most commonly listed as time constraints, difficulty following study procedures, and lack of staff and training.16,17 Other reasons may include lack of pharmaceutical industry funding for antibiotic research due to poor profit margins and fewer grant opportunities through government entities in RCT trials. Our findings were consistent with those evaluated in other specialties that reported low proportions of Level I evidence used in their CPGs, such as cardiology (11%), gastroenterology (15%), and emergency medicine (9%).9,11,18 However, the proportion of IDSA CPG recommendations supported by Level I evidence was markedly lower than those reported for obstetrics and gynecology (30%) and pediatrics (43%).19,20 
This study had several limitations. First, our analysis was based upon the evidence grading scale provided by each CPG which, inevitably, is highly dependent on the reviewer. We recognize that this could have been overcome by evaluating the primary research underlying each recommendation; however, this limitation was somewhat attenuated by our use of two researchers independently evaluating the guidelines and cross-checking results for accuracy. Also, many of the guidelines we evaluated were developed several years ago and may not reflect the volume or quality of research published after the guideline development. Our study may therefore underestimate the quality of research supporting new and updated guideline recommendations. Finally, we recognize that some of individual recommendations may not be suitable for analysis by randomized controlled trials due to concerns for public safety of withholding treatments as necessitated by a control or subjecting patients to experimental arms that will undoubtedly cause harm. 
Conclusions
The proportion of high-quality research used in developing guideline recommendations is low, which suggests that low- and moderate-quality research is being used to develop IDSA recommendations. Although the number of recommendations has increased over time, the proportion of high quality research used in recommendation development has not changed significantly. CPGs should be taken in context with clinical experience, as they are neither all-inclusive nor definitive; that collective and customized approach is particularly evident in the tenets of osteopathic medicine. Future studies should focus on developing randomized controlled trials and strong observational studies to overcome the continued reliance of CPG developers on lower-level evidence and expert opinion. 
  Research funding: None reported.
 
  Author contributions: All authors provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; all authors drafted the article or revised it critically for important intellectual content; all authors gave final approval of the version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
 
  Competing interests: Authors state no conflict of interest.
 
References
Steinberg, E, Greenfield, S, Wolman, M, Mancher, M, Graham, R. Clinical practice guidelines we can trust. 2011; National Academies Press. https://www.awmf.org/fileadmin/user_upload/Leitlinien/International/IOM_CPG_lang_2011.pdf. Accessed July 10, 2020.
Lohr, KN, Field, MJ. (Eds.). Guidelines for clinical practice: from development to use. 1992; National Academies Press. http://guidelines-registry.cn/uploadfile/2016/0914/20160914112724303.pdf. Accessed July 10, 2020.
Grol, R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med care. 2001; 39(8), II-46. https://www.jstor.org/stable/pdf/3767643.pdf?casa_token=xOPog-Wk2_8AAAAA:FCxDp6jXRwuwwpn0PEDWKoTrLJo04jJt6CXFWnZZHFazK_LsZPRSSAYoukV4i67kyyUrFADn8HVEvvBzJ0g874wjJxHfyLWaAu4rwymJH-656Y9fQTM. Accessed July 10, 2020.
Woolf, SH, Grol, R, Hutchinson, A, Eccles, M, Grimshaw, J. Potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318(7182):527-530. doi: 10.1136/bmj.318.7182.527.
Chassin, MR. Practice guidelines: best hope for quality improvement in the 1990s. J occup med. 1990; 32(12), 1199-1206. doi: 10.1097/00043764-199012000-00015.
Audet, AM, Greenfield, S, Field, M. Medical practice guidelines: current activities and future directions. Ann Intern Med. 1990;30:709-714. doi: 10.7326/0003-4819-113-9-709.
IDSA Standards and Practice Guidelines Committee. Clinical practice guidelines development: training and resources. 2019. Retrieved from https://www.idsociety.org/practice-guideline/clinical-practice-guidelines-development-training-and-resources/. Accessed July 28, 2019.
Atkins, D, Best, D, Briss, PA, Eccles, M, Falck-Ytter, Y, Flottorp, S, GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ (Clinical research ed.). 2004; 328(7454), 1490. doi: 10.1136/bmj.328.7454.1490.
Meyer, C, Bowers, A, Wayant, C, Checketts, J, Scott, J, Musuvathy, S, Vassar, M. Scientific evidence underlying the American College of Gastroenterology’s clinical practice guidelines. PloS One. 2018; 13(10), e0204720. doi: 10.1371/journal.pone.0204720.
Fanaroff, AC, Califf, RM, Windecker, S, Smith, SC, Lopes, RD. Levels of evidence supporting American College of Cardiology/American Heart Association and European Society of Cardiology guidelines, 2008-2018. JAMA. 2019; 321(11), 1069-1080. doi: 10.1001/jama.2019.1122.
Venkatesh, AK, Savage, D, Sandefur, B, Bernard, KR, Rothenberg, C, Schuur, JD. Systematic review of emergency medicine clinical practice guidelines: implications for research and policy. PloS one. 2017; 12(6), e0178456. doi: 10.1371/journal.pone.0178456.
Lee, DH, Vielemeyer, O. Analysis of Overall Level of Evidence Behind Infectious Diseases Society of America Practice Guidelines. Arch Intern Med. 2011; 171(1):18-22. doi: 10.1001/archinternmed.2010.482.
Cuschieri, S. The STROBE Guidelines. Saudi Journal of Anaesthesia. 2019; 13(1): S31. doi: 10.4103/sja.SJA_543_18.
McDonald, LC, Gerding, D, Johnson, S, et al.. Clinical Practice Guidelines for Clostridium difficile Infection in Adults and Children: 2017 Update by the Infectious Diseases Society of America (IDSA) and Society for Healthcare Epidemiology of America (SHEA), Clin Infect Dis. 2018; Volume 66, Issue 7, Pages e1–e48. doi: 10.1093/cid/cix1085.
Mills, EJ, Seely, D, Rachlis, B, Griffith, L, Wu, P, Wilson, K, Wright, JR. Barriers to participation in clinical trials of cancer: a meta-analysis and systematic review of patient-reported factors. Lancet Oncol. 2006; 7(2), 141-148. doi: 10.1016/S1470-2045(06)70576-9.
Ross, S, Grant, A, Counsell, C, Gillespie, W, Russell, I, Prescott, R. Barriers to participation in randomised controlled trials: a systematic review. J Clin Epidemiol. 1999; 52(12), 1143-1156. doi: 10.1016/s0895-4356(99)00141-9.
Spaar, A, Frey, M, Turk, A, Karrer, W, Puhan, MA. Recruitment barriers in a randomized controlled trial from the physicians’ perspective–A postal survey. BMC Med Res Methodol. 2009; 9(1), 14. doi: 10.1186/1471-2288-9-14.
Tricoci, P, Allen, JM, Kramer, JM, Califf, RM, Smith, SC. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA. 2009; 301(8), 831-841. doi: 10.1001/jama.2009.205.
Wright, JD, Pawar, N, Gonzalez, JS, Lewin, SN, Burke, WM, Simpson, LL, Herzog, TJ. Scientific evidence underlying the American College of Obstetricians and Gynecologists’ practice bulletins. Obstet Gynecol. 2011; 118(3), 505-512. doi: 10.1097/AOG.0b013e3182267f43.
Isaac, A, Saginur, M, Hartling, L, Robinson, JL. Quality of reporting and evidence in American Academy of Pediatrics guidelines. Pediatrics. 2013; 131(4), 732-738. doi: 10.1542/peds.2012-2027.
Figure 1:
Comparison of guidelines by quality of evidence. Comparison of 29 guidelines using percentage distribution of quality of evidence underlying individual recommendations. ASP, antibiotic stewardship program; HIV, human immunodeficiency virus; OI, opportunistic infection.
Figure 1:
Comparison of guidelines by quality of evidence. Comparison of 29 guidelines using percentage distribution of quality of evidence underlying individual recommendations. ASP, antibiotic stewardship program; HIV, human immunodeficiency virus; OI, opportunistic infection.
Figure 2:
Evolution of guidelines and the underlying quality of evidence. Comparison of three current guidelines with respective previous versions. The total number of recommendations is depicted within each subdivision.
Figure 2:
Evolution of guidelines and the underlying quality of evidence. Comparison of three current guidelines with respective previous versions. The total number of recommendations is depicted within each subdivision.
Table 1:
Strength and quality of research underlying 29 IDSA clinical practice guidelines.
Evidence quality level I Evidence quality level II Evidence quality level III Total
Strength level A 352 620 756 1,728
Strength level B 52 211 357 620
Strength level C 14 97 461 572
Total 418 928 1,574 2,920
Table 1:
Strength and quality of research underlying 29 IDSA clinical practice guidelines.
Evidence quality level I Evidence quality level II Evidence quality level III Total
Strength level A 352 620 756 1,728
Strength level B 52 211 357 620
Strength level C 14 97 461 572
Total 418 928 1,574 2,920
×