Abstract
Context: During the past 20 years, colleges of osteopathic medicine (COMs) have made several advances in research that have substantially improved the osteopathic medical profession and the health of the US population. Furthering the understanding of research at COMs, particularly the factors influencing the attainment of extramural funds, is highly warranted and coincides with the missions of most COMs and national osteopathic organizations.
Objectives: To describe bibliometric measures (numbers of peer-reviewed publications [ie, published articles] and citations of these publications, impact indices) at COMs from 2006 through 2010 and to examine statistical associations between these measures and the amount of National Institutes of Health (NIH) research funds awarded to COMs in 2006 and 2010.
Methods: A customized, systematic search of the Web of Science database was used to obtain bibliometric measures for 28 COMs. For the analyses, the bibliometric measures were summed or averaged over a 5-year period (2006 through 2010). The NIH database was used to obtain the amount of NIH funds for research grants and contracts received by the 28 COMs. Bivariate and multivariate statistical procedures were used to explore relationships between bibliometric measures and NIH funding amounts.
Results: The COMs with 2010 NIH funding, compared with COMs without NIH funding, had greater numbers of publications and citations and higher yearly average impact indices. Funding from the NIH in 2006 and 2010 was positively and significantly correlated with the numbers of publications, citations, and citations per publication and impact indices. The regression analysis indicated that 63.2% and 38.5% of the total variance in 2010 NIH funding explained by the model (adjusted R2=0.74) was accounted for by 2006 NIH funding and the combined bibliometric (ie, publications plus citations), respectively.
Conclusion: Greater scholarly output leads to the procurement of more NIH funds for research at COMs.
Colleges of osteopathic medicine (COMs) have made great strides in research during the past 20 years.
1,2 The total amount of research funding secured by COMs in 2004 was approximately $101.7 million, up from $16.6 million in 1989.
1,2 (When adjusted for inflation in 2010 dollars, these totals are $117.4 million and $29.2 million, respectively.) The American Osteopathic Association (AOA), recognizing that support and encouragement of quality scientific research is critical for the osteopathic medical profession, plays a vital role at the start of the funding process. To stimulate scholarly activity, the AOA Council on Research (formerly the Bureau of Research) provides modest support—often termed “seed grants”—to investigators and trainees at COMs to help them gain visibility in the broader medical community and within federal funding agencies such as the National Institutes of Health (NIH).
3 Rose and Prozialeck
4 reported that awards made by the AOA Bureau of Research between 1995 and 2001 helped grantees secure an additional $5.7 million in extramural funds.
The NIH was the primary funder of research at COMs—as it is for most allopathic medical schools—accounting for $60.4 million of $101.7 million (59%) of the total funds. This amount is more than 7 times that of the next identifiable contributor (“Other Federal”), which gave $8.5 million.
1,5 On a yearly basis, the NIH allocates billions of dollars for health research and typically provides nearly half of its funds for research at US medical schools.
4 In 2009, however, of the approximately $11 billion given to medical schools, only approximately $135 million (1.2%) was granted to COMs
4; COMs accounted for approximately 20% of all medical schools at that time. Given the importance of NIH funding to research at COMs but the disproportionate amount awarded to COMs, an investigation of the factors related to the securing of NIH funding by COMs is highly warranted.
Previous assessments of research productivity at COMs
1,2 have been limited in number and scope; they have focused primarily on describing the characteristics of funded research projects, such as the amount and number of grants. To our knowledge, no systematically conducted scientific reports are available regarding correlates or predictors of research funding at COMs, although Prozialeck
7 offered anecdotal and speculative suggestions about factors that influence research productivity by COM faculty. Furthermore, some researchers
8,9 have underscored the weaknesses (eg, inadequate use of standard statistical methods) of
U.S. News & World Report, the most commonly cited and referenced ranking of the nation's colleges, which may lead to biased outcomes. For example, every COM in the research medical school rankings
10 is listed as “Rank Not Published” or “Unranked” either because they are in the bottom quarter of the research medical schools or because the school did not supply the
Report with enough key statistical data to be numerically ranked.
11
Bibliometrics is an array of methods that examines the influence of research areas, researchers, or research products (eg, journal articles) in a given field of study.
12-14 These methods may, for example, comprise counting total publications (ie, published articles) or counting the average number of citations per publications. Bibliometric measures are used to calculate journal impact factors, develop benchmarks, and coordinate research activities.
12,15,16 Although the object of some criticism (eg, self-citation bias), these measures nevertheless accurately depict scholarly communication patterns, correlate with peer-review ratings, predict emerging fields of research, show disciplinary influences, and map various types of collaboration.
17-20 Germane to a primary motivation for the current study, bibliometric measures appear to be important for informing funding decisions. A majority (64%)
21 of NIH grants result in an article in a mainstream scientific journal, and the number of articles produced is directly related to the amount of NIH awards made to the medical schools. Other studies
5,22,23 also have shown strong linear relationships between bibliometric measures and funding for research. An evaluation of bibliometric measures may provide insights into NIH funding at COMs, especially if several measures derived from sound methodologic approaches are integrated.
5
The objectives of the present study were to determine bibliometric measures at COMs and to examine associations between measures generated between 2006 and 2010 and the amount of research funding COMs received from the NIH in the fiscal years 2006 and 2010. The current study is important because it focuses on bibliometric measures from COMs, which, to our knowledge, have not been previously published. Thus, we hypothesize that the data on bibliometrics could establish benchmarks for more productive research at COMs and thus help COMs improve their ability to procure NIH funds.
All COMs that had inaugural classes in 2006 or before were included in this study (N=28). Institution establishment dates ranged from 1891 to 2005, and 22 of 28 (79%) were privately financed institutions. In 2010, the average enrollment was 450 students and the total number of full-time faculty was 1932.
24
On June 3, 2011, we used the Web of Science database to obtain information about bibliometric measures at COMs. The database indexes more than 12,000 of the world's top-cited journals in all disciplines and provides details regarding their citation data.
25 We analyzed several search strategies and results for each institution in the address field. We searched using a combination of the schools' proper names, common name abbreviations, parent institution names, departmental affiliations, zip codes, and city names to retrieve all relevant data. After searching for research at an institution, we refined the results by 2 criteria available within the “Refine Results” feature. First, we limited the results to 2 document types—articles and reviews—thus excluding such items as letters, proceedings papers, meeting abstracts, editorial materials, notes, news items, reprints, and corrections. After the document type refinement, we limited the results to 2006 through 2010, the publication years within the scope of this study. The use of this 5-year period diminished the possibility of a single year skewing the results, and it allowed up to a 5-year lag between publications and funding.
22
We used the Web of Science database to generate a citation report of 3 bibliometric measures for each year from 2006 through 2010 at each COM: total number of peer-reviewed, published articles (ie, publications); total number of citations to publications (ie, citations); and number of publications with no citations. From these measures we calculated institutional percentages of articles with no citations, the number of citations per publication, and the impact index. The impact index has been described in detail elsewhere.
5,26 It characterizes, by institution, the number of publications with a high number of citations relative to all publications at the same institution. The index was computed as follows:
For this equation,
h is the h-index, or the number of publications (
h) from an institution that have been cited by others at least
h times, and
p is the total number of publications per institution raised to the power law or master curve exponent (
m). Molinari and Molinari
26 set
m at 0.4, deriving that value from the universal growth rate for citations over time for large numbers of publications.
Numbers of publications, citations, and publications with 0 citations were summated to yield 5-year totals (2006 through 2010) for these variables. The average value over the same 5-year period was used for citations per publication, impact index, and institutional percentages of publications with 0 citations. We obtained the information about NIH funding of COMs from a database maintained by the US Department of Health and Human Services (
http://report.nih.gov/).
Descriptive statistics are presented as mean (standard deviation) and sums, if applicable. All 2006 dollar amounts were adjusted for inflation (plus 8.16%) to reflect 2010 dollar values. Prior to data analyses, variable distributions were examined for normality. Deviations from normality (eg, skewness statistics ⩾2.0) were log or square root transformed for use in analyses requiring normality of data. In all cases, the transformation procedure reduced skewness statistics to values of less than 2.0. Multivariate analysis of variance was conducted to compare bibliometric measures between COMs with and COMs without NIH funding in 2010. Bivariate relationships among variables were examined by means of the Pearson product moment correlation. A multiple linear regression model was constructed to determine if bibliometric measures resulting from 5 years of scholarly activity (2006 through 2010) at COMs predict 2010 NIH funding. The independent variables were school type (private=0, public=1), 2006 NIH funding, and a combined bibliometric. The combined bibliometric was calculated as the number of publications from 2006 through 2010 plus citations to these publications (ie, publications plus citations). The variable was created because publications and citations could not exist as separate variables in the same regression model because of their high correlation (
r=0.99) and subsequent problems with multicolinearity.
27 All statistical analyses were conducted using SPSS statistical software (version 17.1; SPSS Inc, Chicago, Illinois) with α set a priori at .05 and with COM (n=28) as the unit of analysis.
Table 1 lists the full and the abbreviated names for all COMs included in this study.
Table 2 contains descriptive statistics for the bibliometric measures summed or averaged for the 5-year period studied and the amount of NIH funding in 2006 and 2010 for the 28 COMs. Over the course of 5 years, COMs with NIH funding in 2010 averaged 60 more publications (
P<.005) and 395 more citations (
P<.001) than COMs without NIH funding; the number of citations per publication, however, did not differ significantly. Funded COMs, compared with nonfunded COMs, produced more publications that were not cited (
P<.01); however, the funded COMs and the nonfunded COMs did so at similar rates, with approximately one-fourth of the publications not being cited regardless of whether they came from funded or nonfunded COMs. The yearly average impact index was 43% higher at COMs with 2010 NIH funding compared with COMs without 2010 NIH funding (
P<.01).
Bivariate correlations among study variables are described by the Pearson product moment correlation coefficients given in
Table 3. As expected, several statistically significant correlations were observed among the bibliometric measures. For example, the numbers of publications and citations were almost perfectly correlated (
r=0.99;
P<.001), and both were directly and statistically significantly related to the impact index and articles with 0 citations. When citations and publications with 0 citations were made relative to the number of publications, their correlations with publications became statistically insignificant. The impact index was the only indicator significantly correlated with all other bibliometric measures. Impact index values were directly related to the number of publications, citations, citations per publication, and publications with 0 citations but inversely associated with the percentage of publications with 0 citations. All bibliometric measures except the percentage of publications with 0 citations were positively and statistically significantly related to 2006 and 2010 NIH funding. Higher levels of NIH funding also were related to status as a public COM.
Results of the stepwise multiple regression analysis are presented in
Table 4. Higher amounts of NIH funding in 2006 and a higher combined bibliometric (ie, publications plus citations) were statistically significantly associated with higher levels of 2010 NIH funding (
P<.005). These 2 independent variables explained 73.6% of the total variance in 2010 NIH funding after adjusting for the standard error. Funding from the NIH in 2006 and combined bibliometric measures accounted for 63.2% and 38.5% of the total variance explained by the model, respectively. School type had a statistically insignificant association with 2010 NIH funding (
P=.65).
The bibliometric data for the 5 years examined in the present study are provided in
Table 5. The 28 COMs are sorted in ascending order by 2010 NIH funding amounts. Approximately $10.5 million of $13.1 million (79%) of the total 2010 NIH funds were accounted for by the first 5 COMs. Likewise, the 5 COMs—despite representing only 18% of COMs—were responsible for disproportionately larger percentages of all publications (699 of 1843 [37.9%]) and citations (4260 of 10,041 [42.4%]). Of the COMs with 2010 NIH funding, 11 of 12 (91.7%) had more than 50 publications between 2006 and 2010. In contrast, only 4 of 16 COMs (25%) in the bottom half of the funding list had more than 50 publications during this period.
In this study, we sought to describe bibliometric data for COMs and to explore their relationships with NIH funding. The current study is unique in its focus on COMs and use of a retrospective, longitudinal study design to determine whether bibliometric measures predict NIH funding. Our primary finding was that previous NIH funding and publishing peer-reviewed journal articles that are cited enhances a COM's success in obtaining NIH support for research. Other noteworthy results include the statistically significant and positive correlations between 2006 NIH funding and 2006 to 2010 scholarly productivity (eg, number of publications); relationships among bibliometric measures; and the clear display of heterogeneity (ie, variance) of NIH and bibliometric data across the 28 COMs examined.
The relationships among bibliometric measures found in this study coincide with those reported by others.
5,28,29 For example, the correlation coefficient between publication and citation counts was
r=0.99 in the present study,
r=0.98 in Hendrix,
5 and
r=0.89 in van Raan.
28 Likewise, uniform findings exist regarding the statistically insignificant association between publication counts and the percentage of publications not cited (correlation coefficient range
r=−0.26 to
r=0.35).
5,28 Druss and Marcus
30 indicated that each NIH R01 grant produced on average 7.6 MEDLINE publications, and Rose and Prozialeck
4 showed that approximately 65% of projects at COMs funded by the AOA result in publications. The results of the present study confirm these findings and those of other investigators
5,22,23,30,31 who have consistently demonstrated that funding is correlated with future scholarship. The current study, however, provides additional insight.
According to the multiple regression analysis, the additive effects of publications and citations also predict NIH funding amounts. Because citations are a good indicator of publication quality,
23 it appears that the best strategy for COMs to garner more NIH funds would be to optimize the balance between the number of publications and the quality of those publications. Results of regression analysis also reveal that a COM might potentially accrue NIH funding based on scholarly output alone. In other words, a track record of obtaining NIH funding (in this study, having obtained NIH funding 5 years previously in 2006 vs 2010) may not be a necessary part of the formula for securing future NIH funding. The finding of a causal relationship between publications plus citations and NIH funding supports the notion that grant application reviews at the NIH are influenced by bibliometric measures. Indeed, reviewers for federal agencies such as the NIH have been found to provide more favorable reviews to senior principal investigators with stronger scholarly records resulting from publishing results of important (ie, high-quality) findings.
32 Furthermore, publications and citations provide greater visibility for scholars and attest to their ability to carry a project through to the dissemination phase.
23,33 According to Ramsden,
34 it is absolutely necessary for academics to have prior publications to be successful in obtaining research grants.
To our knowledge, the present study is the first to provide a detailed description of bibliometric measures and NIH funding for COMs. The 28 COMs averaged $435,000 and $469,000 in 2006 and 2010 NIH funding, respectively. Twelve of the 26 COMs (46.2%) received NIH funding in 2010. On average, each COM produced a total of 13.2 publications and 71.7 citations per year from 2006 through 2010. In comparison, allopathic medical schools (n=123) were awarded, on average, $78 million each year in NIH funding between 1997 and 2005, with 952 publications and 16,288 citations per year between 1997 and 2007; US dental schools were awarded an average of $4.9 million per year in NIH funds between 2005 and 2009.
5,35 These considerable differences are probably the result of several simultaneously active factors. One obvious institutional factor is faculty size. In 2010, COMs averaged 69 full-time faculty compared with an average of 1027 at allopathic medical schools, and this disparity appears to be widening.
1,24,36,37
Comparing allopathic schools with COMs also yields a much higher average citation rate per article (14 vs 6) and impact indices (3.2 vs 2.0).
5 These differences were likely less related to the number of faculty and more to quality issues at the institutional and faculty levels. According to Clearfield et al,
1 an important barrier to scholarly activity at COMs is the inadequate or suboptimal organizational infrastructure or the lack of human resources to cover academic responsibilities not related to research (eg, administration, classroom instruction, service on university committees).The existence of these barriers would reduce the time available for research.
38-40 Ultimately, this could dampen motivation or momentum and foster a negative research culture, resulting in comparatively low levels of scholarship.
7,41 Other barriers to scholarly output include the private status of COMs—which often limits resources and emphasizes training osteopathic physicians—and a long-standing funding bias, whereby COMs do not receive an adequate proportion of the federal monies for infrastructure and research programs.
6,42
A number of actions could be taken to overcome some of the barriers to scholarly productivity at COMs. Investment in research is a proven method to stimulate extramural grant procurement. An investment of $69 million at 1 medical school for basic science research garnered $99.7 million in extramural funding.
43 The importance of this investment is accentuated by the considerable proportion of faculty (up to 55%) who conduct basic science research, because basic science faculty tend to be more focused on research compared with other faculty types, especially clinical faculty.
44 Not surprisingly, when compared with clinical faculty, basic science faculty produce more publications per year (3.7 vs 2.1), receive higher impact scores for their publications (6.7 vs 4.4), and obtain more funding for research (5-fold higher funding amounts).
44 Another approach would be to devote more funds to the development of research initiatives focused on the study of osteopathic manipulative medicine. A prime example of success in this area is The Osteopathic Research Center, which was established with an initial investment of $1.1 million. During the first 4 years of operation, The Osteopathic Research Center was awarded $3.2 million in NIH grants, which it then used to conduct comprehensive studies of osteopathic manipulative medicine, resulting in its faculty and staff generating 35 full-length publications and 30 conference posters or presentations.
45
Scholarly output also can be enhanced through writing interventions. In a review, McGrail et al
46 observed that writing interventions—such as finding support groups, attending courses, and working with coaches—stimulate the publication of research findings. Several studies in this review
46 reported that in addition to improving the number of publications, writing interventions also positively affected the quality of the publications. Other benefits of writing interventions include the formation of multidisciplinary collaborations, which leads to more authors contributing to a manuscript (ie, more publications overall) and the creation of a more attractive work environment, which is important for recruiting research faculty.
46 The former benefit, more authors, could have a substantial impact on manuscript quality and thus funding procurement.
21
There are several limitations that should be considered when interpreting the results of the current study. First, because of the nature of Web of Science's data set and capabilities, research was excluded if linked to institution names that were misspelled or that used unfamiliar variants in the address field. Moreover, citation errors occur in all bibliographic fields in Web of Science; thus, these citations were not accounted for. As stated in the Methods section, no fraction or proportional attribution techniques were applied in the case of multiple authors from different medical schools. Because its coverage is selective, Web of Science does not track the citation histories of thousands of journals, proceedings, technical reports, and patents. Additionally, the collected Web of Science data include self-citations, which may skew the resulting data. Other bibliometric studies
47,48 demonstrated that the inclusion of self-citations proved insignificant to the overall results in macro-level studies. Moreover, all of these potential errors are likely to be randomized across COMs and therefore should not substantially affect the quality of the statistical findings.
The NIH was the only source of grant-funding data used in this study. Monies from private foundations, non-governmental organizations, and other government departments were not taken into account. There is a strong likelihood that there is some base of research funding at the COMs not associated with NIH funding, such as intramural or foundation grants.
31 Further, only NIH funding awarded in 2006 was used to represent past NIH funding history. It is possible that considering NIH funding history from different perspectives (eg, continuous years used, total number of years used) could alter the outcomes.
Because we conducted our analyses at the institutional level, an assortment of bibliometric measures were examined.
5,49 Although this formed a more comprehensive and accurate picture of research at COMs than previously available, other measures—especially qualitative measures such as opinions or perceptions—might have been included.
5 McAllister and Narin
31 found a statistically significant relationship between size-dependent measures, such as number of publications, and faculty perceptions of institutional quality. Further, other factors beyond the methodologic scope (eg, the US economy and its impact on NIH budgets and funding decisions) of this study may have mediated the effects of scholarly activity on NIH funding.
The findings in the current study attest to the need for COMs to improve resources and infrastructure, which are necessary to augment the dissemination of research findings in quality, peer-reviewed journals. This contention aligns with standards ratified by the AOA Commission on Osteopathic College Accreditation, which challenges COMs to contribute more to science and medicine by scaling up their research efforts.
50 Improvements in scholarly quality would also address widely held negative opinions of COMs being overly reliant on tuition to finance operating expenses rather than research.
51 The osteopathic medical profession, and COMs in particular, could ostensibly lead the way with scientific evidence to establish the safety, efficacy, and effectiveness of spinal manipulation in treating low back pain and other musculoskeletal conditions.
51 This would dovetail with the results of the 2007 National Health Interview Survey, which reported that 14.3 million adults use complementary and alternative medicine and 36% of adults had received spinal manipulation within the past year.
51 Further, because of the philosophical ideals that form the basis of osteopathic medicine, COMs are in an advantageous position to contribute substantial research-generated knowledge toward preventing chronic diseases, such as childhood obesity.
52