From:
Berman JJ, Borkowski A, Rachocka H, Moore GW. The impact of unfunded
research in pathology, surgery and medicine.
Southern Medical Journal 88(3):295-299, 1995
THE IMPACT OF UNFUNDED RESEARCH IN MEDICINE, PATHOLOGY, AND SURGERY
Jules J. Berman, PhD, MD, Andrzej Borkowski, MD,
Hanna Rachocka, MD, and G. William Moore, MD, PhD
Grant support: none.
ABSTRACT
The impact of unfunded medical research (i.e., research conducted
with no visible means of support) has received scant attention.
Research contributions from the 10 most cited journals in the
fields of internal medicine, pathology and surgery were counted.
Ten consecutive articles, excluding case reports and review articles,
for the years 1987, 1989 and 1991 from each of ten journals for the
three areas of medicine, were sampled. Unfunded articles accounted
for the majority of contributions (60% of pathology articles, 62% of
internal medicine articles and 74% of surgical articles). In 1987,
funded research articles published received somewhat more citations
(2,961) than unfunded research articles (2,368). Among articles
supported with an NIH grant, the first author of the article was
seldom the grant's principal investigator (38.6%, 26.9% and 16.7%
of funded articles by pathologists, internists and surgeons,
respectively). These results indicate that unfunded research
plays a major role in medical research. (150 words)
INTRODUCTION
Research dollars have not kept pace with the number of grant
applications, and there is now an overall surplus of unfunded
investigators [1,2] Are unfunded researchers productive?
In a prior study, we showed that the majority of research
contributions in the field of pathology are unfunded.
How do medicine departments compare to departments of pathology and
surgery with respect to the influence of unfunded research?
In this report we employed the listings of the Institute for
Scientific Information (ISI) to evaluate the impact of articles
appearing in the most-cited journals in the fields of medicine,
pathology and surgery. The value of articles to the research
community was assessed by comparing the number of citations to
funded and unfunded research articles.
Counting citations is a widely used quantitative measure of the
importance of scientific contributions.[4,5] As in any
attempt to quantitate scientific value, it is flawed. [6]
Although citation counts can be used unfairly to evaluate the
merit of an individual scientist, it may be argued that citation
analysis is a valid device for measuring scientific impact when
comparing categories of researchers within a scientific field,
since factors such as self-citation, scientific marketing, journal
obscurity and reference copying tend to balance out.[7] We feel
that the results obtained from citation analysis relate to the
impact of research on other researchers.
Are funded articles published by funded researchers (principal
investigators)? To determine whether funded articles are authored
by the principal investigator of the listed grant, publicly available
grant listings were searched to determine the name of the principal
investigator of National Institutes of Health (NIH) grants cited. [8]
MATERIALS AND METHODS
We reviewed publications in 10 internal medicine journals, 10 pathology
journals and 10 surgical journals (Table 1). Journals were chosen
from the list of the most cited journals in each area, according to
statistics gathered by the ISI.[9] Ten consecutive research
articles, starting in the month of March in the years 1987, 1989 and
1991, were collected from each journal. In journals with fewer than
20 research articles in the March issue, additional articles were
examined from consecutive issues until a total of twenty articles
was obtained. A "research" article is defined as any journal
contribution other than case reports, review articles, letters
and editorials, which were excluded. An `internist,' `pathologist,'
and `surgeon' are defined here as an author whose affiliation in the
article was listed in the respective departments of medicine, pathology
or surgery. A funded article was defined as an article with a stated
grant number from an outside organization. Granting agencies generally
require grantees to list their grant in any publications resulting from
the research. Articles were searched in both the first page footnotes
and in the last page acknowledgments. Works supported by the institution
where the study was conducted were not counted as funded research
(i.e., the money had to derive from an outside agency grant).
Citations to each article (the number of times that the article
was referenced by any other article) published in 1987 and first-authored
by a member of a medicine, pathology, or surgery department were collected
over the next four years, using the Science Citation Index (ISI).
The year 1987 was chosen, as citations accumulated over the following
4 years (up to September, 1992) could be assembled as a adequate span
for measuring citation frequency.
Articles with the NIH listed as a funding source were examined to
determine the name of the principal investigator for the grant.
NIH funding data were obtained from the Biomedical Index to PHS-Supported
Research.[8]
RESULTS
Among the top 10 peer-reviewed journals in the three major categories
of medicine, there were more unfunded original journal contributions
than funded contributions (Table 2). Overall, unfunded research
contributions accounted for 65.3% of all research contributions.
Unfunded contributions accounted for 62% of internal medicine articles,
60% of pathology articles, and 74% of surgery articles. The proportion
of unfunded articles for each field did not vary statistically in the
period of study (1987 to 1991), by the chi-square contingency test.
Overall, 167(18.6%) articles (of the 900 articles selected) received
NIH funding support (Table 3). NIH-funded articles accounted for
22.3% of internal medicine articles, 23.3% of pathology articles, and
10% of surgery articles. There was no difference in proportion of
NIH-funded articles in the period studied. Among all articles, 5.6%
were first-authored by the person listed as the principal investigator
on the listed grant. First-author NIH-funded articles accounted for
6% of internal medicine articles, 9% of pathology articles, and 1.6%
of surgery articles.
In the field of internal medicine, most citations went to funded
articles. There were 1,916 citations to funded articles compared
with 800 citations to unfunded articles (Table 3). In the fields
of pathology and surgery, unfunded articles accounted for the
majority of citations to the sampled journal articles. There were
967 citations to unfunded pathology articles compared to 664
citations to funded pathology articles. There were 601 citations
to unfunded surgery articles compared to 381 citations to funded
surgery articles. All told, for the three fields of medicine,
there were 2,961 citations to funded articles compared to 2,368
citations to unfunded articles. Therefore, unfunded research
articles accounted for 44% of all citations in the journals
surveyed.
DISCUSSION
From the viewpoint of the medical center administration, unfunded
research is a losing proposition, as it uses resources while
returning nothing to the institution. Unfunded researchers usually
perceive their endeavors as costing little but returning much in
terms of scientific advancement and institutional prestige.
Furthermore, unfunded researchers with active university-based
clinical practices often think that they are in fact paying for
their research with patient care revenues. However, there is no
denying that the contribution of unfunded researchers to medicine
is enormous. Among the top 10 peer-reviewed journals in the three
major categories of medicine, 65.3% of all research contributions
came from unfunded papers. Furthermore, unfunded research articles
accounted for 44% of all the citations. These numbers reflect the
prevalence of unfunded and funded papers in the ten most-cited
journals for the fields of internal medicine, pathology and surgery.
These numbers do not necessarily reflect the overall prevalence of
unfunded papers in the entire literature for these three fields.
Thus, if journals with lower citation rates were included, the
proportion of unfunded papers in these fields might differ from our
findings. Nor do these figures necessarily reflect the prevalence of
funding for investigators in these fields, since there may be a high
rate of funding for investigators in a given field without a resulting
high rate of papers published by funded investigators and appearing in
highly-cited journals. By limiting the study to published papers in
the most-cited journals, issues concerning research productivity
(measured by the number of articles appearing in highly-cited journals)
and the impact of this research (measured by citations to the research)
can be examined.
The results of this study would indicate that the majority of medical
research reports appearing in the most cited research journals are
completed without grant funding. Overall, the majority of the citations
in the scientific medical literature are made to funded research articles
(Table 4). However, in the fields of pathology and of surgery, unfunded
research articles claim the greatest number of citations. In the field
of internal medicine, unfunded research articles receive fewer citations
than do funded research articles, but this probably results from a small
number of funded medical articles that earn a disproportionate number of
citations. In fact, three funded medical articles accounted for 22% of
the total medical citations and 9.6% of the total citations in all fields.
Among the funded research articles, most are not first-authored by the
principal investigator named in the grant that supported the research
endeavor. The term `principal investigator' implies that there is no
other person more involved in the research effort. That being the case,
shouldn't the `principal investigator' be the first author on papers
arising from the research effort? In actuality, the `principal
investigator' often acts as a research manager, coordinating the
activities of other researchers who perform research but who themselves
are not grant holders (i.e., are unfunded). The small number of papers
first-authored by `principal investigators' would suggest that even
funded research is performed by unfunded scientists.
A recent study by Stein et al.[10] reviewing 196 clinical articles,
found that unfunded articles accounted for 23% of the literature
contributions. Although the prevalence of unfunded research in their
study is somewhat less than the prevalence reported in our current study,
both studies support the assertion that unfunded research accounts for a
large portion of journal literature.
Since unfunded research is prevalent and successful (in terms of
citation frequency), what is the importance of research funding?
The answer to this question lies in an understanding of how medical
institutions support research activities.
All research endeavors accrue costs, and the absence of grant funding
for a research project only means that alternative payment mechanisms
must be sought. Federal grant support accounts for the majority of
biomedical research grants in the United States. [11] However, research
support unassociated with investigator-initiated grant awards exists in
many forms, including money obtained through university-based patient
care revenues, charitable contributions and endowments, grants awarded
to an institution rather than to individual investigators, and
industry-supported institution-based contracts.
In addition, investigator-initiated federal grants pay institutions
`indirect costs' to cover services provided by the institution that
cannot be identified directly as line-item costs on the investigator's
grant. Indirect costs may be used for a variety of expenses, such as
library subsumptions and building maintenance costs. Unfunded
researchers enjoy the benefits of grant money when they use equipment
in research laboratories, or enlist the services of librarians, animal
facility personnel and bioengineering shops.
Since unfunded research accounts for the majority of medical research
described in the most cited peer-reviewed medical journals, and since
research contributions to these journals is only rarely contributed by
a first author who is the principal investigator of a grant, it is
tempting to argue that the major benefit of funding dollars is to
provide an infrastructure that supports unfunded research projects.
Unfunded research is a way of life at the National Cancer Institute
(NCI) where budgeted research funds are parceled out to laboratories
with proven records of achievement. At the NCI, peer review consists
of reviewing the accomplishments of groups of researchers rather than
rating the research prospects of individual investigators. Each NCI
laboratory is reviewed by an external group of peers recruited primarily
from universities. The productivity and goals of every researcher in
the laboratory (often consisting of dozens of individual investigators)
are summarized and evaluated. Based in part on recommendations from
the peer reviewers, the intramural research budget is allocated to the
various NCI laboratories.
This system, applied to other academic settings, would allow previously
unfunded but productive researchers to emerge as fully enfranchised
participants in group research efforts. Such a system has been
considered in the past, but has been strongly criticized on the
basis that the distribution of grant funds at the local level would
invite political battles for resource supremacy that the peer review
system is designed to exclude.®MDSU¯12®MDNM¯ The grant review study
sections have, in turn, also been criticized over the years as being
highly political, with one group or another demanding study section
representation more favorable to their own constituency. [13,14]
It would seem only fair that granting agencies take into account the
productivity of unfunded researchers when negotiating an institution's
indirect cost allowances. Institutions generally negotiate with
government agencies to establish indirect cost levels that are applied
to federal grants. Currently, universities may specify particular
indirect costs in Memoranda of Understanding. [15] Since
support of unfunded research is a hidden benefit of indirect costs,
institutions with proven high productivity from unfunded researchers
might receive a monetary incentive for those efforts, possibly via a
comparable mechanism.
In summary, the majority of research contributions in the most-cited
medical journals is contributed by unfunded researchers. Among research
articles supported by NIH grants, the principal investigator for the
research article (i.e., the first author) is seldom the principal
investigator named in the grant. Since all research requires monetary
support, it would seem that a large benefit of research funding is to
provide an infrastructure in which unfunded researchers function.
This would indicate the need for reassessing the relationship between
funding agencies and unfunded researchers.
REFERENCES
1. Palca J: Young investigators at risk. Science 1990;249:351-3
2. Palca J: Researchers declare crisis. Seeking funding solutions.
Science 1990;249:17-18
3. Borkowski A, Berman JJ, Moore GW: Unfunded pathology research:
its frequency and success in the scientific literature. Modern
Pathol 1992;5:577-579
4. Hamilton DP: Publishing by - and for? - the numbers. Science
1990;250:1331-1332
5. Hamilton DP: Research papers. who's uncited now? Science
1991;251:25
6. VonBorstel RC: An analysis of citation analysis. Science
1991;251:1546
7. Seglen PO: Citation frequency and journal impact. Valid
indicators of scientific quality? J Internal Med 1991;229:109-111
8. Biomedical Index to PHS-Supported Research. U.S. Department of
Health and Human Services, Public Health Service, NIH (titled
Research Award Index prior to 1988)
9. Science Citation Index, Journal Citations Report, Subject Category
Listings, Institute for Scientific Information, 1989
10. Stein MD, Rubenstein L, Wachtel TJ: Who pays for published
research?. JAMA 1993;269:781-782
11. Ginzberg E, Dutka AB: The Financing of Biomedical Research.
Baltimore, MD, The Johns Hopkins University Press, 1989
12. Wade N: Peer review system: how to hand out money fairly.
Science 1973;179:158-161
13. Freireich EJ: A study of the status of clinical cancer research
in the United States (1990). J Natl Cancer Inst 1991;83:829-837
14. Silverberg SG. The surgical pathologist as researcher. Am J Clin
Pathol 1981;75(Suppl):453-456
15. Volkers N. Defining the terms of indirect costs. J Natl Cancer
Inst 1991;83:1289®PG¯
Table 1. Journals surveyed for each specialty
Internal Medicine
New England Journal of Medicine
Lancet
Annals of Internal Medicine
Journal of the American Medical Association
Medicine
British Medical Journal
American Journal of Medicine
Archives of Internal Medicine
Journal of Chronic Diseases
Quarterly Journal of Medicine
Pathology
Laboratory Investigation
American Journal of Pathology
Journal of Neuropathology and Experimental Neurology
Journal of Pathology
American Journal of Surgical Pathology
Human Pathology
Histopathology
Journal of Clinical Pathology
Clinical Immunology and Immunopathology
American Journal of Clinical Pathology
Surgery
Transplantation
American Journal of Surgical Pathology
Annals of Surgery
Journal of Neurosurgery
Surgery
Archives of Surgery
Journal of Vascular Surgery
British Journal of Surgery
Annals of Thoracic Surgery
World Journal of Surgery®IP0¯®pg¯
Table 2. Funding for research articles by publication year
and field of medicine
funded unfunded
research research
Internal Medicine
1987 47 53
1989 34 66
1991 33 67
subtotal 114(38%) 186(62%)
Pathology
1987 38 62
1989 42 58
1991 39 61
subtotal 119(40%) 181(60%)
Surgery
1987 29 71
1989 22 78
1991 28 72
subtotal 79(26%) 221(74%)
total 312(35%) 588(65%)®PG¯
Table 3. Research articles funded by U.S. granting agencies
categorized by grant investigator status of the first author
(900 articles sampled).
Number of Number of NIH-funded
NIH-funded articles with first author
articles as Principal Investigator
of grant
Internal Medicine
1987 27 8
1989 20 7
1991 20 3
subtotal 67(22.3%) 18(6%)
Pathology
1987 27* 11
1989 19# 6
1991 24 10
subtotal 70(23.3%) 27(9%)
Surgery
1987 15 2
1989 9 2
1991 6 1
subtotal 30(10%) 5(1.6%)
Total 167(18.6%) 50(5.6%)
*Two articles stating NIH-funding sources were not listed in the
funding directory and are excluded from the table.
#One article stating NIH-funding sources was not listed in the
funding directory and is excluded from the table.
Table 4. Citations to funded and unfunded articles published in 1987.
funded articles unfunded articles
(1987) (1987)
Internal Medicine
total number of articles 47 53
total number of citations 1916 800
average number cited (per article) 40.77 15.1
standard deviation
of the average cited 46.5 13.0
Pathology
total number of articles 40 60
total number of citations 664 967
average number cited (per article) 16.6 15.9
standard deviation
of the average cited 27.1 19.0
Surgery
total number of articles 29 71
total number of citations 381 601
average number cited (per article) 13.1 8.4
standard deviation
of the average cited 12.2 7.9
Total number of citations 2961 2368
Last modified: January 18, 2008