Content area

|

Surgical research remains comic opera and maybe for good reasons

Authors

Jacob Rosenberg1, 2

1) Department of Surgery, Copenhagen University Hospital – Herlev Hospital, 2) Cochrane Colorectal Group

Dan Med J 2021;68(12):A10210792

Abstract

INTRODUCTION The quality of surgical research was criticised 25 years ago in a Lancet commentary, where an analysis showed that papers published in prestigious surgical journals were rarely randomised clinical trials, but typically simple case series. Therefore, the aim of the present paper was to conduct a new publication analysis to establish if this has changed.

METHODS The January issues of the ten surgery and ten general medicine journals with the highest impact factors were analysed. Only original articles were included, and funding and study design were registered.

RESULTS Medicine journals had higher impact factors (median 25.0 (range: 9.0-91.2)) than surgical journals (9.1 (7.0-14.8)) (p = 0.004). More randomised trials and fewer retrospective studies were recorded in medicine than in surgery, and more prospective studies were found among surgery than the medicine publications. Furthermore, funding was more frequently involved in medicine than in surgery papers (94% versus 62%, p < 0.00001).

CONCLUSIONS In the field of surgery, randomised trials remain rare and the field is characterized by more retrospective studies being published than in the field of medicine. The reasons explaining why may be found in knowledge and research traditions, but funding opportunities may also play a role. Furthermore, in clinical trials in surgery, it may be difficult or occasionally impossible to employ a double-blind study design.

FUNDING none.

TRIAL REGISTRATION not relevant.

Twenty-five years ago, Lancet Editor Richard Horton, prepared an interesting commentary entitled “Surgical research or comic opera: questions, but few answers” [1]. Here, he rightfully criticised the field of surgical research for publishing very few randomised controlled trials (RCTs) and pointed out that half of the articles published in nine surgical journals in the January issue of 1996 were case series. Now, 25 years have passed, and it would seem appropriate to revise the current status.

METHODS

The January issues of the surgery journals (category “surgery” in Journal Citation Reports) and the ten general medicine journals (category “medicine, general & internal” in Journal Citation Reports) with the highest impact factors (IF) for 2020 were analysed. The surgical journals were: JAMA Surg (IF = 14.8), Ann Surg (IF = 13.0), J Heart Lung Transplant (IF = 10.2), J Neurol Neurosurg Psychiatry (IF = 10.2), Endoscopy (IF = 10.1), Am J Transplant (IF = 8.1), Dig Endosc (IF = 7.6), Hepatobiliary Surg Nutr (IF = 7.3), Eur J Vasc Endovasc Surg (IF = 7.1), J Hepatobiliary Pancreat Sci (IF = 7.0). The medical journals were: N Engl J Med (IF = 91.2), Lancet (IF = 79.3), JAMA (IF = 56.3), BMJ (IF = 39.9), Ann Intern Med (IF = 25.4), Lancet Digit Health (IF = 24.5), JAMA Intern Med (IF = 21.9), J Cachexia Sarcopenia Muscle (IF = 12.9), PLoS Med (IF = 11.1), J Intern Med (IF = 9.0).

The focus of the present paper was on various types of original research. Therefore, guideline articles, systematic reviews, narrative reviews and papers without an abstract (letters, editorials, etc.) were excluded. Funding was defined as any external funding, and the number of papers receiving specific funding from industry (pharmaceutical or device companies) was registered. Studies were categorised by study design. Thus, data were divided into RCTs, prospective studies (case series, case-control studies, observational studies, Delphi design studies, cohort studies and surveys) and retrospective studies (database studies and retrospective case series).

Impact factors for medical versus surgical journals were compared using the Mann Whitney U test. Data from medical versus surgical papers were compared using the χ2- or Fisher’s exact test, as appropriate. The study was not registered in a public database since it was not a trial, and under Danish law approval from an ethics committee was not required.

RESULTS

The ten most prestigious journals in general medicine had significantly higher impact factors than their surgical counterparts. Thus, the median (range) of the impact factors was 9.1 (7.0-14.8) for the surgical journals and 25.0 (9.0-91.2) for the medicine journals (p = 0.004, Mann Whitney U test). This difference was not due to fewer citable items in the medicine journals, as the total number was 123 in medicine versus 101 in surgery. Thus, the difference in impact factor must be the result of more citations per article.

More RCTs and fewer retrospective studies were registered in medicine than in surgery, and more prospective studies were registered in the surgical than in the medicine journals (Table 1). Furthermore, more “other” funding (e.g. from governmental funding agencies, etc.) and more funding from industry was recorded in medicine than in surgery (Table 1).

DISCUSSION

In the analysis conducted 25 years ago, 7% of the surgical papers were RCTs [1] compared with 10% in the present analysis. In contrast, among papers in medicine journals today, 40% were RCTs. Prospective studies including case series comprised 46% of the surgical papers 25 years ago [1] compared with 54% studies with a prospective study design in surgery and 41% in medicine today. Richard Horton criticised surgical research for relying excessively on non-RCTs such as case series but also noted that case series are inexpensive, quick and easy to perform [1]. Therefore, the present analysis also comprised study funding, and the findings were striking, showing significantly more funding in papers in medicine than in surgery journals. Thus, funding opportunities are much better in medicine research, and this may, at least partly, explain why more RCTs are published in medicine than in surgery.

Surgical research has traditionally used simple study designs such as the prospective cohort or occasionally a case-control design. For some research questions, this will be inadequate, and we do need to be able to conduct large clinical RCTs in surgery. Unfortunately, surgical RCTs are often discontinued, and the most common reasons seem to be poor recruitment and a lack of continued funding [2]. Thus, in surgery, some obstacles need to be addressed. Firstly, the traditional blinded study design may be very difficult or impossible for certain interventions, and the traditional placebo-controlled design carries various special problems in surgery. In some cases, this issue may be overcome by innovative research methods [3], but it cannot always be solved. However, increasing efforts seem to be devoted to conducting blinded placebo-controlled trials in surgery as 66 such trials were published during 2000-2015 period compared with only 19 in the 1950-2000 period [4]. Nevertheless, placebo surgery trials are still characterised by many difficulties as shown recently in a study demonstrating that only few of these trials are completed and published [5]. Secondly, funding is often difficult since traditional surgical research does not involve industry to the same degree as do drug trials. Thirdly, surgical trials often require hundreds or even thousands of patients because of variability in trauma response and ensuing effects on patient outcomes.

Blinding of patients and staff is difficult in surgery. However, years ago, in Denmark we succeeded in conducting the only effectively blinded study of laparoscopic versus open colon resection [6]. The study in question was conducted with a very large postoperative bandage covering the entire abdomen, and therefore neither the patient nor the nursing staff or the inpatient doctors could see what had occurred. The surgical description in the medical record was hidden, and the daily wound inspections were performed by specially dedicated project staff without any involvement with the regular clinical team and also blinded to the patient. The organisation and implementation of this study was complicated, but not undoable. A similar concept was implemented successfully with cholecystectomy [7], but such a design will be extremely difficult to implement in large international multicentre studies comprising thousands of patients. Therefore, the COLOR [8] and COLOR II [9] studies chose not to implement a blinded design. Funding, however, remains a considerable obstacle for many surgical interventional trials, and national or other types of public funding will most probably be necessary to increase overall research quality in surgery in the future.

Research traditions have varied between surgery and medicine, with more focus on (and knowledge about) epidemiology and study design issues in medicine than in surgery. There seems, however, to be a positive trend in this respect as trial registration, disclosure of funding and investigator conflicts have improved markedly over the past ten years in surgical RCTs [10]. Various initiatives are underway, e.g., numerous educational efforts are being made locally in many countries, the Cochrane Collaboration has established review groups focusing on surgical issues [11] and large research collaborations in surgery have been established [12]. The future for research quality in surgery may therefore be bright, provided non-industry funding opportunities may be secured for these studies.

CONCLUSION

In conclusion, surgical research is, indeed, still comic opera. RCTs remain rare, and we rely excessively on retrospective study designs compared with the medical specialties. Reasons may be found in the research traditions, but another significant factor may very well be different funding opportunities, where the medical studies are able to obtain better funding, especially from industry. Furthermore, in surgery, it may be difficult or impossible to blind the patient and investigator to the given treatment. To increase the quality of evidence in surgical research where industry sponsorship may not always be relevant, it will be important to focus on funding for surgery in future national/governmental funding programmes. There should be no doubt that both the desire and the energy needed to raise the quality of research are present within the surgical disciplines.



Correspondence Jacob Rosenberg. E-mail: jacob.rosenberg@regionh.dk
Accepted 12 November 2021
Conflicts of interest none. Disclosure form provided by the author is available with the article at ugeskriftet.dk/dmj
References can be found with the article at ugeskriftet.dk/dmj
Cite this as Dan Med J 2021;68(12):A10210792

Bib ref: 
Dan Med J 2021;68(12):A10210792

References

  1. Horton R. Surgical research or comic opera: questions, but few answers. Lancet 1996;347:984–5.

  2. Chapman SJ, Shelton B, Mahmood H et al. Discontinuation and non-publication of surgical randomised controlled trials: observational study. BMJ 2014;349:g6870.

  3. Wartolowska K, Beard D, Carr A. Blinding in trials of interventional procedures is possible and worthwhile. F1000Res 2017;6:1663.

  4. Wartolowska KA, Beard DJ, Carr AJ. The use of placebos in controlled trials of surgical interventions: a brief history. J R Soc Med 2018;111:177-82.

  5. Bunzli S, Choong E, Shadbolt C et al. Placebo surgery controlled trials: do they achieve what they set out to do? A systematic review. Ann Surg 2021;273:1102-7.

  6. Basse L, Jakobsen DH, Bardram L et al. Functional recovery after open versus laparoscopic colonic resection: a randomized, blinded study. Ann Surg 2005;241:416-23.

  7. Johansson M, Thune A, Nelvin L et al. Randomized clinical trial of open versus laparoscopic cholecystectomy in the treatment of acute cholecystitis. Br J Surg 2005;92:44-9.

  8. Deijen CL, Vasmel JE, de Lange-de Klerk ESM et al. Ten-year outcomes of a randomised trial of laparoscopic versus open surgery for colon cancer. Surg Endosc 2017;31:2607-15.

  9. Bonjer HJ, Deijen CL, Abis GA et al. A randomized trial of laparoscopic versus open surgery for rectal cancer. N Engl J Med 2015;372:1324-32.

  10. Helliwell JA, Shelton B, Mahmood H et al. Transparency in surgical randomized clinical trials: cross-sectional observational study. BJS Open 2020;4:977-84.

  11. Cochrane Colorectal Group. https://colorectal.cochrane.org (20 Oct 2021).

  12. Rosenberg J, Angenete E, Pinkney T et al. Collaboration in colorectal surgical research. Colorectal Dis 2021;23:2741-9.

💬 0 Comments

Add new comment

Right side