An analysis of the productivity and impact of clinical PhD theses from the University of Copenhagen

During the past two decades, the annual number of PhD graduates from Faculties of Health Science across all Danish universities has increased by almost 200% [1]. Consequently, questions have been raised about the quality of the PhD and about the motives behind initiating and concluding a PhD training programme – the latter especially among medical doctors [2, 3]. Data from the Danish Ministry of Higher Education and Science revealed that the majority of foreign assessment committee members rated the quality of Danish PhD theses higher than the international standard [4]. Additionally, an evaluation of Danish Health Science PhD theses concluded in 2016 strongly demonstrated that the quality (i.e., journal impact factor) was, indeed, unchanged or slightly improved from 2004 to 2010 [5] despite a substantial increase in the number of PhD graduates. Interestingly, the evaluation showed that approximately 80% of all manuscripts embedded in PhD theses from 2004 and 2010, respectively, were published in peer-reviewed journals, with no difference between these time points [5]. An investigation of the PhD theses concluded in the field of the pharmaceutical sciences reported that 13% of all manuscripts embedded in PhD theses appeared to remain unpublished. Consequently, up to 20% of all research conducted during PhD training programmes in the field of Health Sciences may never be published [5, 6].
It has recurrently been suggested that a substantial part of medical doctors primarily initiate a PhD training programme for strategic motives, i.e. to advance their medical careers and not owing to an interest in clinical research [2, 3]. However, as previous evaluations have investigated Health Sciences in general, it remains somewhat unclear whether PhD graduates within clinical research areas in particular are prone to graduating from PhD training programmes with inadequate scientific outcomes. Therefore, the aim of the present study was to investigate the scientific outcome of PhD theses conducted at the Department of Clinical Medicine, University of Copenhagen, Denmark, in terms of publication activity and scientific impact.

METHODS

The study was initiated and funded by the Graduate School of Health and Medical Sciences and was approved by the Danish Data Protection Agency and the local ethics committee at Faculty of Health and Medical Sciences, University of Copenhagen (504-0116/19-5000).

Data collection

A total of 848 clinical PhD theses were acquired via the University of Copenhagen journaling database (KMD WorkZone) from a full list of PhD graduates from the Department of Clinical Medicine in the 2013-2017 period, regardless of the graduates’ educational background. This list was provided by the Graduate School of the Faculty of Health and Medical Sciences, University of Copenhagen. In the present analysis, only manuscripts and publications that were an integrated part of the theses were included. Additional produced data and publications during the PhD studies, which were not part of the integrated manuscripts, were not included. The study also included theses in which all embedded data were generated without enrolment to the PhD training programme, which is an option described in Section 15.2 of the Danish PhD Order (Section 15.2-theses). The number of Section 15.2 theses was substantially higher among graduates from 2013 compared with the successive years. Two monographic theses were identified and included in the bibliographic analyses of the theses but excluded in analyses of publication status and bibliometrics. Civil registration number (CPR number) determined the gender of the PhD graduates (female/male).

Identification of published manuscripts

Between 21 March and 18 September 2019, three databases (PubMed, Scopus and Google Scholar) were used for identification of the published manuscripts. The database search principles applied were firstly a manuscript search by title and author name in PubMed using field descriptors ([All fields], [Author], [title], etc.) either alone or in combinations using Boolean operators (AND and OR). Secondly, a search was conducted by title and co-author names using the same search procedures. Lastly, if no identification was made in PubMed, the manuscript was searched in Scopus and/or Google Scholar by title and/or author names. If no identification was made in any of the three databases, the manuscript was considered unpublished.
In order to be considered published, manuscripts had to be published in a peer-reviewed journal. Identification of a published manuscript was achieved by comparing methods (in terms of study design, number of participants and measurements), results and main conclusion.

Bibliometric data
In order to evaluate the impact of the PhD theses, both journal and paper bibliometrics were assessed. The SCImago Journal Rank (SJR) and Quartile ranking covered by the Scopus database were applied for evaluation of journal impact. The SJR measures the scientific influence or “prestige” of a journal through the quantity of citations in a given year compared with publications in a three-year publication timeframe. Importantly, the SJR adjusts the influence of these citations based on the importance of the journals from which the citations originated [7-9]. The Quartile ranking divides journals into four quartiles (Q1-Q4) according to their SJR. Q1 represents the highest quarter, whereas Q4 represents the lowest quarter of SJR. Journals are categorised into several specified research fields, which each have a Quartile ranking attached. Regardless of research field, the best Quartile ranking for each journal was used in the present analysis.
Furthermore, the SJR limits self-citations and includes more journals than Impact Factor (IF, Web of Science database) [9]. Consequently, the SJR was chosen as an indicator of journal impact.
Additionally, the Danish Bibliometric Research Indicator (BFI) provided by the Danish Ministry of Higher Education and Science was applied to further evaluate journal impact. BFI ranks the journals in any given research field 1, 2 or 3, with 3 being the best journals in their respective research field, thereby making comparison between research fields easier. Since BFI 3 was not introduced until 2018, BFI ranking was restricted to 1 and 2 in the present study.
To evaluate paper bibliometrics of the published manuscripts from each thesis, the snowball metrics were registered from the Scopus database at the time of observation (from 21 March to 18 September 2019). The data included citations with and without self-citations, citations/year and a Field-Weighted Citation Impact (FWCI). FWCI represents the ratio between the citations of a publication and the average number of citations received by all similar publications indexed in the Scopus database over a three-year period [10]. An FWCI value greater than 1.00 indicates that the publication is more cited than the world average for publications within its research field. Publications lacking journal and/or paper bibliometrics were excluded from the respective analyses.

Statistics

Data are presented as mean ± ± standard deviation with range, or as counts with percentages. Systematic differences between years were analysed using one-way analysis of variance (ANOVA) with post-hoc Tukey’s multiple comparisons test and distributions were evaluated by descriptive statistics (i.e., boxplots). Differences between years in the distribution of categorical data were analysed with a non-parametric Kruskal-Wallis test by ranks with post-hoc Dunn’s multiple comparisons test. Significance was set at an α-level of 0.05. Statistical analyses were performed and figures constructed in GraphPad Prism 8.2.1.

Trial registration:not relevant.

RESULTS

In total, 841 PhD theses (99%) including 2,845 manuscripts (Table 1) were identified, corresponding to an average of 3.4 manuscripts per thesis (Figure 1A). In total, 60% of the theses were written by female graduates, and overall the graduates spent an average of 3.4 years from enrolment to submission of thesis, including leave of absence (Table 1).

At the time of submission, 56% of all manuscripts were published (Figure 1B) and 89% of all theses included at least one published manuscript (Table 1). Interestingly, 10% of the theses included at least one or more manuscripts published prior to enrolment. On average, the theses included 1.9 published manuscripts at submission (Table 1). At the time of observation, this increased to 3.1 published manuscripts (Table 1), and 92% of all manuscripts were published with no differences between the studied years (p = 0.116) (Figure 1B). Only three theses included no published manuscripts. On average, the manuscripts were written by 7.4 authors, with the PhD graduate serving as first author in 97% of the cases (Table 1). Overall, the author status of the PhD graduate was different in six published manuscripts when compared to the draft version embedded in the respective theses.

The mean SJR was 2.016, with a lower mean SJR in 2017 compared with 2015 (p = 0.027), whereas no differences were observed between the other years (Figure 2A). Overall, 98% of the manuscripts were published in journals ranked in the best half of their respective research fields, and there was no difference in this distribution between years (p = 0.472) (Figure 2B). Furthermore, 71% and 29% of the manuscripts were published in journals with an BFI ranking of 1 or 2, respectively (Figure 2C). On average, the published manuscripts were cited 3.9 times/year across the five-year period (3.1 ± 5.9 (0.0-134.4) citations/year excluding self-citations), and this decreased systematically throughout the period (Figure 3A). On average, FWCI was 102% higher than the world average, and it was higher for the published manuscripts submitted by 2013-graduates than for all subsequent years (p < 0.05) (Figure 3B).

DISCUSSION

The main finding was that the publication activity among the clinical PhD graduates from the University of Copenhagen in the 2013-2017 period was very high, as 92% of all manuscripts were published when examined between 21 March and 18 September 2019.
Furthermore, the quality of the research conducted by the clinical PhD graduates was above the world average within their specific research fields, as 82% of all manuscripts were published in journals with Q1 rankings and as the FWCI was between 71% and 175% above the world average.
Compared to previous reports, the publication activity among the clinical PhD graduates was higher than that of Health Science PhD graduates in general [5, 6], which may also be explained by different traditions and practices in the research areas. Nonetheless, this finding together with the observation of a high embedded manuscript publication activity post thesis submission is indicative of a high interest in publication of research among clinical PhD graduates. Indeed, there may also be a change of opinion about achieving a PhD and about publishing, where more emphasis may now be directed towards ensuring that papers are published and not just prepared in draft version. In addition, the results suggest that the graduates were main contributors to the research conducted as they were first authors in 97% of all published manuscripts [11]; and only in six occasions, the author status changed from the manuscript in the thesis to the published manuscript. However, it remains elusive whether the apparent interest in research is driven by a profound research interest and/or career-advancing motives.
In terms of impact, Figure 2A shows that 75% of all manuscripts were published in journals with an SJR between 0.0 and 2.5, with an equal distribution between the studied years. Nonetheless, the highest SJR varied from 13.951 to 21.979, which may indicate that there is some variation in research quality between years. However, based solely on SJR, it is impossible to compare the research quality between research fields. Thus, the best quartile and BFI-ranking systems are important in order to evaluate research quality; and based on the results from the present study, the quality of the research conducted by clinical PhD graduates was high and remained unaltered during the five-year study period. The number of citations decreased systematically from 2013 to 2017, also when self-citations were excluded. However, this is unsurprising, as it is well known that the majority of citations are achieved several years after a manuscript has been published. This provides a negative bias for the manuscripts published in the latter part of the five-year period. Additionally, it is inadequate to evaluate scientific impact solely based on the number of citations. The citation practices differ between research fields, which makes between-field comparisons difficult if not impossible [12]. The FWCI provides a field-weighted citation score, and hence it is a more qualified measure of the scientific impact of articles. As the FWCI of the published manuscripts investigated in the present study is, on average, more than double of the world average, the scientific impact of the clinical PhD theses seems rather solid. The mean FWCI was higher in 2013 than in the successive years; however, as illustrated in Figure 3B, the median is similar between years and hence the difference is mainly driven by a few very high values in 2013.

CONCLUSIONS

Overall, the present findings suggest that the scientific outcome of clinical PhD theses conducted at the University of Copenhagen was high, as 92% of all manuscripts were published in peer-reviewed journals with a fieldweighted journal ranking and citation impact above the world average. This indicates that the strategic motives behind graduation are not compromising the scientific productivity and quality of the theses.

Future research

From the present study it is not possible to evaluate whether the graduates continue publishing new research after obtaining their PhD. However, a previous study reported that approximately 26% of Health Science PhD graduates were still active publishers five years post-graduation [2], and a similar study on the present population is warranted to establish if publication activity is continued.

Correspondence:Tue Rømer. E-mail:tue.roemer@sund.ku.dk
Accepted:24 March 2020
Conflicts of interest: none. Disclosure forms provided by the authors are available with the full text of this article at Ugeskriftet.dk/dmj
Acknowledgements:The authors would like to express their gratitude toMarie Louise Munch (Graduate School of the Faculty of Health and Medical Sciences, University of Copenhagen) for practical coordination and assistance throughout the study and to Marianne Gauffriau (Copenhagen University Library) for guidance during the planning of the data-acquisition phase of the study.
*) Equal contribution.