Skip to main content

Reply to “Correspondence on ‘cover letters written by ChatGPT-4 or humans’“

Can Deniz Deveci, Jason Joe Baker, Binyamin Sikander & Jacob Rosenberg

18. dec. 2023
3 min.

Initially, it is important to mention that the results of the subjective evaluation of readability did not show that GPT-4 performed better than humans. Rather, the evaluation showed that GTP-4 letters were non-inferior to human-written cover letters, meaning that GTP-4 performed as well as humans, which was what the study aimed to explore. We agree that authors of cover letters bear the final responsibility for any draft written by AI, which was one of the reasons why we adopted a non-inferior design.

The sample size was, as always, a potential limitation, but sample size was determined by way of the sample size calculation, which was made from a thin base of assumptions since the study was one of the first of its kind, and we had no other research that had made similar comparisons that we might lean towards. We also acknowledge that increasing the number of assessors per cover letter could, in theory, contribute to more narrow effect estimates of the results. However, as in any study, we underlined by practical constraints and had to find independent researchers with medical and PhD degrees who would voluntarily spend their free time assessing these cover letters and completing questionnaires. Furthermore, it is important to clarify that we did not include only three assessors for the entire study. Rather, we engaged eight assessors and ensured that each cover letter was assessed by three different individuals.

We agree that other parameters, such as content and formatting, would have been relevant. However, our questionnaire was designed to evaluate impression, readability, criteria satisfaction and details that indirectly addressed these aspects. For example, criteria satisfaction included adherence to content standards, and readability and impression are influenced by the formatting and presentation of the cover letters. We thought that the included parameters were highly relevant, but others may choose to include or exclude different parameters.

Lastly, it is worth noting that future studies comparing other AI systems would be of great interest. However, that would have been a different study objective, and one should consider whether it would be relevant when ChatGPT-4 writes cover letters just as well as human authors who will always be responsible for the final version anyway. While we acknowledge the broader context of AI tools in professional writing, our study provides meaningful insights within its focused scope.

Correspondence Can Deniz Deveci. E-mail: can.d.deveci@gmail.com

Conflicts of interest Potential conflicts of interest have been declared. Disclosure forms provided by the authors are available with the article at ugeskriftet.dk/dmj

Cite this as Dan Med J 2024;71(1):A205178

Open access under Creative Commons License CC BY-NC-ND 4.0

Referencer

  1. Daungsupawong H & Wiwanitkit V. Correspondence on “cover letters written by ChatGPT-4 or humans”. Dan Med J 2024;71(1):A205177
  2. Deveci CD, Baker JJ, Sikander B, Rosenberg J. A comparison of cover letters written by ChatGPT-4 or humans. Dan Med J. 2023 Nov 23;70(12):A06230412.