Letter of concern regarding »Reduction in COVID-19 infection using surgical facial masks outside the healthcare system«

We are concerned about the design and interpretation of “Reduction in COVID-19 infection using surgical facial masks outside the healthcare system,” (1) a randomized controlled trial in Denmark regarding utility of mask-wearing. At this time, a trial registration and protocol paper have been posted, and the results - which we have not seen - are expected to be published shortly. According to the authors, their goal is to: “…study whether a face mask protects the wearer of the mask against SARS-CoV-2 infection. The findings are expected to apply to the present pandemic and to future viral outbreaks and to provide evidence for authority recommendations across the world,” (1). We believe that the design of this study is inappropriate to achieve these stated objectives. What follows is a description of some of the study design elements which limit what the study can tell us about mask wearing and COVID-19.

The authors misleadingly frame their study as “investigat(ing) whether the use of face masks in the community will reduce wearers' risk of SARS-CoV-2 infection”. The protocol paper suggests the two study arms are “masks must be carried outside the home…” vs “masks are not to be used,” [1], and the clinical trial registration suggests they are "randomis(ing participants) to either wear facial masks or not,” [2]. However, this trial is better described as examining whether a delivery of 50 surgical masks plus weekly messages induce mask-wearing behaviour. A secondary question is whether that messaging reduces the risk of SARS-CoV-2 infection. It is essential to distinguish between stated objectives and the actual intervention because we anticipate compliance problems with mask wearing or non-mask wearing. Participants in the non-mask/non-message arm are likely to have worn masks during the study and vice versa. Non-compliance in both study arms will dilute the effect of interest and bias it toward the null with respect to the stated objectives.

The trial is also severely underpowered based on the effect size assumption in the protocol, exacerbated by the low incidence of COVID-19 in Denmark. The sample size calculation assumes at least a 50% relative risk reduction ; this is unreasonably high given that it is the product of both compliance behaviour and mask protective effects. The combined effect size would surely be smaller than that found in three observational studies collectively showing that actually wearing masks (i.e. ignoring the impact of non-compliance) outside the healthcare setting reduced the risk of SARS infection by less than 50% (pooled risk ratio 0·56 (0·40–0·79)) [3]. Further, Denmark almost certainly experienced a much lower than the predicted 2% incidence over the course of this trial [4, 5], although data on this are limited.

Issues around outcome measurement further reduce the effective power of this study. Firstly, participants were only followed for 30 days post randomisation. However, based on the natural history of disease, this is not long enough to actually capture all infections in the study period. While about half of people infected with COVID-19 will report symptoms within five days, it may be two weeks or more until some people develop symptoms or achieve a viral load above the limit of detection [6]. The substantial lag makes detecting cases - and thus the effects of masks - far less likely during this trial: Participants are assigned to their arm; then change their behaviours; then their infection probabilities change; then, if exposed, the virus has a latency period when infection is not detectable; and then finally testing occurs. That leaves, at most, only two to three weeks of effective measurement time to pick up cases. Secondly, properties of the test itself (e.g., probability of false negatives or positive) increase the variance estimate, as well as potential error associated with self-testing procedures. Measurement error in the dependent variable generally inflates standard errors, further reducing power.

In sum, this study has a number of critical design limitations which lead it to being biased toward the null and underpowered to answer the question of interest as stated by the investigators. Statistical insignificance and/or too-small effect measured are the most likely outcome as an artifact of the study’s design, regardless of the true effect of masks. Additionally, the trial only measures protective effects to the wearer, not transmission effects to other individuals (source control). Source control is hypothesised to be the primary way masks work to reduce the spread of SARS-CoV-2 [7].

This study poses a serious risk of mistranslation, in part due to misleading statements about what the study actually measures in the protocol paper and trial registration. To most decision-makers, null or too-small effects will be misinterpreted to mean that masks are ineffective. However, the more accurate translation is that this study is uninformative regarding the benefits (or lack thereof) of wearing masks outside of the healthcare setting. As such, we caution decision-makers and the media from interpreting the results of this trial as being anything other than artifacts of weak design.

Reponse from Bundgaard H et al

Correspondence: Noah A. Haber, Email: noahhaber@stanford.edu
Conflict of interest: Disclosure forms provided by the authors are available with the full text of this coprrespondence at www.danmedj.dk