Letter of concern regarding »Reduction in COVID-19 infection using surgical facial masks outside the healthcare system«
We are concerned about the design and interpretation of “Reduction in COVID-19 infection using surgical facial masks outside the healthcare system,” (1) a randomized controlled trial in Denmark regarding utility of mask-wearing. At this time, a trial registration and protocol paper have been posted, and the results - which we have not seen - are expected to be published shortly. According to the authors, their goal is to: “…study whether a face mask protects the wearer of the mask against SARS-CoV-2 infection. The findings are expected to apply to the present pandemic and to future viral outbreaks and to provide evidence for authority recommendations across the world,” (1). We believe that the design of this study is inappropriate to achieve these stated objectives. What follows is a description of some of the study design elements which limit what the study can tell us about mask wearing and COVID-19.
The authors misleadingly frame their study as “investigat(ing) whether the use of face masks in the community will reduce wearers' risk of SARS-CoV-2 infection”. The protocol paper suggests the two study arms are “masks must be carried outside the home…” vs “masks are not to be used,” , and the clinical trial registration suggests they are "randomis(ing participants) to either wear facial masks or not,” . However, this trial is better described as examining whether a delivery of 50 surgical masks plus weekly messages induce mask-wearing behaviour. A secondary question is whether that messaging reduces the risk of SARS-CoV-2 infection. It is essential to distinguish between stated objectives and the actual intervention because we anticipate compliance problems with mask wearing or non-mask wearing. Participants in the non-mask/non-message arm are likely to have worn masks during the study and vice versa. Non-compliance in both study arms will dilute the effect of interest and bias it toward the null with respect to the stated objectives.
The trial is also severely underpowered based on the effect size assumption in the protocol, exacerbated by the low incidence of COVID-19 in Denmark. The sample size calculation assumes at least a 50% relative risk reduction ; this is unreasonably high given that it is the product of both compliance behaviour and mask protective effects. The combined effect size would surely be smaller than that found in three observational studies collectively showing that actually wearing masks (i.e. ignoring the impact of non-compliance) outside the healthcare setting reduced the risk of SARS infection by less than 50% (pooled risk ratio 0·56 (0·40–0·79)) . Further, Denmark almost certainly experienced a much lower than the predicted 2% incidence over the course of this trial [4, 5], although data on this are limited.
Issues around outcome measurement further reduce the effective power of this study. Firstly, participants were only followed for 30 days post randomisation. However, based on the natural history of disease, this is not long enough to actually capture all infections in the study period. While about half of people infected with COVID-19 will report symptoms within five days, it may be two weeks or more until some people develop symptoms or achieve a viral load above the limit of detection . The substantial lag makes detecting cases - and thus the effects of masks - far less likely during this trial: Participants are assigned to their arm; then change their behaviours; then their infection probabilities change; then, if exposed, the virus has a latency period when infection is not detectable; and then finally testing occurs. That leaves, at most, only two to three weeks of effective measurement time to pick up cases. Secondly, properties of the test itself (e.g., probability of false negatives or positive) increase the variance estimate, as well as potential error associated with self-testing procedures. Measurement error in the dependent variable generally inflates standard errors, further reducing power.
In sum, this study has a number of critical design limitations which lead it to being biased toward the null and underpowered to answer the question of interest as stated by the investigators. Statistical insignificance and/or too-small effect measured are the most likely outcome as an artifact of the study’s design, regardless of the true effect of masks. Additionally, the trial only measures protective effects to the wearer, not transmission effects to other individuals (source control). Source control is hypothesised to be the primary way masks work to reduce the spread of SARS-CoV-2 .
This study poses a serious risk of mistranslation, in part due to misleading statements about what the study actually measures in the protocol paper and trial registration. To most decision-makers, null or too-small effects will be misinterpreted to mean that masks are ineffective. However, the more accurate translation is that this study is uninformative regarding the benefits (or lack thereof) of wearing masks outside of the healthcare setting. As such, we caution decision-makers and the media from interpreting the results of this trial as being anything other than artifacts of weak design.
(1) The trial registration states, "Participants will be instructed in using the facial mask consistently when outside their home (and at home when receiving visits from others. The instruction is given in writing and via an instruction video. The participants will be contacted once weekly to optimise compliance," .
(2) The trial description does not contain a description of how the statistical analysis will be handled for the publication. The above paragraph assumes intention-to-treat analysis, but broadly per-protocol analysis and instrumental variables embedded within the trial are each problematic due to power (discussed later), non-mask behaviour impacts, and issues around self-reporting mask use.
(3) The authors state 2% overall incidence of COVID-19 in the population, and 1% for the mask wearers.
(4) Around the time of initial recruitment in April, seroprevalence (which reflects cumulative incidence, not incidence or reported cases) based on blood donor data was 3.04% . Just under 10,000 cases were detected in Denmark’s population of 5.8 million were detected during the three-month trial period, suggesting < 0.2% cumulative incidence of reported cases during that three-month period (5). While infection incidence is substantially higher than 0.2% due to undetected cases, the 2% monthly incidence rate assumed is implausibly high.
Bundgaard H, Bundgaard JS, Raaschou-Pedersen DET, et al. Face masks for the prevention of COVID-19 - Rationale and design of the randomised controlled trial DANMASK-19. Dan Med J 2020;67(9):A05200363.
Bundgaard H. Reduction in COVID-19 Infection using surgical facial masks outside the healthcare system - Full text view - ClinicalTrials.gov 2020. https://clinicaltrials.gov/ct2/show/NCT04337541 (7 September 2020)
Chu DK, Akl EA, Duda S, et al. Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis. Lancet. 2020;395(10242):1973-87.
Iversen K, Bundgaard H, Hasselbalch RB, et al. Risk of COVID-19 in health-care workers in Denmark: an observational cohort study. Lancet Infect Dis 2020;S1473309920305892.
European Centre for Disease Prevention and Control. Download the daily number of new reported cases of COVID-19 by country worldwide. https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide (7 September 2020).
Lauer SA, Grantz KH, Bi Q, et al. The incubation period of coronavirus disease 2019 (COVID-19) from publicly reported confirmed cases: estimation and application. Ann Intern Med 2020;172(9):577-82.
World Health Organization. Advice on the use of masks in the context of COVID-19: interim guidance, 5 June 2020. Geneva: World Health Organization; 2020. https://apps.who.int/iris/handle/10665/332293 (7 September 2020).