Validation of the Danish National Tympanostomy Tube Insertion Questionnaires

Otitis media (OM) is one of the most frequent infectious diseases during childhood and is associated with a high number of physician appointments [1]. OM comprises acute OM (AOM), OM with effusion (OME) and chronic OM with effusion (COME). According to Danish National Clinical Guidelines (NCG), tympanostomy tube (TT) insertion should be offered to children with recurrent AOM (RAOM) episodes (three or more episodes in six months or at least four episodes in 12 months) and/or bilateral COME with hearing problems [2]. In Denmark, TT insertion is performed in 250/10,000 children aged zero to five years, which makes it the most common surgical procedure performed in children [3]. There is some evidence that TT insertion for RAOM reduces the number of further AOM episodes and that TT insertion for COME improves hearing and language skills [3].

Until recently, no previous studies had measured ear-related symptoms in children with OM undergoing TT insertion or investigated Danish private ENT specialists’ adherence to national guidelines for TT insertion by use of electronic patient-reported outcome (ePRO) data [4, 5].

Since ePRO data have shown significant strength for the monitoring of treatment effects, a group of Danish ENT specialists decided to create a database (the DØNHO database) holding electronic, encrypted and anonymised patient-reported outcome measures in a fast, easily executed, cost-effective and time-efficient manner [5]. The purpose was to use electronic questionnaires to investigate the parent-reported effect of TT insertion on children’s ear symptoms while also assessing adherence to the 2015 NCG on TT insertion in children aged 0-5 years with OM performed by private Danish ENT specialists.

Successful application of a questionnaire depends on its design and psychometric properties. When developing a questionnaire, it is important to assess its validity and reliability to ensure the precision and relevance of the instrument. In general, validity describes how well an instrument is measuring what it is intended to measure. One important type of validity is content validity, which is established through interviews or by consulting specialists in the area of interest. Face validity can be viewed as a subtype of content validity, which is usually assessed after the instrument has been developed. Lack of validity has been shown to result in systematic errors [6-8]. Reliability constitutes “the proportion of the total variance in the measurements which is caused by the true difference between patients” [9]. This is quantitatively investigated by conducting a test/re-test, calculating a correlation parameter and measuring the proportion of individuals providing the exact same answer twice [9, 10].

In the present study, we validated two questionnaires (The Danish National Tympanostomy Tube Insertion Questionnaires (DANTIQ)) used for the collection of ePRO data before and one month after TT insertion. The validation included assessment of the face and content validity of the questionnaires and assessment of their reliability based on a three-day test/retest analysis [6, 7].

METHODS

The DØNHO database is a private database using software that via e-mail automatically forwards and receives questionnaires and subsequently processes data from patients < 12 years of age before TT insertion and periodically up to two years after TT insertion. The post-operative questionnaires are all identical. Currently, 26 ENT specialists in 17 private practices enrol patients into the DØNHO database. An outline description of its function is reported elsewhere [4, 5]. The DANTIQ questionnaires are provided as Appendix 1 and Appendix 2 to the present publication.

The DØNHO database is certified and approved to hold patient health data by the Danish Data Protection Agency (no. 2016-42-3152). Ethical approval was not deemed necessary under Danish Law.

The content validity of the questionnaires was assessed through group discussion among four active ENT specialists (three representing the private sector and one representing the public healthcare sector).

The face and content validity examination was conducted using data from single-person interviews with nine subjects, ranging from 22 to 52 years of age. There were eight females and one male, which corresponds to the sex ratio of the respondents in the DØNHO database. The nine individuals were selected based on a purposeful sampling strategy with a large variety in age and being/not being a parent. This strategy was adopted in an attempt to include current users as well as potential future users. Four participants just had their child scheduled for surgery or had their child included in the post-operative programme. Three had previously had their children treated with TT insertion, and two were within the 22-28-year age group representing possible parents to children needing TT insertion.

All interviews were conducted in the participants’ homes. The participants had unlimited time to read the contents of the questionnaires before the interview started. Before the beginning of each interview, the procedure and purpose of the study were briefly introduced. During the interviews, the participants were asked to describe the contents of each question and whether or not they found the question or multiple-choice answers confusing. The meaning and definition of core nouns and verbs were discussed in each question. Example: “What does tympanostomy tube mean to you?” or “What does ear effusion mean to you?” Lastly, the participants were asked to identify answers or additional questions that they considered relevant to the subject. Interviews were recorded and transcribed. All interviews were conducted in a semi-structured manner using a guidance document by the same researcher (JKT) to ensure consistency. The interviewer did not form part of the group assessing the content validity of the questionnaires. The same researcher who conducted the interviews examined the transcripts to ensure maximum exposure to the data material. Key aspects concerning the understanding of the questions included in the questionnaires were presented in a report to the group of specialists in the field of interest who discuss whether or not and (if yes) how a question should be rewritten. Finally, the four active ENT specialists re-evaluated the questions according to the results of the validity study and made changes accordingly.

The reliability examination was conducted as a three-day test/re-test study involving two groups each counting 117 individuals receiving the DANTIQ questionnaires “before surgery” or “one month after surgery” twice. A sample size of approximately 117 is generally considered an acceptable sample size for robust evaluation [11]. The first “before surgery” questionnaire was accessible for the parents by email 14 days before the date of surgery (day 0). If the timespan between the registration date and the scheduled surgery was less than 14 days, the questionnaire was available on the day of patient registration. The subsequent, identical questionnaire was available three days later (day 3). Questionnaires not fully completed or with a response interval of less than three days or more than 14 days were excluded. The same procedure was used with the questionnaires applied “one month after surgery”.

Statistical analysis

Statistical analysis was performed on data from the test/re-test to ensure question reliability. The proportion of respondents providing identical answers twice in the test/re-test study was determined. One question measuring the total number of symptoms observed in the children was also analysed using Spearman’s test of correlation between the two scores of each questionnaire as the number of symptoms could be counted and measured on a numerical scale. RStudio Team (2016) version 1.1.463 was used for calculation.

The statistical analysis was conducted in collaboration with the Biostatistical Advisory Service at the University Hospital in Aarhus (BIAS), Denmark.

Trial registration: not relevant.

RESULTS

Face and content validity

The single-person interviews showed that, in general, the nine individuals found that both questionnaires were clear and understandable. However, some questions were more complicated to understand than others. The number of available categorical choices was found to be sufficient to cover the relevant aspects of the items.

Examples of answers from validity interviews were: “Can you think of any missing category to this question?” (question about symptoms of OM). Person 2: “No...”. On the same question, Person 4 states: “I think the categories go beyond my understanding of what may be described as “ear problems” … But I don’t think anything is missing”. However, some participants stated that it was difficult to make an assessment of the child regarding the aspects: “delayed development of motor function”, “changes in behaviour” and “delayed development of language”.

The following group discussion between the four ENT specialists about the results of the contents of every separate question in the questionnaire produced rephrasing of the questions concerning “Delayed development of motor function, change in behaviour and delayed development of language”. The question options changed from: “delayed development of motor function” etc. to “sense of delayed development of motor function”, “sense of change in behaviour” and “sense of delayed development of language”.

Reliability testing

A total of 108 of 117 participants completed the “before surgery” questionnaires twice within the time limits. The mean response interval was 4.65 days (standard deviation (SD) = 1.24, range 3-11 days). The proportion of agreement of the included questions between the two tests ranged from 0.98 (confidence interval (CI): 0.95; 1.00) to 0.77 (CI: 0.69; 0.85). Results are presented in Table 1. Analysing the reliability of the total number of symptoms in a child with OM, we found a Spearman’s rank of correlation of 0.93 (p < 0.001).

A total of 112 of the 117 participants completed the “one month after surgery” questionnaire within the time limit. The mean response interval was 4.72 (SD = 1.67, range 4-13 days). The proportion of agreement of the included questions between the two tests ranged from 1.00 to 0.83 (CI: 0.76; 0.90). Results are presented in Table 2. Spearman’s analysis of correlation of the total number of symptoms was 0.84 (p < 0.001)

The proportion of agreement on each specific ear symptom ranged from 0.92 (CI: 0.86; 0.98) to 0.77 (CI: 0.61; 0.93). Results are presented in Table 3.

DISCUSSION

We have developed content-validated instruments to measure Danish private ENT specialists’ adherence to guidelines on TT insertion in children with OM and to determine the nature of the symptom patterns in children undergoing TT insertion in Denmark.

The face validity and the content validity of the tested questionnaires were rated as acceptable, with the questions being easily understandable and the language clear. However, some participants mentioned that it was hard to assess a child correctly with regard to, e.g., “sense of delayed development of motor function” or “sense of change in behaviour”. According to some interviewees, this could be assessed properly only by a physician. This issue was thoroughly discussed by the members of the group of ENT specialists who concluded that symptomatic patterns are highly subjective to each individual patient and will therefore always be reported subjectively. The question about symptomatic patterns was designed to report parents’ or guardians’ “experience of symptoms” and therefore did not affect the face and content validity of the questionnaires. When asked specifically whether additional or other response options should be available to the questions, no suggestions were presented in the interviews, reflecting an acceptable content validity.

Four of nine participants were parents from the target population of children scheduled for TT insertion; three were parents to a child who had had TT insertion; while two did not yet have children. This indicated that face and content validity of the questionnaires will most likely be valid for the present target population and also for possible future users.

The study design included only single-person interviews, which is less ideal than single-person interviews in combination with large group interviews [6]. The test/re-test analysis was conducted online by parents at home. It therefore was not possible to detect whether the parents received assistance in filling out the questionnaire, but, on the other hand, it was also not possible for the ENT specialists or their staff to influence the answers. A response interval of three days carries a risk of recollection bias in the test/re-test study. On the other hand, a longer interval in the test/re-test could possibly compromise the validity of the test/re-test analysis since symptoms related to OM in children may regress within days. The shorter interval chosen in this study falls within limits suggested by other authors [6].

Generally, we found a high proportion of agreement on each individual question, suggesting a high level of reliability within the DANTIQ. The lowest proportion of agreement when analysing the specific symptoms one by one was the assessment of “sense of delayed development of motor function” and “sense of change in behaviour”. This was in accordance with the observations found during the single-person interviews, since these questions included the most difficult symptoms for the respondents to assess.

Looking at the total amount of symptoms as a marker of the severity of disease as evaluated by the respondents, the correlation test suggested a good correlation between the first and the second scores of symptoms in both DANTIQ questionnaires. Hence, the level of reliability was found to be acceptable when assessing the total amount of different ear symptoms in the children.

CONCLUSIONS

The two DØNHO database questionnaires have an acceptable content validity and are reliable for measuring ear-related symptoms in Danish children with OM undergoing TT insertion. The DANTIQ is also valid for measuring Danish private ENT specialists’ adherence to national guidelines in TT insertion using ePRO data. The study revealed that some respondents may have problems assessing certain symptoms, but these findings do not compromise the use of the questionnaires for the purposes mentioned above.

CORRESPONDENCE: Preben Homøe. E-mail: prho@regionsjaelland.dk

ACCEPTED: 1 July 2019

CONFLICTS OF INTEREST: none. Disclosure forms provided by the authors are available with the full text of this article at www.danmedj.dk.

ACKNOWLEDGEMENTS: We take this opportunity to acknowledge and express our gratitude towards our pilot clinics collaborators Pascal Bonvin, Mikkel Bruun, Janus Jespersen, Peder Frandsen, Christian Bak, Jakob Korsholm Nielsen, Jonas Peter Holm, Thomas Qvist Barrett, Malene Sine Rokkjær, Michael Lüscher, Jette Berg, Bo Karlsmose, Andreas Agger, Torben Lilholt/Rikke Haahr, Berit Svenstrup, Pia Juul, Lotte Jung, Peter Koefoed Tingsgaard, Liviu Guldfred, Kim Werther, Tomaas Ravn, Ann-Louise Reventlow-Mourier, Anders Schermacher, Christian Prætorius, Roland Welinder and Mikkel Holmelund for their implementation of patient registration in the DØNHO database. We also greatly appreciate our valuable collaboration with the Danish ENT Specialists Organisation (DØNHO), the Unit of Quality in Specialist Practice (eKVIS) and the Fund for Professional Development in Specialist Practice who have been of great support in setting up and implementing the DØNHO database.