Introduction. This study aimed to conduct a targeted needs assessment to identify and prioritise technical skills and procedures suited for simulation-based training (SBT) in private otorhinolaryngology (ORL) practice in Denmark, including mapping the learning environment related to implementation of SBT.
Methods. A panel of trainers and trainees in private ORL practice was recruited. Using the Delphi method, three rounds of surveys were conducted. Round one consisted of a survey of the learning environment and a brainstorming phase. Round two quantified the frequency of procedures, ranked the importance of procedural competency, impact on patient safety and feasibility for SBT. In round three, panelists eliminated and ranked procedures for final prioritisation.
Results. A total of 26 of 57 invited trainers and trainees accepted participation. The educational environment was described and 136 skills were suggested in the brainstorming phase. “Non-technical” skills were removed, and the remaining 46 technical skills were grouped for appraisal in round two. In round three, panelists reduced these to eight technical skills and procedures which were maintained for final prioritisation for SBT with myringotomy with ventilation tube insertion ranking highest. Trainees and trainers indicated that close supervision and dedicated time for training were major strengths of the learning environment.
Conclusions. Our findings extend the results obtained in a previous general needs assessment and may inform curricular implementation of SBT in private ORL practice. A structured “package” with SBT and assessment for the identified procedures are desired by trainers. This work is already in progress and implementation is facilitated by a positive attitude towards SBT among trainers and trainees alike.
Trial registration. not relevant.
Competency-based education has gained a foothold as the cornerstone of modern health professional training, including surgery [1, 2]. The standardised conditions of simulation introduce various benefits as a learning and assessment environment  providing trainees with a safe training context [4, 5]. A crucial step in the implementation of simulation-based training (SBT) and its integration within the training curricula is establishing a proficiency/mastery level in simulation, which is required before allowing supervised surgery on patients. Establishing this level remains a challenge despite ample evidence in support of its positive effects of SBT [6, 7].
A systematic approach to planning and development is key to successful implementation . The six-step approach suggested by Kern is: 1) General needs assessment, 2), Targeted needs assessment, 3) Goals and objectives, 4) Educational strategies, 5) Implementation and 6) Evaluation and feedback . A targeted needs assessment aims to avoid redundancy by repeating what was already done, using resources efficiently and helping mapping the best approach to implementation. A national general needs assessment in otorhinolaryngology (ORL), which mainly focused on training in a hospital setting, identified thirteen technical skills suited for SBT . The Danish ORL training curriculum includes a six-month rotation in private practice where the educational environment, objectives, challenges and opportunities are very different from those of the hospital setting. This warrants a targeted needs assessment to map the learning environment and resources to establish how SBT may best be implemented. We have identified only a single study reporting on a targeted needs assessment in ORL ; a study which was not conducted in a private practice educational setting.
In this study, we aimed to describe the learning environment, identify technical skills for SBT in private practice ORL and explore barriers and resources in this context.
This study was designed as a survey among educational stakeholders (trainers as well as trainees) in private Danish ORL practices. We utilized the Delphi method and planned three rounds of questionnaires with results from each round being used iteratively to inform subsequent rounds. The Delphi method  serves to establish a consensus among content experts and has previously been used for surgical curriculum development [5, 12].
We conducted rounds one, two and three in June, September and October 2020, respectively. All questionnaires were answered anonymously and handled by an independent part (law firm). In every round, non-responders were contacted within a three-week period and encouraged to participate. In rounds one and two, the questionnaires were distributed digitally or in printed form. For round three, we designed an online survey in SurveyMonkey (SVMK Inc., San Mateo, CA, USA) to facilitate easy elimination and re-ranking of procedures.
The private ORL practices are small clinics with one to four specialists. They typically perform minor surgical procedures such as myringotomy with ventilation tube insertion and adenoidectomy. ORL specialist trainees spend six months in private practice.
Panel participants were recruited aiming to include both trainers and trainees through 1) an established network of 19 trainers in private ORL practice, and 2) 38 trainees through the Danish Association of Junior Otorhinolaryngologists. All were invited by e-mail.
Delphi round one: learning environment
The first part of the questionnaire (https://ugeskriftet.dk/files/a08210630_-_supplementary.pdf) considered demographics, whereas the second part focused on the learning environment with free-text responses related to supervision, learning resources and teaching methodology/pedagogy. We also asked panelists how much time they spent giving/receiving supervision. The third part enquired about past experiences and expectations related to SBT and asked respondents to list in free-text all the technical skills and procedures the trainee needs to be able to competently perform (i.e., safely and independently) at the end of their rotation in private practice. All the suggested technical skills and procedures were then reviewed, and non-technical skills such as communication were excluded.
Delphi round two: technical skills and procedures
In round two, panelists were asked to appraise the procedures identified in round one with respect to 1) frequency of the procedure at their private practice; 2) how important competency is for an efficient consultation where the procedure may be performed rapidly (given that it is performed safely); and 3) whether the procedure is uncomfortable or puts the patient at risk if training is completed on patients (https://ugeskriftet.dk/files/a08210630_-_supplementary.pdf). In the round two questionnaire, we applied a five-point Likert scale allowing the panelists to evaluate the procedures .
We used a modified Copenhagen Academy for Medical Education and Simulation Needs Assessment Formula (CAMES-NAF) to calculate a score for the remaining procedures  after linear data normalisation had been used to transform the Likert scores into percentages.
Delphi round three: elimination and prioritisation for simulation-based training
In this final round, panelists were given the opportunity to revise the ranking order and/or to eliminate procedures from the list compiled after round two (https://ugeskriftet.dk/files/a08210630_-_supplementary.pdf). For the final list, procedures were considered only if two thirds of the panelists had reached a consensus on including the procedure.
Under Danish law, ethical board approval is not required for this type of study (questionnaire study). This study complies with the Helsinki Declaration. All participants were informed and gave their consent for participation.
Trial registration: not relevant.
Delphi round one: learning environment
Eleven of the 19 trainers responded in the first round. The median age was 54 years (range: 42-62 years) and the medium period participants had served as a trainer in private practice was seven years (range: 2-13 years). Three had solo practices and eight served in shared clinics. Fifteen out of the 38 trainees responded. The median age among trainees was 35 years (range: 32-37 years). The response flow is presented in Figure 1.
Trainers reported a median of six hours/week (range: 2.5-10 hours) compared with five hours (range: 2-8 hours) indicated by trainees. Panelists indicated good availability of the following learning resources: supervision and mentorship, access to books and journals, journal clubs, electronic materials and online databases. Half of the panelists indicated that simple simulators were available (e.g. a simple model for myringotomy with ventilation tube insertion). All trainers and trainees reported a generally positive attitude towards SBT and believed that the method was well suited for teaching in private ORL practices.
The most commonly used training method was the principle of "see one, do one" and direct observation with immediate or subsequent feedback. Trainees indicated that protected time for supervision and learning within working hours ranged from 0 to 10 hours/week, with the majority indicating one hour/week. Trainers reported allocating 0-9 hours/week for supervision and teaching for an average of two hours. Only 30% of panelists indicated that structured assessment tools were used.
We found good agreement between trainers’ and trainees’ responses relating to the strengths and opportunities of the educational environment in private ORL practices. All trainees indicated that working in close cooperation with and receiving supervision from the trainer were major strengths in private ORL practice. Trainers similarly found these aspects to be the major strengths. Working close together (“door to door”) with the trainer and having access to high-quality equipment were identified as strengths. Trainees indicated that time pressure in the daily routines and many patients with “trivial” problems (e.g. obstructing cerumen) were weaknesses of the educational environment.
Delphi round two: technical skills and procedures
A total of 136 skills were listed in round one. After grouping of similar skills and exclusion of “non-technical” skills and procedures, 46 unique technical skills and procedures remained (Table 1). All skills and procedures were submitted to the panelists in round two for appraisal. Meanwhile, the authors assessed feasibility and scored the procedures on a five-point Likert scale in relation to availability, cost and suitability for SBT in private practice. A consensus was reached in a virtual meeting. We further eliminated five technical skills and procedures that no panelists indicated were performed in private ORL practice: bronchoscopy, gastroscopy, large-needle biopsy, robotic surgery, and tracheotomy. For the remaining 41 procedures, panelists’ scores along with feasibility scoring were used to calculate the modified NAF score (Table 1).
Delphi round three: elimination and prioritisation for simulation-based training
Among the 41 technical skills and procedures ranked after round two, we further eliminated those with a CAMES-NAF score < 20% (Table 1). Thirty-six skills remained. In round three, we asked panelists to first eliminate technical skills and procedures that they perceived to be safe to train in the clinical setting or which should not be trained during the rotation in private ORL practice. Next, panelists were asked to rank the remaining technical skills and procedures for prioritisation of training using SBT.
Eight technical skills and procedures were included by at least two thirds of the panelists and were therefore prioritised (Table 2): tubulation and myringotomy were ranked highest.
We conducted a targeted needs assessment of private ORL practice involved in postgraduate training. We used the Delphi technique because this structured approach is well-suited for generating ideas and establishing consensus among key educational stakeholders [7, 13].
In our mapping of the learning environment, daily supervision and "door-by-door" collaboration, facilitating easy access to feedback and clinical discussion were identified as primary strengths. Furthermore, a very positive attitude towards using SBT was recorded.
The panelists gave highest priority to SBT related to insertion of ventilation tubes (tubulation). This technical skill is among the most frequently performed paediatric surgical procedures in the ORL curriculum . Myringotomy, local anaesthesia for tubulation and myringotomy, myringoplasty with rice paper, and cleaning of radical cavity were also given a high priority for SBT. These procedures present several challenges to the novice because of the use of otomicroscope and microsurgery instruments. Furthermore, the procedures may be painful for the patient, and delicate structures can be damaged . At present, tubulation is mainly trained through supervision in the operatinf theatre. Fortunately, various ear surgery simulators exist  ranging from technology-enhanced simulators such as visual reality environments  to more “simple” physical models [18, 19]. For assessment of technical skills performance, Schwartz et al. have developed an objective assessment tool for myringotomy and tympanostomy tube insertion . In combination with local SBT, this facilitetes structured competency assessment in private ORL practice.
Three rhinological procedures were also prioritised for SBT: endoscopically guided removal of nasal polyps, insertion of tube in the maxillary sinus and puncture of the maxillary sinus. The panel’s prioritisation of procedures involving the nasal cavity and sinuses underlines the need to develop suitable simulation methods. Such methods currently do not exist.
The general needs assessment of SBT in ORL training gave high priority to several surgical (cricothyroidectomy, tracheotomy, septoplasty) and diagnostic (flexible fibre rhino-pharyngo-laryngoscopy, flexible bronchoscopy and eosophagoscopy) procedures 9.Most of these procedures were also identified by our panel. However, these procedures did not accomplish final priority because (except for flexible fibre rhino-pharyngo-laryngoscopy), they are not performed in private ORL practice in Denmark. Flexible fibre rhino-pharyngo-laryngoscopy is frequently performed, but competency is achieved early in residency prior to rotation in ORL practice. These differences highlight the need for a targeted needs assessment mapping the educational environment and identifying the resources available for implementation.
A limitation of our study is that it was conducted in a Danish context. Therefore the results cannot necessarily be extrapolised directly to other countries. Furthermore, in this study, we only included technical skills; yet we acknowledge the importance of also training communication, teamwork, clinical reasoning and other “non-technical” skills in the training of highly competent medical professionals. Another limitation is panelist drop-out, which was comparable to drop-out reported in other needs assessment studies [9, 10]. However, drop-out did not change the geographical distribution, and we maintained representation of both trainers and trainees, ensuring representative responses as well as data saturation. Also, the positive attitude we found towards SBT in our panel may potentially be a consequence of representation bias, as respondents with an interest in and positive mindset towards SBT may be more likely to participate in a study as this.
Our results enable the planning and implementation of relevant SBT activities that fit the current needs and the learning environment in private ORL practice. To support this, we are currently developing a novel low-cost prototype for SBT in ear surgery including ventilation tube insertion. Also, we are investigating new plastic compound materials that would allow 3D-printing of sinus models. Importantly, we have engaged our community of trainers, who are motivated for implementing SBT and structured competency assessment.
We conducted a targeted needs assessment using the Delphi methodology in order to map central aspects for implementation of SBT in private ORL practice. Our findings extend the results obtained in a previous general needs assessment and underline the importance of continuous educational curriculum development. Our panelists of trainers and trainees prioritised eight technical skills suited for SBT in private ORL practice. The reported findings may be used for implementing SBT in private ORL practice: a process which is already in progress, facilitated by the constructive learning environment and the positive attitude towards SBT among trainers as well as trainees.
Correspondence Michael Lüscher. E-mail: firstname.lastname@example.org
Accepted 9 February 2022
Conflicts of interest Potential conflicts of interest have been declared. Disclosure forms provided by the authors are available with the article at ugeskriftet.dk/dmj
Acknowledgements The authors would like to thank all participating trainers and trainees for their contribution to this project.
References can be found with the article at ugeskriftet.dk/dmj
Cite this as Dan Med J 2022;69(4):A08210630
Carraccio C, Englander R, Van Melle E et al. Advancing competency-based medical education: a charter for clinician-educators. Acad Med. 2016;91(5):645-9.
Ten Cate O. Competency-based postgraduate medical education: past, present and future. GMS J Med Educ. 2017;34(5):Doc69.
Bjerrum F, Sorensen JL, Thinggaard E et al. Implementation of a cross-specialty training program in basic laparoscopy. JSLS. 2015;19(4):e2015.00059.
Bewley WL, O’Neil HF. Evaluation of medical simulations. Mil Med. 2013;178(10 suppl):64-75.
Nayahangan LJ, Schroeder T, Paltved C et al. A national needs assessment to identify technical procedures in vascular surgery for simulation based training. Eur J Vasc Endovasc Surg. 2017;53(4):591-9.
Cook DA, Hatala R, Brydges R et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306(9):978-88.
McGaghie WC, Issenberg SB, Cohen ER et al. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86:706-11.
Thomas PA, Kern DE, Hughes MT, Chen BY, eds. Curriculum development for medical education. Johns Hopkins University Press Books, 2020.
Andersen SAW, Nayahangan LJ, Konge L, Melchiors J. Identifying and prioritizing technical procedures in otorhinolaryngology for simulation-based training: a national needs assessment in Denmark. Eur Arch Otorhinolaryngol. 2019;276(5):1517-24.
Daloğlu M, Alimoğlu MK. What do otolaryngologists want to learn? An educational targeted needs assessment study. Braz J Otorhinolaryngol. 2020;86(3):287-93.
Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Practical Assessment, Research, and Evaluation 2007;12:Article 10. https://scholarworks.umass.edu/pare/vol12/iss1/10 (1 Marts, 2021).
Gustafsson A, Viberg B, Paltved C et al. Identifying Technical procedures in orthopaedic surgery and traumatology that should be integrated in a simulation-based curriculum: A National general needs assessment in Denmark. J Bone Joint Surg Am. 2019;101(20):e108.
Nayahangan LJ, Stefanidis D, Kern DE, Konge L. How to identify and prioritize procedures suitable for simulation-based training: experiences from general needs assessments using a modified Delphi method and a needs assessment formula. Med Teach. 2018;40(7):676-83.
Siggaard LD, Barrett TQ, Lüscher M et al. Parent satisfaction and symptom relief in children with otitis media undergoing tympanostomy tube insertion. Dan Med J. 2019;66(9):A5567.
Rosenfeld RM, Schwartz SR, Pynnonen MA et al. Clinical practice guideline: Tympanostomy tubes in children. Otolaryngol Head Neck Surg. 2013;149(1 suppl):S1-S35.
Mahalingam S, Awad Z, Tolley NS, Khemani S. Ventilation tube insertion simulation: a literature review and validity assessment of five training models. Clin Otolaryngol. 2016;41(4):321-6.
Huang C, Cheng H, Bureau Y et al. Face and content validity of a virtual-reality simulator for myringotomy with tube placement. J Otolaryngol Head Neck Surg. 2015;44:40.
Shenton C, Aucott W. How I do it: a simulator of the ear for developing otomicroscopy skills during the coronavirus disease 2019 pandemic. J Laryngol Otol. (online 21 Oct 2020).
Clark MPA, Nakku D, Westerberg BD. An endoscopic ear trainer for the low-resource setting. J Laryngol Otol. 2019;133(7):571-4.
Schwartz J, Costescu A, Mascarella MA et al. Objective assessment of myringotomy and tympanostomy tube insertion: a prospective single-blinded validation study. Laryngoscope. 2016;126(9):2140-6.