Customization of a tool to assess Danish surgeons’ non-technical skills in the operating room

The operating room (OR) is a complex domain in which clinicians’ clinical knowledge and technical skills are essential. Retrospective studies have reported that many surgical mistakes were caused by insufficient communication and teamwork [1, 2]. Prospective studies have shown that insufficient teamwork in the OR such as -infrequent information sharing has increased the risk of complications [3] and that communication failures have caused procedural errors and delays [4]. It is therefore evident that surgeons and surgical teams need skills in the fields of communication, teamwork, decision making and leadership (so called non-technical skills) to ensure a good outcome for their patients.

Training these skills can enhance communication and collaboration in surgical teams [5], reduce the number of technical errors [6, 7] and reduce surgical mortality [8].

The effects of training should be measured to ensure sufficient quality and cost-effectiveness, but assessment of individuals also serves the purpose of guiding and motivating clinicians [9]. For this purpose valid and reliable tools are needed, such as behavioural marker systems, which have been developed for surgeons [10], anaesthesiologists [11], scrub nurses [12] and surgical teams [6, 13]. The Scottish behavioural marker system, NOTSS (Non-Technical Skills for Surgeons) [10], is a rating tool based on a hierarchical system of skills comprising four categories and 12 elements and examples illustrating good and poor behaviour. It is an observational tool for individual in training assessment that has provided reliable and valid ratings of non-technical skills [14] and has proved usable during real cases [15].

Currently, assessment tools lack in Denmark. Although cultural and organizational differences complicate a direct transfer of tools developed in a different context, elements of such tools are likely to be generic and thus applicable across contexts. The aims of the present study were:

to explore which non-technical skills Danish general surgeons, scrub nurses, anaesthesiologist and nurse anaesthetists find important for the work of general surgeons

to test to which extent these skills fitted the Scottish NOTSS and

to customize the Scottish NOTSS for general surgeons in Denmark as NOTSSdk.

MATERIAL AND METHODS

Sampling

Eight group interviews were conducted at two university hospitals in the Capital Region of Denmark. One hospital mainly deals with acute admissions and gastrointestinal oncology with operations being laparoscopic rather than open, and the other is specialized in upper abdominal surgery and treats patients with surgical complications, and operations are predominantly open. Four groups were interviewed in each hospital: 1) consultant general surgeons, 2) trainee surgeons, 3) scrub nurses and 4) anaesthesiologists and nurse anaesthetists. The sampling was based on convenience sampling, as there was a need to fit the interviews into clinical schedules. Groups 3 and 4 in one hospital were recruited via the head of department who chose interview participants.

Interviews

The interview guide was developed to explore surgeons’ non-technical skills. The interview used openended questions. It was structured around the NOTSS categories: situation awareness, decision making, leadership and communication & teamwork, and around its elements, but it also contained questions to explore if interview participants had anything to add. The researchers spent 30 hours in the OR doing unstructured observations, taking field notes to get acquainted with the culture and work habits in the two hospitals. This helped design the interview guide and analyse the data. The interview guide was pilot tested and minor changes were made. Data from the pilot test were not included.

Interviews were scheduled to last 1-1.5 hours and were conducted at the respective clinical departments in undisturbed surroundings. They were audio-recorded and guided by two interviewers. The discussions among the interview participants concerned experiences of which kinds of surgical behaviour they considered to be either safe and efficient or not safe and efficient in regard to the treatment of the patient.

According to Danish law, ethical approval is not needed for this type of study. Interview participants gave oral informed consent and data were anonymized.

Analysis

The interviews were transcribed verbatim and analysed using directed content analysis [16]. Two interviewers independently identified interview bits containing information about surgeons’ non-technical skills in the intraoperative setting. These bits were categorized and colour-marked according to the four NOTSS categories. Interview bits that contained descriptions of non-technical skills that did not fit the categories were marked with a fifth colour. Each interview bit was only sorted into one category. The interviewers discussed their cat-egorization, and in case of disagreement, the results were discussed with a third group member until consensus was reached.

The colour-marked interview bits from all the interviews were then printed on paper and sorted according to the three NOTSS elements under each category. For each element, the bits were further sorted into subgroups which were subsequently labelled. For each labelled subgroup, one to two paraphrases were written. For each paraphrase, one observable behavioural example was written. The interview bits that did not fit the existing categories were analysed separately. Data were analysed in an iterative process in the research group to ensure adequate sorting. Table 1 illustrates the process from interview quotation to behavioural example in the category of Communication and Teamwork. The other categories were analysed in a similar way.

The prototype of the Danish taxonomy with categories, elements and behavioural examples were discussed in two workshops with a panel of 12 surgeons with educational responsibility representing eight hospitals. The purpose was to check the face validity and generalizability of the behavioural examples. They refined behavioural examples, removed duplicates and wrote new behavioural examples if these were considered lacking. Based on this input, the research group designed the final revision of the taxonomy. For an overview of the research process, see Figure 1.

Trial registration: The study was reported at clinical-trials.gov (NCT01334411).

RESULTS

A total of 72 clinicians participated in the interviews which lasted between 52 and 84 minutes. Groups consisted of: 4 and 10 consultant surgeons, 12 and 7 trainee surgeons, 14 and 7 scrub nurses and 11 and 7 from the anaesthesia team (nurses and doctors), respectively. Table 2 shows the demographic data of the participants.

The majority of the identified behavioural markers were easy to sort according to the four categories and 12 elements of NOTSS, and no categories or elements were left blank. This suggests that the structure of NOTSS also applies in Denmark. Interview bits that did not fit the structure centred on the importance of having an awareness of one’s competence, knowing when to slow down and concentrate, keeping track of the progression of the procedure and having strategies for coping with stress, fatigue and disturbances. These skills were not evident in the Scottish NOTSS, so they were gathered and added as a new cognitive element under situation awareness and labelled “monitoring own performance.” This resulted in NOTSSdk consisting of the four categories and 12 elements from NOTSS, and one additional element and between 3-8 good and 3-6 poor behavioural examples underpinning each element. All behavioural examples were identified from the material. Table 3 shows the categories and elements and one good and one poor example for each element. The full version of NOTSSdk is available as a user guide [17]

Description of the four categories

The leadership category reflects that good surgical behaviour involves providing and maintaining a good and professional working atmosphere in the OR and respecting the competencies and working conditions of team members. A good surgical leader has a strategy, demonstrates enthusiasm and supports team members by providing constructive feedback, supervision and motivation.

Good surgical communication and teamwork involves coordination of the strategy with the anaesthesia team, communicating this clearly and keeping the team informed about the progress of the operation. It also includes creating a shared focus of all team members on the task and to collaborate in order to progress safely and efficiently.

Safe decision making relates to considering all relevant available information and balancing risks, pros and cons before deciding on a course of action. Aspects of teamwork such as “letting the team comment on the decision” and “striving at consensus on the treatment” are also highlighted. Good decision making contains efforts to re-evaluate decisions repeatedly and to recognize when it is time to change strategy.

DISCUSSION

A Danish behavioural marker system for surgeons, NOTSSdk, was developed using the Scottish NOTSS as a template consisting of four categories: situation awareness, decision making, leadership and communication & teamwork, including 13 elements and examples of good and poor observable behaviour.

Respecting team members, creating a good working atmosphere and discussing options and decisions in the surgical team seems more prominent in the Danish taxonomy than in the Scottish. This may reflect cultural differences between Denmark and Scotland assuming that a more flat hierarchy and team working structure dom-inate in the Scandinavian countries. NOTSSdk was also developed five years later than NOTSS and the understanding of these matters may have changed during this interval. Another explanation is that, as opposed to previous studies, we included the perspectives of the full surgical team, including anaesthesia staff and scrub nurses, who provided additional kinds of behaviour that are obvious to the collaborative team members. These are important to include, since the cognitive and social skills of one team member influence the task management of others [4]. Experimental studies have shown that victims of rude behaviour significantly decrease their engagement in cognitive task management Ð and this decrease already shows if people only witness rude behaviour [18]. There is no reason to doubt that the same should apply to multi professional OR teams with their complex tasks and interdependencies. We believe that our approach broadens the perspectives on surgeons’ non-technical skills and thereby expands the overall understanding of what safe and efficient surgical behaviour is.

Several examples of behaviour were derived from interviews with trainee surgeons, scrub nurses or anaesthesiologists. This is not surprising, though, since various studies have documented differences of opinion regarding the perceived quality of teamwork in surgical teams. In general, doctors assess the teamwork more positively than nurses and, similarly, surgeons have a more positive perception of teamwork than anaesthesiologists [5, 19].

The behavioural marker system is not intended to be a checklist and it is not exhaustive. It offers a framework and language to assess and guide feedback on surgeons’ non-technical performance in the OR. The examples are only illustrations. However, one challenge in the cognitive categories of situation awareness and decision making is the assumption that the mental processes are analytical and purely rational, which may not always be the case. Some surgical decision making involves a certain degree of creativity and intuition [20] and in cases of time restraints or inexperience, a more rule-based -approach will be appropriate [20].

Our study has some limitations. The interviews were the empirical foundation of this study. Although pre-interview observations were carried out and the development was fostered on a template of an existing behavioural marker system and driven by experienced clinicians, no systematic observations were conducted to causally relate behaviours to patient outcome, or to ensure the match between what the surgical team think are important non-technical skills and which non-technical skills surgeons actually display.

The interviews were conducted at two hospitals in one of five regions in Denmark. This might not be representative. However, we believe that the large number of participants in the interviews and in the workshops (with participants from two regions) to some extent compensated for this.

We found it important to include the surgical team members’ considerations since the surgeons’ non-technical skills influence the rest of the team and vice versa. There is a risk, though, that we have been looking specifically for these topics when analysing the interviews and potentially over-emphasizing them.

In Denmark, the seven roles of the Canadian Medical Education Directions for Specialists (CanMEDS) framework are implemented, but assessment tools are lacking. NOTSSdk offers a tool to assess skills in the roles of Manager, Professional, Collaborator and Communi-cator based on direct observations and a structure for giving feedback to the surgical trainee. Future studies will explore the reliability and validity of NOTSSdk when used in the simulator and in the clinical setting.

CONCLUSION

An empirically grounded Danish behavioural marker system of general surgeons’ non-technical skills was developed using a Scottish system as a template. The system consists of four categories and 13 elements along with examples illustrating good and poor behaviour. Differences exist between the Danish and the Scottish system providing an argument for a tool developed or customized to the local context in which it is to be used.

Correspondence: Lene Spanager, Dansk Institut for Medicinsk Simulation, Herlev Hospital, 2730 Herlev, Denmark. E-mail: lene.spanager@regionh.dk

Accepted: 3. September 2012

Conflicts of interest:Disclosure forms provided by the authors are available with the full text of this article at www.danmedj.dk.

Acknowledgement: The authors wish to thank Lene Heise Garvey for great help in translating the Scottish NOTSS.