Face validity and inter-rater reliability of the Danish version of the modified Yale Preoperative Anxiety Scale

Anaesthesia and surgery can be frightening to children [1, 2]. International studies have shown that around 65% of all children undergoing anaesthesia and surgery experience significant preoperative anxiety causing adverse effects [3, 4]. Preschool children seem to be the most vulnerable due to their age-related cognitive limitations [5, 6].

Preoperative anxiety is a multi-factorial and complex problem affecting the child before, during and after surgery [7]. Anxiety is associated with adverse clinical, behavioural and psychological outcomes, including delayed recovery, increased postoperative pain, increased need for analgesics and new onset of negative behavioural changes postoperatively [8-10]. These changes include nightmares separation anxiety, problems related to eating as well as increased fear of health professionals, and they can be observed more than six months after discharge [11]. However, the effectiveness of any intervention needs to be evaluated with a standardised, valid and sensitive tool for specific assessment of preoperative anxiety across various age groups. Today, the modified Yale Preoperative Anxiety Scale (m-YPAS) is the most widely used observational tool internationally for this purpose [5, 12]. The original version of the YPAS was developed in 1995 to assess anxiety in children aged 2-6 years in the operating room. It was subsequently modified in 1997 to include assessment of children’s anxiety in the preoperative holding area and to extend the age range up to 12 years [5, 13].

The purpose of this study was to: 1) translate the m-YPAS into Danish and 2) test its face validity and inter-rater reliability in a Danish context.

MATERIAL AND METHODS

Setting and procedure

The study was performed at the Department of Anaesthesiology and Intensive Care, Aarhus University Hospital, Denmark, from 2011 to 2012 in two phases: 1) translation of the m-YPAS from English into Danish and 2) testing of the face validity and inter-rater reliability of the Danish version.

Tool characteristics

The m-YPAS is composed of five categories (activity, emotional expressivity, state of arousal, vocalisation and use of parents) which define 22 specific behaviours indicating anxiety in the children during induction of anaesthetics. The highest level observed in each of the five m-YPAS categories is the score for each separate category. As each category of the m-YPAS has a different number of items (four or six), partial weights are used to calculate the total score. The total weighted score ranges from 23.33 to 100, and higher scores indicate a higher level of anxiety.

The m-YPAS offers a rating period beginning in the preoperative holding area (holding), and the subsequent walk to the operating room (OR) (separation), entry into the OR until mask notification (Induction 1) and when the mask is on face (Induction 2). Observing the whole pathway provides an overview of the child’s level of anxiety during the preoperative process. The rater is neutral and unobtrusive and does not influence the child’s behaviour. The m-YPAS can either be scored “in vivo” where the observer rates the child’s anxiety in real time, or later, if the preoperative process is video recorded. The scale has previously shown good to excellent inter- and intra-observer reliability and validity [5, 12].

Translation process

The original English-language version of m-YPAS was translated into Danish upon written permission from the developer Dr. Kain; Department of Anesthesiology; University of California, Irvine.

The translation was performed according to the WHO guidelines [14] which include four steps: 1) forward translation, 2) expert group back-translation, 3) pre-testing and cognitive interviewing and 4) a final version (Figure 1).

The forward translation was performed by a native Danish-speaking translator familiar with the terminology of English-speaking cultures. The translation was subsequently conciliated and incorporated into the Danish version by an expert group using a consensus procedure. The expert group included nurse anaesthetists with 1-12 years of seniority, paediatric anaesthesiology consultants, the native Danish translator and a child and adolescent psychiatrist experienced in the use and translation of questionnaires measuring psychopathology, including anxiety in children. Minor changes were included in the Danish version of the m-YPAS after consensus in the expert group, approval by the developer of the m-YPAS and before testing it in a clinical setting.

Forward and back translations were performed by two native English-speaking translators without prior knowledge of the m-YPAS.

Face validity

The face validity of the Danish version of m-YPAS was tested by a cognitive interview in a focus group [15, 16]. The focus group counted five certified registered nurse anaesthetists with 1-20 years of seniority.

Inter-rater reliability

The empirical material included 14 video sequences recorded by the first author of children between two and seven years of age scheduled for urological surgery. Each video sequence had a duration of approximately 15 minutes and showed the children from arrival in the OR until they were anaesthetised.

Non-Danish speaking parents and children as well as cognitively and psychically disabled children were excluded. All children and their parents came from a paediatric unit and directly to the preoperative holding area. During the preoperative period, the anaesthesiologist informed the parents and obtained health data on the child. The child entered the OR with one or both parents, and anaesthesia was then induced. When the child was asleep, the parents left the OR.

The video recordings were analysed separately by three raters: a child and adolescent psychiatrist (CUR), a paediatric anaesthesiology consultant (RPD) and a nurse anaesthetist (PSK, the first author). The raters scored the child’s anxiety using the Danish m-YPAS tool 1) when entering the OR (= induction 1) and 2) at introduction of the anaesthesia mask/or IV (= induction 2).

Statistical analysis

All data were analysed using STATA 12 (StataCorp LP).

Reliability on subscales between the three observers was assessed using weighted kappa (κw) statistics for overall chance-corrected agreement. Agreement between the overall weighted scores were calculated by using the intraclass correlation coefficient (ICC) where values between 0 and 1 are the proportion of total variability due to differences among raters. The closer the ICC is to 1, the higher the agreement in the scoring system.

Ethics

The Danish Data Protection Agency of the Central Denmark Region approved the study (j.no.: 2007-58-0010). Study information to parents and children was given by healthcare professionals. Both parents had to give written informed consent.

Trial registration: The Danish Data Protection Agency of the Central Denmark Region approved the study (j.no.: 2007-58-0010).

RESULTS

Translation process

An understandable Danish version adapted to a Danish clinical setting was developed (without changing the meaning of the words or the construction of the scale) (Figure 2). In the original version, there were words that challenged both the expert group and translators. An example was the phrase, “fidgets with hands” which was difficult to translate into a clinically relevant Danish expression. Danish translations of “fidget with hands” are very different depending on whether the translation relates to children or to adults.

The final version of the translation was approved by Dr. Kain.

Face validity

The focus group agreed that the various items captured the essential information of children’s preoperative anxiety and that the items were recognisable and relevant in everyday clinical practice. Furthermore, the focus group suggested that “Intravenous induction” was added to the description detailing when to score the child. In accordance with this, the description of when to start scoring the children in the OR was changed from mask notification to “parents place their child on the operating table”. It was clarified when to score the child (corresponding to induction 1 and 2, respectively) in the Danish version.

Inter-rater reliability

The 14 video recordings were scored independently by the three raters at both induction 1 and induction 2. The weighted ratings ranged from 23.33 to 100, which showed that the whole range of the scale was used. The ICC was 0.93 for induction 1 and 0.92 for induction 2.

The agreement between observers was also calculated separately for each category in both inductions. The agreements were good to very good at both inductions (Induction 1: &kapha;w: 0.63-0.98; induction 2: &kapha;w: 0.72-0.96).

DISCUSSION

This study concerns translation and preliminary validation of the Danish version of the m-YPAS. The translation was performed systematically in accordance with the WHO guidelines and approved by the original developer of the tool. The expert group represented different professionals familiar with children and anxiety in hospital settings. The final Danish version of the m-YPAS showed good face validity as well as inter-rater reliability when applied to the data consisting of 14 video recordings of anaesthetic procedures.

In accordance with the WHO guidelines, the translation process revealed that the cross-cultural understanding was more important than the linguistic equivalence to obtain a useful result [14]. The translation process was challenged by the original scale consisting of short descriptive sentences without further explanations. The composition of the expert group, including professionals with experience in both observing and anaesthetising children, contributed to a broader understanding of words and concepts describing the reality of children’s preoperative anxiety in a Danish hospital setting.

According to Myers et al, face validity concerns whether the items appear on the surface to tap the contents [16]. In the present study, face validity was assessed by a focus group to ensure that the instrument measured the contents area. In the focus group, all five Certified Registered Nurse Anesthetists (CRNAs) agreed that domains and items for scoring preoperative anxiety in the Danish m-YPAS were identifiable and satisfactory. The testing of the final version of the Danish m-YPAS on 14 video recordings showed good to very good inter-rater reliability. This is consistent with the results of a Swedish study describing the translation process of m-YPAS and the testing of reliability and validity of the m-YPAS in Swedish children [12]. Even though our empirical material was small, the rating results revealed that we used the whole range of the scale. The ICC was high in both inductions 1 and 2, which indicates that the raters also agreed on the overall weighted score (Table 1).

Proczkowska-Bjorklund et al stated that m-YPAS presupposes training and experience in paediatric anaesthesiology in order to observe children’s preoperative anxiety [16]. The effect of anaesthesiology experience was also pointed out in a study by Maclaren et al. They reported that experienced anaesthesiologists’ ability to predict children’s anxiety was better than that of less experienced staff [17]. The challenge in using the Danish m-YPAS is that systematic training and education of raters is necessary to avoid unexpected, misleading or invalid results [16]. To ensure consistent use of the m-YPAS, raters must have a basic understanding of the scale and its purpose. Future training of raters may consist of watching video recordings, repeating ratings and discussing the scoring results.

Due to rules on confidentiality, we were not allowed to bring the video recordings from the Danish clinical setting to the USA to be scored by more experienced raters (Dr. Kain’s group), which could have served as a gold standard. Instead, the first author visited Dr. Kain and was systematically trained by his group. The obtained skills and experience were then used in the preparation of the raters performing the inter-rater reliability testing in a Danish setting.

The small empirical sample and few ratings performed should be considered when inter-preting results. Furthermore, the inter-rater reliability testing could be weakened by the fact that the three raters were also members of the expert group in the translation process wherefore the “shared” familiarity with the m-YPAS may have contributed to a similar approach to the ratings. The strength of the rating process was that it was performed by three people with different professional backgrounds and perspectives on observing anxiety in children.

CONCLUSION

Standardised and validated assessment tools are needed to evaluate interventions to reduce preoperative anxiety in children. The m-YPAS is an internationally approved tool in the field of anxiety measurement in children prior to anaesthesia. The translated version of the Danish m-YPAS developed in the present study has a satisfactory face validity and inter-rater reliability based on a small empirical material (Table 2). Further validation is warranted and the development of systematic training of future raters is necessary.

Perspectives

The validated Danish version of the m-YPAS enables the performance of future descriptive surveys of preoperative anxiety in children in a Danish clinical setting and the comparison of results with other Nordic countries with similar cultures and hospital settings. Also, this tool makes it possible to compare and monitor results of future intervention strategies in this area in Denmark.

Correspondence: Pernille Skovby, Anæstesiologisk-Intensiv Afdeling, Aarhus Universitshospital, Brendstrupgårdsvej 100, 8200 Aarhus N, Denmark. E-mail: pernisko@rm.dk.

Accepted: 27 March 2014

Conflicts of interest:Disclosure forms provided by the authors are available with the full text of this article at www.danmedj.dk

Acknowledgements: The authors wish to thank all the participating children and +arents. Special appreciation is extended to the participants of the focus group and to Dr. Kain and his group who took time for the first author’s study visit and provided insightful knowledge and skills in using the m-YPAS tool.