Comparison of Child Behavior Checklist subscales in screening for obsessive-compulsive disorder

Obsessive-compulsive disorder (OCD) is a psychiatric disorder, which according to the International Classification of Diseases (ICD-10) is characterised by recurrent and persistent obsessions and/or compulsions. The condition affects about 0.5-2.0% of the population. Affected persons attempt to ignore or suppress recurrent thoughts or impulses by performing certain rituals to keep interrelated anxiety at bay [1]. In the American diagnostic system (DSM-IV), OCD is classified as an anxiety disorder [2], but otherwise DSM-IV characteristics are generally in line with those mentioned in the ICD-10 classification.

The aetiology is thought to be correlated with an imbalance in serotonergic concentration, changes in glucose metabolism and structural changes in different parts of the brain. A predominant genetic disposition is presumed and studies have indicated that genetics may play an important role in certain subtypes of OCD [3]. Onset of symptoms is seen earlier in boys than girls, possibly due to a more vulnerable nervous system. Boys are also more likely to have a co-morbid externalizing behavioural disorder, which may result in earlier referral [4].

OCD is associated with considerable psychosocial disability and it is essential to recognize, diagnose and treat children and their families as early as possible [4]. However, cases of OCD in childhood are often diagnosed long after onset of symptoms [5]. Psychometric instruments are useful in screening and in the diagnostic process. One of the instruments widely used in screening is the Child Behavior Checklist (CBCL) [6], whereas for classification and treatment evaluation, recommendations in the literature tend to favour Children’s Yale-Brown Obsessional Scale [7] and Leyton Obsessional Inventory-Child Version [8]. In 2007, Uher et al created an OCD scale consisting of seven items: the Short OCD Screener [9].

Few studies on OCD scales derived from the CBCL have been published. From factor analysis of 11 items (OCS-11) presumed to predict OCD, Nelson et al created an OCD scale called the OCS-8 which consisted of eight items [10]. In 2006, Geller et al found that the reliability and validity of the OCS-8 were good [11]. However, Storch et al found that the OCS-8 was not reliable. Instead, a 6-item scale (OCS-6) proved to be valid [12].

In 2006, Hudziak et al found that the optimal cut-off score on the OCS-8 in a general population was five with an area under the curve (AUC) of 0.88, a sensitivity of 92% and a specificity of 67%. In 2008, Ivarsson & Larsson found that two items alone, “Obsessions” and “Compul- sions” (OCS-2) were the strongest predictors of OCD. Sensitivity and specificity were not increased by adding more items [13].

The present study aims to explore the existing CBCL-derived OCD scales to determine whether any of them are markedly better than the others as screening tools. In addition, various single items will be examined.

MATERIAL AND METHODS

Participants

The study sample was derived from a Danish child psychiatric outpatient clinic during the five-year period from 2000 to 2005. Inclusion criteria were an OCD diagnosis based on the ICD-10 criteria and a CBCL questionnaire filled in by parents, which led to a total of 104 included OCD cases. However, 20 children were excluded because of missing CBCL-values, which resulted in an OCD group of 84 children: 39 boys and 45 girls between four and 17 years of age, with a mean age of 11.7 and 12.6 years, respectively. All 84 included cases had OCD as their primary diagnosis. Co-morbidity was not systematically registered, but CBCL scores on different subscales were evaluated in order to take this aspect into consideration.

Both the “anxiety/depression” and the “thought problems” CBCL subscales were scored higher in the OCD group than in the clinical control group. The “social problems” subscale was scored low in the OCD group, which indicated few autistic symptoms. If an autism spectrum diagnosis occurs in combination with OCD, the former will take precedence, and such a child would not be included as an OCD patient.

The clinical control group was constructed by random selection of gender- and age-matched patients from the same clinical data source [14]. In each case, the main diagnosis was identified. The mean age, the diagnostic distribution and the CBCL mean total problem score is presented in Table 1.

Children in the OCD group and the clinical control group were all diagnosed according to the ICD-10 man-ual based on standard clinical assessment. The staff was trained and experienced medical doctors, nurses and psychologists led by a senior child and adolescent psychiatrist who sanctioned all diagnoses.

The general population control group was derived from the material used in the Danish CBCL standardisation in 1999 consisting of 1,300 children aged 4-16 years [15]. These children were not further assessed and some may have been referred for psychopathology.

Trial registration: not relevant.

The Child Behavior Checklist

The CBCL/4-18 is a questionnaire designed to reflect parents’ views of their children’s competencies and difficulties. The first part of the checklist consists of a number of items tapping into leisure activities, school and social competencies. The second part is a problem checklist comprising 118 closed and two openended items that cover a wide range of behavioural and emotional problems. Parents are asked to rate their child’s behaviour over the past six months. Responses are scored on a 0-1-2 scale. 0 is scored if the description is “not true”, 1 if the description is “somewhat or sometimes true”, and 2 if the statement is “very or often true”. This study is based on the 1991 version of the CBCL questionnaire [16] which was standardised in Denmark by Bilenberg in 1999 [15]. In the revised version, the CBCL 2001 [6], five items have been changed of which only one, item (#99 Too concerned with neatness or cleanliness) is included in the analyses made in this study.

Table 2 illustrates how each of the OCD scales tested is composed. The following single items were examined: #9, #66, #84, #85 and #99. These items were selected because of their obvious or possible correlation to the OCD syndrome. The remaining six items of the 11 items listed in Table 2 seem to correlate more with anxiety and nervousness.

Data analysis

1. T-test was used to test significant differences in mean scores for the OCD sample compared with the two control samples.

2. Receiver operating characteristics (ROC) curves were used to find the optimal cut-off score and to evaluate the AUC.

3. Sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV) and correct classification proportions were calculated from 2-by-2-tables.

ROC curves illustrate sensitivity and 1-specificity for a particular diagnostic test along the whole range of the scale [14]. AUC expresses the statistical likelihood that a random pair of one case and one control proband is correctly classified [5]. By means of ROC analysis, the optimal cut-off point can be identified as the value closest to the upper-left corner [12]. The kappa-value is a measure of diagnostic validity: the ability of an item or a scale to predict the diagnosis [14].

RESULTS

Testing of scales

T-tests showed that all four scales significantly differentiated between the OCD, the clinical control and the normative groups at the p < 0.005 level.

Receiver operating characteristics analysis

For OCD versus the clinical control group, the AUC was 0.743 for OCS-11, 0.745 for OCS-8, 0.748 for OCS-6 and 0.787 for OCS-2 (Figure 1).For OCD versus the general population, the AUC was 0.965 for OCS-11, 0.964 for OCS-8, 0.963 for OCS-6 and 0.949 for OCS-2.

For OCD versus the clinical control group, the optimal cut-off scores were nine for OCS-11, eight for OCS-8, six for OCS-6 and three for OCS-2 (Figure 1). For OCD versus the general population, the optimal cut-off scores were seven for OCS-11, five for OCS-8, four for OCS-6 and two for OCS-2. The 95% confidence intervals of the AUCs overlapped, which indicates that none of the tested scales proved to be significantly superior to the others.

Table 3 illustrates the psychometric values for each single item tested and for the optimal cut-off scores for each of the scales. Item #9 Can’t get mind off thoughts, scored 1 or 2, had the best sensitivity at 90.5%. However, with a specificity of 35.7%, only about one third of the controls were screened negative by this single item. The high sensitivity explains why the NPV was best in item #9 as well: a 78.9% likelihood of correct negative screening (Table 3). Specificity was highest in item #99 Too concerned with neatness or cleanliness, scored 2. It was 95.2% and 100% for OCD cases versus clinical controls and general population, respectively. The PPV was equally good for every item (all at score 2).

Item #9, scored 1 or 2, classified 89.3% OCD cases correctly when used among general population controls. Among clinical controls, item #66 had the best correct classification rate (CCR) of 75.0%. And finally, the kappa-values for item #9 and item #66 ranged from 0.500-0.786 (Table 3).

DISCUSSION

There is a strong need for validated psychometric instruments for easy and valid discrimination of children with OCD from a general population and from referred children with other diagnoses. Based on factor analyses of the CBCL problem items, a number of different scales have been suggested.

Four CBCL-based OCD subscales, named OCS-2, -6, -8, -11, were compared. As expected, the mean scores on all four scales were significantly higher in the OCD group than in the clinical and population-based control samples.

The results of the ROC analysis showed an almost equal AUC for all four scales when discriminating OCD patients from the general population. If population screening is the aim, there is a small but non-significant advantage to be gained by adding a number of non-specific symptoms to the OCD core symptoms. On the other hand, when screening OCD cases in a clinical group, using only the core symptoms (OCS-2) seems superior; however, not significantly so.

Storch et al, who developed the OCS-6, based their results on control groups with externalizing and internalizing disorders [12]. They found an optimal cut-off point at five with the following values (present-study results in brackets): sensitivity 86.8% (82.1%), specificity 56.1% (externalizing disorders) and 67.3% (internalizing disorders), respectively, (53.6%). In the present study, we recommend a cut-off point at six when screening among clinical patients in whom the sensitivity is 73.8% and the specificity is 65.8%. In the study by Storch et al, the AUCs were 0.658 and 0.732, respectively. In the present study, the AUC was 0.787 when discriminating OCD patients from clinical controls.

Nelson et al, who constructed the OCS-8, achieved the following results in the general population group: sensitivity 75.3% (92.9%), specificity 97.3% (88.1%), PPV 96.5% (88.6%) and NPV 79.8% (92.5%). Results observed in our clinical control group had lower scores than the corresponding group in the Nelson study: sensitivity 75.3% (70.2%), specificity 87.7% (69.0%), PPV 85.9% (69.4%) and NPV 78.1% (69.9%) [17]. One explanation of the poorer, but still acceptable, results could be a different composition of the clinical groups. Nelson et al had a group of 73 children, of whom 34 had attention deficit hyperactivity disorder (ADHD), 23 conduct disorders and 29 an emotional disturbance. Compared to the clinical group in this study, there are proportionally more externalising diagnoses, which have been shown to be more easily differentiated from OCD [10].

Overall sensitivity, specificity, PPV, NPV, CCR and kappa-values were found to be very alike across the four scales at the optimal cut-off points. The ability to discriminate OCD from the general population was very good for all scales, with kappa-values ranging from 0.810 (OCS-6 and OCS-8) to 0.857 (OCS-2 and OCS-11).

Kappa-values measuring discrimination of OCD cases versus clinical controls were almost identical, around 0.4 for OCS-11, OCS-8 and OCS-6 at the best cut-off points, which is moderately good (0.40-0.60). OCS-2 attained a kappa-value of 0.476. These results suggest that a scale comprised of the two core symptoms is more accurate as a clinical screening tool than the remaining scales. This supports the findings of Ivarsson & Larsson [13].

The OCS-2 scale also yielded slightly better results in both groups in all areas except for sensitivity in the clinical control group, which was 64.3% compared with a sensitivity of 75.0% for OCS-11. However, despite the superior kappa score of 0.476, the OCS-2 suitability as a screening tool should be questioned. Due to its low sensitivity, there is a risk that too many OCD patients may be disregarded.

One way to construct a more optimal scale could be to use multiple informants. Nevertheless, no significant correlation was found between the self-report questionnaire Spence Children’s Anxiety Scale and the parent-rated CBCL internalizing subscale [18]. This indicates either a mismatch between children and parents as reliable informants and/or a substantial difference in reporting of the same symptoms. Another study concluded that teachers were better than parents at spotting internalizing symptoms in children [19].

The co-morbidity often present in OCD children is likely to have a great influence on the ability of the scales to discriminate OCD from other internalising diagnoses.

Moreover, the CBCL only evaluates the presence and the frequency of symptoms, not their severity. Since symptoms of OCD are also present as part of normal development, there is a risk that typically developing children are screened positive. This study has not invest-gated co-morbidity in the OCD group, and the general population control group was not assessed for psychopathology. Undetected diagnoses, particularly internalising symptoms, may have influenced the results. Moreover, the random selection of the clinical control group may not represent the actual psychopathological distribution.

When diagnosing OCD, no scale can be a substitute for thorough diagnostic assessment. Nevertheless, bearing in mind the better prognosis associated with early detection and the relatively low costs of distribution and use of questionnaires, this speak in favour of further development and use of screening tools.

CONCLUSION

None of the CBCL based OCD scales tested were significantly better than the others at differentiating OCD patients from controls. However, using a two-item scale tapping into the core symptoms was almost as effective in population screening and seems even better at discriminating OCD patients in a mixed clinical population than the scales with more items.

Correspondence: Pia Aaron Skovby Andersen, Børne- og Ungdoms- psykiatrisk Afdeling, Odense, Psykiatrien i Region Syddanmark, 5000 Odense C, Denmark. E-mail: pianrs@rm.dk

Accepted: 24 August 2012

Conflicts of interest:Disclosure forms provided by the authors are available with the full text of this article at www.danmedj.dk