Poor predictive ability of the risk chart SCORE in a Danish population

Cardiovascular disease (CVD) remains one of the leading causes of mortality in developed countries and it is the number one cause of death globally [1]. Comprehensive cardiovascular risk assessment models have consequently been developed to assist prevention and ensure early identification and treatment of persons at high risk of dying from CVD. The European risk chart Systematic COronary Risk Evaluation (SCORE) is such a risk assessment model.

It is derived from data from > 200,000 individuals pooled from 12 European cohort studies collected in the 1980-90s (from Denmark: several population studies from Research Centre for Prevention and Health), and constructed by the European Society of Cardiology (ESC) with the aim of providing better predictive ability for European individuals. SCORE estimates the ten-year risk of a fatal CVD event for individuals in the age range 40-65 years without diabetes [2].

The ESC guidelines for CVD prevention in clinical practice advocate the use of SCORE for more rigorous identification of asymptomatic individuals who are at increased risk of CVD [3]. The Danish Society of Cardiology has embraced these guidelines, and since 2004 the systematic use of SCORE in primary practice has been recommended in Danish guidelines on CVD prevention [4].

Like in many other European populations, cardiovascular mortality has declined in Denmark during recent decades [5]. Risk estimates based on cohort studies that started more than a decade ago are thus likely to estimate cardiovascular risk incorrectly. The applicability of SCORE has been evaluated in several European countries. Studies show that SCORE generally overestimates the CVD risk [6-10], and so the chart has been recali-brated for a number of European countries [11]. Despite national guidelines dating back to 2004 advising systematic use of the risk chart in general practice in Denmark [4], the predictive ability of SCORE in a Danish population has never been investigated.

The individual SCORE risk estimation is based on gender, age, smoking status, systolic blood pressure, and total cholesterol. A series of other CVD risk factors are classified as ‘qualifiers’. Social deprivation is such a qualifier [3, 12]. This means that risk may be higher than indicated by SCORE in socially deprived individuals. Yet, evidence is limited on the performance of SCORE in different socioeconomic groups.

The aim of this study, therefore, was to use population-based data to assess the predictive ability of SCORE in Denmark by the socioeconomic factors of income and education.

MATERIAL AND METHODS

Copenhagen City Heart Study

The Copenhagen City Heart Study (CCHS) is a prospective cardiovascular study of 20,000 women and men aged 20 years and older, randomly drawn from the Copenhagen Population Register [13]. Four studies were conducted (1976-78, 1981-83, 1991-94, 2001-2003). The third study (1991-1994) of the CCHS was selected for this study because it is the most recent study for which the prerequisite 10-year follow-up was available from The Danish Register of Cause of Death (data were avail-able up to 31-12-2009). Information on examination procedures and distribution of risk factors is described in Schnohr et al, 2001 [13].

Study sample

From a total study sample of 10,135 individuals from the third study by the CCCHS, 4,224 subjects were included in the analyses. Reasons for exclusion were: failure to fulfil the age restriction of SCORE (40-65 years) (n = 5,441, 54%), previous CVD events (n = 229, 2.3%), diabetes (n = 131, 1.3%) and missing values on covariates (n = 110, 1.1%).

Definition of endpoints

The study population was traced by means of the unique identification number of all Danish citizens which was linked to the Danish Cause of Death Registry. The ICD codes of death certificates used to create SCORE [2] were applied, i.e. International Classification of Diseases (ICD)-10 codes I10-I25, I44-I51, I61-I73 and R96-R96.1 and the corresponding ICD-8 codes 401-414, 426-443 (with the exception of 426.7, 429, 430, 432.1, 437.3, 437.4, 437.5) and 798.1, 798.2

Statistical analyses

The predictive ability of SCORE was studied by means of calibration measures, expected-to-observed (E/O) mortality ratios and discrimination measures. Calibration is the extent to which predicted and observed events coincide. For this the Hosmer-Lemeshow goodness-of-fit statistic was used. Low values indicate good calibration, and values > 20 indicate significant lack of calibration (p < 0.01) [14]. The E/O-ratios supplemented the calibration by assessing the (dis)agreement in subgroups of age, sex and socioeconomic indicators. Discrimination is the model’s ability to correctly rank individuals according to risk. For this sensitivity, specificity and predictive values at the risk threshold of SCORE (≥ 5%) as well as the area under the receiver operator characteristic (AUROC) curve was assessed. The area under the curve is the probability that a person who experiences an event will have a higher risk score than a person who does not experience an event.

In 2012 Denmark was re-categorized from a high CVD risk region to a low CVD risk region and hence now uses the low-risk version of SCORE (Figure 1). Consequently, the number of expected CVD events estimated by both SCORE high-risk and low-risk was calculated for comparison. This calculation used the same methodology, i.e. risk factors (age at baseline, sex, smoking yes/no, systolic blood pressure (mmHg) (measured in a sitting position on the left upper arm after five minutes of rest, determined by a London School of Hygiene sphygmomanometer [13]) and total cholesterol (mmol/l) (determined by cholesterol oxidase-phenol-aminophenazone (CHOD-PAP) method [13]) were used) and risk coefficients as in SCORE [2]. Socioeconomic position was defined by income, measured as the total gross household income (categorized into low (< 100,000 DKK), lower middle (100,000-199,000 DKK), upper middle (200,000-399,000 DKK) and high (≥ 400,000 DKK)) and education measured as vocational training (categorized into none, ≤ 1 year, 1-3 years, > 3 years). The expected number of CVD events was calculated as the sum of individual absolute risks [6].

The statistical analyses were performed using SAS version 9.2. Data are presented as means ± standard deviations and percentage frequencies.

Trial registration: not relevant.

RESULTS

The baseline characteristics of the study population by sex, age, income and education are given in Table 1. Of the 4,224 subjects in our study population, 157 (114 men and 43 women) developed a fatal cardiovascular event over the course of ten years. The number of events predicted by SCORE was 212 for the high-risk model and 122 for the low-risk model. When looking at the distribution of the actual, observed events by the SCORE risk classification of the study population (Table 2), there was good agreement for men: the majority experiencing a fatal CVD event were classified by SCORE as high-risk subjects (86.8% and 70.2% of all events, respectively, for SCORE high-risk and SCORE low-risk). The agreement was less satisfactory for women: of the 43 women experiencing an event, 23 (53.5 %) and 27 (62.8%), respectively, were classified in the intermediate risk of 1-4%.

Hosmer-Lemeshow χ2 statistic was ± 33 (p < 0.01), exceeding the limit of 20, and thereby indicating significant lack-of-fit for both models. However, when stratifying by sex, both models showed a better fit for women, slightly better for SCORE low-risk (p ≈ 0.19, χ2 ≈ 11.2) than for SCORE high-risk (p ≈ 0.13, χ2 ≈ 12.4). For men both models still lacked fit with the data, with SCORE low-risk (p < 0.01, χ2 ≈ 34.4) performing slightly poorer than SCORE high-risk (p ≈ 0.01, χ2 ≈ 31.2).

In Table 3 the E/O ratios are stratified by age, sex, income and education. Overall, SCORE high-risk overestimated the number of events, especially among women. SCORE low-risk was a relatively good predictor among women and among men in the age range 45-54 years, but underestimated the risk among men aged ≥ 55 years. SCORE high-risk was a better predictor among the lower income and education groups, while SCORE low-risk was a better predictor among the higher income and education groups. This was applicable for both women and men – though most markedly for men.

In Table 4, we quantify and compare the capacity of the models to predict events by means of the ROC curve and by comparing sensitivities, specificities and predictive values at the recommended risk threshold for intervention (≥ 5%). With areas under the ROC-curve of ≈ 0.7-0.8 for both models, the discriminative predictive ability of SCORE in the study population was acceptable. Sensitivity was best among men, and, conversely, specificity was best among women. The 5% threshold showed good balance between sensitivity and specificity. The balance was better among men than among women.

DISCUSSION

Overall, we found that SCORE has good discriminative ability. However, calibration is inadequate and E/O ratios vary with age, sex and socioeconomic status in this Danish population. This is in line with the common observation that SCORE performs relatively well in different populations in terms of discrimination, while calibration varies widely [2, 8]. SCORE high-risk, which until 2012 was recommended in Denmark, substantially overestimated CVD events in the population. This finding is in accordance with observations in other Nordic countries [6, 7, 9]. Besides being a better predictor in women, SCORE low-risk, which is today’s recommended version, did not perform much better when predicting absolute risks.

The discriminative values match those previously reported in the literature (AUROC = 0.71-0.84) [15]. Few studies have reported sensitivity and specificity, but for those which have, our findings are comparable [8, 9]. A lower sensitivity among women (and conversely higher specificity) is also in line with previous studies [16]. The low positive predictive values reflect the low prevalence of fatal CVD events in the study population. Regarding calibration, the performance of SCORE is obviously affected by the secular changes in CVD mortality between the time when the function was derived and the time at which it was applied.

Our analyses also point to substantial variation in CVD risk according to sex and age, and show that this variation exceeds what may be accounted for by SCORE. This result is in line with an Icelandic study of SCORE. In this study, the risk for younger men tallied with that in low-risk European countries and diverged towards that in high-risk countries with increasing age, while the risk for women was identical to that for women in low-risk countries [9]. Our analyses also showed that the predictive ability of SCORE varied by socioeconomic position. SCORE high-risk was a better predictor of risk in lower socioeconomic groups, while SCORE low-risk was a better predictor in higher socioeconomic groups. This finding is similar to the outcomes reported in a recent study in British men which found that overestimation by SCORE was particularly marked in high socioeconomic classes [17]. This finding also supports the classification of social deprivation as a SCORE qualifier [3] , and is in accordance with Danish literature on socioeconomic differences in CVD [18].

Some limitations of this study need to be addressed, one of which is that the study data date back to 1991-1994. This is an inherent limitation to any such study since ten-year follow-up is required. Another limitation is the relatively low number of events, particularly among women (n = 44), which needs to be noted for reliability interpretation purposes, especially in age-stratified subgroups. A main strength of the study is the quality of the data used, both from the Danish Register of Cause of Death and the CCCHS.

The purpose of this study was to assess the predictive ability of SCORE in a Danish population. This study demonstrated that there is a need for recalibration of SCORE to risk levels and risk factor distribution in the Danish population. Because of secular time changes in CVD risk, a viable solution would be a continuous update of SCORE, which is feasible in Denmark where recent mortality statistics and epidemiological studies of risk factor distribution are accessible. What truly matters when assessing risk prediction models is the performance close to the levels at which decisions are made, not at the extremes when decisions are obvious. SCORE’s classification of women at risk in the study population is therefore disappointing. This result adds to the growing evidence in favour of lowering the thresholds in female populations [16, 19] and amplifies the gain of a recalibration.

While estimating total risk seems eminently logical, it remains unknown whether an individual risk estimation approach results in risk factor – and ultimately morbidity – reduction [12], and research is warranted regarding clinical benefits and cost effectiveness. An innate consideration is the number needed to treat to achieve desirable outcomes. Finally, an important challenge for the future will be to introduce efficient public health policies with an emphasis on a population approach to support the current high-risk approach for cardiovascular prevention [20].

Correspondence: Madina Saidj, , Forskningscenter for Forebyggelse og Sundhed, Koncern Plan Udvikling og Kvalitet, Afsnit 84/85, Glostrup Hospital, 2600 Glostrup, Denmark. E-mail: m@saidj.dk

Accepted: 14 February 2013

Conflicts of interest:None. Disclosure forms provided by the authors are available with the full text of this article at www.danmedj.dk.