Validation study of the population included in the Danish Schizophrenia Registry

Schizophrenia is a complex and often disabling neuropsychiatric illness, but effective evidence-based treatment regimens exist to reduce the burden of the illness both for afflicted subjects and their caregivers [1]. However, the patients’ illness insight is often poor. In combination with side effects associated with currently available medications, this poor insight makes it challenging to establish a good rapport and treatment adherence [2]. To optimise and continuously improve the quality of treatment and care for patients with schizophrenia, a national schizophrenia registry was founded in Denmark in 2003 as part of the Danish Quality Improvement Programme [3]. Henceforth, it has been mandatory for hospital-based psychiatric services (both inpatient and outpatient services) to collect data for this registry pertaining to several treatment and care quality measures. The Danish Schizophrenia Registry (DSR) currently covers 19 clinical quality measures relating to the following domains: diagnostic evaluation, antipsychotic medication including adverse reactions, cardiovascular risk factors including laboratory values, family intervention, psychoeducation, post discharge mental health care, assessment of suicide risk in relation to discharge, and assessment of global functioning [3]. Several of the indicators pertain to the incident population. Therefore, valid registration of the incident patient is of importance.

All Danish citizens with a schizophrenia diagnosis since 2004 are included in the registry and data are transferred from the Danish National Patient Registry which is the primary data source on utilisation of hospital-based mental health care services [4]. For a disease-specific registry, such as the DSR, to maintain its integrity, it is of utmost importance that data are valid as this allows clinicians and researchers to maintain trust in the conclusions drawn based on the registry. The aim of this study was to assess if the schizophrenia diagnoses in the DSR are in agreement with the clinical diagnoses in medical records, and to investigate if subjects are classified correctly as belonging to either the incident or the prevalent population. Thus, we did not aim to validate the clinical schizophrenia diagnosis. This would have required a diagnostic interview.

METHODS

Denmark provides tax-funded universal access to primary and secondary healthcare, with no out-of-pocket expenses except for pharmaceuticals and a limited number of other services. Individual level data from all Danish registries can be linked via the unique personal identifier (CPR number) which is assigned to all Danish residents at birth or upon immigration and recorded in the Danish Civil Registration System [5].

We identified all subjects registered in the DSR (International Classification of Diseases, tenth version (ICD-10) codes F20.0 to F20.9) in 2014-2015. The data used in the DSR are transferred from the Danish National Patient Registry. The observation period was chosen to cover the need for a renewed validation of
included diagnoses after the DSR was based on the National Patient Registry in 2011. All subjects in the DSR have had a minimum of one contact in an outpatient or inpatient setting within a year, with a main (A) diagnosis or an auxiliary (B) diagnosis of schizophrenia.

We drew a random sample of 390 subjects, ensuring that each of the five geographic health regions in Denmark were represented in proportion to its number of inhabitants. Subjects were identified by their CPR number.

In total, 24 hospital departments were invited to participate in the validation study. We asked the management of the healthcare services in each region to distribute the CPR numbers to the medical doctors who they had selected to perform a medical record review of the subjects in the study sample. None of the identified subjects were contacted in person; only register data and existing medical record information were used.

An information collection sheet specifying which information to collect when reviewing the medical record was used. The collected data included contact date, department, a yes/no indication of whether the subjects fulfilled the diagnostic criteria for schizophrenia as judged by the experienced clinicians, based on the medical record data, and if this was the case, the date of diagnosis as assessed from the medical record. In the registry, incidence status is defined as the first two first years after diagnosis. The medical doctors performing the reviews were required to be specialists in psychiatry or residents in psychiatry completing the final part of their training (i.e., less than 18 months from specialist approval). They received payment corresponding to one hour of work per medical record review. It was not possible to blind the assessing doctors to the identity of the selected patients or the treating care givers.

This study was conducted as part of DSR quality enhancement efforts and was, as such, approved by the Danish Data Protection Authority.

Statistical analysis

We calculated the positive predictive value (PPV) and corresponding 95% confidence intervals (CI) of the schizophrenia diagnoses registered in the DSR, using the patient’s medical record as a reference. PPV was
defined as the number of patients with a confirmed schizophrenia diagnosis according to the medical record review, divided by the total number of patients with a schizophrenia diagnosis in the DSR that we were able to validate against a medical record. The PPV was calculated overall and stratified by health region to examine variability in diagnosis based on the location of the validation.

Several of the health regions introduced a new/
revised medical record system as from 2008, and therefore we could only reliably assess the year of diagnosis after 1 January 2009. The correspondence of the year of diagnosis was calculated as the proportion of cases with the same year of diagnosis in the DSR and the medical record review, divided by the total number of patients. We calculated a 95% CI around each proportion. We assessed both the 2009-2015 period and the most recent two-year period, 2014-2015, to take into account any changes in registration practice.

Trial registration: not relevant.

RESULTS

In total, we received medical record review data on 325 subjects out of the 390 subjects requested corresponding to an 83% response rate. The requested information was provided by 20 of 24 hospital departments
initially approached. Table 1 summarises the overall results concerning PPV and presents results by health region. In Denmark overall, the schizophrenia diagnosis in the DSR was confirmed by medical record review in 297 out of 325 subjects, yielding a PPV of 91% (95% CI: 88-94%). PPVs varied between regions from 85% to 100%. Results from Region Zealand were associated with uncertainty due to a high rate of missing data. Inpatient wards or outpatient clinics that did not return the requested information were registered as missing. Three regions had no missing data, for the Capital Region of Denmark 13% (20 out of 160) of the requested data were missing and for Region Zealand 60% (45 out of 75) of the requested data were missing. Due to the high rate of missing data in Region Zealand, we performed a sensitivity analysis in which we calculated the overall national PPV without Region Zealand, which yielded a PPV of 91% (95% CI: 87-93%), which was in line with the main result.

In total, 115 patients were diagnosed after 1 January 2009. Out of these, the same year of diagnosis was reported for 85 patients: 74% (95% CI: 65-81%) in the DSR and the medical record review, respectively, in the 2009-2015 period. For patients diagnosed in the most recent two-year period, 2014-2015, the correspondence between year of diagnosis in the DSR and the medical record review was 93% (95% CI: 80-97%) (37/40).

DISCUSSION

This study confirmed that subjects included in the DSR – and thus included in the evaluation of quality of treatment and care associated with individual psychiatric departments and outpatient clinics – fulfil the criteria for a clinical diagnosis of schizophrenia in the majority of cases. Period of incidence is defined in the registry as the first or the two first years after the diagnosis (depending on the quality performance measure), and the results from the present study confirmed that the classification into incident and prevalent populations is in gross accordance with the reference used.

The results of this study are in line with those of previous studies examining the validity of the schizophrenia diagnosis in the Danish National Patient Registry. In 2013, Uggerby et al found that the validity of the registry diagnosis (compared using medical record review as a reference) for 300 subjects with first-time schizophrenia was 97.5% and a worst-case scenario estimate (with all uncertainties going against the diagnosis) showed 89.7% [6]. Another study investigated the reliability of the schizophrenia diagnosis in
a random sample of 100 subjects from the Danish Psychiatric Biobank with a clinical assessment using
the semi-structured interview Operational Criteria Checklist for Psychotic Illness as a reference [7]. This study found a PPV of 87% for schizophrenia and 98% for schizophrenia spectrum disorders. Thus, despite a heterogeneous clinical picture, the reliability and validity of the schizophrenia diagnosis is high according to current diagnostic classifications.

We observed minor variation among the geographic regions with the lowest PPV estimate found in the South Denmark Region (85%) which is close to the ranges reported in the above-cited studies. Possible reasons for the lack of 100% accordance between the DSR and the medical record review diagnosis include the following: 1) medical records are not updated with sufficient clinical information when the diagnosis is decided upon, 2) some patients simply do not fulfil the criteria for schizophrenia, but are diagnosed anyhow due to lack of clinical expertise or due to misunderstandings in the previous diagnostic history, 3) schizophrenia presents with a heterogenous clinical picture and clinicians might disagree on when schizophrenia criteria are fulfilled and when they are not. All three reasons may play a role in producing the lack of full
diagnostic validity recorded for the DSR.

Since the termination of the study, we have modified slightly the inclusion criteria of the DSR. Previously, all subjects with a schizophrenia diagnosis made as from 2004 were included regardless of the diagnosis for which they were currently receiving treatment. To be included in the DSR after the modification of the inclusion criteria, we request that an active diagnosis of schizophrenia be made within the past year, i.e., at least one contact (inpatient admission, outpatient or emergency room visit) to a secondary mental healthcare facility. This is to embrace the rising awareness that schizophrenia is not always a chronic disease and that a proportion of the subjects recover after their first episode [8].The DSR monitors the quality of treatment of delivered care. Hence, including subjects not participating in this care for whatever reason would not add much information relevant for quality inprovement purposes. However, this matter should not disturb the current analyses because herein we only estimated the diagnostic validity at the time of the diagnosis.

Limitations of the study include a lack of analysis of the negative predictive value, i.e. the proportion of subjects with a schizophrenia diagnosis as judged by the reference standard but not included in the DSR. This would have required access to medical records for patients with a diagnosis rather close to schizophrenia and would have introduced a risk of a having missed schizophrenia diagnosis. However, due to Danish legislation and the restricted approval associated with the DSR, such an approach was not feasible. Another limitation is the high proportion of missing data from Region Zealand which hampers the estimate of the PPV from this region which becomes very uncertain. It became apparent that the lack of returned data on 45 patients in Region Zealand and 20 patients in the Capital Region of Denmark was due to work pressure why managers could not find any available employees to perform the medical record reviews. Due to Danish
legislation, the author group themselves did not have access to the necessary medical record information. Additionally, patients in the health region were validated by the treating doctors, and this may have introduced a bias when evaluating the symptoms described in the medical record. This potential bias could have been minimised if it had been possible to blind the assessing medical doctors who performed the chart review. However, due to the legislation guiding the use of clinical quality registries, it was not feasible to perform a blinded assessment. Furthermore, in this study we had no data to further evaluate the underlying cause for the assessment ‘not suffering from schizophrenia’ based on the medical record review. Thus, this assessment included both cases where the medical record data clearly refuted the schizophrenia diagnosis and medical record data that were insufficient to establish the diagnosis with certainty. Finally, we did not record the proportion of medical record reviews performed by specialists or residents in psychiatry, and each medical record review was performed by only one physician. Ideally, two assessors would have been assigned to perform independent reviews of each medical record to minimise any bias and chance error in the evaluation process. However, this was not possible within the practical limits of this study.

CONCLUSIONS

The validity of the population included in the DSR is high. The registry represents a valuable source for enhanced monitoring and improved quality of treatment and care among people with schizophrenia. Furthermore, when linking the registry to other central national registers, the DSR holds unique research properties linking quality of care to pragmatic outcome measures.

CORRESPONDENCE: Lone Baandrup. E-mail: lone.baandrup@regionh.dk

ACCEPTED: 6 August 2019

CONFLICTS OF INTEREST: none. Disclosure forms provided by the authors are available with the full text of this article at Ugeskriftet.dk/dmj

ACKNOWLEDGEMENTS: Charlotte Cerqueira, Danish Clinical Registries, contributed to the design of the study and the application for funding.