INTRODUCTION. Information on data validity is essential for understanding the precision of studies based on data from the Danish National Patient Registry (DNPR). Thus, the aim of this study was to validate the quality of ankle fracture data in the DNPR.
METHODS. We identified all patients from four hospitals with a surgically treated ankle fracture between 1 January 2018 and 31 December 2018. The positive predictive value (PPV) was estimated for a random sample of 10% of patients with both a relevant ankle fracture diagnosis code and a relevant procedure code, as well as for patients with only a relevant ankle fracture diagnosis code or a relevant ankle fracture procedure code. We collected data from medical records and X-rays. Two consultants independently validated the ankle fracture diagnosis and procedure codes reported to the DNPR.
RESULTS. Among the four centres, 651 patients were identified with both an ankle fracture diagnosis and a procedure code. Among these, data from 65 (10%) patients were extracted for validation. For these patients, the PPV for an ankle fracture was 0.95 (95% confidence interval (CI): 0.88-0.99). The PPV for the diagnosis code was 0.89 (95% CI: 0.79-0.95), and for the procedure code, the PPV was 0.82 (95% CI: 0.70-0.90). For patients with only an ankle fracture diagnosis code or only a surgical procedure code, the PPV for an ankle fracture was 0.77 (95% CI: 0.64-0.87).
CONCLUSION. This study showed that ankle fracture diagnosis and procedure codes registered in the DNPR are of a high quality and thus constitute a valuable data source for research on ankle fractures.
TRIAL REGISTRATION. The Danish Data Protection Agency approved the study (journal number 2015-18/62866).
Ankle fractures are common and often require surgery that carries a high risk of post-operative complications [1, 2]. Current evidence is primarily based on smaller studies, and high-volume data are needed to identify risk factors and improve treatment. Data from national patient registers are potentially a valuable source of risk factors. Prior to conducting studies based on data from patient registers, however, gaining insight into their data validity is essential .
Data on diagnoses made and procedures performed in Danish hospitals are transferred to the Danish National Patient Registry (DNPR). Therefore, the DNPR contains prospective and routinely collected data of all patients who have sustained an ankle fracture in Denmark. Overall, the DNPR has a high completeness rate, with 99.4% of all hospital discharges being recorded . The Danish Health and Medicines Authority published an evaluation of diagnosis codes in 1993, which reported a positive predictive value (PPV) of orthopaedic diagnosis codes of 83% . A 2006 study by Lass et al. on patients in the North Denmark region found that the PPV for orthopaedic procedure codes in the DNPR was 63.2% . No other validation studies on ankle fractures have been conducted for national patient registries in Denmark or in other countries.
Therefore, the aim of this study was to validate the diagnosis and procedural codes for surgically treated ankle fractures in the DNPR by estimating the PPV.
This was a validation study on register data from the DNPR. Data were extracted from four hospitals. The Danish Data Protection Agency approved the study (journal number 2015-18/62866). The Head of each of the four departments involved in the study approved the medical record review. Trial registration was not relevant.
Healthcare is fully tax funded in Denmark, allowing free and equal access for the country’s 5.7 million inhabitants. In Denmark, the Civil Registration System assigns all residents a ten-digit personal identifier, enabling linkage of individual patient data across health and administrative systems .
DNPR: The registry documents patient contacts in both private and public hospitals in Denmark. The Danish National Health and Medicine authority reimburses departments for diagnoses and surgical procedures reported to the register. This motivates departments to report all cases to the register, ensuring a high completeness of both discharge diagnoses and surgical procedures . Several variables are reported to the register, including the dates of admission, surgery and discharge. The discharge diagnosis reported to the registry includes one primary and up to 20 secondary diagnoses. Surgical procedures are coded according to the Danish version of the Nordic Medico-Statistical Committee Classification of Surgical Procedures .
The study population included patients > 18 years of age admitted to one of the four hospitals with a surgically treated ankle fracture from 1 January 2018, to 31 December 2018. For a representative geographical distribution across Denmark, one hospital on the island of Funen (Odense University Hospital), one on the island of Zealand (Hvidovre Hospital) and two on the Jutland peninsula were selected (Aarhus University Hospital and Kolding Hospital). The four hospitals were chosen as representative of Denmark as a country overall and for the category of hospital. Two are university hospitals and the other two regional hospitals. The number and selection of the hospitals included were based on an aim to establish a balance between ensuring geographical distribution and representing different types of hospitals with an acceptable use of resources.
We identified all ICD-10 diagnosis codes that refer to ankle fracture – hereafter referred to as “relevant diagnosis code” – and all NOMESCO procedure codes that represent a surgical procedure for an ankle fracture – hereafter referred to as “relevant procedure code” (Table 1).
All patients admitted to one of the four hospitals with either one of the relevant diagnosis codes or relevant procedure codes were identified from the hospitals’ healthcare databases. These databases automatically and directly transfer data to the DNPR. Therefore, data in the hospitals’ healthcare databases and the DNPR are identical. From each of the identified patients in the databases, two samples were extracted for validation. Two samples were chosen because of the higher probability that the entries were correct if both diagnosis and procedure codes were used to identify cases than if entries with only diagnosis codes were used. Consequently, one sample was extracted in which both relevant diagnosis and procedure codes were reported. From this sample, only data for 10% of patients were extracted, as we assumed that patients with both a diagnosis and a procedure code had a high validity (Figure 1). The 10% proportion is in accordance with guidelines for validation published by the Danish Clinical Quality Program – National Clinical Registries . A second sample was extracted in which only the relevant diagnosis code (and no relevant procedure code) or only the relevant procedure code (and no relevant diagnosis code) was available (Figure 2). From this second sample, all patients were extracted for validation.
To validate that information in healthcare databases of the hospitals and the DNPR were identical, we identified ten patients by fracture date, surgery date and side from each hospital and matched them to the DNPR in order to study whether the diagnosis and procedure code were identical in the databases and the register.
Four trained physicians entered data from patients’ medical records, X-rays and the radiologist’s report on a standardised form developed specifically for this study using EpiData Software . The four trained physicians (MH, YC, SOM and NB) were provided with the civil registration number, operation side as reported to the database, date of admission and discharge, and surgical procedure and diagnosis codes. The trained physicians extracted information on operation side (left/right) and the X-ray. Furthermore, the surgeon's level of experience was recorded to investigate whether more experienced surgeons were more precise than less experienced surgeons when reported the correct diagnosis and procedure codes.
The gold standard was defined as the classification of each case from two orthopaedic consultants (PHG and FBD), who independently reviewed the X-rays and the X-ray descriptions from the radiologist and medical records. The consultants classified each patient with regard to the following questions:
1. Does the patient have an ankle fracture?
2. Are the diagnosis codes correct?
3. Are the procedure codes correct?
4. Is the operation side correctly reported to the DNPR?
In the event of disagreement, a third author (BV) was consulted, followed by the entire author group.
The operation side reported to the DNRP was compared with the side reported on the X-ray and in the report from the radiologists, which were defined as the gold standard.
For the two groups – patients with both a relevant diagnosis code and a relevant procedure code; and patients with only a relevant diagnosis code or only a relevant procedure code – the PPV of whether the patients could be classified as having an ankle fracture (question 1) by the two consultants was estimated. Furthermore, the PPV was estimated for whether or not the specific diagnosis code (question 2), procedure code (question 3) and operation side were correct (question 4).
All data management and statistical analyses were conducted with STATA version 15.0 (StataCorp). Results for the PPV are provided as estimates with 95% confidence interval (CI) .
Trial registration: The Danish Data Protection Agency approved the study (journal number 2015-18/62866).
Patients with a relevant diagnosis code and a relevant procedure code
Among the four centres, 651 patients with both a relevant diagnosis code and a relevant procedure code were included in the source population. Among these patients, 57% were female and the mean age was 54 years.
Among these, 65 (10%) were extracted for validation, including 66% females; the mean age was 56 years (Figure 1).
For patients with both a relevant diagnosis code and a relevant procedure code, the PPV of an ankle fracture was very high at 0.95 (95% CI: 0.88-0.99), and for the specific diagnosis and surgical procedure code, a similar high PPV was found (Table 2).
In 32% of the operations in this sample, the consultant was the primary surgeon and thus the responsible person for choosing the correct diagnosis and procedure code. The PPV for diagnosis codes was 0.85 (95% CI: 0.64-0.97) for consultants and 0.91 (95% CI: 0.78-0.97) for non-consultants, with no statistically significant difference between them (p = 0.53). For the surgical procedure codes, the PPV was 0.90 (95% CI: 0.70-0.99) for consultants and 0.77 (95% CI: 0.62-0.89) for non-consultants. The differences were not statistically significant (p = 0.22).
Patients with only a relevant surgical procedure code or only a relevant diagnosis code
Eleven patients were registered with only a relevant diagnosis code and no relevant surgical procedure code, whereas 45 patients had only a relevant procedure code and no relevant diagnosis code.
Of the 56 patients in this sample (8% of the total cohort), 41% were female, and the mean age was 51 years. The PPV for an ankle fracture was 0.77 (95% CI: 0.64-0.87).
If only an ankle fracture diagnosis was registered, the PPV of the patient with an ankle fracture was high 0.91 (95% CI: 0.59-1.00), whereas it was somewhat lower if only a relevant surgical procedure code was registered 0.73 (95% CI: 0.58-0.85).
After excluding ten patients with missing information on operation side in the databases, the PPV was 0.93 (95% CI: 0.86-0.97) for the entire population, 0.94 (95% CI: 0.84-0.98) for patients with a relevant diagnosis code and a relevant procedure code, and 0.92 (95% CI: 0.79-0.98) for patients with only a relevant surgical procedure code or only a relevant diagnosis code (Table 2).
Identical diagnosis and procedure code in the healthcare databases and the DNPR
The diagnosis and procedure codes were identical in the DNPR and the healthcare databases for all 40 patients who were matched between the DNPR and the healthcare databases by fracture date, surgical date and side.
In this study, we found that 95% (95% CI: 88-99%) of the patients registered in the DNRP with an ankle fracture diagnosis and an ankle fracture surgical procedure code could be identified as having a fracture of the ankle. For patients reported to the DNRP with only an ankle fracture diagnosis code or only an ankle fracture surgical procedure code, the likelihood that the patients had sustained an ankle fracture was only 0.77 (95% CI: 0.64-0.87). Therefore, we recommend that in future research on surgically treated ankle fractures, both the relevant diagnosis and relevant surgical procedure codes are used to identify patients with an ankle fracture.
A significantly higher PPV was detected for patients with both a relevant diagnosis code and a relevant procedure code than for patients with only a relevant surgical procedure code or only a relevant diagnosis code. The disadvantage of using a combination of ankle diagnosis codes and surgical procedure codes is that a lower completeness is achieved, as 8% of the ankle fractures were only registered with either a relevant ankle fracture diagnosis code or procedure code. This lower completeness has also been found in studies on hip fractures in which the diagnosis code and procedure code used in combination resulted in an underestimation, whereas using them separately resulted in an overestimation [10, 11]. We believe that the lower completeness reported for a combination of diagnosis and procedure codes is largely outweighed by the higher PPV.
Consultants were not more precise in reporting the correct procedure and diagnosis codes than were other practitioners. Therefore, problems with correct reporting are likely not the result of a lack of experience. That consultants were not more meticulous in their patient reports corresponds well with the results from a study of the inter-observer reliability of ankle fracture classifications . We believe that errors leading to a decrease in PPV are not due to a lack of experience but a lack of meticulousness. This assumption is supported by the fact that for ten patients in this study, the operation side was not reported; and for the remaining 100 patients, the PPV for the operation side was only 0.93 (95% CI: 0.86-0.97). This means that in 7% of cases, the surgeon reported the incorrect side to the register, even though reporting this variable correctly is straightforward. Incorrect reporting to registers has also been found for other simple and straightforward variables . Because the personal identification number and operation side are often used to identify subsequent revisions in the register, such errors – along with the ten patients with missing information on operation side – are a concerning finding.
Our study has several limitations. The most critical limitation is that we did not extract our data directly from the DNPR. The preferred method would be to extract a random sample of patients from the DNPR and then gather information from the hospitals in Denmark where these patients were treated. However, this strategy would be laborious, expensive and time consuming. Therefore, the method used in this study was chosen to ensure the best possible validation at an acceptable use of resources.
Another major limitation is that the method by which we identified relevant patients only allows us to estimate the PPV. We were unable to estimate the negative predictive value, sensitivity and specificity, as we did not have data on the patients that had a malleolar fracture but were reported with a different diagnosis code other than the ones we used to identify cases. Furthermore, we were unable to estimate the completeness of data regarding malleolar fractures in the DNPR, but as the completeness of the DNPR in generally high, we consider this a minor limitation.
A high PPV for ankle fractures in the DNPR was found, especially if a combination of ankle fracture diagnosis and procedure codes was used. Therefore, data on ankle fractures in the DNPR are of a high quality and thus constitute a valuable data source for research on ankle fractures. We recommend that future research on surgically treated ankle fractures uses the combination of relevant diagnosis and relevant surgical procedure codes, even though this combination reduces completeness.
Correspondence Per Hviid Gundtoft, E-mail: firstname.lastname@example.org
Accepted 27 September 2022
Conflicts of interest Potential conflicts of interest have been declared. Disclosure forms provided by the authors are available with the article at ugeskriftet.dk/dmj
Cite this as Dan Med J 2022;69(12):A01220032
Elsoe R, Ostgaard SE, Larsen P. Population-based epidemiology of 9767 ankle fractures. Foot ankle Surg. 2018;24(1):34-9.
Macera A, Carulli C, Sirleo L et al. Postoperative complications and reoperation rates following open reduction and internal fixation of ankle fracture. Joints. 2018;6(2):110-5.
Varnum C, Pedersen AB, Gundtoft PH et al. The what, when and how of orthopaedic registers: An introduction into register-based research. EFORT Open Rev. 2019;4(6):337-43.
Schmidt M, Schmidt SAJ, Sandegaard JL et al. The Danish National Patient Registry: a review of content, data quality, and research potential. Clin Epidemiol. 2015;7:449-90.
Lass P, Lilholt J, Thomsen L et al. The quality of diagnosis and procedure coding in Orthopaedic surgery Northern Jutland. Ugeskr Læger. 2006;168(48):4212-5.
Pedersen CB, Gøtzsche H, Møller JO et al. The Danish Civil Registration System. A cohort of eight million persons. Dan Med Bull. 2006;53(4):441-9.
NOMESCO. Classification of surgical procedures. NOMESCO, 2009. norden.diva-portal.org/smash/get/diva2:970548/FULLTEXT01.pdf.
RKKP. Vejledning for validering af data fra kliniske kvalitetsdatabaser. RKKP, 2017. www.rkkp.dk/siteassets/de-kliniske-kvalitetsdatabaser/projekter/valideringmanuel_rkkp_-final_06062017.pdf (20 Jun 2020).
EpiData Software. epidata.dk (20 Jun 2020).
Høiberg MP, Gram J, Hermann P et al. The incidence of hip fractures in Norway -accuracy of the national Norwegian patient registry. BMC Musculoskelet Disord. 2014;15:372.
Hjelholt TJ, Edwards NM, Vesterager JD et al. The positive predictive value of hip fracture diagnoses and surgical procedure codes in the Danish Multidisciplinary Hip Fracture Registry and the Danish National Patient Registry. Clin Epidemiol. 2020;12:123-31.
Viberg B, Haidari TA, Stork-Hansen J et al. Reproducibility of the stability-based classification for ankle fractures. Eur J Orthop Surg Traumatol. 2019;29(5):1125-9.
Kosy JD, Kassam AAM, Hockings M. National Joint Registry data inaccuracy: a threat to proper reporting. Br J Hosp Med (Lond). 2013;74(12):691-3.