Register-based algorithm to detect post-operative complications in patients with ovarian cancer

Approximately 500 ovarian cancer (OC) cases are diagnosed annually in Denmark [1]. OC is the most fatal gynaecological cancer, with almost 400 annual OC-related deaths [1]. The standard treatment of OC combines surgery and chemotherapy [2, 3]. In advanced OC classified by the International Federation of Gynaecology & Obstetrics (FIGO) as stage III-IV, surgery is extensive and associated with a risk of severe post-operative complications. The severity of complications varies from simple urinary infections to severe sepsis or leakage from bowel anastomosis requiring intensive care and is associated with a risk of death [4, 5]. Surgical complications are traditionally classified using the Clavien-Dindo (CD) grading system [5].

Medical history, surgery and early post-operative complications were registered manually by clinicians online in the clinical database, the Danish Gynaecological Cancer Database (DGCD) [1]. From January 2023, direct manual registration in the DGCD was replaced by data capture from other registers such as the Danish National Patient Registry (NPR) [6, 7]. Data in the NPR are retrieved from patient charts and linked to a unique social security number for each citizen in Denmark. The transformation from prospective manual registration to data capture from various registers will expectably change data quality. Regarding post-operative complications, data quality will likely improve because of the previously insufficient manual registration. Thus, it is of utmost importance to develop validated algorithms to ensure a correct import of data to maximise the benefits of retrieving data from registers and avoid significant non-measurable changes in quality.

The present study aimed to develop and validate an algorithm for identifying post-operative complications in OC patients using data registered in the NPR.

Methods

We developed an algorithm to detect 30-day post-operative complications based on data from the NPR between 2007 and 2012. The algorithm was validated using a cohort of patients with advanced OC from a previous study conducted at Odense University Hospital, Denmark (OUH), as the reference standard [8].

The Danish National Patient Registry

The NPR is a nationwide registry that was established in 1977. Every Danish citizen who comes into contact with Danish healthcare is registered in the NPR, and data are linked by a civil registration number (CPR number). Data consists of the International Classification of Diseases 10 codes (ICD-10) and the derived Danish Medical Classification System codes (SKS codes), including date of surgery, clinical procedures, admissions and date of hospital discharge [6, 7]. Survival status is registered in the CPR registry with the date of death. In Denmark, every SKS code is linked to a diagnosis-related group (DRG), with an associated fee. Thus, the SKS codes are the basis for reimbursement to the department upon discharge.

The cohort

The cohort consisted of 330 patients with OC with FIGO stage IIIC-IV who underwent surgery at the OUH between 2007 and 2012. All medical files were reviewed for data collection on treatment, surgical complexity and post-operative complications. The presence of a specific post-operative complication was determined as either “yes” or “no.” In cases with more than one post-operative complication, the most severe complication defined the CD grade. A previous study described the validation cohort in more detail [8].

The algorithm was evaluated on this cohort, and all derived results were validated by reviewing patients’ files and defined as a validation cohort.

Algorithm

The algorithm was developed by a working group consisting of five medical doctors, two of whom are experts in oncologic gynaecology. Two doctors reviewed all SKS and ICD-10 codes and classified them into two groups: possible/not possible complications of OC surgery. All possible complications were then subdivided into 26 groups (for details of the groups, please see the Supplementary material). All relevant SKS and ICD-10 codes were then classified under one of these groups.

The algorithm was designed to include only SKS codes registered from the primary surgery date to 30 days after surgery. The procedure and diagnostic codes were applied to the relevant complication groups to grade the severity of each complication. If coupled with a specific diagnosis code, the same procedure code could appear in multiple subgroups.

Data regarding stays at intensive care units from the date of primary surgery to 30 days after surgery were added to the algorithm. Thus, if a complication was linked to an intensive care unit stay, the complication was graded CD 4. However, staying at an intensive care unit up to 24 hours after the surgical procedure was not graded as CD 4, as routine intensive monitoring is most often planned for the first day after extensive surgery. Finally, if death occurred within 30 days of surgery, the complication was registered as CD 5.

Statistics

Data were retrieved from the NPR using the patient’s unique CPR number. The algorithm was applied to the patients in the cohort, thereby capturing and categorising data regarding complications and their severity.

We compared frequencies between complications identified by the NPR algorithm and complications manually registered in the validation cohort according to the CD grade to assess agreement and evaluate the accuracy of the algorithm; sensitivity (SN), specificity (SP), positive predictive value (PPV) and negative predictive value (NPV) between groups. We subdivided complications into mild (CD 0-2) and severe (CD 3-5). Sub-analyses of three relevant and commonly selected complications and reoperations were made: intra-abdominal abscess, wound dehiscence and bowel perforation. Finally, the Cohen’s kappa coefficient (κ) test was used to determine agreement, taking chance agreement into account, between the algorithm and the validation cohort. For the interpretation of agreement, Landis and Koch’s scale was used [9]. Data were analysed using SAS version 25, and SPSS version 28.

Trial registration: not relevant.

Results

A total of 330 patients from the validation cohort were included in the study, and thus, the NPR algorithm was applied to these 330 patients. Most patients (79.4%) had no major comorbidities according to the Charlson Comorbidity Index when using the algorithm [10], 236 (71.5%) were FIGO stage IIIC and 94 (28.5%) were FIGO stage IV (Table 1).

In the validation cohort, 133 (40.3%) patients had a complication registered compared with 90 (27.3%) patients when applying the NPR algorithm to the cohort. The severity of complications graded according to the CD classification is presented in Table 2. According to severity, 66 (20.0%) in the validation cohort and 56 (17.0%) in the NPR algorithm presented with severe complications (CD 3-5).

The overall SN and SP of the algorithm were 56.4% (95% confidence interval (CI): 48.0-65.0%) and 92.4% (95% CI: 86.5-93.0%), respectively, with a κ of 0.43, PPV was 83.3% (95% CI: 75.6-91.0%) and NPV was 75.8% (95% CI: 70.4-81.2%). The SN and SP for the algorithm regarding severe complications (CD 3-5) were 74.2% (95% CI: 67.4-83.6%) and 97.4% (95% CI: 95.5-99.4%), respectively, with a κ of 0.65. The PPV was 87.5% (95% CI: 78.8-96.2%) and the NPV was 93.8% (95% CI: 90.9-96.7%).

Sub-analyses for severe complications (CD 3-5) were made for selected complications: intra-abdominal abscess, wound dehiscence and bowel perforation. SN and SP ranged from 33.3 to 69.0% and from 91.9 to 100%, respectively, with a κ range of 0.46-0.74. PPV ranged from 87.0 to 100% and NPV ranged from 79.1 to 91.2% (Table 3). A sub-analysis for reoperation (yes or no) showed an SN of 89.2% (95% CI: 79.9-94.7%), and an SP of 99.0% (95% CI: 97.6-99.6%), with a κ of 0.89. The PPV was 91.7% (95% CI: 82.7-100%); the NPV, 98.6% (95% CI: 97.2-100%).

Discussion

Overall, the NPR-based algorithm performed moderately compared to prospectively registered complications in a department-specific database. The algorithm improved to substantial agreement regarding severe (CD 3-5) complications.

Several reasons for only moderate agreement could be suggested. In Denmark, the initial treatment of OC is centralised at four specialised gynaecological oncological centres. Registration of post-operative complications relies on the individual surgeon. If complications are typed as text in the medical files without accurate registration, this may result in non-transfer and under-reporting of data to the NPR.

Complications may arise after discharge from the specialised centres and be handled by general practitioners or regional hospitals. In such cases, the awareness of registration of specific OC-related post-operative complications may be limited. In agreement, a systematic review by Martin et al., comprising 119 studies investigating short-term outcomes after pancreatectomy, oesophagectomy and hepatectomy, highlighted inconsistencies in reporting complications. They concluded that the inconsistencies were due to incomplete medical records, incomplete registration and treatment of complications at different locations, including specialised facilities, local hospitals and general practitioners [11]. In contrast to the NPR-based algorithm, all medical files were intensively scrutinised for any intra- and postoperative complications in the validation cohort.

When interpreting sub-analyses related to reoperations in the present study, it became evident that addressing complications across various healthcare institutions poses significant challenges. The algorithm demonstrated a high SN (89.2%) and SP (99.0%) in detecting reoperations. A minor subset of cases (n = 3) exhibited a discrepancy, where reoperations were documented in the NPR but not in the validation cohort. This discrepancy may likely be attributed to patients undergoing reoperations at local hospitals.

The SN and SP of the algorithm were higher for severe complications. This reflects incomplete registration of mild complications, which may be considered unimportant by the healthcare professional, and not coded correctly when the patient's data are registered upon discharge.

A notable observation is the lower total number of CD 3-4 complications registered in the NPR compared to the validation cohort. While the overall SN for detecting CD 3-5 complications was 74.2%, our sub-analysis of three selected complications indicated an SN ranging from 33.3 to 69.0%. This variance confirms our suspicion regarding the selective registration of only the most severe complications and highlights potential inadequacies in capturing a comprehensive spectrum of complications using the present data entry strategy. This trend aligns with findings in the systematic review by Martin et al. [11]. Their research revealed that CD 5 was the only completely reliably registered complication across the 119 studies. Therefore, caution is currently advised in relying on the algorithm to precisely detect specific complications, which may be significantly underreported.

Registration could be improved by standardised coding guidelines, education in registration practices, implementation of specifically required registration variables in the electronic patient files, and yearly quality reports and audits. Algorithms can enhance awareness of the importance of correct registration. Correct data registration is crucial for future research and quality assessment. We propose implementing the algorithm in the future registry-based DGCD for prospective validation and yearly assessment of the quality of data capture from the NPR.

Strengths and limitations

This study used a large validation cohort from the OUH as the reference standard. The correctness of the complication registration in this cohort was thoroughly confirmed via review of medical records. However, the medical records were only reviewed at one hospital where the patients underwent surgery. The data were not registered in the validation cohort if a patient was treated for a complication by their general practitioner or a regional hospital after discharge. The validation cohort consisted of patients treated between 2007 and 2012. The coding in the NPR has not changed significantly since then. However, awareness regarding minor changes in registration practice cannot be discarded. Surgical procedures have changed to be more extensive today, which may potentially indicate an increased risk of post-operative complications. The algorithm was not validated on an external dataset, which may pose a potential bias and problem with generalisability and reliability.

The cohort consisted of patients with FIGO stage IIIC-IV only, so our findings may not apply to patients diagnosed in a lower stage. However, it is anticipated that the algorithm will perform equally well in identifying severe complications in the lower FIGO stages.

Five medical doctors reviewed all complications incorporated in the algorithm. However, given that the complications are indexed and discussed subjectively, there might be misclassified or non-classified complications in the study. The lack of standardisation in the registration of SKS codes may compromise comparative analyses of specific complications.

Several other Danish clinical databases placed under the Danish Clinical Quality Program – National Clinical Registries (RKKP) are changing from manual registration in specific databases to automatic data transfer from the NPR. The present algorithm, if improved, may be usable for all the RKKP-based clinical databases, especially the surgery-based ones.

Conclusions

Overall, the present NPR-based algorithm performed moderately with substantial improvement regarding severe (CD 3-5) complications. Standardised coding guidelines, a focus on education in coding practice and implementing specifically required registration variables in the electronic patient files are likely to improve the performance of an NPR-based algorithm for future quality assurance and research.