INTRODUCTION. The hypothesis of this study was that the patient-reported outcome (PRO) triage algorithm may be used as a screening tool to discriminate between patients who do and do not need a visit in the outpatient epilepsy clinic. The purpose of this study was to evaluate the validity of the triage algorithm by comparing it to an assessment made by a neurologist subspecialised in epilepsy.
METHODS. A neurologist reviewed the answers to the PRO questionnaires, and, based on the severity of the answers, the neurologist assessed whether or not the patients should be reclassified into another triage colour group.
RESULTS. We found a significant difference between the triage of the neurologist and that of the PRO algorithm. The neurologist and PRO algorithm agreed regarding triage colour in 48.6% (37.4-59.9%) of the questionnaires (p < 0.0001). The neurologist and the PRO algorithm identified the same triage colour in 64.3% (38.8-83.7%) of the green baseline questionnaires (p < 0.0001), 46.7% (32.9-60.9%) of the yellow baseline questionnaires (p < 0.0001) and 38.5% (17.7-64.5%) of the red baseline questionnaires (p = 0.0016).
CONCLUSIONS. The neurologist assessment reduced the number of patients in the yellow triage group and increased patient numbers in the green and the red triage groups compared with the PRO triage algorithm. Major differences between the expert assessment and the PRO results were found. Improvement of PRO triage is needed before it may be used as a clinical management tool for patients with epilepsy.
TRIAL REGISTRATION. not relevant.
Patients with epilepsy often report problems with thinking, memory and mood disorders related to both the disease and adverse events from medical therapy [1, 2]. Depression, dysthymia, anxiety and suicidal thoughts and behaviour may affect activities of daily living and quality of life, and patients with epilepsy have a three times higher risk of committing suicide than controls [1, 3]. These symptoms can be monitored with patient-reported outcomes (PROs).
PROs are systematically collected information about the patient’s symptoms, mental and physical health, and health-related quality of life [4, 5]. PROs provide information about the patient’s wellbeing that supplements data from technology and observers and is of special value in diseases where quality of life plays an essential role . PROs are reported directly by the patient without interference by a clinician or other persons [6, 7], and subsequently the results are interpreted by healthcare professionals. Owing to a growing desire for a patient-centred healthcare system, PROs are gaining more importance in the clinical world .
PROs are used in healthcare to improve patient-clinician communication, increase patients’ influence on their own treatment, optimise the setting of priorities and possibly reduce time-consumption on the part of patients and clinicians alike . The use of PROs in outpatient follow-up may potentially help identify patients in need of a consultation, thus promoting a better distribution of resources by minimising visits that neither the patient nor the clinician find necessary . One of the eight national goals of the Danish Ministry of Health is to increase patients’ influence on their own treatment and to promote patient-centred healthcare . Thus, a committee was established to ensure an extensive and standardised implementation of PROs in Danish healthcare . One study has shown a high reliability of a PRO triage algorithm, but no study has yet validated this PRO triage algorithm or assessed how patients with epilepsy should be followed up .
The hypothesis of this study was that the PRO triage algorithm would be able to discriminate between patients in need of a consultation in the outpatient epilepsy clinic and those not needing contact. The aim of the study was to evaluate the validity of the triage algorithm by comparing it to an assessment made by a neurologist subspecialised in epilepsy, which was considered the gold standard.
Denmark has a government-financed healthcare system that is free of charge and in which all citizens have equal access to healthcare . The present study used data from patients treated in an outpatient epilepsy clinic where a PRO triage algorithm was used to determine how patients should be followed up. The algorithm triages patients into three categories (green, yellow or red) based on the severity of their condition as reflected in their answers to the PRO questionnaire. The PRO questionnaire consists of questions about side effects, compliance, drug abuse, seizures, suicidal thoughts, pregnancy and wish for contact with the clinic (Appendix 1: https://ugeskriftet.dk/files/filefield_paths/a12210915-supplementary.pdf).
A specialised nurse reviews each patient questionnaire. If the patient is triaged green by the algorithm, nothing further is usually done until the next questionnaire is sent out. Patients who are triaged yellow or red by the algorithm are assessed individually by a nurse. In some cases, the patient is invited to attend a telephone or an outpatient consultation. In other cases, the nurse concludes that the patient’s epilepsy remains unchanged, and that nothing further needs to be done until the next questionnaire is sent out. Patients are triaged red by the algorithm if they report increased seizure frequency, the highest level of suicidal thoughts, pregnancy or a wish for contact with the clinic. Patients are triaged yellow by the algorithm if they report seizures or less severe symptoms such as decreased libido, fatigue or headache.
The patients receive the PRO questionnaire at predefined intervals, usually every six months, but the nurse may change the intervals to suit each patient’s individual need. Therefore, some patients receive the questionnaire every three months, and others only annually. The first questionnaire completed by the patient is referred to as the baseline questionnaire, and the follow-up questionnaires are referred to by their number. The patients’ answers are automatically uploaded to the eHealth Portal used in the Region of Southern Denmark where the specialised nurse can assess them .
All patients were evaluated individually regarding eligibility by a specialised nurse or physician before inclusion in PRO-based follow up. Patients were excluded if unable to understand written Danish or to use a computer/tablet.
Data were collected retrospectively for the period from September 2017 to August 2019. Information was obtained from the eHealth Portal in the Region of Southern Denmark.
All patients who attended the epilepsy outpatient clinic during the study period were eligible for study inclusion if they had submitted the baseline PRO questionnaire at some point during the study period. Patients who failed to submit the baseline questionnaire were excluded from the study. A total of 72 patients submitted a baseline questionnaire and were therefore included in the study. A detailed study flowchart is presented in Figure 1.
A neurologist subspecialised in epilepsy reviewed the patients’ answers to the PRO questionnaires and, based on the severity of the answers, assessed whether or not the patients should be reclassified into another triage colour group.
Triage colour reclassification by the neurologist
The neurologist pooled the answers in need of the most urgent action to the red triage colour. Answers requiring action, although less urgent, were pooled in the yellow triage colour. The neurologist used the green triage colour for answers not requiring action other than a new questionnaire at the predefined interval. The neurologist reclassified patients to red if they reported seizures, increases in their seizure frequency, suicidal thoughts, poor medicine compliance or drug abuse; or were pregnant or were planning a pregnancy. Patients who reported problems such as side effects and emotional difficulties that were considered to be treatable by the neurologist, e.g. by adjusting or changing the patient’s medication, were reclassified to yellow. Patients without distinct difficulties were reclassified as green.
GraphPad Prism 8 was used for the statistical analyses. Continuous variables were compared using the Wilcoxon test and categorical variables were compared using the χ2-test. All results are shown as means followed by 95% confidence interval (CI) in parentheses, except for gender and triage colour which are presented as proportions and the CI is calculated using the Wilson/Brown method.
Trial registration: not relevant.
Among the 179 questionnaires submitted, 72 were baseline questionnaires (Figure 1). Table 1 shows the patient characteristics at baseline, no difference was found between subgroups.
Validation of the algorithm
The percentages of patients in the green triage group were 19.4% (12.0-30.0%) before and 27.8% (18.8-39.0%) after the neurologist reviewed the questionnaires. In the yellow group, the percentages were 62.5% (50.6-72.8%) before and 43.1% (32.3-54.6%) after; and in the red group, the percentages were 18.1% (10.9-28.5%) before and 29.2% (19.9-40.5%) after reclassification. No statistical difference between the number of patients in each triage group before and after reclassification was found (p > 0.9999) (Figure 2A).
In the green triage baseline group, the neurologist reclassified 35.7% (16.3-61.2%) of the patients as yellow and 0% (0.0-21.5%) as red. In the yellow triage baseline group, the neurologist reclassified 17.8% (9.3-31.3%) of the patients as green and 35.6% (23.2-50.2%) of the patients as red. In the red triage baseline group, the neurologist reclassified 23.1% (8.2-50.3%) of the patients as green and 38.5% (17.7-64.5%) as yellow (Figure 2B). We found a difference between the neurologist and the PRO algorithm regarding triage colour in 64.3% (38.8-83.7%) of the green baseline questionnaires (p < 0.0001), 46.7% (32.9-60.9%) of the yellow baseline questionnaires (p < 0.0001) and 38.5% (17.7-64.5%) of the red baseline questionnaires (p = 0.0016) (Figure 2C). Overall, the neurologist agreed with the PRO algorithm regarding triage colour in 48.6% (37.4-59.9%) of the questionnaires, which was statistically different (p < 0.0001) (Figure 2D).
Our study demonstrated differences between triage results from the PRO algorithm and the triage assessment by a senior consultant neurologist. Regardless of the triage colour given by the algorithm, we found a 50% risk that an audit by a neurologist would lead to changed triage colour. Our findings are relevant from both the patient’s and the clinician’s perspective, and also at the socioeconomic level since PRO triage algorithms are being implemented as the basis for outpatient follow-up at the national level. Misclassification by the triage may impact the care of patients with epilepsy since half of the patients may be triaged differently based on the PRO algorithm compared with good clinical practice. As a consequence, nursing time is potentially spent on telephone consultations and extra screening of patients who currently have well-treated epilepsy. Likewise, senior consultant time may be spent on seeing patients requesting outpatient consultations, even if these patients report stable-phase epilepsy. Correct prioritising is essential, especially in a healthcare system in which time and resources are scarce.
The discrepancy in triage may be due to the neurologist having a more clinical approach and prioritising patients with symptoms and difficulties that can be treated by adjusting or changing the patient’s medication, whereas the algorithm to a greater extent prioritises patient requests. Due to the way in which the PRO algorithm performs the triage, patients who have reported substantial difficulties or symptoms may not be awarded a consultation in the outpatient clinic for years. Patients who do not use PROs as a basis for outpatient follow-up get prescheduled consultations at intervals that match their individual needs as evaluated by a neurologist. As mentioned previously, patients with epilepsy are vulnerable and carry a higher risk of committing suicide than controls. The patients are only triaged red by the algorithm if they rate themselves as having the highest score regarding suicidal thoughts. Patients at high risk of committing suicide may therefore be neglected by the algorithm. One might therefore argue that patients who are in PRO-based outpatient follow-up do not get the same level of quality healthcare as patients who receive a more traditional outpatient follow-up. This concern is supported by a study conducted among clinicians working with PRO-based outpatient follow-up in epilepsy in the Central Denmark Region .
The PRO questionnaire contains questions regarding different symptoms, of which only a fraction can be treated by the neurologist. It may be frustrating for patients to repeatedly report symptoms if nothing is done to relieve them. Mejdahl et al. reported that some patients felt that it was a waste of time to complete the PRO questionnaires since they often received no response from the clinicians; and if they did, they felt abandoned and alone as the response was very superficial. These patients also stated that a lack of response to the problems they reported eroded their motivation for completing PROs . It is possible that response rates could be improved if patients were trained in how to reply and how the epilepsy clinic will respond to PROs.
A study by Stafford et al. stated that many of the developments on PROs in epilepsy are opportunistic, and that many of the PRO measures are poorly developed and largely unvalidated . Our study demonstrated that PROs may be used as an explanatory/supplementary questionnaire for patients with epilepsy, but further improvement of the PRO triage is necessary before the results can contribute to better clinical practice by nurses and physicians in outpatient epilepsy clinics.
This study was based on data from a single-centre epilepsy clinic. This may influence the generalisability of the results since the population in the hospital catchment area is different from other hospitals in the country. Patient selection for the PRO algorithm was not standardised but based on an individual decision by the physician, and only one unblinded neurologist assessed the algorithm. Observer bias and confirmation bias may therefore have influenced the results. Further studies validating PRO triage in a multicentre setting are warranted. In future studies, we recommend algorithm assessment by several blinded neurologists to accommodate intra- and inter-rater variability.
Our study found significant differences between triage results from the PRO triage algorithm and validation by a consultant neurologist. Regardless of the triage colour given by the algorithm, the risk of misclassification was 50% when compared with a senior neurologist assessment. These results are important as the PRO triage algorithm is currently used in the clinic to determine how epilepsy patients should be followed up. Misclassification by triage may impact patient care. To our knowledge, this study is the first validation study of the PRO triage algorithm in outpatient follow-up in epilepsy. The PRO triage algorithm may be useful as a supplementary management tool for patients with epilepsy, but significant improvement of the triage is needed before it may be used as the main management tool for patients with epilepsy in the outpatient clinic.
Correspondence Eva Myrdal. E-mail: email@example.com
Accepted 10 May 2022
Conflicts of interest none. Disclosure forms provided by the authors are available with the article at ugeskriftet.dk/dmj
References can be found with the article at ugeskriftet.dk/dmj
Cite this as Dan Med J 2022;69(7):A12210915
Mayo Clinic. Epilepsy. https://www.mayoclinic.org/diseases-conditions/epilepsy/symptoms-causes/syc-20350093 (10 Aug 2019).
Russel BM, Kristensen JK, Hansen BL. Epilepsi. https://www.sundhed.dk/sundhedsfaglig/laegehaandbogen/neurologi/tilstande-og-sygdomme/kramper/epilepsi/ (7 Apr 2017).
Christensen J, Vestergaard M, Mortensen PB et al. Epilepsy and risk of suicide: a population-based case-control study. Lancet Neurol. 2007;6(8):693-8.
PRO-sekretariatet. PRO. http://pro-danmark.dk/da/pro (23 Mar 2019).
Mejdahl CT, Schougaard LMV, Hjollund NH et al. PRO-based follow-up as a means of self-management support - an interpretive description of the patient perspective. J Patient Rep Outcomes. 2018;2:38.
Deshpande PR, Rajan S, Sudeepthi BL, Nazir CPA. Patient-reported outcomes: a new era in clinical research. Perspect Clin Res. 2011;2(4):137-44.
PRO-sekretariatet. Hvad er PRO? http://pro-danmark.dk/da/pro/hvad-er-pro (21 Aug 2019).
PRO-sekretariatet. Info til sundhedsfaglige. http://pro-danmark.dk/da/pro/info-sundhedsfaglig (9 Jan 2018).
Schougaard LMV, Larsen LP, Jessen A et al. AmbuFlex: tele-patient-reported outcomes (telePRO) as the basis for follow-up in chronic and malignant diseases. Qual Life Res. 2016;25(3):525-34.
Danish Ministry of Health. 8 nationale mål for sundhedsvæsenet. https://www.sum.dk/Temaer/Nationale-maal-for-sundhedsvaesenet.aspx (2 Dec 2019).
PRO-sekretariatet. Den Nationale Styregruppe for PRO. https://pro-danmark.dk/da/pro/den-nationale-styregruppe-for-pro (7 Mar 2022).
Schougaard LMV, de Thurah A, Christiansen DH et al. Patient-reported outcome (PRO) measure-based algorithm for clinical decision support in epilepsy outpatient follow-up: a test-retest reliability study. BMJ Open. 2018;8(7):e021337.
Danish Health Authority. Det danske sundhedsvæsen. https://www.sst.dk/da/udgivelser/2017/det-danske-sundhedsvaesen (22 May 2020)
Region of South Denmark. Mit sygehus. https://www.regionsyddanmark.dk/wm509838 (23 May 2020).
Mejdahl CT, Schougaard LMV, Hjollund NH et al. Exploring organisational mechanisms in PRO-based follow-up in routine outpatient care - an interpretive description of the clinician perspective. BMC Health Serv Res. 2018;18(1):546.
Stafford M, Gavriel S, Lloyd A. Patient-reported outcomes measurements in epilepsy. Expert Rev Pharmacoecon Outcomes Res. 2007;7(4):373-84.