Content area


A feasible method to study the Danish out-of-hours primary care service

Lone Flarup1, Grete Moth1, Morten Bondo Christensen1, Mogens Vestergaard1, 3, Frede Olesen1 & Peter Vedsted1, 2, 1) Research Unit for General Practice, Department of Public Health, Aarhus University2) Danish Research Centre for Cancer Diagnosis in Primary Care (CaP), Research Unit for General Practice, Aarhus University3) Section of General Practice, Department of Public Health, Aarhus University


Introduction: The primary care out-of-hours (OOH) service is of considerable importance; it is the main provider of freely accessible medical advice outside daytime hours, and it covers 75% of the active time in the healthcare system. Although the OOH handles three million contacts annually, only little is known about the reasons for encounter, the performed clinical work and the patient perspectives.

Material and methods: During a one-year period (2010-2011), data on patient contacts were collected using pop-up questionnaires integrated into the existing IT system. The questions explored the contents and characteristics of patient contacts. A paper-based questionnaire was sent to the included patients.

Results: Of all 700 general practitioners (GP) on duty, 383 (54.7%) participated at least once, and the participating GPs were representative of all GPs. In total, 21,457 contacts were registered; and the distribution of patient, contact and GP characteristics in OOH contacts was similar to the background contacts. Telephone consultations were most often offered to children and home visits primarily to elderly patients. The patient response rate was 51.2%. Females comprised the majority of the included contacts and of the respondents in the patient survey.

Conclusion: The method was highly feasible for generating a representative sample of contacts to OOH services. The project has formed a substantial and valid basis for further studies and future research in the OOH service.

Funding: Financed by the Central Denmark Region, the Danish National Research Foundation for Primary Care and the Health Foundation.

Trial registration: not relevant.

Danish general practitioners (GPs) are the frontline of Danish health care, 24 hours a day, seven days a week. Services include the primary care out-of-hours (OOH) service which is run by GPs on a rota basis on weekdays and throughout weekends and public holidays [1, 2].

The OOH service is organised in five regional cooperatives, covering from 0.6 to 1.8 million inhabitants; and nearly three million contacts are made per year in a fee-for-service system [1-4]. Calls to the OOH service via one single telephone number per region are answered and triaged by GPs. The GPs either complete the call as a telephone consultation or refer the patient to either a clinic consultation or a home visit. The OOH service is part of a fully computerised patient record system, and all patients are registered by their unique personal identification number (CPR) [5]. An electronic copy of the OOH record is sent to the patient’s own GP, and data are transmitted to the regional administration for remuneration purposes and to the Danish National Health Service Register for Primary Care [6].

Few international studies have focused on the reasons for encounter (RFE) [7, 8], and none have targeted specifically the present Danish OOH organisation. Thus, in June 2010, we launched a comprehensive, prospective research study “Kontakt- og sygdomsmønsteret i lægevagten – LV-KOS 2011” (LV-KOS) focusing on the clinical factors and the patient perspective on the OOH service in the Central Denmark Region (RCJ).

This paper aims to describe the sampling method and the characteristics of GPs and patients in the LV-KOS cohort study. We used a computerised questionnaire in the existing electronic patient record system in the Danish OOH primary care service for our data collection.


Design and setting

The was a cross sectional study in which data collection was performed from 1 June 2010 to 31 May 2011. Pop-up questionnaires were integrated into the existing electronic patient record system. The GPs were invited when logging on to a duty session. Only one GP could participate per contact type (telephone triage, clinic consultation or home visit) in each eight-hour shift. The pop-up questionnaire appeared after every 10th telephone contact, after every third clinic consultation and after each home visit. For each contact, a paper questionnaire focusing on the experience of the encounter was sent to the patient.

Contact registrations

The GP questionnaires were formulated to fit each contact type, including ad hoc questions developed for the study. Telephone contacts were subdivided into telephone consultations or referrals. Pilot testing using cognitive interviewing of 12 GPs was made to improve the face validity of the survey.

The questionnaires comprised the following themes: new event or exacerbation, duration of symptom, severity, possible diagnosis (in text) and estimated relevance of the contact. Background data on contact, time, date and prescribed medication as well as information on the GPs were obtained from the operational computer system. The GPs received a basic remuneration for their participation (20 Euro) and for each registered contact (4 Euro).

Patient survey

Data on name, home address and CPR for all registered patients were securely transferred to a research database. Patient records were manually reviewed for exclusion criteria such as death, discretionary reasons (e.g. psychotic behaviour, dementia) or publically recorded protection from being contacted by researchers (Figure 1). The questionnaires were mailed with postage paid return envelopes. Questionnaires regarding contacts with children below 18 years of age were sent to the parents. A reminder was sent in case of no response after 14 days.

The patient questionnaire comprised pilot tested ad hoc items in combination with items from previous studies [9]. In questionnaires for adults, we added validated items from the Patient Experience Questionnaire, the 12-Item Short Form Health Survey (SF-12) and the Symptom Checklist (SCL-13) anxiety/depression scale [10-13].

Data analysis

RFEs and diagnoses (in text) were manually coded using the International Classification of Primary Care – 2nd Edition (ICPC-2) [14]. The coding was performed by trained research assistants closely supervised by one of the authors (LF). To ensure the validity of the coding procedures, approximately 5% of the coding was continuously audited.

Frequency data are presented as percentages with 95% confidence intervals (95% CI) and continuous numeric data as means, standard deviations and intervals. χ2- and Wilcoxon tests were used to test differences between groups. Data were analysed using STATA 11.0 (StataCorp LP, College Station, TX, USA). p-values of 0.05 or less were regarded as statistically significant.


The project was approved by the Danish Data Protection Agency (j. no. 2009-41-4069) and by the Danish Health and Medicines Authority (j. no. 7-604-04-2/122/EHE). According to Danish law, approval by the National Committee on Health Research Ethics was not required as the project did not include intervention.

Trial registration: not relevant.


Participating general practitioners

In total, 700 GPs had at least one OOH shift; and 383 (54.7%) of these participated in the LV-KOS at least once with a median frequency of 328 registrations per GP (25-75-quartiles: 175, 510). Of all duty periods, 95.5% of the telephone contacts were covered in the study. The participating GPs were comparable with the non-participating GPs (Table 1), although fewer duties in the LV-KOS were staffed by trained GPs compared with all the OOH duties in the RCJ (p < 0.001).

Contacts to the out-of-hours service

In total, 21,457 contacts were included, making up 2.4% of all contacts to the OOH service during the study period (Table 2). Due to the varying pop-up interval of registrations, the distributions of registered contacts for the four types of contacts are not comparable to the distribution of all contacts to the OOH service. In total, 7,810 contacts (36.4%) were registered as telephone contacts of which 4,620 (59.1%) were completed by telephone, whereas 6,973 (32.5%) contacts resulted in consultations and 6,674 (31.1%) in home visits. Despite almost similar point estimates for distribution of gender and mean age across all contact types, the large number of contacts resulted in statistically significant differences in gender and mean age for telephone consultations (p < 0.001) and in age groups for telephone referrals (p = 0.021) and clinic consultations (p = 0.005) (Table 2). The mean age was higher for patients receiving home visits and lower for clinic consultations. Children aged 0-4 years and adults aged 18-60 years more often had telephone consultations and clinic consultations than other age groups. For all contact types, females comprised a higher proportion than males.

Patient survey

Of a total of 19,852 unique patients, 16,434 (82.8%) were included in the patient survey (Figure 1). A total of 136 (0.7%) were excluded due to previous inclusion, and 3,282 (16.5%) were excluded due to unknown addresses or for discretionary reasons. In total, 8,410 (51.2%) patients returned completed questionnaires. Generally, respondents were younger than non-respondents and excluded patients (Table 3).

Females comprised a higher proportion than males in all groups. Parents of patients aged 0-4 years comprised the highest proportion of respondents. The largest proportion of excluded patients was adults aged 18-40 years and patients with a publically recorded protection from being contacted by researchers.


Main findings

Our study showed a feasible method of integrating an on-time, randomly-activated questionnaire into an existing patient administration IT system in the OOH service. Nearly all duties were represented with a high GP response rate. This gave representative, complete and detailed data on randomly selected contacts comprising data from nearly 20,000 unique patients. Of all patient calls, 59% were completed by telephone. Telephone consultations and clinic consultations were primarily offered to children and the youngest group of adults aged 18-40 years. Home visits were most often offered to patients aged 75 years or more. In total, 16,434 patients were included in the subsequent self-administered paper patient survey providing data on the GP-registered contacts in a patient perspective. The response rate was just over 50%. Patients with a publically recorded protection from being contacted by researchers formed the majority of exclusions from the survey (78.5%).

Discussion of methods

General practitioner registrations

The data comprise a large sample of contacts to the OOH service yielding a high statistical precision. The contacts were randomly included and therefore not affected by selection bias from GPs or from patients. Data were collected from nearly all duty sessions, which implies that the data represent a full picture of the patients’ needs and the medical activities of the OOH service. All data were transferred electronically without missing data as each registration was linked to the GPs’ remuneration. This is emphasised by the large agreement in patient characteristics between the LV-KOS and all OOH patient contacts in the RCJ.

The coverage of shifts was high considering the high workload. We found the same distribution in age and gender in the participating group of GPs as in the non-participating group. However, the proportion of trained GPs was lower in the participating group, which indicates that the trainees were more inclined to participate in the LV-KOS. However, the difference was small and considered insignificant, which was also supported by the similar rates of completed telephone consultations in the LV-KOS and in all OOH contacts (59.1% versus 59.3%).

The electronic questionnaire in combination with the automated and electronic data transmission ensured complete and valid data, including background information on the contacts obtained independently through the GPs on duty. The questionnaire was pilot-tested; and the pilot was followed by interviews to ensure the unambiguity and intelligibility of the items.

The data quality depended on the details provided by the GP records and on the individuals performing the encoding. This dependency may be a source of coding inconsistency and may thus influence the inter-rater variability. However, the quality of the ICPC-coding process was enhanced by the meticulous review of all text passages during which all the information present was considered. ICPC coding performed by the participating GPs may have improved the encoding validity as coding uncertainty due to ambiguous texting may have introduced some misclassification. However, this solution was not feasible since the GPs do not presently perform ICPC coding of RFEs in the OOH service. Such requirement may have increased the workload and may thus have lowered the participation rate and challenged the coding completion.

The pop-up frequencies of the questionnaires in the various contact types were not similar to the distribution of all OOH contacts in the study period. This setup was made to ensure sufficient numbers of registered clinic consultations and home visits to make valid estimates and yet avoid a considerable increase in the GP workload. Therefore, the analyses must be performed for each contact type separately or in a weighted analysis. The statistically significant differences between all contacts in the region and our registered contacts can be attributed to the high number of patients, but was considered to be of no clinical significance.

Patient survey

The electronic transfer of patient registrations ensured completeness and validity of data. Manual review of patient records for exclusion criteria was chosen, even though this method introduced a risk of selection bias. Because of the increased attention to sensitive contacts, we might have excluded more patients than needed. However, in view of the low number of exclusions, this did not seem to affect the validity of the inclusion.

A paper-based rather than a web-based questionnaire was chosen due to the diversity of the population with regard to age, health conditions and RFEs. The response rate of 51.2% is considered acceptable for a survey in a heterogeneous population [15-17]. The fairly low response rate among patients aged 18-40 years (including parents to patients under 18 years) is not easy to explain, but may result from reluctance to complete time-consuming paper questionnaires among patients of this age since this group is generally pressed for time due to work and child care. Moreover, this group tends to be in good health and may find the questionnaire less relevant compared with other patients. This means that interpretations of the results for future studies have to take into account that the group of adult patients responding is older and may therefore have more chronic disease and that study results related to children may be affected by the high response rate of the youngest children. It may be worth considering if the response rate could have been increased if respondents had been offered a free choice between a paper-based or web-based questionnaire. However, previous studies did not find that a web-based method significantly increased the response rate [18, 19]. The finding of a lower response rate among older patients is in line with other study results, which is not surprising since this group may be less resourceful and more ill [16]. In our patient survey, we used validated questionnaire scales because of their high validity and specificity in combination with ad hoc developed study questions.

Comparison with other studies

We found only one previous study describing a method for sampling research data in the primary care OOH service by means of electronic pop-up questionnaires and one study describing a method for generating data from an existing administrative system. Christensen also obtained a high GP participation rate through an electronic data collection method targeting frequent attenders [9]. Rebnord et al described an electronic method for using laboratory tests in the OOH service in Norway [20]. By using electronic registrations in the national remuneration system, they generated an almost complete and valid dataset consistent with the method used in our study.


Integration of electronic questionnaires into the existing OOH primary care patient administration system was highly feasible. The described method of registering patient contacts to the OOH service may form the basis for future research into the OOH service, into patients’ perspectives and into treatment pathways. The possible selection bias due to a low patient response must be addressed. The presented computerised method and the patient sampling can be used for further studies and could easily be scaled up to national level or could be modified for other purposes. This study provides a solid basis for further research into quality issues as well as clinical and patient-oriented perspectives. Future studies should present a broader perspective of the results from the LV-KOS study.

Correspondence: Lone Flarup, Institut for Folkesundhed, Forskningsenheden for Almen Praksis, Bartholins Allé 2, Aarhus Universitet, 8000 Aarhus C, Denmark. E-mail:

Accepted: 18 March 2014

Conflicts of interest:Disclosure forms provided by the authors are available with the full text of this article at

Acknowledgements: The authors wish to thank the GPs who participated in the pilot test and the study. We also wish to thank the regional board of GPs in charge of the OOH administration for their kind support of our collaboration on this study. Finally, we wish to thank EG Data Inform for the development of the electronic pop-up questionnaire.

Bib ref: 
Dan Med J 2014;61(5):A4847


  1. Christensen MB, Olesen F. Out of hours service in Denmark: evaluation five years after reform. BMJ 1998;316:1502-5.

  2. Olesen F, Jolleys JV. Out of hours service: the Danish solution examined. BMJ 1994;309:1624-6.

  3. Vedsted P, Christensen MB. The effect of an out-of-hours reform on attendance at casualty wards. The Danish example. Scand J Prim Health Care 2001;19:95-8.

  4. Statistics Denmark. StatBank Denmark. (25 May 2012).

  5. Pedersen CB, Gøtzsche H, Møller JØ et al. The Danish Civil Registration System. Dan Med Bull 2006;53:441-9.

  6. Lægeforeningen. Forhandlingsaftale mellem RTLN og PLO. (6 Jun 2012).

  7. Huibers LA, Moth G, Bondevik GT et al. Diagnostic scope in out-of-hours primary care services in eight European countries: an observational study. BMC Fam Pract 2011;12:30.

  8. Welle-Nilsen LK, Morken T, Hunskaar S et al. Minor ailments in out-of-hours primary care: an observational study. Scand J Prim Health Care 2011;29:39-44.

  9. Christensen MB. Frequent attenders at the out of hours service in Denmark - Implementation of change in general practice. A stratified cluster randomised controlled trial among frequent attenders at the out of hours service in the County of Northern Jutland, Denmark. Aarhus: Faculty of Health Sciences, University of Aarhus; 2001.

  10. Riddle DL, Lee KT, Stratford PW. Use of SF-36 and SF-12 health status measures: a quantitative comparison for groups versus individual patients. Med Care 2001;39:867-78.

  11. Steine S, Finset A, Laerum E. A new, brief questionnaire (PEQ) developed in primary health care for measuring patients’ experience of interaction, emotion and consultation outcome. Fam Pract 2001;18:410-8.

  12. Dansk Institut for Klinisk Epidemiologi. Modelspørgeskema til undersøgelse af sundhed og sygelighed i befolkningen. Copenhagen: Dansk Institut for Klinisk Epidemiologi, 1994.

  13. Bjørner JB, Thunedborg K, Kristensen TS et al. The Danish SF-36 Health Survey: translation and preliminary validity studies. J Clin Epidemiol 1998;51:991-9

  14. Bentsen BG. International classification of primary care. Scand J Prim Health Care 1986;4:43-50.

  15. Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol 1997;50:1129-36.

  16. Millar MDD. Improving response to web and mixed-mode surveys. Pub Opinion Quart 2011;75:249-69.

  17. Murphy C. Increasing the response rates reluctant professionals to mail surveys. App Nurs Res 1993;6:137-41.

  18. van den Berg MH, Overbeek A, van der Pal HJ et al. Using web-based and paper-based questionnaires for collecting data on fertility issues among female childhood cancer survivors: differences in response characteristics. J Med Internet Res 2011;13:e76.

  19. Vergnaud AC, Touvier M, Mejean C et al. Agreement between web-based and paper versions of a socio-demographic questionnaire in the NutriNet-Sante study. Int J Public Health 2011;56:407-17.

  20. Rebnord IK, Sandvik H, Hunskaar S. Use of laboratory tests in out-of-hours services in Norway. Scand J Prim Health Care 2012;30:76-80.


Add new comment

Right side