What factors influence patient participation in an artificial intelligence-based initiative to optimise referrals from primary to specialist haematology care? A multicentre retrospective observational study in four Spanish hospitals.

STRENGTHS AND LIMITATIONS OF THIS STUDY

The multicentre design and large sample size increase the external validity and robustness of the findings.

The retrospective nature of the study limits control over confounding and restricts causal inference.

The relatively low participation rate may introduce selection bias and limit the representativeness of the evaluated initiative.

Introduction

The global demand for specialist haematology care is rising, 1 2 partly due to an increase in the incidence of haematological malignancies, which cause significant morbidity and mortality worldwide. However, the full spectrum of blood disorders includes many other highly prevalent conditions, such as iron deficiency anaemia. 3 4 The initial diagnostic workup for many suspected blood disorders may be performed in primary care. However, in some healthcare systems, primary care physicians lack direct access to the complete spectrum of available tests necessary to rule out conditions which merit specialist haematology care. On the other hand, time constraints, patient pressure and the often-overwhelming workload of primary care may lead to inappropriate referrals. 5 6 Sometimes, testing for nonspecific symptoms such as fatigue leads to a specialist referral due to incidental findings. 7 Increasing backlogs for specialist care, coupled with increasing demand, make reducing unnecessary use of healthcare a priority to attain sustainable care for patients with haematological disorders.

Among strategies aimed towards reducing inappropriate testing and referrals to specialist care, educational initiatives and guideline development have both proven effective. 8 However, these initiatives alone are insufficient to solve the specialist backlog, and further strategies are necessary to optimise the referral process from primary to specialist care. Pre-appointment management for specialty referral has demonstrated improved adherence to clinical guidelines with reductions in unnecessary referrals, as well as high patient satisfaction. 9 Other pre-appointment approaches geared towards providing specialists with all the information necessary to reach a diagnosis and initiate treatment (if appropriate) at the first visit could potentially save valuable time and follow-up visits booked merely to review the results of tests ordered at the first encounter with the patient. Thus, the implementation of guidelines and protocols for comprehensive testing prior to a first specialist appointment could lead to rapid resolution of non-complex cases at the first encounter, reducing unnecessary follow-up visits booked for review of test results. A recent study of the effects of incorporating a natural language processing algorithm for automatic placing of standardised test order sets on low-value appointments and attendance rates demonstrated an increase in the number of participating patients discharged back to primary care with a diagnosis and treatment plan at their first specialist visit, as well as lower rates of follow-up and non-attendance after the project’s implementation. 10 This publication evaluated the system-level impact of implementing an automated algorithm for standardised pre-appointment test ordering using an interrupted time series design. Its primary focus was on temporal changes in attendance rates and low-value follow-up appointments before and after implementation of the intervention across participating hospitals, without distinguishing individual patient participation or exploring reasons for non-participation.

However, the effects of automatically placed test orders prior to first haematologist appointments on discharge rates and low-value follow-up have yet to be described.

The present manuscript addresses distinct and complementary research questions at the level of a specific medical specialty (haematology). This study aims to describe the implementation of automatised standardised test order sets for common presenting complaints in specialist haematology care, analysing differences in participants and non-participants including demographic and clinical characteristics and the number of patients discharged at first appointment, as well as common reasons for not participating. We also describe the most frequent diagnoses for each of the included presenting complaints, including the absence of haematological disorders.

Methods

Study design

A retrospective observational multicentre study was performed. We included all patients referred for a first appointment at the four haematology outpatient clinics of the Fundación Jiménez Díaz University Hospital (Madrid, Spain), Infanta Elena University Hospital (Valdemoro, Spain), Rey Juan Carlos University Hospital (Móstoles, Spain), and General Villalba University Hospital (Villalba, Spain), from May 2021 to December 2023, with the following presenting complaints: abnormal serum protein electrophoresis; anaemia; blood clotting disorder; bleeding diathesis; deep vein thrombosis/pulmonary embolism; eosinophilia; leucocytosis/lymphocytosis; leucopenia; lymphopenia; polycythaemia; thrombocytopenia; and thrombocytosis. Ineligible presenting complaints and conditions included: acute lymphoblastic leukaemia, acute myeloid leukaemia, amyloidosis, anticoagulant treatment assessment, arterial thrombosis, bicytopenia, chronic lymphocytic leukaemia, congenital coagulopathy, cutaneous lymphoma, dendritic cell neoplasm, general check-up or medical exam, hereditary thrombophilia, Hodgkin lymphoma, hyperferritinaemia, hyperhomocysteinaemia, inversion of the leucocyte formula, lymphadenopathy, macrocytosis, mastocytosis, microcytosis, monocytosis, multiple myeloma, myelodysplastic syndrome, myeloproliferative neoplasms, neutropenia, neutrophilia, non-Hodgkin's lymphoma, other haematologic disorders associated with pregnancy, pancytopenia, pregnancy and anaemia, pregnancy and haemorrhagic diathesis, pregnancy and thrombocytopenia, pregnancy and thrombophilia, signal abnormality on MRI, splenomegaly, thalassaemia, thrombocytopathy and vitamin B12 disorder. Patients on active chemotherapy attending the haematology day clinic were excluded, as were patients attending first haematology appointments for other presenting complaints. Patients were referred to haematology outpatient care either by primary care physicians, emergency care specialists or other specialists.

Intervention

The intervention formed part of a network initiative for automatic placing of standardised test order sets to reduce low-value follow-up appointments. A full description of the initiative has been published in a previous report. 10 The rationale for the project was to provide specialists with all information necessary to make clinical decisions at first appointment, thus reducing the number of follow-up appointments made merely to review test results ordered at the first patient encounter. Briefly, a multidisciplinary team built on big data to analyse tests ordered by physicians in first appointments for different presenting complaints. Tests ordered in 90% or more of first appointments were considered for inclusion in standardised test order sets, and a panel of experts reviewed and decided on the final version of the test order sets for each presenting complaint. The test order sets were embedded in the network’s electronic health record (EHR), Casiopea and connected to the SCAE primary care database ( online supplemental file 1 ). A natural language processing pipeline was designed to analyse referral notes and generate the relevant test order sets automatically. Patients were contacted by the participating centres’ admissions departments prior to booking the tests in order to confirm consent to participate. Patients referred from primary care physicians outside the four participating hospitals’ catchment areas and those under 18 years old were excluded from the initiative.

Objectives and measurements

Our primary endpoint was to characterise demographic and clinical characteristics (age, sex and presenting complaint) between participants and eligible non-participants (control group), as well as common motives for not participating in a subset of non-participating patients. The secondary endpoint was to describe differences in the number of patients discharged back to primary care with a diagnosis and/or treatment plan between the two groups. The electronic health record prompts haematologists to select a response from a dropdown menu at each consultation to determine patient workflow. Responses include ‘discharge—yes’ or ‘discharge—no’. The variable ‘discharge—yes’ was used to identify those patients discharged back to primary care after first specialist appointment. Finally, we aimed to describe the ten most frequent diagnoses for each presenting symptom included in the intervention using International Classification of Diseases, tenth revision (ICD-10) coding and to calculate the overall percentage of patients with a diagnosis of ‘haematological disorder ruled out’ as an indicator of potentially avoidable referrals to specialist care. In the context of the Spanish National Health System, administrative staff use ICD-10 coding to identify diagnoses and procedures carried out during inpatient care, while ICD-10 coding is not usually carried out for care episodes taking place in the outpatient setting. However, as part of a departmental initiative, haematologists at the four centres included in this study must select a suspected ICD-10 diagnosis from a drop-down menu when assessing outpatients as part of routine procedure. The selected diagnoses are automatically recorded in patients’ electronic health records. If the diagnostic suspicion is the absence of haematological disease, the diagnosis ‘haematological disorder ruled out’ should be selected. Data were extracted from the network EHR, while data for the subset of non-participants used to analyse motives for non-participation was extracted by the network Big Data department from the admissions department database. This information was recorded by admissions staff when phoning patients to invite them to participate in the initiative. Patients declining to participate were invited to state their motive for not participating. Motives for not participating were coded by administrative staff as A: the patient is explicitly unwilling to participate; I: administrative staff are unable to contact the patient; P: The patient has a history of recent tests performed; and NULL: administrative reasons, including the patient is outside the catchment area, or unavailable slots for test appointments before first specialist appointment, or patients younger than 18 years.

Patient and public involvement

No patients or members of the public were involved in the design of this initiative, although one of the objectives of the project was to improve patient experience by reducing the number of unnecessary outpatient visits.

Statistical analysis

Categorical variables were expressed as number (percentage) and continuous variables as median (IQR) due to their non-normal distribution. Differences in categorical variables were analysed with a χ 2 test, while differences in continuous variables were analysed with a Mann-Whitney U test. A p value of 0.05 was set as the threshold for statistical significance. Data analysis was performed with Python V.3.1.

Results

During the study period, a total of 43 950 patients were referred to the haematology outpatient clinic for a first visit, of whom 18 190 (41%) were referred for presenting complaints included in the intervention (‘eligible patients’). During the study period, 612 (3.3%) patients agreed to participate in the intervention. Significant differences in age and prevalence of different presenting complaints were observed between participants and non-participating patients ( table 1 ).

To describe common motives for not participating, we analysed the available data from the admissions department from a subset of non-participating patients (n=8149, 46%), presented in table 2 . Interestingly, the most common motives for not participating included administrative reasons (6268, 76.9%), such as patients not belonging to the participating centres’ catchment areas, or unavailable slots for test appointments before the first specialist appointment. Other reasons for not participating included a history of recent tests performed (1044, 12.8%), being unable to contact the patient (715, 8.8%), and patients’ explicit unwillingness to participate (122, 1.5%).

A significant increase in the number of patients discharged with a diagnosis and treatment plan, where applicable, upon first appointment was observed for the study group compared with the control group (146 (23.9%) vs 3375 (19.36%); p=0.041). Upon analysing numbers of patients discharged for each presenting complaint ( table 3 ), however, no significant differences were found except for ‘thrombocytosis’, in which the number of patients discharged with a diagnosis and treatment plan was higher for the control group.

Finally, we analysed the ten most frequent coded diagnoses for each presenting complaint during the study period, including both first and follow-up haematology appointments (n=87, 397). The diagnosis ‘haematological disorders ruled out’ was constantly observed as one of the ten most common diagnoses for all presenting complaints except for deep vein thrombosis/pulmonary embolism ( online supplemental file 2 ), accounting for 1440 (1.6%) of patients overall, and up to 4.6% of patients for presenting complaints such as leucopenia.

Discussion

This study describes characteristics of patients participating in an initiative to optimise referrals from primary to specialist haematology care in four academic hospitals in Madrid, Spain. Participants in the initiative were significantly younger than non-participants. Common motives for not participating included mostly administrative reasons, such as availability of a first haematologist appointment before the tests could be performed. We observed that the intervention was associated with a significant increase in the number of patients discharged back to primary care at first appointment. The ICD code ‘haematological disorders ruled out’ made up a 1.6% of total diagnoses made by haematologists and was consistently among the top ten diagnoses for included presenting complaints.

The increased demand for specialist haematology care due to higher rates of haematological malignancies and the ageing population underlines the importance of reducing low-value appointments on the grounds of health system sustainability. 11 From the patient and caregiver’s perspective, unnecessary healthcare is linked to lower quality of life due to days missed at work, time spent in transportation and related stress. 12 13 At the same time, low-value clinical acts are associated with higher rates of burnout among healthcare professionals. 14 15 On a system-wide level, unnecessary outpatient appointments consume valuable resources that other patients need, as well as contributing to costs of healthcare. The implementation of standardised test order sets prior to first haematologist appointments demonstrated an increase in the number of patients successfully discharged with a diagnosis and treatment plan where appropriate, pointing to a potential reduction in low-value follow-up programmes to review test results. Varying results for the different standardised test order sets opens the door to careful review of included tests for certain presenting complaints such as anaemia, as the broad nature of these complaints may require a fine-tuning of the natural language processing algorithm to order more specific test order subsets according to patients’ clinical and demographic characteristics. On the other hand, for presenting complaints such as thrombocytosis, the standardised test order sets seem to be appropriate for the general population.

The growing body of literature on e-consult systems offers important insights into the implementation of digitally supported specialist triage models. Several studies in haematology have demonstrated the potential of e-consults to improve access to care, reduce unnecessary in-person visits and enhance care coordination between primary and specialty services. 16 , 19 For example, e-consult programmes have been associated with shorter wait times for specialist evaluation and high levels of satisfaction among both clinicians and patients. 16 18 However, challenges such as privacy issues, insufficient reimbursement and unclear workflows for follow-up testing have also been identified as barriers to broader adoption and long-term sustainability. 20 21 Our model addresses some of these pitfalls through a structured and data-driven approach to pre-appointment test ordering, informed by clinical protocols and supported by deterministic natural language processing. Unlike traditional e-consults that often rely on asynchronous communication and individualised specialist responses, our system standardises the diagnostic process upstream, helping to ensure that key tests are completed prior to the initial specialist visit. This not only improves efficiency but also preserves specialist resources for cases requiring in-depth evaluation. Our findings complement existing literature by demonstrating an alternative strategy for improving outpatient workflows and diagnostic readiness, while also highlighting the need for continued monitoring, protocol refinement and integration of clinical judgement.

Our results show that the coded diagnosis ‘suspected diseases or conditions ruled out’ was one of the ten most frequent diagnoses for common presenting complaints made by haematologists, indicating a high potential number of avoidable first appointments and pointing to the need for other, complementary strategies aimed at reducing the number of unnecessary referrals. Educational programmes for primary care physicians have demonstrated efficacy in improving the quality of referrals from primary to secondary care, 22 23 and the combination of initiatives such as this study and educational initiatives is a field for further research.

The use of natural language processing to analyse referral notes has been described in several studies. 24 , 26 However, to the best of our knowledge, this is the first report of the successful, large-scale implementation of a NLP pipeline to process referral notes from primary care providers to haematology care. Regarding participant characteristics, we found that patients participating in the initiative were significantly younger than non-participants, in line with studies demonstrating that age is often a barrier towards the adoption of digital innovations in healthcare. 27 28 Participants were also predominantly female, although this difference was not statistically significant. Interestingly, the major barriers towards participation were administrative reasons, pointing to the relevance of other factors such as communication and scheduling in the referral process, as well as high overall patient acceptance, in line with previous reports. 10

This study has various limitations, mostly stemming from its retrospective design. First, we observed significant heterogeneities in study and control groups due to lack of randomisation. However, we considered that a non-randomised design was the most appropriate for analysing differences between participants and non-participants. Although we were able to describe common motives for non-participation, this information was only available for a subset of patients, and there was no specific data for those patients who were explicitly unwilling to participate; neither were we able to include demographic information such as level of education, rurality index or language spoken. Also, haematologists did not receive specific information on the use of ICD-10 coding to classify diagnoses in outpatient care, and so we cannot rule out an under or over-estimation of the frequency of different diagnoses. Additionally, our research protocol did not include data extraction for second visits, and so we were unable to determine if some patients were still being kept in the system primarily for lab follow-up despite the intervention. Finally, the study design failed to include participating haematologists’ perspectives on the innovation, and this remains a field for further research. Comparing the use of NLP to facilitate timely pre-appointment test ordering and that of fixed order panels based on ICD 10 codes represents an important area for future research. Likewise, evaluating the cost-effectiveness and operational sustainability of maintaining an NLP algorithm is highly relevant in the context of rising computational costs. While central to understanding broader implementation implications, these questions fall beyond the scope of the present study, which focuses specifically on patient level participation, reasons for non-participation and diagnostic outcomes in specialist haematology care. We fully recognise the value of these lines of inquiry and intend to explore them in future work.

Conclusions

Natural language processing of referrals from primary to specialist haematology care with automated placing of standardised test orders can decrease low-value follow-up appointments. Although most eligible patients did not participate in the initiative, non-participation was usually due to administrative barriers, pointing to the importance of factors such as scheduling and communication in this kind of initiative. Explicit refusal to participate was low. Participants tended to be younger than non-participants, which underlines the relevance of designing strategies to target the older population in order to improve participation. Haematological disorders were ruled out in almost 2% of all haematology appointments, indicating that there is potential scope for improvement of the quality of referrals to haematology care.

Supplementary material

Footnotes

Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Prepublication history and additional supplemental material for this paper are available online. To view these files, please visit the journal online ( https://doi.org/10.1136/bmjopen-2025-115195 ).

Provenance and peer review: Not commissioned; externally peer reviewed.

Patient consent for publication: Not applicable.

Ethics approval: This study was carried out according to the principles of the Declaration of Helsinki. All data were pseudoanonymised prior to analysis. The requirement for informed consent was waived due to the retrospective nature of the study. The project received approval from the Fundación Jiménez Díaz University Hospital Ethics Committee; Reference id EO103-21_FJD. SQUIRE 2.0 guidelines were followed when writing this manuscript.

Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Data availability statement

Data are available upon reasonable request.

References

1. Jäger U, Chomienne C, Cools J, et al. Blood disorders stepping into the limelight. Haematologica. 2016;101:101–3. doi: 10.3324/haematol.2016.142018.

2. Masselink LE, Erikson CE, Connell NT, et al. Associations between hematology/oncology fellows’ training and mentorship experiences and hematology-only career plans. Blood Adv. 2019;3:3278–86. doi: 10.1182/bloodadvances.2019000569.

3. Le CHH. The Prevalence of Anemia and Moderate-Severe Anemia in the US Population (NHANES 2003-2012) PLoS One. 2016;11:e0166635. doi: 10.1371/journal.pone.0166635.

4. Levi M, Rosselli M, Simonetti M, et al. Epidemiology of iron deficiency anaemia in four European countries: a population-based study in primary care. Eur J Haematol. 2016;97:583–93. doi: 10.1111/ejh.12776.

5. Latinovic R, Gulliford M, Ridsdale L. Headache and migraine in primary care: consultation, prescription, and referral rates in a large population. J Neurol Neurosurg Psychiatry . 2006;77:385–7. doi: 10.1136/jnnp.2005.073221.

6. Kushnir T, Greenberg D, Madjar N, et al. Is burnout associated with referral rates among primary care physicians in community clinics? Fam Pract. 2014;31:44–50. doi: 10.1093/fampra/cmt060.

7. Kang SK, Berland LL, Mayo-Smith WW, et al. Navigating Uncertainty in the Management of Incidental Findings. J Am Coll Radiol. 2019;16:700–8. doi: 10.1016/j.jacr.2018.09.042.

8. Akbari A, Mayhew A, Al-Alawi MA, et al. Interventions to improve outpatient referrals from primary care to secondary care. Cochrane Database Syst Rev. 2008 doi: 10.1002/14651858.CD005471.PUB2/INFORMATION/EN.

9. Harrington JT, Dopf CA, Chalgren CS. Implementing guidelines for interdisciplinary care of low back pain: a critical role for pre-appointment management of specialty referrals. Jt Comm J Qual Improv. 2001;27:651–63. doi: 10.1016/s1070-3241(01)27055-2.

10. Álvaro de la Parra JA, Del Olmo Rodríguez M, Caramés Sánchez C, et al. Effect of an algorithm for automatic placing of standardised test order sets on low-value appointments and attendance rates at four Spanish teaching hospitals: an interrupted time series analysis. BMJ Open. 2024;14:e081158. doi: 10.1136/bmjopen-2023-081158.

11. Sorenson C, Japinga M, Crook H, et al. Building A Better Health Care System Post-Covid-19: Steps for Reducing Low-Value and Wasteful Care. NEJM Catal. 2020;1 doi: 10.1056/CAT.20.0368.

12. Golics CJ, Khurshid M, Basra A, et al. The impact of patients’ chronic disease on family quality of life: An experience from 26 specialties. Int J Gen Med. 2013;6:787–98. doi: 10.2147/IJGM.S45156.

13. Smith AC, Youngberry K, Christie F, et al. The family costs of attending hospital outpatient appointments via videoconference and in person. J Telemed Telecare. 2003;9 Suppl 2:S58–61. doi: 10.1258/135763303322596282.

14. Shanafelt TD. Enhancing meaning in work: a prescription for preventing physician burnout and promoting patient-centered care. JAMA. 2009;302:1338–40. doi: 10.1001/jama.2009.1385.

15. Lambden JP, Chamberlin P, Kozlov E, et al. Association of Perceived Futile or Potentially Inappropriate Care With Burnout and Thoughts of Quitting Among Health-Care Providers. Am J Hosp Palliat Care. 2019;36:200–6. doi: 10.1177/1049909118792517.

16. Cecchini M, Rose MG, Wong EY, et al. The implementation of electronic hematology consults at a VA hospital. Blood. 2016;127:1610–1. doi: 10.1182/blood-2015-09-672113.

17. Dosani T, Xiang J, Wang K, et al. Impact of Hematology Electronic Consultations on Utilization of Referrals and Patient Outcomes in an Integrated Health Care System. JCO Oncol Pract. 2022;18:e564–73. doi: 10.1200/OP.21.00420/SUPPL_FILE/DS.OP.21.00420.PDF.

18. Fogel A, Khamisa K, Afkham A, et al. Ask the eConsultant: Improving access to haematology expertise using an asynchronous eConsult system. J Telemed Telecare. 2017;23:421–7. doi: 10.1177/1357633X16644095.

19. Santopietro A, Bekendam RH, Dibibar A, et al. Evaluating efficiency of electronic consult requests in an outpatient classical hematology clinic. JCO Oncol Pract. 2023;19:46. doi: 10.1200/OP.2023.19.11_suppl.46.

20. King HL, Benedetti GB, Barisic S, et al. The impact of hematology electronic consultations on the management of iron deficiency. Eur J Haematol. 2024;113:614–22. doi: 10.1111/ejh.14276.

21. Osman MA, Schick-Makaroff K, Thompson S, et al. Barriers and facilitators for implementation of electronic consultations (eConsult) to enhance access to specialist care: a scoping review. BMJ Glob Health. 2019;4:e001629. doi: 10.1136/bmjgh-2019-001629.

22. Heimly V, Nytro O. Clinical guidelines as decision support for referrals from primary care. Proceedings of the 2011 IEEE International Conference on Information Reuse and Integration, IRI 2011; 2011. pp. 406–11.

23. Lucassen A, Watson E, Harcourt J, et al. Guidelines for referral to a regional genetics service: GPs respond by referring more appropriate cases. Fam Pract. 2001;18:135–40. doi: 10.1093/fampra/18.2.135.

24. Potočnik J, Thomas E, Killeen R, et al. Automated vetting of radiology referrals: exploring natural language processing and traditional machine learning approaches. Insights Imaging. 2022;13:1–8. doi: 10.1186/S13244-022-01267-8/TABLES/3.

25. Doan S, Maehara CK, Chaparro JD, et al. Building a Natural Language Processing Tool to Identify Patients With High Clinical Suspicion for Kawasaki Disease from Emergency Department Notes. Acad Emerg Med. 2016;23:628–36. doi: 10.1111/acem.12925.

26. Todd J, Richards B, Vanstone BJ, et al. Text Mining and Automation for Processing of Patient Referrals. Appl Clin Inform. 2018;9:232–7. doi: 10.1055/S-0038-1639482/ID/OR170189CR-10/BIB.

27. Evangelista L, Steinhubl SR, Topol EJ. Digital health care for older adults. The Lancet. 2019;393:1493. doi: 10.1016/S0140-6736(19)30800-1.

28. Portz JD, Bayliss EA, Bull S, et al. Using the Technology Acceptance Model to Explore User Experience, Intent to Use, and Use Behavior of a Patient Portal Among Older Adults With Multiple Chronic Conditions: Descriptive Qualitative Study. J Med Internet Res. 2019;21:e11604. doi: 10.2196/11604.