Impact of Kidney Failure Risk Prediction Clinical Decision Support on Monitoring and Referral in Primary Care Management of CKD: A Randomized Pragmatic Clinical Trial.

Plain-Language Summary Early chronic kidney disease does not have symptoms. Patients and their primary care physicians are often not aware of the risk of progression to kidney failure and potential need for dialysis. This project tested an electronic tool that automatically detected abnormal laboratory results in the patient’s electronic health record and calculated a risk score for the patient. The risk score was shown to the primary care physician. Recommendations for further laboratory testing or referral to a kidney specialist were available if the physician clicked on a link. The results of the study showed that the link was clicked only 122 times out of 2,794 cases where it was available. There was no significant effect on laboratory testing.

Chronic kidney disease (CKD) is increasingly prevalent, affecting 14.9% of Americans. 1 National data and data from our own institution show opportunities for improvement in stage-appropriate monitoring, treatment, and referral in CKD. 1 , 2

The majority of patients with early CKD receive care in primary care practices; however, specialty care is essential for patients with progressive CKD. If patients with CKD are referred to nephrology even 3 months before kidney failure, there is a significant impact on morbidity, mortality, and cost outcomes. 3 , 4 However, early referral does not always take place. 2 , 5 Late referral is associated with patient factors such as age, comorbid conditions, non-White race, a lack of insurance, low socioeconomic status, and low education levels. 6 , 7 , 8 In addition, delayed referral is likely related to the variable rate with which CKD progresses to kidney failure. In 1 cohort study, half of patients had nonlinear progression of their CKD. 9 It has also been found that two-thirds of patients with kidney failure experienced a steady decline in the 2 years before receiving dialysis, whereas one-sixth declined rapidly. 10

To address the uncertainty about the rate of decline, risk prediction models have been developed; for example, an 8-variable model developed by Tangri et al, 11 which incorporates age, sex, estimated glomerular filtration rate (eGFR), urinary albumin-creatinine ratio (UACR), calcium, phosphorous, albumin, and bicarbonate to estimate 5-year risk of kidney failure. The kidney failure risk equation (KFRE) has not yet been widely adopted in primary care settings. This lack of uptake may be at least partly because of the time it takes to locate the calculator online and enter the laboratory values. 12 , 13 One possible strategy to overcome this barrier is to implement clinical decision support (CDS) that automatically calculates the KFRE value and delivers the risk estimate along with recommendations for monitoring and referral. CDS, particularly interruptive CDS, has been effective in improving stage-appropriate monitoring in CKD but not in improving referral. 14 , 15 Theoretically, noninterruptive CDS would lead to less alert fatigue, garner stakeholder support, and still be effective. 16

Our objective was to determine whether CDS could have a positive impact on stage-appropriate monitoring and referral. There were 3 main goals of the clinical trial. The first was to determine whether noninterruptive point-of-care CDS would be noticed and acted upon by primary care physicians (PCPs). The second was to determine whether recommendations regarding laboratory testing would be followed. The third was to determine whether automatic calculation of the 5-year KFRE value would increase referral to nephrology in high-risk patients.

Methods

Overall Study Design

As described previously, we developed and implemented a noninterruptive point-of-care CDS tool. 17 We evaluated the effect of the tool in comparison with that of usual care in a block-randomized pragmatic clinical trial. The trial began on December 4, 2015 and included a 12-month intervention period with a subsequent 6-month follow-up period. The trial was registered on ClinicalTrials.gov ( NCT02990897 ). The protocol was approved by the Partners Human Research Committee under approval #2015P001366. The requirement for informed consent was waived because patients were enrolled automatically through the electronic health record (EHR) in this pragmatic clinical trial and the research presents no more than a minimal risk of harm to participants and involves no procedures for which written consent is normally required outside the research context.

Eligibility and Setting

We identified eligible patients who were older than 18 years and had a primary care visit during the 12-month intervention period. To identify patients with CKD, we identified 2 outpatient eGFRs of <59 mL/min/1.73 m 2 at least 90 days apart. We used the CKD Epidemiology Collaboration creatinine equation. 18 Eligibility was determined by querying the EHR on an individual patient basis by accessing an interoperability data standard called the Continuity of Care Document ( Fig 1 ). 17 The trial was conducted in a primary care network consisting of 11 clinics affiliated with a regional hospital in greater Boston. Figure 1 Clinical decision support application flow for patient screening, enrollment, randomization, and delivery of intervention. Abbreviations: CCD, Continuity of Care Document; CDS, clinical decision support; CKD, chronic kidney disease; CO2, carbon dioxide; eGFR, estimated glomerular filtration rate; PCP, primary care physician.

Block Randomization Procedure

Because the number of patients with CKD varied between PCPs, we opted to randomize at the level of the patient. We decided to block randomize patients in order to minimize the effect of PCP-level practice patterns with respect to outcomes. The end result was that each PCP had an equal number of intervention arm and control arm patients. 2 , 19 We chose a block size of 2. There was no blinding.

CDS Intervention

Patients in the control arm received usual care ( Fig 2 ). For patients in the intervention arm, the noninterruptive CDS appeared on the main screen of the EHR ( Fig 3 ). This risk estimate appeared on the summary screen for patients in the intervention arm 2 days before a scheduled primary care visit. The CDS risk estimate text—for example, “This patient has a 12% risk of end stage renal disease in the next 5 years”—was highlighted to indicate the presence of a hyperlink. The linked webpage included the following: (1) the stage of CKD based on the eGFR and UACR; (2) the KFRE risk estimate; (3) CDS recommendations to order laboratory tests and/or refer the patient to nephrology if risk is >10%; and (4) the laboratory data used in the calculation of the risk estimate, along with data entry fields to allow the PCP to update the values and recalculate the risk estimate ( Fig 1 , Figs S1-S3 ). Based on user feedback, the serum creatinine appeared next to the eGFR ( Fig S2 ). The Continuity of Care Document contains up to 3 values from the preceding 5 years, and any available values were shown. However, the CDS only prompted the PCP to repeat any laboratory tests that were performed more than 1 year ago. Figure 2 Consolidated Standards of Reporting Trials (CONSORT) flowchart: participant enrollment, allocation, exclusion, and analysis. Abbreviations: KFRE, kidney failure risk equation. Figure 3 Partners electronic health record patient summary page showing clinical decision support hyperlink and rollover text.

The development and implementation of the CDS application have been described previously. 17 Briefly, the CDS consisted of a single-page application, web server, database, and application programming interface. The CDS application extracted the laboratory results needed to calculate and display the patient’s 5-year KFRE value from the patient’s Consolidated Clinical Document Architecture interoperability standard Continuity of Care Document. 20 The application repeated the data extraction and calculation 1 day before the visit. This was to mitigate the possibility that the PCP ordered laboratory tests 1 day before the visit.

The CDS appeared at multiple visits per patient over the 12-month intervention period; however, the primary and secondary outcomes were only assessed once per patient at a data collection timepoint 6 months after the first PCP visit during the intervention period. The CDS link appeared to all users of the EHR; however, only users accessing the chart from one of the primary care clinics included in the study were able to open the separate webpage (the Internet Protocol addresses of the included clinics were whitelisted).

No formal training materials were provided to PCPs; however, information sessions were held at the clinics, which were presented by the Principal Investigator (LS) with support from the local nephrologist clinical champion (KB).

Data Collection

The CDS application included a database that stored the assignment of each patient to the intervention or control arm. The database also stored data on how often PCPs clicked the CDS link.

Because of the conversion of the primary care network from a homegrown EHR ( Fig 2 ) to Epic in January 2017, we collected baseline data on the included cohort at the end of the intervention period. Data on outcomes were collected from either the homegrown EHR or Epic, depending on when the 6-month data collection timepoint occurred for each patient.

Missing Data

Our approach toward missing data and data outside of the risk prediction model’s calibration was described in an appendix to the previous publication. 17 Briefly, values were substituted for missing data to calculate a low-risk estimate and a high-risk estimate for each patient. One value was chosen based on the normal range for that laboratory test, and a second value outside of the normal range was chosen based on the characteristics of the original cohort included in the development and validation study conducted by Tangri et al, 11 as described in the appendix to the publication. In other words, we presented a range of risk—for example, “This patient has an 8.9%-11.3% risk of end stage renal disease in the next five years.”

Outcomes and Data Analyses

Patient demographics and clinical characteristics were aggregated in both arms of the study and assessed for differences at baseline using χ 2 tests 21 for categorical variables and robust generalized estimating equations z-tests, 22 , 23 both accounting for clustering by PCP.

Primary Outcome

The primary outcome was the completion of the laboratory tests necessary to calculate the 8-variable KFRE value. For the primary outcome, patients who had undergone the laboratory tests necessary for the KFRE in the preceding 12 months were not included in the analysis ( Fig 2 ). For example, if a patient attending a primary care visit had undergone all of the laboratory tests necessary to calculate the KFRE value in their electronic record, no CDS recommendation was displayed ( Fig S2 ; this patient would not be included in the analysis of the primary outcome). The primary outcome was assessed at a 6-month timepoint after the primary care visit, which allowed time for patients to follow through on the PCP’s orders for the laboratory tests. A 3-month and a 12-month timepoint were considered; however, 3 months was deemed to be too short because of patient delays in visiting the laboratory and 12 months was deemed to be too long because of the effect that intervening visits could have (ie, washing out the effect of the intervention).

The association of the intervention with the primary outcomes was assessed with logistic regression, 22 accounting for clustering by PCP. Because of a lack of differences at baseline (as expected because of patient-level randomization), only unadjusted analyses were performed.

Secondary Outcomes

The UACR is a component of the KFRE but was examined separately as a secondary outcome because it is recommended for stage-appropriate monitoring in CKD. The recommendation to order the UACR may have appeared as the only recommendation or may have accompanied other recommendations ( Fig S1 ). The outcome was also assessed only in those patients who did not have a UACR in the previous 12 months, and the outcome was also assessed at a 6-month timepoint after the primary care visit.

Nephrology referral was examined as a secondary outcome in the subgroup of patients with a risk estimate of >10% ( Fig S3 ) or an estimated range of risk exceeding 10% ( Fig S4 ). The outcome was assessed in patients regardless of whether they had seen a nephrologist in the preceding 12 months and was assessed at a 6-month timepoint after the primary care visit ( Fig S5 ).

Formal and Informal Feedback from PCPs

In a previously published qualitative study, we interviewed PCPs about CKD management and the potential for a CKD CDS system. 24 We developed the application in parallel with the qualitative study; thus, we were able to show PCPs a screenshot of the CDS application to gather feedback formally. We also gathered informal feedback during site visits to the clinics included in the clinical trial.

Results

The CDS application requested and processed 569,533 Continuity of Care Documents during the period between December 4, 2015, and December 3, 2016. Of these, 41,842 (7.3%) documents led to a diagnosis of stage 3, 4, or 5 CKD in a total of 5,590 patients.

There were no differences in the patients with CKD at baseline ( Table 1 ). Patients were predominantly White and more than half were women. The mean age was 77 years. The majority of patients used public insurance, and the median Charlson comorbidity index was 2.00. Table 1 Demographic and Clinical Characteristics at Baseline Patient characteristics Total (N = 5,590) Intervention (N = 2,794) Control (N = 2,796) P Value Age (y), mean (SD) 77.20 (10.90) 77.25 (10.90) 76.96 (10.90) 0.29 Male sex, n (%) 2,173 (38.87%) 1,088 (38.94%) 1,085 (38.81%) 0.90 Race/ethnicity, n (%) White 5,397 (96.56%) 2,700 (96.67%) 2,697 (96.46%) 0.67 Non-White 192 (3.44%) 93 (3.33%) 99 (3.54%) Marital status, n (%) Married 2,544 (45.51%) 1,275 (45.63%) 1,269 (45.39%) 0.84 Single or other 3,046 (54.49%) 1,519 (54.37%) 1,527 (54.61%) Tobacco use, n (%) Current 379 (6.78%) 192 (6.87%) 187 (6.69%) 0.85 Former 2,434 (43.54%) 1,225 (43.84%) 1,209 (43.24%) Never 2,777 (49.68%) 1,377 (49.28%) 1,400 (50.07%) Insurance status, n (%) Public 4,229 (75.65%) 2,133 (76.34%) 2,096 (74.96%) 0.38 Private 1,354 (24.22%) 658 (23.55%) 696 (24.89%) Self-pay 7 (0.13%) 3 (0.11%) 4 (0.14%) Diabetes 1,773 (32%%) 879 (32%%) 894 (32%) Charlson comorbidity index, median (IQR) 2.00 (1.00-4.00) 2.0 (1.0-4.0) 2.0 (1.0-4.0) 0.82 KFRE characteristics KFRE risk, median (IQR) 2.30 (1.10-5.60) 2.20 (1.10-5.50) 2.30 (1.20-5.70) 0.35 KFRE risk >10%, n (%) 840 (15.03%) 419 (15.00%) 421 (15.06%) 0.95 Abbreviations: IQR, Interquartile range; KFRE, kidney failure risk equation; SD, standard deviation.

Upon enrollment, only 319 (319 of 2,794 = 11%) patients in the intervention arm and 357 (357 of 2,796 = 13%) patients in the control arm had the laboratory test values necessary to calculate the KFRE value at baseline. One of the laboratory tests necessary to calculate the KFRE value, the UACR, was only present in 874 (874 of 2,794 = 31%) patients in the intervention arm and 931 (931 of 2,796 = 33%) patients in the control arm at baseline. For the rest of the patients who did not have the laboratory test values necessary to calculate the KFRE value, it was estimated using methods to substitute 2 possible values for each missing value to produce a range of risk, as described in the Methods section.

There was no difference in the median KFRE risk or the proportion of patients with a KFRE risk of >10% between the 2 arms ( Table 1 ).

The CDS link was displayed for intervention patients to 139 PCPs. Overall, the CDS link was clicked just 122 times by 57 PCPs. Only 14 PCPs clicked the link more than once, with the heaviest user clicking the link 20 times and the second heaviest user clicking the link 19 times. The intervention patients for whom the link was clicked included 26 (21%) who had the KFRE laboratory tests done at baseline (not included in analysis), 23 (19%) whose PCP subsequently ordered the KFRE laboratory tests, and 73 (60%) who neither had the KFRE laboratory test values at baseline nor in the subsequent 6 months. Only 22 of these 122 intervention patients for whom the link was clicked had a KFRE risk of >10%. Three (14%) patients had seen a nephrologist in the preceding 12 months; 6 patients (27%) were subsequently referred to nephrology in the following 6 months, and 13 (60%) were not referred before the study or in the subsequent 6 months.

Primary Outcome

There was no significant impact of the intervention on the primary outcome. The proportion of patients for whom the PCP ordered the laboratory tests necessary to calculate the KFRE value was minimally higher at 6 months in the intervention arm than in the control arm; however, the difference was not significant: 6.95% (95% confidence interval [CI], 5.57%-8.33%) versus 6.56% (95% CI, 5.22%-7.90%), P = 0.61 ( Fig 4 A); difference in proportions, 0.39% (95% CI, −1.11% to 1.89%). Figure 4 Comparison of the proportion of patients with completion of tests or referrals. (A) Primary outcome: completion of tests necessary for calculation of the kidney failure risk equation in the intervention arm versus in the control arm patients. (B) Secondary outcome: completion of urinary albumin-to-creatinine ratio test in the intervention arm versus in the control arm patients. (C) Secondary outcome: completion of nephrology consultation for patients with a risk estimate of >10% in the intervention arm versus in the control arm patients.

Secondary Outcomes

In the 1,920 intervention arm patients who had not had a UACR in the preceding 12 months compared with the 1,865 control arm patients who had not had a UACR in the preceding 12 months, the proportion of patients for whom the PCP ordered the UACR test was minimally higher in the intervention arm than in the control arm; however, the difference was not significant: 14.90% (95% CI, 11.90%-17.89%) versus 14.32% (95% CI, 11.26%-17.38%); P = 0.54 ( Fig 4 B); difference in proportions, 0.58 (95% CI, −1.3% to 2.5%).

For 840 patients with a KFRE risk of >10%, there was a statistically significantly lower proportion of nephrology referral in the intervention arm than in the control arm: 118 of 419 (28.16%; 95% CI, 22.48%-33.84%) versus 161 of 421 (38.24%; 95% CI, 32.82%-43.67%; P = 0.01) ( Fig 4 C); difference in proportions, −10.1% (95% CI, −3.0% to −17.2%).

Formal and Informal Feedback from PCPs

As described in the Methods section, we collected formal feedback from PCPs in a qualitative study and informal feedback during visits to the clinics involved in the study. 24 In the qualitative study, when we showed PCPs a screenshot of the CDS application, all 12 PCPs thought that the CDS would raise their awareness of a patient’s increased risk of developing worsening CKD. Many PCPs appreciated the exact percentage estimate for 5-year kidney failure risk. Some PCPs requested additional information about the patient’s context for example, the results of recent serum creatinine, potassium, eGFR, hemoglobin A 1C , hemoglobin, and urinalysis, comorbid conditions such as diabetes, hypertension (including most recent blood pressure reading), multiple myeloma, and history of kidney transplant, or relevant medications. In addition, several PCPs made comments about their lack of familiarity with the KFRE and what type of information could make it more useful. For example, 1 PCP said “I think the concept of what the five year kidney failure risk is, is totally foreign,” then went on to compare the concept to the Framingham study risk estimate, saying “definitely, that number prompts action whether my patients’ risk of a heart attack is 4% or 10% over the next decade.” Another PCP made a similar comparison, “At the beginning I wouldn’t have a feel for what the five year risk is. But we’ve gotten better at that for cardiovascular disease,” and went on to say that knowing which risk factors could lower risk would be helpful. Several PCPs mentioned that the risk estimate would not prompt referral for elderly patients or those with active malignancy.

During the course of the study, the Principal Investigator visited 10 of the 11 primary care clinics included in the clinical trial either for a noon conference or to spend a half day observing clinic workflow. There were 2 examples of unique clinic workflows: 1 clinic had a point-of-care UACR machine and another clinic had a team-based care model where nonphysician staff members reviewed CDS recommendations and prefilled test orders. During the visits, PCPs gave informal feedback to the Principal Investigator. Generally, PCPs cited lack of time and lack of familiarity with the KFRE as reasons for not clicking the CDS link. One PCP said that she was afraid to click the link because she was not sure if it would exit the patient’s chart. Another PCP said that she did not have time to click the link, but she copied the risk estimate and pasted it into her note. She also mentioned that the appearance of a range of risk alerted her to the fact that there were missing laboratory test values.

Discussion

Our study found no effect of the intervention on stage-appropriate monitoring. The only statistically significant finding was that, within the subgroup of patients with a KFRE risk of >10%, patients in the control arm had a higher proportion of nephrology referral than patients in the intervention arm, an absolute difference of 43 patients. Although this is a small absolute number of patients, it is worthwhile to consider possible explanations. For one, the subgroup of control arm patients with a KFRE risk of >10% may have had higher KFRE risk than that in the parallel subgroup of intervention patients or may have had more severe illness in other, unmeasured ways. Arguing against this possibility is the fact that the proportion of patients with KFRE risk was balanced across the arms, and the median KFRE risk was equal across the arms. Another possibility is that the intervention detracted attention from appropriate management, thereby discouraging PCPs from referring high-risk patients to nephrology. However, this is unlikely given that the CDS recommendations were only viewed 122 times out of the 2,794 cases where it was available.

Multiple factors may have contributed to the lack of observed effect on the primary outcome. Given the low number of times that PCPs clicked on the link, we know that the CDS recommendations were not viewed often. Interruptive CDS is generally more effective but contributes to pop-up fatigue; hence, alternate designs for CDS are necessary. 25 , 26

It is worth highlighting that the overall rate of UACR monitoring was low at baseline, around 30%, and that the rate of new orders for UACR tests over the subsequent 6 months in both the control and intervention arms was ∼15%. The proportion of patients with UACR at baseline in the development cohort for the original Tangri publication was only 1,723 of 3,449 (50%). 11 In the subsequent multinational validation study, the proportion of patients with UACR ranged from 9%-99% across 31 cohorts, totaling 721,357 patients. 27 Of note, in a more recent study conducted in the Mass General Brigham health system of 61,546 patients, the UACR rate was also 30%. 28

The lack of intended effect of the CDS can be examined in the context of the Five Rights of CDS: the right information, to the right person, in the right format, through the right channel, at the right point in the workflow. 29 In the case of this unsuccessful intervention, the last 2 “rights,” the right channel and the right point in the workflow, were lacking. Informal observation in the primary clinics also revealed at least 1 instance where the CDS was not viewed by the right person. A second factor contributing to the lack of effect is that we included all patients with stage 3-5 CKD who were seen in this network of primary care clinics, which resulted in an elderly population with a considerable level of comorbid conditions, some of whom were palliative care patients. PCPs are appropriately disinclined to refer patients to specialists in such a population. Even for those younger patients with stage 3 CKD, as we learned through the qualitative study, PCPs are inclined to focus on managing comorbid conditions such as diabetes and hypertension, which are, in turn, relevant to the management of CKD progression. 24 Therefore, inclusion of other clinical outcomes like hemoglobin A 1C and blood pressure may have shown other downstream effects of increased awareness of the risk of kidney failure. Third, in reference to nephrology referral, the formal feedback from PCPs pointed to a lack of familiarity with the KFRE and reluctance to use the information in the decision to refer a patient. Fourth, there are many patient-level and PCP-level barriers that cannot be addressed solely through CDS. For example, the majority of patients with CKD are not aware of the diagnosis. 30 There are many barriers in primary care, including a lack of time to counsel and educate patients, the challenge of staying current with guidelines for multiple chronic conditions, and the unintended burden of EHRs. 31

Our findings are similar to the results of other studies of noninterruptive CDS that have failed to show a significant impact on process outcomes and clinical outcomes in CKD. 32 , 33 In particular, a study of passive alerts for CKD monitoring and referral in a primary care setting failed to show an effect on nephrology referral, although it did show an effect on the UACR. 14 The authors attributed the positive effect to additional education and quality improvement initiatives, suggesting that CDS should be part of a multicomponent intervention. Multicomponent interventions should be studied within an Implementation Science framework to determine which components are consistently deployed and which components have the most impact.

The trial’s limitations include our choice to block randomize patients rather than providers to ensure balance of patients in each arm. This approach leads to contamination. We also used clustering to account for PCP-level effects, which biases the study result toward null. We designed the CDS application to calculate the 8-variable version of the KFRE, although a 4-variable version has been validated in a multinational set of patient cohorts and is currently more commonly used. However, if the trial result had been positive, updating the CDS application would have required a trivial amount of reprogramming. We used a pragmatic clinical trial approach in a primary care setting, which allowed higher enrollment and less biased enrollment than that possible with a traditional clinical trial. However, this enrollment approach is still limited and does not represent prevalent cases of CKD overall. The reasons are as follows: PCPs may not have ordered stage-appropriate monitoring tests, patients may not have completed the tests, or patients may not have had regular primary care visits within the timeframe captured by the CCDA document. Specifically, we only included patients who had at least 2 eGFR results in the previous 5 years, which lowered the generalizability of the study to the large reservoir of patients who do not benefit from consistent monitoring of kidney function. From a public health perspective, resources should be focused on identifying groups at a high risk of developing CKD who are not engaged in primary care to realize the greatest benefit of tools such as the KFRE. 34

In conclusion, this pragmatic randomized controlled trial of noninterruptive CDS delivering the KFRE to PCPs failed to show an effect on stage-appropriate monitoring and referral. Future efforts should include multicomponent interventions, community-based screening of high-risk populations, and population health management of patients at a high risk of kidney failure.

References

1 United States Renal Data System USRDS 2020 Annual Data Report: Atlas of Chronic Kidney Disease and End-Stage Renal Disease in the United States 2020 National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases https://adr.usrds.org/2020/

2 Samal L. Wright A. Waikar S.S. Linder J.A. Nephrology co-management versus primary care solo management for early chronic kidney disease: a retrospective cross-sectional analysis BMC Nephrol 16 2015 162 10.1186/s12882-015-0154-x 26458541 PMC4603818

3 Chan M.R. Dall A.T. Fletcher K.E. Lu N. Trivedi H. Outcomes in patients with chronic kidney disease referred late to nephrologists: a meta-analysis Am J Med 120 12 2007 1063 1070 10.1016/j.amjmed.2007.04.024 18060927

4 Stroupe K.T. Fischer M.J. Kaufman J.S. Predialysis nephrology care and costs in elderly patients initiating dialysis Med Care 49 3 2011 248 256 10.1097/MLR.0b013e31820192ba 21224743

5 Greer R.C. Liu Y. Cavanaugh K. Primary care physicians’ perceived barriers to nephrology referral and co-management of patients with CKD: a qualitative study J Gen Intern Med 34 7 2019 1228 1235 10.1007/s11606-019-04975-y 30993634 PMC6614220

6 Baer G. Lameire N. Van Biesen W. Late referral of patients with end-stage renal disease: an in-depth review and suggestions for further actions NDT Plus 3 1 2010 17 27 10.1093/ndtplus/sfp050 25949401 PMC4421547

7 Navaneethan S.D. Aloudat S. Singh S. A systematic review of patient and health system characteristics associated with late referral in chronic kidney disease BMC Nephrol 9 2008 3 10.1186/1471-2369-9-3 18298850 PMC2291456

8 Norton J.M. Moxey-Mims M.M. Eggers P.W. Social determinants of racial disparities in CKD J Am Soc Nephrol 27 9 2016 2576 2595 10.1681/ASN.2016010027 27178804 PMC5004663

9 Li L. Astor B.C. Lewis J. Longitudinal progression trajectory of GFR among patients with CKD Am J Kidney Dis 59 4 2012 504 512 10.1053/j.ajkd.2011.12.009 22284441 PMC3312980

10 O’Hare A.M. Batten A. Burrows N.R. Trajectories of kidney function decline in the 2 years before initiation of long-term dialysis Am J Kidney Dis 59 4 2012 513 522 10.1053/j.ajkd.2011.11.044 22305760 PMC3312937

11 Tangri N. Stevens L.A. Griffith J. A predictive model for progression of chronic kidney disease to kidney failure JAMA 305 15 2011 1553 1559 10.1001/jama.2011.451 21482743

12 Rigatto C. Sood M.M. Tangri N. Risk prediction in chronic kidney disease: pitfalls and caveats Curr Opin Nephrol Hypertens 21 6 2012 612 618 10.1097/MNH.0b013e328359072f 23042027

13 Kidney failure risk equation (8 variable). QxMD https://qxmd.com/calculate/calculator_125/kidney-failure-risk-equation-8-variable

14 Abdel-Kader K. Jhamb M. EHR-based clinical trials: the next generation of evidence Clin J Am Soc Nephrol 15 7 2020 1050 1052 10.2215/CJN.11860919 32094245 PMC7341790

15 Allen A.S. Forman J.P. Orav E.J. Bates D.W. Denker B.M. Sequist T.D. Primary care management of chronic kidney disease J Gen Intern Med 26 4 2011 386 392 10.1007/s11606-010-1523-6 20922494 PMC3055964

16 Phansalkar S. van der Sijs H. Tucker A.D. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records J Am Med Inform Assoc 20 3 2013 489 493 10.1136/amiajnl-2012-001089 23011124 PMC3628052

17 Samal L. D’Amore J.D. Bates D.W. Wright A. Implementation of a scalable, web-based, automated clinical decision support risk-prediction tool for chronic kidney disease using C-CDA and application programming interfaces J Am Med Inform Assoc 24 6 2017 1111 1115 10.1093/jamia/ocx065 29016969 PMC6580936

18 Levey A.S. Stevens L.A. Schmid C.H. A new equation to estimate glomerular filtration rate Ann Intern Med 150 9 2009 604 612 10.7326/0003-4819-150-9-200905050-00006 19414839 PMC2763564

19 Samal L. Linder J.A. Bates D.W. Wright A. Electronic problem list documentation of chronic kidney disease and quality of care BMC Nephrol 15 1 2014 70 10.1186/1471-2369-15-70 24885821 PMC4021481

20 HL7 Implementation guide: S&I framework transitions of care companion guide to consolidated-CDA for meaningful use stage 2, release 1–US realm. HL7 International https://www.hl7.org/implement/standards/product_brief.cfm?product_id=374

21 Irimata KE, He Y, Cai B, Shin HC, Parsons VL, Parker JD. Comparison of quarterly and yearly calibration data for propensity score adjusted web survey estimates. Surv Methods Insights Field. Published online October 12, 2020. https://doi.org/10.13094/SMIF-2020-00018 10.13094/SMIF-2020-00018 PMC7976170 33748097

22 Lipsitz S.R. Fitzmaurice G.M. Orav E.J. Laird N.M. Performance of generalized estimating equations in practical situations Biometrics 50 1 1994 270 278 8086610

23 Liang K.Y. Zeger S.L. Longitudinal data analysis using generalized linear models Biometrika 73 1 1986 13 22

24 Gulla J. Neri P.M. Bates D.W. Samal L. User requirements for a chronic kidney disease clinical decision support tool to promote timely referral Int J Med Inform 101 2017 50 57 10.1016/j.ijmedinf.2017.01.018 28347447 PMC5497591

25 Elias P. Peterson E. Wachter B. Ward C. Poon E. Navar A.M. Evaluating the impact of interruptive alerts within a health system: use, response time, and cumulative time burden Appl Clin Inform 10 5 2019 909 917 10.1055/s-0039-1700869 31777057 PMC6881214

26 Shah S.N. Amato M.G. Garlo K.G. Seger D.L. Bates D.W. Renal medication-related clinical decision support (CDS) alerts and overrides in the inpatient setting following implementation of a commercial electronic health record: implications for designing more effective alerts J Am Med Inform Assoc 28 6 2021 1081 1087 10.1093/jamia/ocaa222 33517413 PMC8661393

27 Tangri N. Grams M.E. Levey A.S. Multinational assessment of accuracy of equations for predicting risk of kidney failure: a meta-analysis JAMA 315 2 2016 164 174 10.1001/jama.2015.18202 26757465 PMC4752167

28 Ahmed S. Mothi S.S. Sequist T. Tangri N. Khinkar R.M. Mendu M.L. The kidney failure risk equation score and CKD care delivery measures: a cross-sectional study Kidney Med 4 1 2022 100375 10.1016/j.xkme.2021.08.010 PMC8767093 35072040

29 Osheroff J.A. Teich J.M. Levick D. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide 2nd ed. 2012 HIMSS Publishing

30 Tuot D.S. Plantinga L.C. Hsu C.Y. Chronic kidney disease awareness among individuals with clinical markers of kidney dysfunction Clin J Am Soc Nephrol 6 8 2011 1838 1844 10.2215/CJN.00730111 21784832 PMC3156423

31 Sperati C.J. Soman S. Agrawal V. Primary care physicians’ perceptions of barriers and facilitators to management of chronic kidney disease: a mixed methods study PLOS ONE 14 8 2019 e0221325 10.1371/journal.pone.0221325 PMC6705804 31437198

32 Lo H.G. Matheny M.E. Seger D.L. Bates D.W. Gandhi T.K. Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care J Am Med Inform Assoc 16 1 2009 66 71 10.1197/jamia.M2687 18952945 PMC2605597

33 Seidling H.M. Phansalkar S. Seger D.L. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support J Am Med Inform Assoc 18 4 2011 479 484 10.1136/amiajnl-2010-000039 21571746 PMC3128393

34 Samal L. Linder J.A. The primary care perspective on routine urine dipstick screening to identify patients with albuminuria Clin J Am Soc Nephrol 8 1 2013 131 135 10.2215/CJN.12681211 22917702 PMC12895568

Supplementary Material

Supplementary File (PDF) Figures S1-S5.

Article Information

Authors’ Full Names and Academic Degrees

Lipika Samal, MD, MPH, John D. D’Amore, MS, Michael P. Gannon, ScM, John L. Kilgallon, BA, Jean-Pierre Charles, BS, Devin M. Mann, MD, Lydia C. Siegel, MD, MPH, Kelly Burdge, MD, Shimon Shaykevich, MS, Stuart Lipsitz, ScD, Sushrut S. Waikar, MD, MPH, David W. Bates, MD, MSc, and Adam Wright, PhD

Authors’ Contributions

Research idea and study design: LS, JDD, DMM, SSW, DWB, AW; software development: JDD; conduct of the clinical trial: LS, LCS, KB; data acquisition: LS, JDD, AW; data analysis/interpretation: LS, SS, SL, JLK, MPG, JPC; statistical analysis: SS, SL; supervision or mentorship: DMM, SL, SSW, DWB, AW. Each author contributed important intellectual content during manuscript drafting or revision and accepts accountability for the overall work by ensuring that questions pertaining to the accuracy or integrity of any portion of the work are appropriately investigated and resolved.

Support

Research reported in this publication was supported by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health under award K23DK097187. The funder had no role in the study design; collection, analysis, and interpretation of data; writing the report; or decision to submit the report for publication.

Financial Disclosure

Dr Samal has received funding from IBM through an investigator-initiated grant on COVID-19 health services research. The remaining authors declare that they have no relevant financial interests.

Acknowledgements

We would like to acknowledge Joy Gulla, BA, for data collection and administrative support, Julie Fiskio, BS, for data acquisition, Zoe Burns, MPH, for administrative support and project management, E. Francis Cook, ScD, for expertise in epidemiology, Anthony Komaroff, MD, for expertise in conduct of research studies, Navdeep Tangri, MD, PhD, for publicly sharing the model specifications without which we could not have developed this CDS application and for collaboration throughout the project, and Jeffrey Linder, MD, MPH, for mentorship on grant preparation, manuscript preparation, and, generally, on the use of health information technology to measure population health and to conduct large-scale, pragmatic, randomized clinical trials in primary care settings.

Peer Review

Received January 7, 2022 as a submission to the expedited consideration track with 2 external peer reviews. Direct editorial input from the Statistical Editor and the Editor-in-Chief. Accepted in revised form April 24, 2022.

Complete author and article information provided before references.

Supplementary File (PDF)

Figure S1 : CDS recommendation to order UACR and serum phosphorus for a patient who had not received these tests in the past 12 months.

Figure S2 : No CDS recommendation for a patient who had received all of the labs necessary to calculate KFRE in the past 12 months. This patient would have been excluded from the analysis of the primary outcome because no action was recommended to the PCP.

Figure S3: CDS recommendation to refer patient to nephrology due to risk estimate >10%.

Figure S4: Two CDS recommendations to 1) refer patient to nephrology due to estimated range of risk which exceeds 10% and 2) order lab tests necessary to calculate KFRE.

Figure S5 : CONSORT flowchart: Participant enrollment, allocation, exclusion, and analysis for secondary outcome of nephrology referral.