Does the SORG Machine-learning Algorithm for Extremity Metastases Generalize to a Contemporary Cohort of Patients? Temporal Validation From 2016 to 2020.

Текст сохранён. Часть иллюстраций пока доступна только в первоисточнике; загрузка будет проверена повторно.

Introduction

Metastatic bone disease affects between 280,000 and 400,000 people each year in the United States [ 7 , 8 , 14 , 30 ]. Metastatic bone disease is a major source of morbidity and mortality and is a growing economic burden; its cost was estimated to be USD 12.6 billion in 2007, or roughly 17% of all direct medical costs in cancer care [ 20 ]. Survival after the diagnosis of a bone metastasis may be measured in months, or it may be measured in years. Knowing which patients are likely to survive for long periods after this diagnosis is important to help them make personal plans and determine care decisions, such as whether to preemptively stabilize impending pathologic fractures. However, patients with an estimated survival of less than 3 months may have a relative contraindication to prophylactic stabilization because morbidity, the short-term complications of stabilization, and costs must be weighed against that of fracture risk during this shorter time span [ 16 ]. Those expected to live more than 1 year must contend with the risks of longer-term complications such as nonunion or implant failure as well as short-term complications and morbidity, which could heavily influence the choice of implant or the choice between temporal stabilization or arthroplasty [ 10 ].

Several attempts have been made to create predictive models for survival in these patients. Methods have included classic statistics [ 27 ] and more complex ones such as boosting algorithms [ 11 ] and Bayesian belief networks [ 4 ]. Machine learning has also been applied in order to answer this question [ 24 ]. Machine learning is an area of artificial intelligence research that can be used to develop predictive models based on large datasets. These methods have been used extensively in areas from diagnostic radiology to orthopaedic surgery [ 1 , 5 , 17 , 18 , 26 ]. We, the Skeletal Oncology Research Group (SORG), previously developed a machine-learning algorithm (MLA) to predict 90-day and 1-year survival of surgically treated patients with extremity bone metastasis [ 24 ]. This was based on 1090 patients treated at two tertiary institutions between 1999 and 2016. The 15 variables used in this model were age, primary tumor, visceral or brain metastases, use of systemic therapy, and 10 laboratory values including albumin and lymphocyte count. However, prediction models based on these historical cohorts are at risk of becoming outdated, because there have been major advancements in cancer biology and treatment over the past decade. It is therefore important to assess how these models perform in patients from more recent time periods to determine whether these models are still relevant. Although the SORG MLA has been validated internationally [ 22 , 29 ], it has not been tested on a contemporary cohort.

We therefore sought to evaluate the model’s performance on a newer cohort of patients who were surgically treated for extremity metastatic disease of bone between 2016 and 2020. In this study, we asked: Does the SORG-MLA accurately predict 90-day and 1-year survival in patients who receive surgical treatment for a metastatic long-bone lesion in a more recent cohort of patients treated between 2016 and 2020?

Patients and Methods

Study Design and Setting

This retrospective, comparative study was performed at a high-volume, urban tertiary care center in the northeastern United States (Massachusetts General Hospital and Brigham and Women’s Hospital, both located in Boston, MA, USA) in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis [ 15 ] and the Strengthening the Reporting of Observational Studies in Epidemiology [ 2 ] guidelines.

Patients

Between 2017 and 2021, we identified 674 patients who were 18 years and older through the ICD codes for secondary malignant neoplasm of bone and bone marrow and the CPT codes for completed pathologic fractures or prophylactic treatment of an impending fracture (Fig. 1 ). Exclusion criteria were no surgery (n = 118); metastases in a location other than the femur, humerus, tibia, ulna, or radius (n = 72); treatment other than intramedullary nailing, endoprosthetic reconstruction, plate-screw fixation, or dynamic hip screw (n = 23); revision surgery (n = 23); no tumor present (n = 17); and loss to follow-up within 1 year (n = 15). Completed pathologic fractures and impending pathologic fractures were included. A multidisciplinary team of medical oncologists, anesthesiologists, and orthopaedic surgeons were responsible for assessing patients’ ability to undergo surgery. An impending pathologic fracture was assessed by an orthopaedic surgeon and radiologist with the use of clinical and radiographic features of the lesion. Applying the inclusion and exclusion criteria resulted in 406 patients who underwent surgical treatment for a long-bone metastasis (Fig. 1 ).

Baseline Characteristics

The median patient age in the cohort was 67 years (IQR 59 to 74 years), and 54% (221 of 406) of patients were female (Table 1 ). Regarding primary tumor histology, 34% (137 of 406) of the patients were in the slow-growth group, 32% (131 of 406) were in the moderate-growth group, and 34% (138 of 406) were in the rapid-growth group. Most lesions were in the lower extremity (79% [321 of 406]). Most patients were treated with an intramedullary nail (58% [237 of 406]), 28% (114 of 406) of the patients received endoprosthetic reconstruction, 8% (31 of 406) of the patients were treated with plate-screw fixation, 5% (21 of 406) of the patients had multiple implants, and 1% (3 of 406) were treated with a dynamic hip screw.

Forty-five percent (182 of 406) of patients had visceral metastases and 16% (63 of 406) had brain metastases. Twenty-three percent (92 of 406) died within 90 days and 51% (208 of 406) died within 1 year. Patients in the validation cohort were older than those in the development cohort (median age 67 years [IQR 59 to 74] versus 63 years [IQR 54 to 72]) (Table 1 ). Patients in the validation cohort also presented with slow-growth tumors less often than those in the development cohort (34% [137 of 406] versus 42% [460 of 1090]) and more often with moderate growth tumor types (32% [131 of 406] versus 24% [263 of 1090]). Additionally, patients in the validation cohort presented with a pathologic fracture less often than those in the development cohort (48% [194 of 406] versus 55% [595 of 1090]).

Outcome and Explanatory Variables

As in the developmental cohort of the SORG-MLA, the primary outcomes were 90-day and 1-year mortality. Mortality was defined as the time between surgical treatment for a long-bone metastasis and death of any cause. The date of death or date of the last follow-up was retrieved from electronic healthcare records and the Social Security Death Index.

We manually obtained the 15 required input variables used in the SORG-MLA by reviewing operative notes, medical records, radiology reports, and pathology: sex, age, histologic subtype of the primary tumor (classified into the following three groups: slow growth, moderate growth, and rapid growth as classified by Katagiri et al. [ 13 ]), tumor location, visceral metastases, brain metastases, previous systemic therapy, and 12 preoperative laboratory values. Systemic therapy was defined as having received at least one of the following: chemotherapy, targeted therapy, hormone therapy, or immunotherapy. Preoperative laboratory values comprised absolute lymphocyte count (x10 3 /uL), absolute neutrophil count (x10 3 /uL), albumin level (g/dL), alkaline phosphatase (IU/L), calcium (mg/dL), creatinine (mg/dL), hemoglobin level (g/dL), neutrophil-to-lymphocyte ratio, platelet count (x10 3 /uL platelet-to-lymphocyte count, sodium (mg/dL) levels, and white blood cell count (x10 3/ uL).

Ethical Approval

Ethical approval for this study was obtained from the institutional review board at Massachusetts General Hospital, Boston, MA, USA (2018P000688).

Statistical Analysis

We assessed differences in baseline characteristics between the development cohort [ 24 ] and the temporal validation cohort (patients from the same institution with the same inclusion and exclusion criteria, except the temporal validation cohort included patients from a more recent period) using the chi-square test for categorical variables and the Mann-Whitney U test for continuous variables.

After running the algorithm’s code on the dataset, we analyzed and compared the predictions of all individual patients with actual 90-day and 1-year mortality rates. Algorithm performance was assessed by the same metrics used in the developmental study and as recommended by the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis guidelines [ 15 ]: discrimination using the area under the curve (AUC); calibration by using a visual plot, intercept, and slope; overall performance using a Brier score and a null model Brier score; and a decision curve analysis.

The AUC, which is a measure of the ability of the model to distinguish (discriminate) between positive and negative outcomes (in this study, the occurrence of death is the positive outcome) was visualized by plotting receiver operating characteristic curves. The AUC ranges from 0.5 (no better than a coin toss) to 1.0 (perfect discrimination). In general, an AUC of 0.9 to 1.0 represents excellent discrimination, 0.8 to 0.9 good, 0.7 to 0.8 reasonable, 0.6 to 0.7 fair, and 0.5 to 0.6 poor, where 0.75 is generally accepted as a prediction model that can aid physicians in their predictions [ 3 ]. Calibration of a prediction model represents the agreement between the predicted versus the actual outcome (in this case, 90-day and 1-year mortality). This was visualized by plotting the predicted probabilities (x-axis) against the observed outcome (y-axis). The calibration curve provides an intercept and a slope. The calibration intercept indicates the extent to which predictions are systematically too low or too high. A negative intercept indicates overestimation of the observed outcome. The calibration slope indicates agreement between the predicted probability and observed outcome. In a model that is calibrated perfectly, the slope equals 1. However, if the calibration slope is less than 1, it suggests that the predicted risks were overly extreme, overestimating for high-risk patients and underestimating for low-risk patients.

The Brier score measures the overall performance of the algorithm by calculating the mean squared difference between predicted probabilities and actual outcomes. A lower score indicates more accurate predictions, while a high score suggests the predicted probabilities are unreliable. In a binary classification, a perfect model would have a Brier score of 0 and a completely random model would have a score of 0.25. When evaluating the Brier score, the prevalence of outcome must also be considered, which is why a null-model Brier score was calculated by assigning a probability equal to the prevalence of the outcome to each patient to serve as a reference point. Decision curve analysis provided a framework to judge the relative value of benefits and harms associated with the prediction model. It aims to determine whether the benefits of using a model outweigh the costs of either treating all or no patients. Decision curve analysis charts the net benefit of a model across a range of threshold probabilities that represent the minimum likelihood of an event that would warrant a specific treatment or intervention. By doing so, decision curve analysis provides useful insights into the clinical impact of using a model to guide decision-making, helping healthcare providers decide whether to adopt a model in their practice.

Lastly, an F1 score was calculated for both models. The F1 score is a performance metric commonly used in evaluating classification models. It considers precision and recall and reflects how well the model can correctly identify positive cases while minimizing the number of false positives. A higher F1 score generally indicates a more accurate and reliable model, while a lower score suggests that the model may need improvement.

The missForest method [ 13 ] was used to impute missing values for the following variables: albumin (17% [70 of 406]), alkaline phosphatase (17% [70 of 406]), calcium (4% [17 of 406]), creatinine (4% [17 of 406]), hemoglobin (3% [12 of 406]), lymphocyte count (14% [56 of 406]), neutrophil count (14% [56 of 406]), platelet count (3% [13 of 406]), sodium (4% [16 of 406]), and white blood cell count (3% [12 of 406]). Vital status at 90 days was missing in 1% (4 of 406) and in 4% (15 of 406) at 1 year.

All statistical analyses were performed using Python (Python Software Foundation). A two‐tailed p value less than 0.05 was considered significant.

Results

Four-hundred six patients were included in the final analysis. The median age was 67 years (IQR 59 to 74 years) and 54% (221 of 406) of the patients were female. Forty-eight percent (194) of the patients presented with a pathologic fracture, and the mortality percentage was 23% (92) for the 90-day timepoint and 51% (208) for the 1-year timepoint.

Ability of the SORG-MLA to Predict 90-day and 1-year Survival in Temporal Validation

The SORG-MLA showed a decreased ability to accurately predict the postoperative 90-day and 1-year mortality. AUC, which is a measure of discrimination (a perfect model has an AUC of 1 and 0.5 represents a model that can discriminate no better than a coin toss) was 0.78 (95% CI 0.72 to 0.82) for 90-day mortality and 0.75 (95% CI 0.70 to 0.79) for 1-year mortality (Table 2 ). Generally speaking, a model with an AUC of 0.75 is considered suitable for clinical use. The calibration analysis provided an intercept of -0.66 (95% CI -0.94 to -0.39) (indicating overestimated predictions) and slope of 0.71 (95% CI 0.53 to 0.89) (suggesting overly extreme predictions) for 90-day mortality prediction. For 1-year mortality, calibration showed an intercept of -0.67 (95% CI -0.90 to -0.43) and a slope of 0.73 (95% CI 0.56 to 0.91) (Fig. 2 ). The actual 90-day and 1-year mortality rates in the temporal validation cohort were lower than the predicted values (90 days: 23% versus 28%; p < 0.001, and 1 year: 51% versus 59%; p < 0.001), which are represented by the negative calibration intercepts. Calibration assesses the agreement between predicted and actual outcomes, with a perfect model having a slope of 1.

The Brier score measures the overall performance of the algorithm, with a lower score indicating more accurate predictions. The prevalence of the outcome should also be considered when evaluating the Brier score, and a null-model Brier score serves as this reference point. Ninety-day mortality prediction was 0.16 (95% CI 0.14 to 0.18) compared with a higher null-model Brier score of 0.18, indicating greater performance of the SORG-MLA. For 1-year mortality prediction, the Brier score was 0.22 (95% CI 0.19 to 0.24), which was also lower than the null-model Brier score of 0.25 (Table 2 ). The decision curve analysis showed greater net benefit at all predicted probabilities compared with default strategies of changing management for all patients or no patients. Above the threshold of 0.7, the predictions of the SORG-MLA resulted in a larger net benefit when predicting 1-year mortality (Fig. 3 C and 3 D).

Overestimation of Mortality

The negative calibration intercept of the SORG-MLA suggests overestimation of mortality for 90-day and 1-year survival (Fig. 3 ). The SORG-MLA tended to overestimate 90-day and 1-year mortality in 9% (38 of 406) of patients (Table 3 ). Overestimation was in this regard defined as every false-positive prediction across our patient population. A subgroup analysis of these patients showed they were more often treated with novel treatment strategies such as PD-(L)1 inhibitors and cyclin-dependent kinase inhibitors (84% [32 of 38] in the overestimated group versus 55% [204 of 368] in the nonoverestimated group; p = 0.001). Moreover, the response of these patients to the novel treatment regimens was also better; 76% (29 of 38) of patients with overestimated mortality had a good response to the immunotherapy, compared with 23% (86 of 368) of the patients with a normal mortality prediction (p < 0.001) (Table 3 ).

Discussion

The growing incidences of cancer and simultaneous advances in cancer treatment have led to a larger number of patients surviving for longer periods and developing bone metastases. The increased proportion of patients with bone metastases has important implications for prognosis and care. Care is primarily focused on pain relief and quality of life. In addition to being painful, bone lesions pose a risk for pathologic fracture. In patients with a longer life expectancy, prophylactic stabilization may be a suitable preventive measure. However, for patients with an estimated survival of less than 3 months, the morbidity, cost, and risk of short-term complications after stabilization must be weighed against that of fracture risk during this shorter time span. Recently, we developed a machine-learning algorithm based on demographics, clinical characteristics, and laboratory values to assess these differences on life expectancy. On geographic validation in the United States, Taiwan, and the Netherlands, the algorithm showed great promise, with AUCs ranging from 0.76 to 0.84 [ 22 , 29 ]. However, in addition to geographic external validation, as cancer treatment regimens evolve over time, these algorithms should also be temporally validated. In this study, we evaluated the SORG-MLA on a more recent (2016 to 2020) dataset. The discriminatory performance of the model (the extent to which it can distinguish between the different outcomes) was reasonable (0.78 for 90-day survival and 0.76 for 1-year survival). However, the calibration intercept and slope showed values that indicate overestimation of mortality at the 90-day and 1-year timepoints, highlighting the need for ongoing temporal validation of the algorithm. The SORG-MLA is freely available at https://sorg-apps.shinyapps.io/extremitymetssurvival/ .

Limitations

First, the retrospective design and its accompanying restrictions pose a practical limitation to this study. Because of missing data in records, missing data needed to be imputed, and so our analysis was partly based on assumptions rather than on measurements. However, the missForest method for multiple imputation is considered an appropriate approach to handle missing data [ 15 , 28 ]. Second, our subgroup analysis was limited by a small sample size and the retrospective nature of our data. Because of this, comparison of the various treatment regimens and patients’ response to this treatment could not be closely investigated. Ideally, the response to treatment should have been categorized because there are several gradients of response to cancer treatment. Third, the use of immunotherapy should also have been categorized, because some treatment regimens may have a more profound effect on prognosis. Considering this limitation, readers should not conclude that this finding is the sole reason for the degraded performance of the SORG-MLA. However, we could conclude from our data that the risk of mortality of patients who were treated with novel immunotherapeutic regimens was overestimated often, indicating that the MLA provides additional information. Fourth, it is not always clear whether patients will receive immunotherapy in the future. This poses a practical limitation to these machine-learning models because one can only use data that are available to the clinician at the point of decision-making. Immunotherapy is sometimes used as a last-resort medication, which means that patients have exhausted all other treatment modalities and therefore have greater morbidity than patients who have not yet received chemotherapy or hormonal therapy. Clinicians should consider that, in the end, the treatment effect is the most important for the patients’ prognoses, and it is therefore important that clinicians adjust the model’s prediction based on their own clinical experience. Lastly, the validation cohort originated from two hospitals in an urban area in the Northeastern United States. Our results are therefore more generalizable to academic hospitals in developed countries. To confirm geographic and temporal generalizability, validation should be performed using data from multiple institutions on multiple continents. Ideally, we would integrate these into our electronic medical records and provide a constant feedback loop to evaluate diagnostic accuracy with every patient evaluated by the algorithm. To date, none of the SORG algorithms have been incorporated into electronic medical records because of many challenges, such as institutional review board and legal regulations, ethical considerations, and data and model governance [ 25 ]. Because continuous retraining is not yet possible, we recommend frequent (at least every 5 years) validation and retraining of these models. Expected deflections in the outcome, changes in the patient population, or innovations in treatment strategies should be the leading driver for deciding this timeframe.

Performance of the SORG-MLA on Temporal Validation

We found that the SORG-MLA showed decreased discriminative ability in this temporal cohort consisting of 406 patients. The most important reason for this is that overall mortality has decreased in this cohort from a more recent period (28% to 23% for 90-day mortality and 59% to 51% for 1-year mortality). Because of this change, the data on which the model was trained do not mirror the target population, and a decrease in performance seems logical. Another possible reason for this is that the algorithm overestimated the risk of 90-day and 1-year mortality in 38 patients. In the past decade, the use of checkpoint inhibitors (CDK inhibitors, PD-[L]1 inhibitors, and CTLA4 inhibitors) has increased in cancer treatment and they have been proven to be effective for improving patients’ survival in a wide range of malignant diseases [ 9 , 12 ]. Surprisingly, the patients in the validation cohort were older than those in the developmental cohort, and generally had more aggressive tumors. The fact that the algorithm still overestimated the risk of death suggests that the survival of patients with metastatic bone disease of the extremities has increased even more than what we observed in the current study. A subgroup analysis of patients who had overestimated predictions of mortality showed that these patients were more often treated with novel immunotherapy and had better responses to this treatment. Unfortunately, the current model does not consider these more modern treatment regimens, which could be why some patients’ risk of mortality was overestimated by the algorithm. Apart from the overestimated patients, the overall mortality of patients in the temporal validation cohort was decreased at both timepoints. This could be because of improvements in diagnostics, medical treatment, and surgical and anesthetic techniques, leading to better prognoses for a wide range of cancer types. Because the primary tumor grouping used to develop the model evaluated in the current study is based on patients treated from 2005 to 2008 [ 13 ], an evaluation or even revision of this scoring system is warranted. Furthermore, patients in the validation cohort were less often treated for a pathologic fracture than those in the developmental cohort; this could indicate that diagnostics are indeed improving and it could also have contributed to lower mortality, because patients with a pathologic fracture often have a worse prognosis than prophylactically treated patients [ 6 , 19 ]. Researchers are encouraged to investigate additional possible causes of these improvements and to investigate whether these novel treatment regimens can have added value to models such as the SORG-MLA. Because of a low sample size, we did not attempt this here.

Additionally, we saw some differences in laboratory values; calcium levels were lower and alkaline phosphatase levels were higher. These differences in bone turnover markers could have led to an overestimated risk of death in these patients. There are several possible reasons for this. In a retrospective study, members of our research group found that higher levels of alkaline phosphatase are associated with lower survival in patients with extremity metastatic disease [ 23 ]. The threshold for this lower odds of survival was calculated to be 100 IU/L. However, because this falls within the normal range of alkaline phosphatase in the blood (44 to 147 IU/L), it does not directly mean that it is clinically relevant. The algorithm might have overweighed the importance of this variable. For one of the patients for whom the risk of mortality was overestimated, the high level of alkaline phosphatase was attributed as having a higher impact on the risk of mortality than the presence of brain metastases and tumor origin. However, when the alkaline phosphatase levels are extremely high (Supplemental Fig. 1; http://links.lww.com/CORR/B114 ), patients might have extensively disseminated metastatic bone disease, which can provide more prognostic value than a small brain lesion that does not cause brain swelling and is therefore of little prognostic value.

Conclusion

The SORG-MLA showed decreased performance in this temporal cohort from the developmental institution. Additionally, the risk of mortality was overestimated in patients in whom immunotherapy was used compared with patients in whom immunotherapy was not used, where the MLA delivered more accurate estimations of mortality. Clinicians are therefore advised to adjust the model’s predictions based on their own experience, considering that patients who are treated at tertiary care facilities in developed countries will have a better prognosis than is depicted by the SORG-MLA. Importantly, patients treated with immunotherapy are more prone to this overestimation. The results of this study show that systematic validation and retraining of these MLAs is of paramount importance because the predictive performance declines over time as treatment regimens evolve. Patients with otherwise poor chances of survival only a few years ago now show better response to treatment. Future researchers should attempt to investigate the usefulness and possible added value of treatment with novel regimens as a variable, preferably in prospective, multicenter approaches, in which these changes can be documented and implemented in the algorithm in real time. The SORG-MLA is available as a freely accessible internet application that can be found at https://sorg-apps.shinyapps.io/extremitymetssurvival/ .

Supplementary Material

This study was supported by “Stichting de Drie Lichten,” the Michael van Vloten Fund, the Hendrik-Muller fund, and the Nijbakker Morra foundation from the Netherlands. Each author certifies that there are no funding or commercial associations (consultancies, stock ownership, equity interest, patent/licensing arrangements, etc.) that might pose a conflict of interest in connection with the submitted article related to the author or any immediate family members.

All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.

Ethical approval for this study was obtained from the institutional review board at Massachusetts General Hospital, Boston, MA, USA (2018P000688).

This work was performed at Massachusetts General Hospital, Boston, MA, USA.

References

1. Alsoof D McDonald CL Kuris EO Daniels AH . Machine learning for the orthopaedic surgeon: uses and limitations . J Bone Joint Surg Am . 2022 ; 104 : 1586 - 1596 . 35383655 10.2106/JBJS.21.01305

2. Cuschieri S . The STROBE guidelines . Saudi J Anaesth . 2019 ; 13 : S31 - S34 . 30930717 10.4103/sja.SJA_543_18 PMC6398292

3. Fan J Upadhye S Worster A . Understanding receiver operating characteristic (ROC) curves . CJEM . 2006 ; 8 : 19 - 20 . 17175625 10.1017/s1481803500013336

4. Forsberg JA Eberhardt J Boland PJ Wedin R Healey JH . Estimating survival in patients with operable skeletal metastases: an application of a Bayesian belief network . PLoS One . 2011 ; 6 : e19956 . 21603644 10.1371/journal.pone.0019956 PMC3094405

5. Groot OQ Bongers MER Ogink PT Does artificial intelligence outperform natural intelligence in interpreting musculoskeletal radiological studies? A systematic review . Clin Orthop Relat Res . 2020 ; 478 : 2751 - 2764 . 32740477 10.1097/CORR.0000000000001360 PMC7899420

6. Groot OQ Lans A Twining PK Clinical outcome differences in the treatment of impending versus completed pathological long-bone fractures . J Bone Joint Surg Am . 2022 ; 104 : 307 - 314 . 34851323 10.2106/JBJS.21.00711

7. Herbst RS Morgensztern D Boshoff C . The biology and management of non-small cell lung cancer . Nature . 2018 ; 553 : 446 - 454 . 29364287 10.1038/nature25183

8. Hernandez RK Adhia A Wade SW Prevalence of bone metastases and bone-targeting agent use among solid tumor patients in the United States . Clin Epidemiol . 2015 ; 7 : 335 - 345 . 26229504 10.2147/CLEP.S85496 PMC4514316

9. Hussaini S Chehade R Boldt RG Association between immune-related side effects and efficacy and benefit of immune checkpoint inhibitors – a systematic review and meta-analysis . Cancer Treat Rev . 2021 ; 92 : 102134 . 33302134 10.1016/j.ctrv.2020.102134

10. Janssen SJ Kortlever JTP Ready JE Complications after surgical management of proximal femoral aetastasis: a retrospective study of 417 patients . J Am Acad Orthop Surg . 2016 ; 24 : 483 - 494 . 27227983 10.5435/JAAOS-D-16-00043

11. Janssen SJ van der Heijden AS van Dijke M Marshall Urist Young Investigator Award: prognostication in patients with long bone metastases: does a boosting algorithm improve survival estimates? Clin Orthop Relat Res . 2015 ; 473 : 3112 - 3121 . 26155769 10.1007/s11999-015-4446-z PMC4562931

12. Karlsson AK Saleh SN . Checkpoint inhibitors for malignant melanoma: a systematic review and meta-analysis . Clin Cosmet Investig Dermatol . 2017 ; 10 : 325 - 339 . 10.2147/CCID.S120877 PMC5580705 28883738

13. Katagiri H Okada R Takagi T New prognostic factors and scoring system for patients with skeletal metastasis . Cancer Med . 2014 ; 3 : 1359 - 1367 . 25044999 10.1002/cam4.292 PMC4302686

14. Li S Peng Y Weinhandl ED Estimated number of prevalent cases of metastatic bone disease in the US adult population . Clin Epidemiol . 2012 ; 4 : 87 - 93 . 22570568 10.2147/CLEP.S28339 PMC3345874

15. Moons KGM Altman DG Reitsma JB Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration . Ann Intern Med . 2015 ; 162 : W1 - W73 . 25560730 10.7326/M14-0698

16. Nathan SS Healey JH Mellano D Survival in patients operated on for pathologic fracture: implications for end-of-life orthopedic care . J Clin Oncol . 2005 ; 23 : 6072 - 6082 . 16135474 10.1200/JCO.2005.08.104

17. Ogink PT Groot OQ Karhade AV Wide range of applications for machine-learning prediction models in orthopedic surgical outcome: a systematic review . Acta Orthop . 2021 ; 92 : 526 - 531 . 34109892 10.1080/17453674.2021.1932928 PMC8519550

18. Oosterhoff JHF Gravesteijn BY Karhade AV Feasibility of machine learning and logistic regression algorithms to predict outcome in orthopaedic trauma surgery . J Bone Joint Surg Am . 2022 ; 104 : 544 - 551 . 34921550 10.2106/JBJS.21.00341

19. Philipp TC Mikula JD Doung Y-C Gundle KR . Is there an association between prophylactic femur stabilization and survival in patients with metastatic bone disease? Clin Orthop Relat Res . 2020 ; 478 : 540 - 546 . 32168065 10.1097/CORR.0000000000000803 PMC7145091

20. Schulman KL Kohles J . Economic burden of metastatic bone disease in the U.S. Cancer . 2007 ; 109 : 2334 - 2342 . 17450591 10.1002/cncr.22678

21. Skalitzky MK Gulbrandsen TR Groot OQ The preoperative machine learning algorithm for extremity metastatic disease can predict 90‐day and 1‐year survival: an external validation study . J Surg Oncol . 2022 ; 125 : 282 - 289 . 34608991 10.1002/jso.26708

22. Stekhoven DJ Bühlmann P . MissForest–non-parametric missing value imputation for mixed-type data . Bioinformatics . 2012 ; 28 : 112 - 118 . 22039212 10.1093/bioinformatics/btr597

23. Thio QCBS Karhade AV Notman E Serum alkaline phosphatase is a prognostic marker in bone metastatic disease of the extremity . J Orthop . 2020 ; 22 : 346 - 351 . 32921951 10.1016/j.jor.2020.08.008 PMC7475062

24. Thio QCBS Karhade AV Ogink PT Development and internal validation of machine learning algorithms for preoperative survival prediction of extremity metastatic disease . Clin Orthop Relat Res . 2020 ; 478 : 322 - 333 . 31651589 10.1097/CORR.0000000000000997 PMC7438151

25. van de Sande D Van Genderen ME Smit JM Developing, implementing and governing artificial intelligence in medicine: a step-by-step approach to prevent an artificial intelligence winter . BMJ Health Care Inform . 2022 ; 29 : e100495 . 10.1136/bmjhci-2021-100495 PMC8860016 35185012

26. Wang S Summers RM . Machine learning and radiology . Med Image Anal . 2012 ; 16 : 933 - 951 . 22465077 10.1016/j.media.2012.02.005 PMC3372692

27. Willeumier JJ van der Linden YM van der Wal CWPG An easy-to-use prognostic model for survival estimation for patients with symptomatic long bone metastases . J Bone Joint Surg Am . 2018 ; 100 : 196 - 204 . 29406340 10.2106/JBJS.16.01514

28. Wolff RF Moons KGM Riley RD PROBAST: a tool to assess the risk of bias and applicability of prediction model studies . Ann Intern Med . 2019 ; 170 : 51 - 58 . 30596875 10.7326/M18-1376

29. Yang J-J Chen C-W Fourman MS International external validation of the SORG machine learning algorithms for predicting 90-day and one-year survival of patients with spine metastases using a Taiwanese cohort . Spine J. 2021 ; 21 : 1670 - 1678 . 33545371 10.1016/j.spinee.2021.01.027

30. Yu H-HM Tsai Y-Y Hoffe SE . Overview of diagnosis and management of metastatic disease to bone . Cancer Control . 2012 ; 19 : 84 - 91 . 22487970 10.1177/107327481201900202