Artificial Neural Network Prediction of Mortality in Cancer Patients Presenting for Radiation Therapy at a Multisite Institution.

Introduction

Accurate prediction of mortality in cancer patients is important in deciding the optimal management strategy and individualized treatment plan while taking into consideration quality of life. At present, most management decisions are based on patient age/functional status, tumor characteristics, and standardized treatment guidelines formulated by the National Comprehensive Cancer Network (NCCN) and ASTRO, which are based on scientific evidence such as phase I-III trials. Present prognostic systems use TNM staging, which incorporates a limited number of histopathologic variables (i.e., tumor size, degree of regional lymph node involvement, and presence of distant metastasis), to predict patient survival over a specified period of time. However, these systems stratify the population of cancer patients into extremely broad categories and thus cannot provide a more accurate projected outcome that incorporates all relevant prognostic data in a patient’s record [ 1 ]. In order to improve upon our ability to more accurately predict patient outcomes, additional patient variables must be taken into consideration. Consequently, there is potential for continued growth with the incorporation of artificial intelligence (AI) and machine learning (ML) for prognostication.

AI/ML algorithms can explore large datasets with many inputs, identify complex patterns and relationships between multiple variables, and generate models to predict outcomes. ML has progressed significantly since Arthur Samuel first coined the term in the 1950s. ML is divided into supervised learning, unsupervised learning, and reinforcement learning [ 2 ]. Supervised learning is a subcategory of ML and AI, defined by its use of labeled datasets with inputs and correct outputs to train algorithms to classify data or predict outcomes accurately. In comparison, unsupervised learning algorithms infer patterns from a dataset without reference to known or labeled outcomes.

Several commonly used supervised machine learning models will be described briefly. Logistic regression is a technique that uses a logistic model with multivariate input and outputs a binary value (i.e., 1 or 0). It performs computations comparable to those of a single-node neural network. Random forest is a machine learning technique that uses an ensemble of individual decision trees, with each “tree” outputting a binary output, with the most common output being the consensus decision. A support vector machine (SVM), also known as a support vector classifier (SVC), is a non-probabilistic binary linear model commonly used in classification problems that sort data into two groups by maximally separating the data points of different classes in distinct regions of space. It can handle linearly as well as non-linearly separable data. Xtreme gradient boost (XGBoost) is another method using classification trees and is popular due to its scalability and high accuracy [ 3 - 5 ].

Artificial neural networks (ANNs) are a class of statistical learning models that have demonstrated success in the domains of image and speech recognition and have recently become widespread in processing biomedical data [ 6 ]. An ANN is composed of computational nodes ("neurons"), which each take a weighted summation over a series of inputs and pass that sum through a nonlinear transfer function (e.g., the sigmoid function) to produce an output. Multiple neurons are then assembled in layers, whose sequential outputs are passed to each other until a terminal output node or nodes are reached. The advantage of an ANN is its ability to produce a complex, nonlinear representation of a large number of input variables for use in classification or regression problems [ 7 ].

One study showed that ANNs incorporating the type of radiation treatment as input were able to predict mortality in patients who received radiotherapy for breast cancer [ 8 ]. Radiation therapy (RT) is a mainstay of cancer treatment and is used for both palliative and curative measures. Radiation treatment decisions, including radiation dose, fractionation, and treatment intent, are based on established standards of care regimens, the tolerability of treatment, potential acute and chronic side effects, and the impact on quality of life. An interesting question is whether the fine-tuned parameters of radiation therapy delivery and patient demographics can be used to augment a predictive model of patient mortality. This study aims to evaluate the efficacy of ANNs in predicting six-month, one-year, three-year, and five-year mortality in a dataset of patients undergoing radiation therapy for various malignancy types. Throughout this study, we employ several machine learning techniques to create baseline models against which ANN performance is compared.

Materials and methods

Institutional review board (IRB) approval was obtained to collect individual patient data from institutional electronic medical records. We collaborated with ONCORA, an integrated system that extracts data from electronic medical records across our institutional network. Patient cases were selected from the data source using the following criteria: primary cancers of the prostate, breast, gynecological, thoracic, gastrointestinal, spine, and brain. The dataset included both cases of palliative and curative intent (stages I-IV). Appendix A enumerates the data files collected.

Clinically, KPS (Karnofsky Performance Status) is a critical variable for assessing prognosis. During the initial dataset analysis, we discovered 2407 missing values for the KPS score. Therefore, we categorized patients into two separate datasets: the KPS dataset (patients with KPS) or the NKPS dataset (all patients with the KPS variable removed), which captures all cases of the original dataset. We organized the data in such a fashion because predictive models expect non-empty values for each variable.

The code written for this study was developed and tested on three separate computers: (i) an i7-3520M 2.90 GHz CPU and no dedicated graphics card (GPU); (ii) an i7-4790 3.60 GHz CPU and Nvidia GeForce GTX 745 GPU; and (iii) an i7-8750H CPU and Nvidia GeForce GTX 1070. The environments used to test the runnability of this study were Ubuntu 18.04 (64-bit), Windows 10 Home (64-bit, v.1809), Windows 10 Pro (64-bit, v.1809), Windows 10 Pro (64-bit, v.1903), Windows Subsystem for Linux running Ubuntu 18.04 (64-bit), and macOS Mojave 10.14.6. Python 3 was used to write the code for this study. The version of Keras used was 2.2.4, with underlying TensorFlow 1.14.0 (CPU and GPU packages). The version of Pandas used was 0.24.2. The version of NumPy used was 1.16.4 ( https://numpy.org /). The version of Scikit-Learn used was 0.21.2 [ 9 - 13 ].

Table 1 includes a summary of the ML algorithms used in this study, including the advantages and disadvantages of each.

Parameter selection

The starting parameters for the model included sex, race, cancer site, treatment site, stage, cancer histology, recurrence, attending physician, diagnosis type, disease system, treatment intent, treatment system, age at first RT fraction, weight, religion, marital status, KPS score, Eastern Oncology Cooperative Group (ECOG) grade, TNM stage, treatment energy, treatment modality, total dose, total fractions, dose per fraction, elapsed treatment days, toxicities by grade, and complications including dysphagia, nausea, depression, and dermatitis. A complete enumeration of raw data variables and a dictionary of possible values for the categorical variables are provided in Appendices B and C.

The following variables were excluded: case_id, attending_physician, date_first_fraction, cb_death, date_of_death, main_diagnosis_text, primary_diagnosis_text, secondary_diagnosis_text. Case ID would have no predictive value due to being an artifact of the electronic health record. We chose not to include the physician, even though this may have an impact on outcomes due to the large number of providers. Any variables that would proactively impart death information, including time of death and complications of treatment by death, were dropped due to the target variable of interest being mortality. Freetext variables were not included since they would not be able to be encoded to serve as useful model inputs. In addition, we dropped kps_score in the NKPS dataset only for the reasons described above.

Data preprocessing

The raw data underwent a cleaning and preprocessing step in Python. The possible target variables in the dataset were mortality within six months, one year, three years, and five years. In the second step, preprocessing of the data was performed. This step consisted of one-hot encoding, binarization of the target, and minimum-maximum scaling. We used a Pandas built-in function to perform one-hot encoding of all categorical variables. Using Scikit-Learn (a machine learning library for Python programming), a minimum-maximum scaler was fitted to the continuous variables of the data and was used to scale all continuous variables to a range of 0 to 1.

In our neural networks, we did not manually adjust the parameters of the networks. Instead, almost all hyperparameters, as well as the number of neurons in the hidden layers, were grid-searched using Scikit-Learn to determine the optimal network design. We configured the network to use either one or two hidden layers with variable numbers of neurons in each, depending on the grid search parameters. The full training and evaluation processes described here were repeated for each of the four mortality variables for both datasets (KPS and NKPS), totaling eight self-contained experiments. Internal to the grid search algorithm, training vectors of cancer cases were passed to a Keras neural network. Keras used its TensorFlow engine backend to compute the output at the final neuron and used the Adam optimizer to update the weights of the network to minimize the loss function, binary cross-entropy.

GridSearchCV’s algorithm created a neural network for each set of 54 parameters, performed training using Keras, and finally ran three-fold cross-validation across several metrics: binary accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Each metric was outputted with an average and standard deviation for that network combination. In order to obtain a baseline for comparison, other classification algorithms were applied to the data: logistic regression, random forest, and SVM, as implemented in Python’s Scikit-Learn library; and extreme gradient boosted trees, as implemented in Python’s XGBoost module.

The models were constructed with default library parameters, except the random forest classifier, which was created with 1000 classification trees. The datasets used for this section were the NKPS and KPS datasets, as described previously. Training and evaluation were performed using an 80:20 split of training to testing data ratio.

Results

Dataset characteristics

The initial dataset consisted of 6625 patients; however, 30 cases were excluded due to blank values in patient variables. The dataset meeting inclusion criteria totaled 6595 patients. Survival data were available for all patients. Table 2 describes the patient characteristics, and Table 3 describes mortality rates in the dataset.

There were 4214 (63.9%) females and 2381 (36.1%) males, with a median age of 75 years (IQR = 8.87 years) and an age range of 2.5-102 years. The cumulative mortality rates within six months, one year, three years, and five years were 18.2%, 27.1%, 39.8%, and 43.9%, respectively. The racial distribution of the dataset was 1934 (29.3%) Hispanic, 1151 (17.5%) African American, 468 (7.1%) White, 592 (9.0%) Asian, and the remainder were not known or “other.” Among the primary sites of malignancy, thoracic cancers were most common, with 1614 (24.5%) cases, followed by genitourinary and gastrointestinal malignancies.

Neural network performance

The best performing ANN for the NKPS dataset demonstrated a sensitivity of 62.71% ± 5.28%, 72.53% ± 5.67%, 76.78% ± 2.17%, and 79.84% ± 5.22% at six months, one year, three years, and five years, respectively. With respect to specificity, the ANN achieved 90.39% ± 3.46%, 80.89% ± 1.59%, 78.09% ± 7.02%, and 70.95% ± 9.73% at six months, one year, three years, and five years, respectively. The best-performing ANN for the KPS dataset demonstrated a sensitivity of 61.76% ± 8.79%, 69.11% ± 6.07%, 78.45% ±7.33%, and 83.00% ± 4.89% at six months, one year, three years, and five years, respectively. With respect to specificity, the ANN achieved 92.35% ± 3.92%, 85.00% ± 6.17%, 61.89% ± 2.13%, and 61.39% ±6.66% at six months, one year, three years, and five years, respectively. The trend over time can be visualized in Figures 1 - 2 .

The sensitivity at all mortality time points for both datasets using ANNs was compared to that of random forest, XGBoost, SVM, and logistic regression algorithms (Figures 3 - 4 ).

Tables 4 - 9 show the full results for neural network models at all mortality time points for both datasets, with respect to sensitivity, specificity, positive predictive value, negative predictive value, and binary accuracy. Tables 10 - 17 show the performance of comparison models.

Discussion

As the advancements in AI and ML continue to progress, we are in the midst of technology that can potentially revolutionize the field of radiation oncology through enhancing decision-making processes. We summarize a few studies that have utilized AI for outcome prediction in malignancies treated with radiation therapy in Table 18 [ 14 - 29 ].

Predictive models can be created by applying advanced algorithms and analyzing large datasets to assist in personalized treatment planning and decision-making. With hyperparameter grid-searching, network configurations optimizing the algorithm for sensitivity were found for six-month, one-year, three-year, and five-year mortality. The ANN was able to achieve a higher sensitivity than other machine learning algorithms across all output target variables, demonstrating the utility of an ANN model for mortality prediction on the provided dataset.

The implemented ANNs were successfully able to perform binary prediction of mortality with an accuracy greater than random chance. The neural networks consistently demonstrated higher average sensitivities and lower standard deviations than all other models across both datasets and all mortality variables. This is an indication that the neural networks were able to extrapolate patterns and heuristics from the variables present in the clinical dataset. Interestingly, the neural networks trained on the NKPS dataset appeared to perform with more precision in terms of sensitivity and specificity compared to the results from the KPS dataset. We believe that this result may be attributable to the removal of 36.33% of the total cancer cases in the KPS dataset, leading to less reliable learning.

The neural network was able to achieve higher sensitivity for mortality prediction over a longer time frame compared to a shorter time frame. A plausible reason this occurred was the increased number of positive cases per mortality variable over a five-year time span. Aside from the imbalance of positive cases, the prediction of mortality in a shorter time frame is more difficult because there is a greater influence of random chance. Prediction over a longer time frame is a more tractable problem for the algorithm since the influence of random chance is reduced while the impact of the prognostic variables used as inputs takes precedence. In contrast to sensitivity, specificity was seen to decrease over time due to the neural network being more prone to making positive predictions with a lower detection threshold. As the “excitability” of the network increases, more type 1 errors occur, thus lowering the specificity. This is typical of the trade-off between sensitivity and specificity.

Binary accuracy was seen to decrease with each mortality time point, which can be explained in the context of data dynamics. As time increases, the mortality proportion of the dataset naturally increases, which leads to increased sensitivity and decreased specificity, as noted before. Additionally, there is increased uncertainty due to a greater number of variables or events that could affect mortality over time, such as comorbid conditions. Therefore, if the model does not perfectly capture the relevant temporal dynamics of the dataset, accuracy will decrease with time. We interpret binary accuracy as a less useful evaluator of neural network performance because of its dependence on target variable prevalence. For example, if there were zero cases of mortality at six months, the network could achieve a binary accuracy of 100% with 100% specificity by outputting only negative predictions with 0% sensitivity. Similar to binary accuracy, PPV and NPV are less useful evaluators of performance because of their similar dependence on target variable prevalence. Therefore, PPV and NPV, in addition to binary accuracy, are subject to the distribution of the data we are using to train the model.

The ratio of positive and negative samples differs greatly amongst the mortality variables at different follow-up dates, so it would be preferable to use a metric that can be consistently applied to all target variables. Sensitivity is intrinsic to the classifier system and cannot be affected by prevalence. However, it is significant to note that, in practical use, PPV is the most important metric in assessing the probability of mortality in a patient given a positive output from any model.

The clearest limitation of this study was the relatively small (N = 6595) dataset of cancer cases. Although this is considered a large medical cohort, especially when compared to other published studies using ML, it is small for machine learning. With more training samples, the networks could possibly develop a more nuanced understanding of the variables to achieve higher sensitivity and specificity. Another limitation specific to this dataset was the class imbalance inherent to mortality outcomes. There were significantly more survivors at six months compared to deceased individuals, which is not ideal for training a classification algorithm. A super-sampling technique such as the synthetic minority oversampling technique (SMOTE) may be employed in the future to achieve a balanced dataset to feed into the machine learning algorithm at training time and potentially improve performance [ 30 ]. An internally obvious limitation was the lack of powerful hardware to run this study. Early attempts were made to run the grid search across hundreds of parameters (in excess of 700), but after several days of training, it became apparent that such program run-times were unacceptable. If greater computing power were available, a wider variety of network architectures could be explored; for instance, deeper networks with three or more layers.

Conclusions

The use of ANNs resulted in a successful mortality prediction model and was shown to outperform other machine learning algorithms with greater sensitivity and specificity. The unique features of our study, when compared to similar studies, include a relatively large and longitudinal sample size as we are a multisite institution composed of both academic and community medical centers, a broad set of pathological diagnoses, and high regional diversity. Moving forward, we believe there is room for improvement in our current model through the refinement of prognostic variables that were used as network inputs, such as the inclusion of radiomics (imaging data in the form of CT or MRI images). A convolutional neural network architecture that incorporates both prognostic and imaging data for each patient could theoretically increase performance by a vast margin. Future projects involving imaging data will require significantly more computing power than what was available in this study.

Acknowledgments

Elan Shahrabani and Michael Shen contributed equally to work.

Appendices

Disclosures

Human subjects: Consent was obtained or waived by all participants in this study.

Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following:

Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work.

Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work.

Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.

References

1. Neural networks in the prediction of survival in patients with colorectal cancer. Grumett S, Snow P, Kerr D. Clin Colorectal Cancer. 2003;2:239–244. doi: 10.3816/CCC.2003.n.005.

2. Artificial intelligence and machine learning in precision and genomic medicine. Quazi S. Med Oncol. 2022;39:120. doi: 10.1007/s12032-022-01711-1.

3. Introduction to deep learning: minimum essence required to launch a research. Wataya T, Nakanishi K, Suzuki Y, Kido S, Tomiyama N. Jpn J Radiol. 2020;38:907–921. doi: 10.1007/s11604-020-00998-2.

4. Machine learning applications in cancer prognosis and prediction. Kourou K, Exarchos TP, Exarchos KP, Karamouzis MV, Fotiadis DI. Comput Struct Biotechnol J. 2015;13:8–17. doi: 10.1016/j.csbj.2014.11.005.

5. Applications of machine learning in cancer prediction and prognosis. Cruz JA, Wishart DS. http://pubmed.ncbi.nlm.nih.gov/19458758/ Cancer Inform. 2007;2:59–77.

6. Artificial intelligence (AI) and big data in cancer and precision oncology. Dlamini Z, Francies FZ, Hull R, Marima R. Comput Struct Biotechnol J. 2020;18:2300–2311. doi: 10.1016/j.csbj.2020.08.019.

7. Artificial neural networks improve the accuracy of cancer survival prediction. Burke HB, Goodman PH, Rosen DB, et al. Cancer. 1997;79:857–862. doi: 10.1002/(sici)1097-0142(19970215)79:4<857::aid-cncr24>3.0.co;2-y.

8. Predicting post-treatment survivability of patients with breast cancer using Artificial Neural Network methods. Wang TN, Cheng CH, Chiu HW. Annu Int Conf IEEE Eng Med Biol Soc. 2013;2013:1290–1293. doi: 10.1109/EMBC.2013.6609744.

9. Python. [ Jul; 2023 ]. 2023. https://www.python.org/ https://www.python.org/

10. Keras: Deep learning for humans. [ Jul; 2023 ]. 2023. https://keras.io/ https://keras.io/

11. Pandas. [ Jul; 2023 ]. 2023. https://pandas.pydata.org/ https://pandas.pydata.org/

12. NumPy 2.0 released! [ Jul; 2023 ]. 2023. https://numpy.org https://numpy.org

13. Scikit-learn: machine learning in python. [ Jul; 2023 ]. 2023. https://scikit-learn.org/ https://scikit-learn.org/

14. Magnetic resonance imaging features of tumor and lymph node to predict clinical outcome in node-positive cervical cancer: a retrospective analysis. Park SH, Hahm MH, Bae BK, et al. Radiat Oncol. 2020;15:86. doi: 10.1186/s13014-020-01502-w.

15. Predicting pathological complete response in rectal cancer after chemoradiotherapy with a random forest using (18)F-fluorodeoxyglucose positron emission tomography and computed tomography radiomics. Shen WC, Chen SW, Wu KC, et al. Ann Transl Med. 2020;8:207. doi: 10.21037/atm.2020.01.107.

16. Can a machine-learning model improve the prediction of nodal stage after a positive sentinel lymph node biopsy in breast cancer? Madekivi V, Boström P, Karlsson A, Aaltonen R, Salminen E. Acta Oncol. 2020;59:689–695. doi: 10.1080/0284186X.2020.1736332.

17. An artificial neural network-based model for predicting annual dose in healthcare workers occupationally exposed to different levels of ionizing radiation. Mortazavi SM, Aminiazad F, Parsaei H, Mosleh-Shirazi MA. Radiat Prot Dosimetry. 2020;189:98–105. doi: 10.1093/rpd/ncaa018.

18. Multi-objective ensemble deep learning using electronic health records to predict outcomes after lung cancer radiotherapy. Wang R, Weng Y, Zhou Z, Chen L, Hao H, Wang J. Phys Med Biol. 2019;64:245005. doi: 10.1088/1361-6560/ab555e.

19. Imaging-based individualized response prediction of carbon ion radiotherapy for prostate cancer patients. Wu S, Jiao Y, Zhang Y, et al. Cancer Manag Res. 2019;11:9121–9131. doi: 10.2147/CMAR.S214020.

20. Optimization of treatment strategy by using a machine learning model to predict survival time of patients with malignant glioma after radiotherapy. Mizutani T, Magome T, Igaki H, Haga A, Nawa K, Sekiya N, Nakagawa K. J Radiat Res. 2019;60:818–824. doi: 10.1093/jrr/rrz066.

21. Toward predicting the evolution of lung tumors during radiotherapy observed on a longitudinal MR imaging study via a deep learning algorithm. Wang C, Rimner A, Hu YC, et al. Med Phys. 2019;46:4699–4707. doi: 10.1002/mp.13765.

22. MRI-based radiomics nomogram may predict the response to induction chemotherapy and survival in locally advanced nasopharyngeal carcinoma. Zhao L, Gong J, Xi Y, et al. Eur Radiol. 2020;30:537–546. doi: 10.1007/s00330-019-06211-x.

23. Neo-adjuvant chemoradiotherapy response prediction using MRI based ensemble learning method in rectal cancer patients. Shayesteh SP, Alikhassi A, Fard Esfahani A, Miraie M, Geramifar P, Bitarafan-Rajabi A, Haddad P. Phys Med. 2019;62:111–119. doi: 10.1016/j.ejmp.2019.03.013.

24. Prediction of local relapse and distant metastasis in patients with definitive chemoradiotherapy-treated cervical cancer by deep learning from [(18)F]-fluorodeoxyglucose positron emission tomography/computed tomography. Shen WC, Chen SW, Wu KC, et al. Eur Radiol. 2019;29:6741–6749. doi: 10.1007/s00330-019-06265-x.

25. A machine-learning-based prediction model of fistula formation after interstitial brachytherapy for locally advanced gynecological malignancies. Tian Z, Yen A, Zhou Z, Shen C, Albuquerque K, Hrycushko B. Brachytherapy. 2019;18:530–538. doi: 10.1016/j.brachy.2019.04.004.

26. Machine learning for prediction of chemoradiation therapy response in rectal cancer using pre-treatment and mid-radiation multi-parametric MRI. Shi L, Zhang Y, Nie K, et al. Magn Reson Imaging. 2019;61:33–40. doi: 10.1016/j.mri.2019.05.003.

27. The value of nodal information in predicting lung cancer relapse using 4DPET/4DCT. Li H, Becker N, Raman S, Chan TC, Bissonnette JP. Med Phys. 2015;42:4727–4733. doi: 10.1118/1.4926755.

28. Support vector machine classification of brain metastasis and radiation necrosis based on texture analysis in MRI. Larroza A, Moratal D, Paredes-Sánchez A, Soria-Olivas E, Chust ML, Arribas LA, Arana E. J Magn Reson Imaging. 2015;42:1362–1368. doi: 10.1002/jmri.24913.

29. Support vector machine-based prediction of local tumor control after stereotactic body radiation therapy for early-stage non-small cell lung cancer. Klement RJ, Allgäuer M, Appold S, et al. http://10.1016/j.ijrobp.2013.11.216 . Int J Radiat Oncol Biol Phys. 2014;88:732–738. doi: 10.1016/j.ijrobp.2013.11.216.

30. LVQ-SMOTE - Learning vector quantization based Synthetic Minority Over-sampling Technique for biomedical data. Nakamura M, Kajiwara Y, Otsuka A, Kimura H. BioData Min. 2013;6:16. doi: 10.1186/1756-0381-6-16.