A Scalable Radiomics- and Natural Language Processing-Based Machine Learning Pipeline to Distinguish Between Painful and Painless Thoracic Spinal Bone Metastases: Retrospective Algorithm Development and Validation Study.

Introduction

Overview

Most patients with cancer with bone metastasis (BM) experience pain [ 1 ] and most receive radiotherapy to control it [ 2 ]. But, it has been shown that due to the subjective and qualitative nature of the pain, clinicians often underestimate pain [ 3 ]. As a result, many patients with BM receive radiotherapy after their pain has already become debilitating [ 4 ].

Although patient-reported outcomes can be used to obtain pain scores directly from patients themselves, the efficacy of these pain scores is limited due to the fact that these ratings are highly qualitative and subjective [ 5 ]. Because of this, it is desirable to have pain scoring systems that are more objective. The goal of this study was to explore ways to automatically and objectively quantify pain associated with BMs using computed tomography (CT) images.

We hypothesized that tumor features extracted from CT images of BMs contain imaging biomarkers that may be used to objectively identify BM-associated pain. These pain biomarkers may provide the opportunity to develop objective pain scoring tools to aid in the diagnosis, treatment, understanding, and prognosis of BM pain.

Background

The search for imaging and nonimaging pain biomarkers has been the focus of numerous studies [ 5 - 12 ]. Various studies [ 13 - 21 ] have shown how artificial intelligence (AI), including machine learning and radiomics, can be used to understand and quantify pain. For example, Mashayekhi et al [ 22 ] showed that radiomic features extracted from the CT images of the pancreas can help to identify functional abdominal pain in patients. Vedantam et al [ 23 ] explored the viability of using radiomics features extracted from magnetic resonance images to detect pain following percutaneous cordotomy. At least 1 study [ 13 ] has reported using radiomics to identify painful metastatic lesions in radiographic images. However, we found no reports in the literature of a scalable approach that can be used efficiently on a large set of unlabeled patient data. To the best of our knowledge, our work is the first to combine natural language processing (NLP) and radiomics to enable an efficient and scalable pain identification pipeline using unstructured data.

A fundamental challenge in developing any AI model for use in medicine is the need to obtain sufficient patient data for training and testing. For example, the data set used by Wakabayashi et al in the study that we mentioned earlier [ 13 ], was limited to 69 patients. One limiting factor is obtaining standard patient-reported pain scores for use as ground-truth data, and another limiting factor is obtaining segmented images from which to extract tumor biomarkers. For the work reported in this paper, we overcame the data set size limitation by using 2 novel strategies. First, by combining NLP with radiomics, we quickly mined pain scores from clinical notes and used these NLP-extracted scores to label our radiomics features for supervised learning. Second, by asking our clinical colleagues to pinpoint only the center points of BM lesions in radiotherapy simulation CT images, we maximized the number of lesions identified in the time available. In the medical field, NLP has shown promising results in extracting biomedical information and clinical outcomes such as pain from unstructured text data [ 24 - 26 ]. Moreover, as we reported previously [ 21 ], by automatically delineating geometrical regions around BM lesion center points, it is possible to successfully extract radiomics features for robust BM lesion detection. In this study, we report how our combined radiomics-NLP machine learning pipeline can successfully identify pain in radiotherapy simulation CT images of patients with cancer with BMs.

Methods

Ethical Considerations

This retrospective study was approved by the research ethics board of the McGill University Health Centre (2020-5899) with the waiver of informed consent. We confirm that the entire research was performed in accordance with research ethics board’s guidelines and regulations.

Data Selection

Our patient-selection process is outlined in Figure 1 . The initial number of 200 pairs of radiation oncology consultation notes and CT images of patients with spinal BM were included in this study based on the minimum sample size calculation as explained in Section A.1 in Multimedia Appendix 1 [ 27 ]. In total, 120 of the notes and all 200 of the CT images from this study were independently used in 2 studies we previously reported on [ 21 - 25 ]. The first [ 25 ] of these studies showed the feasibility of extracting pain from consultation notes of patients with cancer, using NLP. The second [ 21 ] demonstrated the feasibility of using lesion center point–based radiomics models to differentiate healthy and metastatic bone lesions in CT scans of patients with BMs. This study combined the data and results from these 2 prior studies and expanded upon them to build an NLP- and radiomics-based model to detect pain using the CT scans of patients.

We searched our institution’s Oncology Information System for the radiotherapy plans of patients diagnosed with a “secondary malignant neoplasm of bone” between January 2016 and September 2019. From the retrieved list, we selected those who were treated for thoracic spinal BM. Then, we retrieved the corresponding consultation notes and simulation CT images. A note-image pair was included if (1) the note was in English, (2) pain was documented, (3) the simulation CT image was taken up to 10 days post consultation, and (4) simulation CT revealed BM lesions in the thoracic spine. Patients with multiple but nonoverlapping note-image pairs were considered independent samples. We only considered the same patients as new participants if they had CT scans and associated consultation notes for BM lesions in different areas of their spines. As a result, each BM lesion was included only once in our study. Also, it should be noted that palliative patients normally have their simulation CT scan (for treatment planning) on the same day or within a few days after the consultation, and radiotherapy is delivered on the same day or within a few days after treatment planning. To assure that there is no change in the BM lesion structure or pain status, we did not allow more than a 10-day gap between the two. Figure A1 in Multimedia Appendix 1 displays the distribution of the time interval between the radiotherapy consultation and CT acquisition dates.

We randomly assigned note-image pairs to the training or cross-validation set (approximately 70%) or the holdout test set (approximately 30%). We used stratified randomization to preserve the original sample ratio between pain labels in each sample set. In addition, we performed a paired t test and a chi-square analysis [ 28 ] to ensure that there was no systematic bias in any of our sample sets regarding gender, age, or primary cancer type. Patient demographics are presented in Table 1 .

NLP-Extracted Pain Labels

Due to the absence of patient-reported pain scores in our Oncology Information System, we extracted physician-reported pain scores from patients' radiation oncology consultation notes using our previously reported NLP pipeline [ 25 ]. While pain scores were typically reported as part of the “history of the present illness” in our hospital, for the sake of generalizability, we extracted pain scores from the entire note.

Our NLP pipeline first processed the text with MetaMap [ 29 ] and mapped it to the UMLS (ie, Unified Medical Language System) Metathesaurus [ 30 ] in order to identify pain terminologies and their severity scores. Next, it applied rules to filter out hypothetical, conditional, and historical references to pain in order to focus solely on references to pain at the time of the consultation. Then, it calculated the average pain intensity (API) in each note by averaging the pain scores therein. Finally, it assigned each note a “verbally declared pain” (VDP) label, as VDP=“no pain” (if API 0), and VDP=“pain” (if API0). These pain labels were used to train, validate, and test our radiomics model.

Expert-Extracted Pain Scores

To evaluate the effect of NLP-extracted pain labels on the performance of our pipeline, we also generated best-available ground-truth pain labels using expert-annotated pain scores. To do so, our radiation oncologists used the texTRACTOR [ 31 ] pain labeling application to manually read consultation notes and label valid pain scores in our training and test data sets using a 4-grade verbal rating scale (no pain, mild, moderate, and severe). A mention of pain was regarded as valid if it reflected the status of pain at the metastatic sites for which treatment was planned at the time of the consultation. Table A1 in Multimedia Appendix 1 contains all the NLP- and expert-extracted pain scores, and Figure A2 in Multimedia Appendix 1 illustrates the level of agreement between them. Due to the quality of the documented pain scores and lack of interrater agreement among experts (Fleiss κ=0.43), as explained by Naseri et al [ 25 ], we subsequently defined a binary pain score as “no pain” and “pain” in order to establish satisfactory interrater agreement (κ=0.66) [ 25 ]. To create binary ground-truth pain labels comparable to the NLP-extracted labels, we assigned notes scored as “no pain” to “no pain” and notes scored as “mild,” “moderate,” and “severe” pain to “pain.” These expert-extracted pain scores were used to measure how well the NLP pipeline works.

Center Point Identification of BM Lesions

BM lesion center points were identified by a team comprising a staff radiation oncologist (SS) with 10 years’ experience, a radiation oncology fellow (MT), and 3 third-year radiation oncology residents (J Khriguian, PR, and MF). Simulation CT DICOM (ie, Digital Imaging and Communications in Medicine) files were exported from the radiotherapy treatment planning software and deidentified. Then, the CT images were randomly divided into 5 sets and loaded into the diCOMBINE [ 32 ] application for BM lesion center point identification. Our experts were blinded to patients’ pain statuses and identities. We requested each expert to label center points for all visually identifiable BM lesions in all CT images within 1 of the 5 sets, and another expert was assigned to validate their labels. A key benefit of this radiomics pipeline [ 21 ] is that it does not require full lesion segmentation, making it feasible to engage busy clinicians.

Segmentation of Regions of Interest

Using our previously reported methodology [ 21 ], we automatically segmented lesion center point–based nested spherical (SP) regions of interest (ROIs). To do this, we first delineated nested spherical ROIs around the identified BM lesion center points (see Textbox 1 , top panel). ROI diameters ranged from 7 mm (3×3 voxels) to 50 mm (average size of the vertebral body) [ 33 ]. Then, in addition to what was reported by Naseri et al [ 21 ], we used Hounsfield units thresholding to exclude fat and air regions from the delineated ROIs. For this, motivated by Deglint et al [ 34 ] and Ulano et al [ 35 ], we applied a threshold to remove voxels with negative Hounsfield units from our ROIs. Hounsfield units of <0 are associated with fat and air [ 34 ]. We used OpenCV [ 36 ] (version 4.4.0) for Hounsfield units thresholding and applied a Gaussian filter to reduce noise. Then, we used pynrrd [ 37 ] (version 0.4.2) to export each ROI as a 3D binary mask and store it as a.nrrd [ 38 ] file. Finally, we aggregated these nested ROI masks to form ensemble ROIs. In this study, we examined 2 contrasting ensemble (EN) ROIs as shown in Textbox 1 (bottom panel): one with small size and 3 layers (EN3) and the other with large size and 6 layers (EN6). Wakabayashi et al [ 13 ] and Naseri et al [ 21 ] have shown that radiomics-based machine learning models trained on ensemble ROIs have better classification performance than single ROI–based models.

The characteristics of the spherical and ensemble regions of interest (ROIs) used in this study.

Nested spherical (SP) ROIs with Hounsfield units (HUs) intensity thresholds (HU>0):

SP7 (diameter 7 mm)

SP10 (diameter 10 mm)

SP15 (diameter 15 mm)

SP20 (diameter 20 mm)

SP30 (diameter 30 mm)

SP50 (diameter 50 mm)

Ensemble (EN) ROIs:

EN3 (ROI SP7+SP10+SP15)

EN6 (ROI SP7+SP10+SP15+SP20+SP30+SP50)

Radiomics Models

Our radiomics pipeline is illustrated in Figure 2 . We essentially used our previously reported pipeline [ 21 ] but with our NLP- and expert-extracted pain labels to train and test it. We made one improvement to the pipeline by incorporating Imbalanced-learn [ 39 ] (version 0.7.0) as a resampling step to account for imbalance (see below).

Radiomics features were extracted from each CT image using masks composed of the ensemble ROIs listed in Textbox 1 . Then, the feature space was scaled using z score normalization [ 40 ], and the associated NLP-extracted binary pain labels (pain=1, no pain=0) were incorporated. A single NLP-extracted pain score was assigned to all the lesions extracted from a given paired CT image.

Due to the nature of BM pain [ 41 ], there was a large imbalance between the number of painful and painless lesions (493 pain, 93 no pain). Therefore, we used the Synthetic Minority Oversampling Technique (SMOTE) [ 42 ] in the training phase as it has been shown to be the best-performing resampling method for radiomics [ 43 ]. We did not apply resampling to our test set in order to maintain the original sample imbalance. Then, the Least Absolute Shrinkage And Selection Operator [ 44 ] feature selection method was applied to the feature space to remove noninformative features. Least Absolute Shrinkage And Selection Operator is a commonly used feature selection method in radiomics studies [ 45 , 46 ]. Finally, we examined the Gaussian process regression, linear support vector machine, random forest, and neural networks classifiers, as they were the best performing machine learning classifiers in our previous work. We evaluated the performance of our models on the training set using 5-fold cross-validation. Final evaluation was performed on the test set. The receiver operating characteristic (ROC) [ 47 ] curve, area under the ROC curve (AUC), precision, sensitivity, specificity, and F 1 -score metrics were used to report the performance of our models on the training and test sets. We also trained and tested our best performing pipeline using the expert-extracted pain scores (best-available ground-truth) to evaluate the impact of NLP-extracted pain labels.

Results

Patient Demographics

A total of 176 pairs of radiotherapy consultation notes and simulation CT images of patients with thoracic spinal BM were included in this study. As summarized in Table 1 , a total of 121 sample pairs (mean patient age 63, SD 14 years; males: n=65, mean age 67, SD 14 years; P =.20) were included for training and cross-validation, and 55 sample pairs (mean patient age 64, SD 12 years; males: n=25, mean age 64, SD 13 years; females: mean age 64, SD 23 years; P =.50) were included in the test set. The sample selection procedure and data quantities are presented in Figure 1 . The demographics of the patients in the training and test sets are presented in Table 1 . The most common primary cancer sites were the lungs (n=52), breasts (n=34), and prostate (n=24).

A total of 586 BM center points were identified by our experts on the training (n=423 lesions) and test (n=163 lesions) data sets. In the training set, 357 (84%) lesions were labeled by the NLP pipeline as painful and 66 lesions were labeled as painless. In the test set, 136 (83%) lesions were identified by the NLP pipeline as painful, and 27 lesions were labeled as painless. This represented a significant but equal imbalance in our training and test sets.

Segmented ROIs

Examples of segmented ROIs with the Hounsfield units threshold applied are presented in Figure 3 for painful and painless BMs.

Testing Our Radiomics Models

In total, 107 radiomics features were extracted from each of the 6 nested ROIs. Then, they were aggregated to form feature spaces for the EN3 (with 321 features) and EN6 (with 642 features) ensemble ROIs. Figure 4 shows the ROC curve of each model in the training (black lines) and test (red squares) data sets using the EN3 and EN6 ROIs. On the training set, the gray range represents the mean (SD) AUC of the 5-fold cross-validation. The AUC and F 1 -score grids are presented in Table 2 .

The precision, accuracy, sensitivity, specificity, F 1 -score, and AUC values of our best-performing pipeline (neural networks with the EN6 ROI) are presented in Table 3 . The performance of this pipeline (trained and tested) on the data set of expert-extracted pain labels (best-available ground-truth) is provided as a quality measurement. The performance of the model described previously by Wakabayashi et al [ 13 ] is also provided for comparison.

Discussion

Underestimation and undertreatment of cancer pain can significantly diminish the quality of life of patients with cancer. Accordingly, systems that can objectively measure cancer pain have the potential to improve quality of life. In this study, we created a scalable NLP-radiomics pain identification pipeline. Our pipeline is designed for palliative treatment for patients with cancer undergoing radiotherapy therapy, for whom there are typically just 2 contemporaneous sources of relevant medical information at the time of the treatment: consultation notes and simulation CT images. We used an NLP pipeline to extract physician-reported pain scores from radiotherapy consultation notes. NLP-extracted pain scores are appropriate, when structured patient-reported pain scores are unavailable (as is the case for at least 25% to 35% of all patients with cancer [ 13 , 48 ] and for all patients with cancer receiving palliative care who are treated with radiotherapy at our institution at the time the data were used in this study). Our lesion center point–based spherical ROI delineation method significantly sped up the ROI segmentation procedure, enabling us to rapidly delineate BM center points in 176 images in this study. For comparison, the radiomics pipeline that was developed by Wakabayashi et al [ 13 ] required full 3D segmentation of each ROI (69 images).

Due to the unbalanced nature of BM pain, our data set contained significantly fewer “no pain” samples. In order to better train our models, we applied SMOTE resampling to the training set to balance the number of samples with the NLP-extracted “pain” and “no pain” labels. We did not apply any resampling techniques to our test (hold out) set to maintain the original sample imbalance. Therefore, while our training set was balanced, our test set had 5 times more “pain” cases than “no pain” cases (136 pain versus 27 no pain cases). This caused a significant change in the pipeline’s performance between the training and test sets. It has been shown that oversampling improves the overall performance of machine learning models, but the effect is stronger on the training set due to the inclusion of replicated samples in the cross-validation subsets [ 49 ]. Moreover, the imbalance in our test set led to high specificity (ability to properly identify pain instances) and low sensitivity (ability to correctly identify no pain cases) in the performance evaluation. For comparison, the sample imbalance reported by Wakabayashi et al [ 13 ] was 2:1, resulting in a more balanced relationship between the sensitivity and specificity of their model.

The performance of our pipeline did not improve much when we trained and tested it using expert-extracted pain labels (best-available ground-truth). This might be the case because, in the first experiment, we both trained and tested our pipeline using NLP-extracted pain labels, and in the second experiment, we both trained and tested our pipeline using expert-extracted pain labels. Consequently, after being trained with one set of labels (NLP- or expert-extracted), our pipeline performed well on the test set that was labeled using the same method (NLP or expert). We also demonstrated that our pipeline’s performance is comparable to that of Wakabayashi et al [ 13 ], who achieved their results using patient-reported pain labels.

Our pipeline performed significantly better on the EN6 ROIs than on the EN3 ROIs. This could be the case because in comparison to EN3, our EN6 ROIs include additional ROIs with sizes of 20, 30, and 50 mm. From visual inspection, we suspect that, in addition to the characteristics of the BM lesion itself, its location (eg, its proximity to the spinal cord) may be a significant contributor to the BM pain. As a result, larger ROIs enable our algorithm to extract characteristics from outside the BM lesion. Wakabayashi et al [ 13 ] also demonstrated the effectiveness of using ROIs outside of the BM lesion.

We are unable to offer a convincing explanation as to why neural networks outperformed random forest and support vector machine classifiers in our analysis. Notwithstanding, it has been demonstrated that neural network classifiers perform better when applied to more difficult problems and larger data sets, while random forest and support vector machine classifiers typically perform well with smaller data sets [ 46 , 50 , 51 ].

Our pipeline was successful in extracting radiomics biomarkers capable of distinguishing between painful and painless BM lesions. These biomarkers potentially provide the opportunity to objectively identify clinical pain-related indicators that may aid in the diagnosis, treatment, and understanding of BM pain.

Our work has several limitations. First, we used data from a single center for this retrospective study. A multicenter study with a larger data set is necessary to assess the generalizability of our radiomics pipeline for pain quantification. We anticipate that the performance of our NLP-radiomics pipeline will vary based on the pain scoring systems of the cohorts tested. Second, by using lesion center point–based geometrical ROIs, we ignored lesion characteristics such as size and shape, which may be important in the context of pain. Although we used Hounsfield units intensity thresholding to preserve some tumor information, we are considering implementing deep learning–based ROI segmentation in the future as it may better account for full tumor and surrounding tissue characteristics. Lastly, we used SMOTE resampling to address the issue of class imbalance. An alternative solution might be to develop cost-sensitive machine learning classifiers that account for the cost of misclassifying minority samples [ 52 ]. However, there is no clear consensus in the literature on whether cost-sensitive learning outperforms resampling [ 53 ]. A model that can differentiate between painful and painless lesions from medical imaging is a critical component of any possible radiomics-based pain quantification pipeline. This work not only shows the feasibility of developing a pain quantification tool, but also it removes some of the barriers to its development. As a result, our future work will be to apply our pipeline to patients’ past and current CT images and consultation notes in order to develop a longitudinal model of pain. Such a model should take into account not only images (taken before, during, and after delivering radiotherapy) but also other internal and external parameters that can influence how pain evolves over time (such as primary cancer type, radiation dose, other treatments, and pain medications). Also, it will include patient-reported pain scores to provide more accurate ground-truth pain labels in order to develop a more robust deep learning–based NLP pipeline [ 24 , 54 ]. This, however, is beyond the scope of this investigation.

In conclusion, we demonstrated that our NLP and radiomics-based machine learning pipeline can effectively differentiate between painful and painless BM lesions in simulation CT images using ensemble lesion center point–based geometrical ROIs. Using NLP-extracted pain labels in conjunction with lesion center point–based radiomics features is time efficient. This helps to pave the way for the development of quickly trained and efficient clinical AI-based decision-making tools that can objectively measure cancer pain. Such a tool may help alleviate the burden of pain management and improve the quality of life of patients with BMs.

This research was supported by the startup grant of J Kildea at the Research Institute of the McGill University Health Centre (RI-MUHC), the Ruth and Alex Dworkin scholarship award from the Faculty of Medicine and Health Sciences at McGill University, an RI-MUHC studentship award, a Grad Excellence Award-00293 from the Department of Physics at McGill University, Fonds de recherche du Québec - Santé (FRQS), and by the CREATE Responsible Health and Healthcare Data Science (SDRDS) grant of the Natural Sciences and Engineering Research Council. The authors would like to thank Dr Luc Galarneau for his help with statistical analysis.

Conflicts of Interest: None declared.

Sample data.

Abbreviations

artificial intelligence

average pain intensity

area under the receiver operating characteristic curve

bone metastasis

computed tomography

ensemble

natural language processing

receiver operating characteristic

region of interest

Synthetic Minority Oversampling Technique

spherical

verbally declared pain

1 van den Beuken-van Everdingen MH Hochstenbach LM Joosten EA Tjan-Heijnen VC Janssen DJ Update on prevalence of pain in patients with cancer: systematic review and meta-analysis J Pain Symptom Manage 2016 06 51 6 1070 1090.e9 10.1016/j.jpainsymman.2015.12.340 27112310 S0885-3924(16)30048-3 27112310

2 McQuay Henry J Collins Sally L Carroll Dawn Moore R Andrew Derry Sheena WITHDRAWN: radiotherapy for the palliation of painful bone metastases Cochrane Database Syst Rev 2013 11 22 2013 11 CD001793 10.1002/14651858.CD001793.pub2 24271498 24271498 PMC6564087

3 Grossman SA Undertreatment of cancer pain: barriers and remedies Support Care Cancer 1993 3 1 2 74 78 10.1007/bf00366899 7511475

4 Cleeland CS Janjan NA Scott CB Seiferheld WF Curran WJ Cancer pain management by radiotherapists: a survey of radiation therapy oncology group physicians Int J Radiat Oncol Biol Phys 2000 04 01 47 1 203 208 10.1016/s0360-3016(99)00276-x 10758325 S0360-3016(99)00276-X 10758325

5 Tracey I Woolf CJ Andrews NA Composite pain biomarker signatures for objective assessment and effective treatment Neuron 2019 03 06 101 5 783 800 10.1016/j.neuron.2019.02.019 30844399 S0896-6273(19)30154-0 30844399 PMC6800055

6 Xu X Huang Y Objective pain assessment: a key for the management of chronic pain F1000Res 2020 1 23 9 35 10.12688/f1000research.20441.1 32047606 F1000 Faculty Rev-35 PMC6979466 32047606

7 Niculescu AB Le-Niculescu H Levey DF Roseberry K Soe KC Rogers J Khan F Jones T Judd S McCormick MA Wessel AR Williams A Kurian SM White FA Towards precision medicine for pain: diagnostic biomarkers and repurposed drugs Mol Psychiatry 2019 04 24 4 501 522 10.1038/s41380-018-0345-5 30755720 10.1038/s41380-018-0345-5 30755720 PMC6477790

8 Diaz MM Caylor J Strigo I Lerman I Henry B Lopez E Wallace MS Ellis RJ Simmons AN Keltner JR Toward composite pain biomarkers of neuropathic pain-focus on peripheral neuropathic pain Front Pain Res (Lausanne) 2022 5 11 3 869215 10.3389/fpain.2022.869215 35634449 35634449 PMC9130475

9 Furfari A Wan BA Ding K Wong A Zhu L Bezjak A Wong R Wilson CF DeAngelis C Azad A Chow E Charames GS Genetic biomarkers associated with pain flare and dexamethasone response following palliative radiotherapy in patients with painful bone metastases Ann Palliat Med 2017 12 6 Suppl 2 S240 S247 10.21037/apm.2017.09.04 29156912 apm.2017.09.04 29156912

10 Gunn Joshua Hill Melissa M Cotten Bradley M Deer Timothy R An analysis of biomarkers in patients with chronic pain Pain Physician 2020 01 23 1 E41 E49 32013287 32013287

11 Marchi A Vellucci R Mameli S Rita Piredda A Finco G Pain biomarkers Clinical Drug Investigation 2009 29 Supplement 1 41 46 10.2165/0044011-200929001-00006 19445554

12 Ota Y Connolly M Srinivasan A Kim J Capizzano AA Moritani T Mechanisms and origins of spinal pain: from molecules to anatomy, with diagnostic clues and imaging findings Radiographics 2020 07 40 4 1163 1181 10.1148/rg.2020190185 32501739 32501739

13 Wakabayashi K Koide Y Aoyama T Shimizu H Miyauchi R Tanaka H Tachibana H Nakamura K Kodaira T A predictive model for pain response following radiotherapy for treatment of spinal metastases Sci Rep 2021 06 18 11 1 12908 10.1038/s41598-021-92363-0 10.1038/s41598-021-92363-0 34145367 10.1038/s41598-021-92363-0 34145367 PMC8213735

14 Carlson LA Hooten WM Pain-linguistics and natural language processing Mayo Clin Proc Innov Qual Outcomes 2020 06 4 3 346 347 10.1016/j.mayocpiqo.2020.01.005 32542226 S2542-4548(20)30032-1 32542226 PMC7283550

15 Dave Amisha D Ruano Gualberto Kost Jonathan Wang Xiaoyan Automated extraction of pain symptoms: a natural language approach using electronic health records Pain Physician 2022 03 25 2 E245 E254 35322976 35322976

16 Tighe Patrick J Sannapaneni Bharadwaj Fillingim Roger B Doyle Charlie Kent Michael Shickel Ben Rashidi Parisa Forty-two million ways to describe pain: topic modeling of 200,000 PubMed pain-related abstracts using natural language processing and deep learning-based text generation Pain Med 2020 11 01 21 11 3133 3160 10.1093/pm/pnaa061 32249306 5816322 32249306 PMC7685694

17 Matsangidou M Liampas A Pittara M Pattichi CS Zis P Machine learning in pain medicine: an up-to-date systematic review Pain Ther 2021 12 26 10 2 1067 1084 10.1007/s40122-021-00324-2 34568998 10.1007/s40122-021-00324-2 34568998 PMC8586126

18 Neijenhuijs KI Peeters CFW van Weert H Cuijpers P Leeuw IV Symptom clusters among cancer survivors: what can machine learning techniques tell us? BMC Med Res Methodol 2021 08 16 21 1 166 10.1186/s12874-021-01352-4 34399698 10.1186/s12874-021-01352-4 34399698 PMC8369803

19 Hong JH Jung J Jo A Nam Y Pak S Lee S Park H Lee SE Kim S Development and validation of a radiomics model for differentiating bone islands and osteoblastic bone metastases at abdominal CT Radiology 2021 06 299 3 626 632 10.1148/radiol.2021203783 33787335 33787335

20 Sun W Liu S Guo J Liu S Hao D Hou F Wang H Xu W A CT-based radiomics nomogram for distinguishing between benign and malignant bone tumours Cancer Imaging 2021 02 06 21 1 20 10.1186/s40644-021-00387-6 33549151 10.1186/s40644-021-00387-6 33549151 PMC7866630

21 Naseri H Skamene S Tolba M Faye MD Ramia P Khriguian J Patrick H Andrade Hernandez AX David M Kildea J Radiomics-based machine learning models to distinguish between metastatic and healthy bone using lesion-center-based geometric regions of interest Sci Rep 2022 06 14 12 1 9866 10.1038/s41598-022-13379-8 10.1038/s41598-022-13379-8 35701461 10.1038/s41598-022-13379-8 35701461 PMC9198102

22 Mashayekhi R Parekh VS Faghih M Singh VK Jacobs MA Zaheer A Radiomic features of the pancreas on CT imaging accurately differentiate functional abdominal pain, recurrent acute pancreatitis, and chronic pancreatitis Eur J Radiol 2020 02 123 108778 10.1016/j.ejrad.2019.108778 31846864 S0720-048X(19)30428-0 31846864 PMC7968044

23 Vedantam A Hassan I Kotrotsou A Hassan A Zinn PO Viswanathan A Colen RR Magnetic resonance-based radiomic analysis of radiofrequency lesion predicts outcomes after percutaneous cordotomy: a feasibility study Oper Neurosurg (Hagerstown) 2020 06 01 18 6 721 727 10.1093/ons/opz288 31665446 5607417 31665446

24 Devlin J Chang MW Lee K Toutanova K BERT: pre-training of deep bidirectional transformers for language understanding arXiv Preprint posted online October 11, 2018 10.48550/arXiv.1810.04805

25 Naseri H Kafi K Skamene S Tolba M Faye MD Ramia P Khriguian J Kildea J Development of a generalizable natural language processing pipeline to extract physician-reported pain from clinical reports: Generated using publicly-available datasets and tested on institutional clinical reports for cancer patients with bone metastases J Biomed Inform 2021 08 120 103864 10.1016/j.jbi.2021.103864 34265451 S1532-0464(21)00193-3 34265451

26 Elbattah M Arnaud É Gignon M Dequen G The role of text analytics in healthcare: a review of recent developments and applications 2021 14th International Joint Conference on Biomedical Engineering Systems and Technologies - Scale-IT-up February 11-13, 2021 Online 10.5220/0010414508250832

27 Smith TMF Cochran WG Sampling techniques, second edition Applied Statistics 1964 13 1 54 10.2307/2985224

28 Freedman D Pisani R Purves R Statistics: Fourth International Student Edition 2007 New York, NY W.W. Norton & Company

29 Aronson A R Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program Proc AMIA Symp 2001 17 21 11825149 D010001275 11825149 PMC2243666

30 McCray A T Aronson A R Browne A C Rindflesch T C Razi A Srinivasan S UMLS knowledge for biomedical language processing Bull Med Libr Assoc 1993 04 81 2 184 194 8472004 8472004 PMC225761

31 hn617/texTRACTOR: texTRACTOR Zenodo 2021 2023-04-18 https://zenodo.org/record/4649625

32 hn617/diCOMBINE: diCOMBINE Zenodo 2021 2023-04-18 https://zenodo.org/record/5218743

33 Busscher I Ploegmakers JJW Verkerke GJ Veldhuizen AG Comparative anatomical dimensions of the complete human and porcine spine Eur Spine J 2010 07 26 19 7 1104 1114 10.1007/s00586-010-1326-9 20186441 20186441 PMC2900026

34 Deglint HJ Rangayyan RM Ayres FJ Boag GS Zuffo MK Three-dimensional segmentation of the tumor in computed tomographic images of neuroblastoma J Digit Imaging 2006 8 25 20 1 72 87 10.1007/10278-006-0769-3 16937021

35 Ulano A Bredella MA Burke P Chebib I Simeone FJ Huang AJ Torriani M Chang CY Distinguishing untreated osteoblastic metastases from enostoses using CT attenuation measurements Am J Roentgenol 2016 08 207 2 362 368 10.2214/ajr.15.15559 27101076

36 Smoothing Images OpenCV 2023-09-18 https://docs.opencv.org/4.x/d4/d13/tutorial_py_filtering.html

37 mhe/pynrrd: v0.4.3 Released Zenodo 2022 2023-04-18 https://zenodo.org/record/6501810

38 Nearly Raw Raster Data 2022-09-04 http://teem.sourceforge.net/nrrd/index.html

39 Lemaître G Nogueira F Aridas CK Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning J Mach Learn Res 2017 18 1 5

40 Low E Review of Understanding Basic Statistics Am Stat 1998 52 2 198 10.2307/2685480

41 Torvik Karin Hølen Jacob Kaasa S Kirkevold ? Holtan A Kongsgaard U Rustøen Tone Pain in elderly hospitalized cancer patients with bone metastases in Norway Int J Palliat Nurs 2008 05 14 5 238 245 10.12968/ijpn.2008.14.5.29491 18563017 18563017

42 Chawla NV Bowyer KW Hall LO Kegelmeyer WP SMOTE: synthetic minority over-sampling technique J Artif Intell Res 2002 06 01 16 321 357 10.1613/jair.953

43 Xie C Du R Ho JW Pang HH Chiu KW Lee EY Vardhanabhuti V Effect of machine learning re-sampling techniques for imbalanced datasets in F-FDG PET-based radiomics model on prognostication performance in cohorts of head and neck cancer patients Eur J Nucl Med Mol Imaging 2020 11 06 47 12 2826 2835 10.1007/s00259-020-04756-4 32253486 10.1007/s00259-020-04756-4 32253486

44 Tibshirani R Regression shrinkage and selection via The Lasso: a retrospective J R Stat Soc Series B Stat Methodol 2011 73 3 273 282 10.1111/j.1467-9868.2011.00771.x

45 Yin P Mao N Chen H Sun C Wang S Liu X Hong N Machine and deep learning based radiomics models for preoperative prediction of benign and malignant sacral tumors Front Oncol 2020 10 16 10 564725 10.3389/fonc.2020.564725 33178593 33178593 PMC7596901

46 Shur JD Doran SJ Kumar S Ap Dafydd Derfel Downey K O'Connor James P B Papanikolaou N Messiou C Koh D Orton MR Radiomics in oncology: a practical guide Radiographics 2021 10 41 6 1717 1732 10.1148/rg.2021210037 34597235 34597235 PMC8501897

47 Fawcett T An introduction to ROC analysis Pattern Recognit Lett 2006 6 27 8 861 874 10.1016/j.patrec.2005.10.010

48 Fleischman RJ Frazer DG Daya M Jui J Newgard CD Effectiveness and safety of fentanyl compared with morphine for out-of-hospital analgesia Prehosp Emerg Care 2010 03 03 14 2 167 175 10.3109/10903120903572301 20199230 20199230 PMC2924527

49 Fernández A García S Galar M Prati RC Krawczyk B Herrera F Learning from imbalanced data streams Learning from Imbalanced Data Sets 2018 Cham Spirnger

50 Sun Q Lin X Zhao Y Li L Yan K Liang D Sun D Li Z Deep learning vs. radiomics for predicting axillary lymph node metastasis of breast cancer using ultrasound images: don't forget the peritumoral region Front Oncol 2020 1 31 10 53 10.3389/fonc.2020.00053 32083007 32083007 PMC7006026

51 Lisson CS Lisson CG Mezger MF Wolf D Schmidt SA Thaiss WM Tausch E Beer AJ Stilgenbauer S Beer M Goetz M Deep neural networks and machine learning radiomics modelling for prediction of relapse in mantle cell lymphoma Cancers (Basel) 2022 04 15 14 8 2008 10.3390/cancers14082008 35454914 cancers14082008 35454914 PMC9028737

52 Thai-Nghe N Gantner Z Schmidt-Thieme L Cost-sensitive learning methods for imbalanced data 2010 The 2010 International Joint Conference on Neural Networks (IJCNN) July 18-23, 2010 Barcelona 10.1109/ijcnn.2010.5596486

53 Liu A Martin C La Cour B Ghosh J Stahlbock R Crone S Lessmann S Effects of Oversampling Versus Cost-Sensitive Learning for Bayesian and SVM Classifiers Data Mining. Annals of Information Systems (volume 8) 2010 Boston, MA Springer

54 Tamang S Humbert-Droz M Gianfrancesco M Izadi Z Schmajuk G Yazdany J Practical considerations for developing clinical natural language processing systems for population health management and measurement JMIR Med Inform 2023 01 03 11 e37805 10.2196/37805 36595345 v11i1e37805 36595345 PMC9846439