With advances in cancer treatment, the five-year survival rate for all cancer patients is on the rise, and it is now possible to survive for a long period of time after being diagnosed with cancer. 1 In line with this trend, the number of patients diagnosed and treated for bone metastasis is also increasing, and the duration of treatment tends to lengthen as well. 2 If not properly diagnosed and treated, bone metastasis can cause bone-related adverse events such as pathologic fracture and spinal cord injury, adversely impacting the quality of life (QOL) of cancer patients. 3 – 6 It has been reported that metastatic epidural spinal cord compression is the first manifestation of cancer in ∼20% of spinal metastasis patients. 7 To reduce such bone-related adverse events, it is important to detect bone metastases at an early stage and to initiate appropriate therapeutic interventions such as radiotherapy, bone-modulating medications, or cement augmentation.
Early detection of bone metastasis and early therapeutic intervention can lead to maintenance or improvement in activities of daily living and QOL and even extend life expectancy. 8 , 9
There are multiple imaging modalities for bone metastasis, including computed tomography (CT), magnetic resonance imaging, and positron emission tomography, each with different advantages. 10 – 12 CT is most frequently used as surveillance in cancer patients because of its superior spatial resolution and bone structure and its relative ease of use. 11 , 12 However, image diagnosis of bone metastasis is often difficult, even for orthopedic surgeons and diagnostic radiologists.
Thus, the purpose of the present study was to improve the accuracy of image diagnosis of bone metastasis by developing an artificial intelligence (AI)-aided detection model utilizing CT image data obtained during the treatment of patients with bone metastases at the TMDU Hospital.
MATERIALS AND METHODS
Data Acquisition
All CT images were obtained retrospectively from the clinical databases of a single institution, TMDU Hospital, acquired consecutively between 2016 and 2022. Both noncontrast and intravenous contrast-enhanced CT scans, which were performed for the diagnosis or follow-up of malignancy, were included.
The inclusion criteria for bone metastasis-positive scans in the data sets were as follows: (a) presence of at least one bone metastasis lesion in the thoracolumbar spines; (b) availability of slice images of 1 to 5 mm thickness; and (c) presence of a bone metastasis lesion 5 mm or more in diameter. Bone metastasis lesions <5 mm were not included because of the difficulty in confirming their diagnosis. Two or more CT scans were included in 31 patients because the radiologic appearance of bone metastases changed substantially. Simultaneously, CT scans from patients without bone metastasis were also collected as a bone metastasis-negative control to allow the AI to identify normal bones. Negative control cases with malignancy were selected to match the distribution of age, sex, and primary lesions.
As presented in a flowchart outlining data collection and division (Fig. 1 ), in total, 5991 positive axial slices in 283 positive CT scans from 255 patients and 88,799 negative axial slices in 192 negative CT scans from 192 patients were utilized in this study. The training and validation data sets included 86,276 (positive: 5519, negative: 80,757) and 5211 (positive: 308, negative: 4903) axial slices in 410 and 41 CT scans, respectively. CT scans originating from the same patients were assigned to the same data sets. The test data set included 3303 (positive: 164 and negative: 3139) axial slices in 40 CT scans from 40 patients. These 40 CT scans for a test data set were separately collected from the training and validation data sets. The characteristics of the data sets used in the present study are summarized in Tables 1 and 2 .
TABLE 1
Characteristics of the Patients and Collected CT Scans
CT indicates computed tomography.
| Metastases group | Control group | ||
| Per patient | |||
| N | 255 (283 CT scans) | 192 (192 CT scans) | |
| Age (Mean ± SD) | 74.4 ± 4.5 | 65.3 ± 12.8 | |
| Sex | |||
| Male | 176 | 118 | |
| Female | 79 | 74 | |
| Use of contrast agent | |||
| Plain | 219 | 87 | |
| Contrast-enhanced | 64 | 105 | |
| Slice thickness | |||
| 1.0 mm | 89 | 104 | |
| 2.0 mm | 118 | 9 | |
| 5.0 mm | 76 | 79 | |
| Scan range | |||
| Chest to pelvis | C7-L5 | 135 | 144 |
| Neck to pelvis | C1-L5 | 19 | 29 |
| Abdomen | Th10-L5 | 1 | 1 |
| Neck to chest | C1-L2 | 1 | 2 |
| Chest | C7-L2 | 97 | 12 |
| Brain to pelvis | C1-L5 | 12 | 2 |
| Brain to chest | C1-L2 | 1 | 0 |
| Abdomen to pelvis | Th10-L5 | 15 | 1 |
| Abdomen to lower limbs | Th10-L5 | 2 | 1 |
TABLE 2
Primary Malignancy Sites of the Subject Patients
| Metastases group | Control group | |
| Lung | 65 | 27 |
| Kidney | 31 | 32 |
| Colon | 24 | 33 |
| Esophagus | 18 | 18 |
| Breast | 17 | 7 |
| Liver | 13 | 2 |
| Multiple myeloma | 13 | 0 |
| Prostate | 12 | 16 |
| Pancreas | 11 | 10 |
| Oral | 10 | 3 |
| Skin | 9 | 2 |
| Pharynx | 6 | 3 |
| Stomach | 4 | 14 |
| Bladder | 4 | 7 |
| Uterus | 3 | 5 |
| Unknown primary | 3 | 2 |
| Thyroid | 3 | 1 |
| Salivary gland | 2 | 1 |
| Adrenal gland | 2 | 0 |
| Thymus | 1 | 0 |
| Testis | 1 | 2 |
| Larynx | 1 | 4 |
| Small intestine | 1 | 3 |
| Ovary | 1 | 0 |
This retrospective study was approved by the Ethics Committee of TMDU (#M2020-139). Formal consent was waived by the Ethics Committee due to the retrospective nature of data collection.
Preparation and Annotation of Ground Truth Labels
For the whole data set, ground truth labels were established with manual semantic segmentation using MD.ai (MD.ai Inc., New York, NY), a platform for medical AI. Manual annotation was performed by a board-certified general orthopedic surgeon (M.M.: 10 years of experience) with consensus by at least one board-certified bone and soft tissue oncologists (S.S. and Y.F.: 24 and 19 years of experience).
After manual semantic segmentation, to extract only osteolytic image portions, the annotated data were modified by density-based spatial clustering of applications with noise clustering 13 on the annotated portions. When the average CT value of each cluster was <200, the portion was considered osteolytic bone metastasis, and the remaining clusters were considered nonbone metastasis. After density-based spatial clustering of applications with noise clustering, the board-certified general orthopedic surgeon reviewed and annotated all slices again to confirm that all annotated labels were ground truth labels for osteolytic bone metastasis (Fig. 2 A).
Deep Learning-Based Algorithm
All positive slices with ground truth annotations were classified into groups of training data, validation data, and test data. After 164 positive slices were collected as test data, the remaining slices were randomly sorted 9:1, with 9 as training data and 1 as validation data. To create a new AI model, deep learning (DL) was performed by importing these training data into a segmentation model called “DeepLabv3+“ (Fig. 2 B). DeepLabv3+ was proposed by Chen and colleagues compared with models, such as U-Net, commonly used for semantic segmentation, DeepLabv3+ can extract a wider range of image features by applying a special convolutional layer called Atrous Spatial Pyramid Pooling and can detect objects with high accuracy, even for objects with smaller areas or regions with unclear boundaries. 13 – 15 The maximum number of training cycles was set at 200,000, and Adam was used as the optimizer. 16
Evaluation
In the present study, sensitivity, precision, F1-score, and specificity were calculated. 17 , 18 To evaluate the performance of object detection by comparing the ground truth annotated areas to the AI-predicted area, intersection over union (IoU) was used. IoU is commonly used to evaluate medical image segmentation. 17 – 19 For slices with ground truth labels, an IoU >0.1 was considered a true positive, and an IoU <0.1 was considered a false negative. For slices with no ground truth label, no detection of bone metastasis by the AI model was considered a true negative, and any detection, even a single pixel, of bone metastasis by the AI model was considered a false positive (FP).
Observer Study
An observer study by 12 observers was conducted to evaluate the clinical utility of our AI model. The 12 observers consisted of three board-certified orthopedic surgeons (25, 24, and 22 yr of experience), three board-certified radiologists (14, 10, and 10 yr of experience), three resident orthopedic surgeons (3 yr of experience each), and three resident radiologists (3 yr of experience each). The observers evaluated 40 CT scans (positive: 20 CT scans, 2634 slices; negative: 20 CT scans, 3139 slices) of the test data set without the AI model and marked suspicious bone metastasis lesions on the images by boxing them using MD.ai. A dedicated image viewer with axial slices and window level and width modification functions was used to view the CT images. For training in how to use the viewer, observers were given the opportunity to practice a couple of demonstration cases before the actual study. The observers were blinded to all clinical data except for the age and sex of each patient. In addition, the second observer study was performed by six resident observers. They evaluated the same 40 CT scans again with the reference of the results predicted by the AI model. The interval between the two observer studies was set to be at least 12 weeks. After the observer studies, sensitivity, precision, F1-score, and specificity by observers were calculated and compared with those without the AI model.
Statistical Analysis
To evaluate significant differences in sensitivity, precision, F1-score, and specificity between observers, statistical analyses were performed using a t test or Wilcoxon signed-rank test. P <0.05 was considered statistically significant.
RESULTS
The evaluation of detection by our AI model is shown in Table 3 . For the test data sets including 40 CT scans in total, the sensitivity, precision, and F1-score were 0.78, 0.68, and 0.72 per slice and 0.75, 0.36, and 0.48 per lesion, respectively. Table 4 shows the confusion matrix (per slice or per lesion) of the binary classification. The number of FP slices was 61 of a total of 3303 slices, which is equivalent to an average of 1.525 slices per case. In contrast, the number of FP lesions was 55 of a total of 91 lesions, resulting in lower precision and F1-score in per lesion evaluation than per slice evaluation. Representative images of true positive lesions are shown in Figure 3 . The AI detection of relatively large osteolytic lesions in the spinal vertebral body was sufficient. Representative images of false negative lesions and FP lesions are shown in Figure 4 A and B. The detection of lesions in the posterior elements of the spine, such as the vertebral arch, transverse process, and spinous process, was relatively difficult (Fig. 4 A). However, Schmoll’s nodes, degenerated intervertebral discs, and horizontal fissures on the posterior aspect of the vertebral body (Hahn’s clefts) 20 caused several FPs (Fig. 4 B).
TABLE 3
Evaluation of Detection by the AI Model and 12 Observers
* P <0.05 versus six residents (ORTHO and RAD) without AI, t test. † P <0.01 versus six residents (ORTHO and RAD) without AI, t test. ‡ P <0.05, versus six residents (ORTHO and RAD) without AI, Wilcoxon signed-rank test. AI indicates artificial intelligence; ORTHO, orthopedic surgeons; RAD, radiologists.
| Evaluation of detection (per slice) | Evaluation of detection (per lesion) | ||||||
| Sensitivity | Precision | F1-score | Specificity | Sensitivity | Precision | F1-score | |
| AI model | 0.78 | 0.68 | 0.72 | 0.98 | 0.75 | 0.36 | 0.48 |
| 12 observers (ORTHO and RAD) | 0.72 | 0.97 | 0.82 | 0.99 | 0.61 | 0.88 | 0.72 |
| 6 Residents (ORTHO and RAD) | 0.66 | 0.95 | 0.78 | 0.99 | 0.54 | 0.87 | 0.66 |
| 3 Residents (ORTHO) | 0.61 | 0.92 | 0.73 | 0.99 | 0.50 | 0.80 | 0.61 |
| 3 Residents (RAD) | 0.71 | 0.98 | 0.82 | 0.99 | 0.58 | 0.94 | 0.72 |
| 6 Experts (ORTHO and RAD) | 0.78 * | 0.99 | 0.87 | 0.99 | 0.69 † | 0.89 | 0.78 * |
| 3 Experts (ORTHO) | 0.80 | 0.98 | 0.88 | 0.99 | 0.71 | 0.89 | 0.79 |
| 3 Experts (RAD) | 0.76 | 0.99 | 0.86 | 0.99 | 0.66 | 0.89 | 0.76 |
| 6 Residents with AI (ORTHO and RAD) | 0.73 | 0.96 | 0.82 | 0.99 | 0.66 ‡ | 0.83 | 0.73 |
| 3 Residents with AI (ORTHO) | 0.64 | 0.95 | 0.76 | 0.99 | 0.63 | 0.85 | 0.72 |
| 3 Residents with AI (RAD) | 0.81 | 0.96 | 0.88 | 0.99 | 0.69 | 0.80 | 0.74 |
TABLE 4
Confusion Matrix, Per Slice and Per Lesion
| Confusion matrix (per slice) | Confusion matrix (per lesion) | |||||
| AI model | 12 observers (average) | |||||
| AI model | Predicted label | Total number of true lesions | 51 | |||
| Positive | Negative | |||||
| True label | Positive | 127 | 36 | Total number of predicted lesions | 91 | 26 |
| Negative | 61 | 3079 | ||||
| 12 observers (average) | Predicted label | True lesions matched to at least one predicted lesion | 38 | 35 | ||
| Positive | Negative | |||||
| True label | Positive | 120 | 43 | Predicted lesions matched to at least one true lesion | 36 | 24 |
| Negative | 2 | 3138 |
The results of the first observer study are shown in Table 3 . Regarding the results of CT image interpretation by 12 observers, the sensitivity, precision, and F1-score were 0.72, 0.97, and 0.82 per slice and 0.61, 0.88, and 0.72 per lesion, respectively. When the 12 observers were divided into two groups, six experts, and six residents, the evaluation indices of the experts tended to be higher than those of the residents. In addition, although radiologist experts and othopaedic experts showed similar values in these indices, radiologist residents showed higher values than orthopedic residents. When compared with the AI model, the sensitivity of our AI model was comparable to that of experts and much higher than that of residents. In contrast, the AI model showed lower precision and F1-score than observers because of the high number of FPs.
Twelve weeks or more after the first observer study, the second study was performed by six residents, who were allowed to see the results predicted by the AI model. By referring to the AI results, the sensitivity and F1-value were improved in most residents, suggesting that our AI model is useful for improving the interpretation skills to detect bone metastasis, particularly for younger doctors.
DISCUSSION
The thoracolumbar spine is one of the most common sites of bone metastasis. Osteolytic lesions are more fragile than sclerotic or mixed types of lesions and are more likely to cause pain, pathologic fracture, spinal cord paralysis, and worsened QOL. In the clinical setting, it is very important to regulate such osteolytic lesions. Early detection of osteolytic lesions allows early therapeutic interventions, such as radiotherapy or other bone-modifying agent administration, resulting in the prevention of fracture and neurological injury.
Several attempts have been made to develop a computer-aided detection system for detecting bone metastases on CT scans. Burns et al. 21 used a combination of a watershed segmentation algorithm and a support vector machine classifier. Hammon et al. 22 used three consecutive random forest classifiers, each processing local image features, to assess candidate regions for bone metastases. Roth et al. 23 applied a deep convolutional neural network to the output of a preexisting computer-aided detection system and demonstrated its efficacy in FP reduction. Chmelik et al. 24 used a deep convolutional neural network for voxelwise segmentation, followed by a random forest classifier for FP reduction. In our study, DL was performed by using the segmentation model “Deep lab V3+“ (Fig. 2 B).
In the present study, the F1-score per slice of the AI model was 0.72. This value is not high compared with previous reports. 18 , 25 , 26 Although more lesions with simple shapes in the data sets lead to a higher F1-score, we included a variety of osteolytic metastatic lesions with various complex forms in the training data sets to be applied to actual clinical practice. That is why our AI model showed a slightly lower F1-score.
Regarding the sample size of positive scans in the training data sets, the number of CT scans with bone metastasis ranged from 40 CT scans in the work of Faghani and colleague and 79 CT scans in that of Koike and colleagues to 269 CT scans and Noguchi and colleagues 18 , 25 , 26 . We used a total of 475 CT scans in our data sets. Although it is generally considered that increased sample size in the training data improve the accuracy of the AI model, according to our results, the change in the number of CT scans from 291 to 475 did not meaningfully alter the F1-score (Supplementary Fig. 1, Supplemental Digital Content 1, http://links.lww.com/BRS/C344 ). Therefore, we considered that our sample size was sufficient for the current work.
Our model showed comparable sensitivity to orthopedic or radiology experts and improved the diagnostic accuracy of residents. In the second observer study, the sensitivity and F1-value were improved in most residents by referring to the AI results. However, the improved scores did not reach the expert level, and the precision and F1-score declined conversely in a few residents. The reason for this is assumed to be that, due to the high number of FPs in the AI model, more annotations were assigned to areas where there was no bone metastasis. Still, this may indicate that the model may also be suited to those who are not experts in orthopedics or radiology, including other medical specialties.
Finally, this study had several limitations. First, our data sets included images from only a single institution. The generalizability of the AI model needs to be assessed with a multi-institutional external data set. Second, our model can detect only osteolytic thoracolumbar spinal bone metastasis on CT images. The detection of osteosclerotic lesions remains a separate issue at this time. Third, lesions <5 mm were not included in the data sets. Fourth, the ground truth labels were established carefully by two or more experts, but the existence of metastatic cells in the selected lesions was not confirmed through histologic analysis.
CONCLUSION
We developed a novel DL-based AI model to automatically detect osteolytic bone metastases in the thoracolumbar spines using conventional CT scans. Our model showed comparable sensitivity to orthopedic or radiology experts and improved the diagnostic accuracy of residents. Although further improvement in accuracy is needed, the current AI model may be applicable to current clinical practice.
Key Points
We developed a novel deep learning-based artificial intelligence model to automatically detect osteolytic bone metastases in the thoracolumbar spines using conventional computed tomography scans.
The clinical availability of our developed artificial intelligence model was evaluated through observer studies involving six orthopedic surgeons and six radiologists.
Our model showed comparable sensitivity to orthopedic or radiology experts and improved the diagnostic accuracy of residents.
Supplementary Material
SUPPLEMENTARY MATERIAL
Acknowledgments
The authors thank Asami Nakada, Kunihiro Suga, Tomoaki Uemichi, Masaaki Kawanishi, Subaru Higuchi, and Yusuke Kawasaki for contributing to the first and second observer studies as orthopedic surgery residents or diagnostic radiology residents. The authors also thank Tabu Gokita, Hirotaka Koyanagi, Kentaro Yamada, Junichi Tsuchiya, Koichiro Kimura, and Ken Yamagiwa for participating in the first observer study as orthopedic surgery experts or diagnostic radiology experts.
Footnotes
The authors report no conflicts of interest.
Supplemental Digital Content is available for this article. Direct URL citations are provided in the HTML and PDF versions of this article on the journal’s website: www.spinejournal.com
Contributor Information
Masataka Motohashi, Email: masataka.mo.09@gmail.com.
Yuki Funauchi, Email: funauchi.orth@tmd.ac.jp.
Takuya Adachi, Email: tadachi806@gmail.com.
Tomoyuki Fujioka, Email: tomoyukifujioka@hotmail.com.
Naoya Otaka, Email: Naoya.Otaka@nttdata.com.
Yuka Kamiko, Email: Yuka.Kamiko@nttdata.com.
Takashi Okada, Email: takashi.okada@nttdata.com.
Ukihide Tateishi, Email: ttisdrnm@tmd.ac.jp.
Atsushi Okawa, Email: okawa@yokohama.jrc.or.jp.
Toshitaka Yoshii, Email: yoshii.orth@tmd.ac.jp.
Shingo Sato, Email: satoshin.phy2@tmd.ac.jp.
References
1. Siegel RL, Miller KD, Fuchs HE, et al. Cancer Statistics, 2021. CA Cancer J Clin. 2021;71:7–33.
2. D’Oronzo S, Coleman R, Brown J, et al. Metastatic bone disease: pathogenesis and therapeutic options: up-date on bone metastasis management. J Bone Oncol. 2019;15:4.
3. Coleman RE. Clinical features of metastatic bone disease and risk of skeletal morbidity. Clin Cancer Res. 2006;12:6243s–6249s.
4. Selvaggi G, Scagliotti GV. Management of bone metastases in cancer: a review. Crit Rev Oncol Hematol. 2005;56:365–378.
5. Coleman RE. Bisphosphonates: clinical experience. Oncologist. 2004;9(suppl 4):14–27.
6. Cole JS, Patchell RA. Metastatic epidural spinal cord compression. Lancet Neurol. 2008;7:459–466.
7. Schiff D, O’Neill BP, Suman VJ. Spinal epidural metastasis as the initial manifestation of malignancy: clinical features and diagnostic approach. Neurology. 1997;49:452–456.
8. Costa L, Badia X, Chow E, et al. Impact of skeletal complications on patients’ quality of life, mobility, and functional independence. Support Care Cancer. 2008;16:879–889.
9. Coleman RE, Rubens RD. The clinical course of bone metastases from breast cancer. Br J Cancer. 1987;55:61–66.
10. Macedo F, Ladeira K, Pinho F, et al. Bone metastases: an overview. Oncol Rev. 2017;11:321.
11. O’Sullivan GJ, Carty FL, Cronin CG. Imaging of bone metastasis: an update. World J Radiol. 2015;7:202–211.
12. Heindel W, Gübitz R, Vieth V, et al. The diagnostic imaging of bone metastases. Dtsch Arztebl Int. 2014;111:741–747.
13. Chen L-C, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation. Proc Eur Conf Computer Vision. 2018:801–818.
14. Sun Y, Shi C. Liver tumor segmentation and subsequent risk prediction based on Deeplabv3+. IOP Conf Ser Mater Sci Eng. 2019;612:022051.
15. Song Y, Hu J, Wu Q, et al. Automatic delineation of the clinical target volume and organs at risk by deep learning for rectal cancer postoperative radiotherapy. Radiother Oncol. 2020;145:186–192.
16. Kingma DP, Ba J. Adam: a method for stochastic optimization. CoRR. 2014;abs/1412:6980.
17. Shen T, Huang F, Zhang X. CT medical image segmentation algorithm based on deep learning technology. Math Biosci Eng. 2023;20:10954–10976.
18. Herrmann P, Busana M, Cressoni M, et al. Using artificial intelligence for automatic segmentation of CT lung images in acute respiratory distress syndrome. Front Physiol. 2021;12:676118.
19. Koike Y, Yui M, Nakamura S, et al. Artificial intelligence-aided lytic spinal bone metastasis classification on CT scans. Int J Comput Assist Radiol Surg. 2023;18:1867–1874.
20. Schuller A, Martinez Fernandez A. Hahn’s clefts or fissures of the vertebrae. Rev Clin Esp. 1961;81:198–199.
21. Burns JE, Yao J, Wiese TS, et al. Automated detection of sclerotic metastases in the thoracolumbar spine at CT. Radiology. 2013;268:69–78.
22. Hammon M, Dankerl P, Tsymbal A, et al. Automatic detection of lytic and blastic thoracolumbar spine metastases on computed tomography. Eur Radiol. 2013;23:1862–1870.
23. Roth HR, Lu L, Liu J, et al. Improving computer-aided detection using convolutional neural networks and random view aggregation. IEEE Trans Med Imaging. 2016;35:1170–1181.
24. Chmelik J, Jakubicek R, Walek P, et al. Deep convolutional neural network-based segmentation and classification of difficult to define metastatic spinal lesions in 3D CT data. Med Image Anal. 2018;49:76–88.
25. Faghani S, Baffour FI, Ringler MD, et al. A deep learning algorithm for detecting lytic bone lesions of multiple myeloma on CT. Skeletal Radiol. 2023;52:91–98.
26. Noguchi S, Nishio M, Sakamoto R, et al. Deep learning-based algorithm improved radiologists’ performance in bone metastases detection on CT. Eur Radiol. 2022;32:7976–7987.