Обучение в области продольного многоразового склероза сегрегации на предметной передачи.

Introduction

Multiple sclerosis (MS) is a chronic and disabling neurological disease affecting more than 2 million people worldwide [ 1 ], and nearly 1 million in the USA, alone [ 2 ]. Multiple focal lesions in white matter (WM) characterized by inflammatory demyelination are the hallmarks of MS and can be visualized through magnetic resonance imaging (MRI). The presence of new lesions on MRI is considered a prognostic and predictive biomarker for the disease course [ 3 ]. As manual lesion segmentation is time‐consuming and prone to inter‐ and intra‐rater variability, many automatic segmentation methods have been developed [ 4 ]. After their successful application of deep learning models in computer vision, several convolutional neural networks (CNN) have been applied to the automatic segmentation of MS lesions [ 5 , 6 , 7 , 8 ]. A key difference in these models is that the major effort shifts from the development of image processing algorithms to the segmentation of a gold standard. With both increased computational power and a greater number of datasets for model training, these CNN models (e.g., ALL‐Net [ 9 ], Tiramisu [ 10 ], and ensembled approach [ 11 ]) are highly accurate and ranked in top levels in MS lesion segmentation challenges [ 12 ].

Nonetheless, these deep learning‐based models often lack generalizability and performance may not be optimal for unseen datasets, which may have variations in image acquisition (MRI vendor, scanner models, sequence parameters, image contrasts, noise levels, and resolution). In clinical settings, such variations in acquisition parameters are a recurring issue and limit the general applicability of the deep learning models. Transfer learning, where a pretrained model for different image datasets is fine‐tuned to perform a task for the unseen dataset, can address this issue [ 13 , 14 ]. Ghafoorian et al. [ 14 ] have shown that the performance of MS lesion segmentation using a CNN is improved after transfer learning, compared to a model trained without transfer learning that uses small training samples. Valverde et al. [ 13 ] have evaluated the performance of MS lesion segmentation using CNN with transfer learning, including variation in training data size and the number of layers used to fine‐tune. They have also shown that transfer learning using very few images or even a single image can achieve segmentation results comparable to fully trained CNN.

Patients with established MS undergo repeated MRIs for disease surveillance. New and enlarging lesions on follow‐up MRIs are considered possible breakthrough diseases and may require changes in disease‐modifying therapy [ 15 ]. Therefore, the segmentation of T2 lesions needs to be optimized not only for accuracy but also for consistency. However, due to a lack of generalizability, based on our knowledge, no deep learning‐based T2 lesion segmentation model has been deployed in the clinical setting. Instead, in most MS research studies, semiautomatic segmentation is common, as fully automatic methods are prone to segmentation inaccuracy. However, manual correction to automatic segmentation can improve consistency as manual segmentation alone may have poor inter‐/intra‐reader variability [ 12 ]. Furthermore, as automatic methods tend to make the same errors in the scans from the same subject, the manual correction may become redundant for the serial MRIs. We hypothesize that transfer learning at the individual level (subject‐wise transfer learning) can be used to overcome these issues and develop a deep learning‐based pipeline for MS lesion segmentation in longitudinal datasets with minimal manual labeling.

In this work, we propose a new pipeline to segment MS lesions for longitudinal MS studies based on the transfer learning of CNN to each subject. This subject‐wise transfer learning process can incorporate the subject's structural variability. We compare the performances of the proposed subject‐wise transfer learning to both traditional transfer learning‐based and without transfer learning methods.

Methods

The details of traditional transfer learning, proposed transfer learning methods, deep learning model architecture, datasets, experiment details, and evaluations are provided in this section.

Transfer Learning Methods

Traditional Transfer Learning

Transfer learning is a method of domain adaptation where model parameters of a pretrained model are fine‐tuned for new unseen datasets. Here, we are referring to this supervised process as traditional transfer learning (Traditional‐TL). This methodology is efficient and popular as it requires a small number of labeled data (e.g., manual lesion segmentations) for the unseen dataset. In a Traditional‐TL–based segmentation pipeline, the deep learning model is first trained on a large image dataset in which the model learns about the image features. In general, the first few layers learn about low‐level features such as edges, corners, and simple blob‐like structures, whereas the deep layers learn more complicated abstract features based on these simple features.

During the transfer learning, these learned image features help the model to adapt to a new dataset with minimal labeling. However, it is not clear how much data in the target domain is required to achieve sufficient accuracy. The pipeline of Traditional‐TL is shown in Figure  1a .

Depiction of different types of transfer learning. We have denoted subjects with S and time‐point as TP. For example, third time‐point of second subject is denoted as S2 TP3. As we need to find optimal performance within a subject's time‐points, we used only first time‐point (TP1) scans of each subject for training and validation dataset, and follow‐up time‐point scans (from time‐point 2 [TP2] to the last time‐point [TPN] where N is the number of time‐points) for testing. The first column represents traditional transfer learning where only once fine‐tuning is done for the unseen dataset. The second column represents the proposed single scan‐based transfer learning, where fine‐tuning is done for each subject and only the TP1 scan is used for fine‐tuning. The third column represents the proposed multiple‐scan based transfer learning, where fine‐tuning is done for each subject and scans from the pretraining dataset along with the TP1 scan is used for fine‐tuning.

Proposed Subject‐Wise Transfer Learning

In the proposed subject‐wise transfer learning methodology, the initial step is the same as Traditional‐TL, and the deep learning model is trained using a large dataset with lesion segmentation. Then, instead of fine‐tuning the pretrained model using the scans from all subjects in the unseen dataset, we fine‐tune each subject's first scan and then apply the fine‐tuned model to the follow‐up scans from the same subject to obtain follow‐up segmentation. As we are fine‐tuning the pretrained model to learn the features from each subject separately, we will need to repeat the fine‐tuning process for each subject within the unseen dataset. During each fine‐tuning, the first time‐point (TP1) scan is used in both training and validation. The model is learning custom, individualized features from each subject in this fine‐tuning process, and this custom fine‐tuning process may lead to improved segmentation performance for other scans from the same subject (e.g., follow‐up scans in longitudinal studies).

As a single scan from the unseen dataset is used for fine‐tuning, we proposed and compared two types of subject‐wise transfer learning to handle the effect of overfitting: (1) single‐scan transfer learning (SSTL) where the pretrained model is fined‐tuned using only TP1 scan of each subject from the unseen dataset, and the same TP1 scan is used for validation. Since the model can provide near‐perfect results on itself, we set the stopping criteria to be Dice ≥ 0.99. (2) Multi‐scan transfer learning (MSTL): Since SSTL uses the same image for both training and validation, there is a potential for overfitting. To avoid overfitting, the scans from the original pretraining dataset are included during the subject‐wise transfer learning in addition to the subject's TP1 scan. As in SSTL, the TP1 scan is used for validation, and the model is fine‐tuned until the model provides near‐perfect (Dice > 0.99) segmentation for TP1. The SSTL and MSTL pipelines are shown in Figure  1b,c .

Dataset

Three datasets were used for training and evaluating the segmentation pipelines.

MS Dataset for Initial Training (Dataset A)

The dataset consists of standardized MRI scans from 67 MS patients (68 studies) acquired in across multiple 3‐Tesla scanners (three Skyra, one Prisma; Siemens, Erlangen, Germany). The relevant scans were fluid‐attenuated inversion recovery (FLAIR, echo time [TE] = 393 ms, repetition time [TR] = 5000 ms, inversion time [TI] = 1800 ms, flip angle [FA] = 120°, 1‐mm isotropic voxel size), T1‐weighted pre‐contrast (TE = 2.98 ms, TR = 2300 ms, TI = 900 ms, FA = 9°, 1‐mm isotropic voxel size) and T2‐weighted (TE = 105–110 ms, TR = 6000–6100 ms, FA = 150°, in‐plane resolution 0.55–0.66 and slice thickness 4 mm) images.

We performed standard preprocessing including N3 intensity nonuniformity correction [ 16 ], in‐house intra‐subject registration using normalized mutual information, standard space linear registration [ 17 ] with ICBM template [ 18 ], skull extraction using brain extraction based on nonlocal segmentation technique [ 19 ], and intensity normalization using MINC toolkit's volume pool (MINC Toolkit, version 1.9.16, Montreal Neurological Institute, Montreal, Quebec, Canada, http://bic‐mni.github.io/ ). Template registration was further refined by ANTS nonlinear registration (2.1.0.post232‐gb152d, Philadelphia, Pennsylvania, USA, https://stnava.github.io/ANTs/ ). The template's probability maps (gray matter, WM, cerebrospinal fluid, lateral ventricles, and lesion) were nonlinearly and inversely transformed to the subject's image space. Since the signal intensity of lesions can be similar to that of the cortex on both FLAIR and T1‐weighted images, we included the priors to help identify lesions. After preprocessing, T2 lesions were manually segmented by two trained readers with above 10 years of segmentation experience, primarily on FLAIR scans. T1‐weighted scans were visually checked for the boundary with the cortex. T2‐weighted scans were also checked for lesions with long T2 relaxation time and lesions in areas where artifacts were common (e.g., brainstem). This dataset was used to pretrain the models.

Longitudinal MS Public Dataset for Transfer Learning (Dataset B)

We used the publicly available International Symposium on Biomedical Imaging (ISBI) 2015 longitudinal lesion segmentation challenge dataset (Dataset B) [ 12 ]. The dataset consists of 21 scans from five MS patients with at least four time‐points. The details of MRI acquisition and patients are described elsewhere [ 12 ]. Briefly, MRIs were acquired on a 3T Philips scanner from four relapsing‐remitting MS (RRMS) and one primary progressive MS (PPMS). The scan parameters for T1‐weighted were TR = 10.3 ms, TE = 6 ms, flip angle = 8°, and 0.82 × 0.82 × 1.17 mm 3 voxel; T2‐weighted image was a part of a dual echo sequence with TR = 4177 ms, TE = 80 ms, and 0.82 × 0.82 × 2.2 mm 3 voxel; and FLAIR with TI = 835 ms, TE = 68 ms, and 0.82 × 0.82 × 2.2 mm 3 voxel size. A total of 16 MR scans were used. We used FLAIR, T2‐weighted and T1‐weighted MRIs and applied the same preprocessing as in Dataset A. Dataset B contained manual lesion segmentation masks from two different raters. We used the manual lesion segmentation from the rater with 17 years of experience in MS MRI analysis.

Longitudinal MS Dataset (Dataset C)

The last dataset consists of 937 MS patients (3210 scans, same MRI parameters as Dataset A) seen in an observational setting. As a part of routine examinations, the patients underwent performance tests (manual dexterity test [MDT], walking speed test [WST], and processing speed test [PST]) as well as patient‐reported outcomes such as patient‐determined disease steps (PDDS). From these measures, confirmed disability progression (CDP) was determined as worsening in any measure confirmed at least over 6 months. Worsening was defined as a 20% increase for MDT and WST, a four‐point decrease in PST, and any increase in PDDS. The patients were grouped into those with CDP or nonprogressors. MRI scans acquired within 90 days of evaluations were included. We used FLAIR, T2‐weighted, and T1‐weighted MRIs. We investigated differences in statistical power in differentiating the lesion volume and assessed the association of CDP and lesion volume using a linear mixed effect (LME) model.

Experimental Details

Deep Learning Architecture

We used UNet [ 20 , 21 ] for the segmentation of T2 lesions because it provided robust results for different biomedical image segmentation [ 22 , 23 , 24 ], but in particular, lesion segmentation [ 25 , 26 ]. The detailed architecture is given in Figure  2 . UNet is a fully convolutional encoder–decoder network architecture with skip connections between the encoding and decoding modules for better information flow. The proposed UNet took 2D slices from co‐registered images (FLAIR, T2‐weighted, and T1‐weighted) and five probability maps (gray matter, WM, cerebrospinal fluid, lateral ventricles, and lesion probability maps) as an input, and it produced as its output a voxel‐wise binary segmentation mask of T2 lesions. The proposed model was taken from our previously published work [ 26 ], where we have shown that this set of inputs to UNet provides good contrast‐enhancing lesion segmentation. The architecture contained five depths of encoder and decoder layers, and each depth consisted of two convolution layers, each followed by batch normalization and a rectified linear unit (ReLU) activation, except for the last decoder layer. The output of the last decoder layer was attached with a sigmoid activation function, which produced the probability map of T2 lesions. The number of convolutional filters was 32, 64, 128, 256, and 512, respectively, at each depth of encoder. The up‐sampling path of the decoder network consisted of 512, 256, 128, 64, and 32 filters at each depth, respectively. The total number of parameters for UNet was 7.8 million. The network was implemented using Keras (version 2.2.2, California, USA, https://keras.io ) and Tensorflow (version 1.10.0, CA, USA, https://www.tensorflow.org/ ) framework. We trained and tested the models on a Linux machine with  12 GB of RAM and an NVIDIA Titan Xp GPU.

UNet architecture. The model has five depths of encoder and decoder. Blue arrows represent two‐dimensional convolution layers with batch normalization and rectified linear unit; red arrows represent the maximum pooling layer, and green arrows represent up‐sampling layer. Brown arrows skip connections between the encoder and decoder network. MRI contrasts and probability maps are the input, and the segmented lesion map is the output.

Initial Model Training Without Transfer Learning

Two basic UNet models were derived from Datasets A and B and served as comparators as well as initial pretraining.

First, a 2D‐UNet (UNet‐BaseA) was trained using Dataset A. This model is later used as the starting weight for transfer learning and as a reference to calculate the performance improvement of transfer learning pipelines. For the training of initial models, data from Dataset A were randomly split into 49 training subjects (50 scans) and 18 validation subjects (18 scans). The co‐registered MRIs, along with the probability maps, were given as the inputs, and the segmentation was used as the output. The loss function was the cross‐entropy loss, and the Adam optimizer (momentum value of 0.5 and a learning rate of 10 −3 ) was used. The batch size was 16, and the maximum number of epochs was 1000. At the end of each epoch, the model performance was evaluated, and the optimal model parameters, which gave the highest Dice coefficient on the validation data, became the final trained parameters. These optimal parameters were used as the initial parameters in the transfer learning models.

UNet‐BaseB was separately trained from Dataset B without pretrained weights, thus, without transfer learning. This model used the same architecture, loss function, and hyperparameters as UNet‐BaseA, but differed in that its training data came from Dataset B, which had longitudinal manual segmentation. As the goal of our study was to improve lesion segmentation for longitudinal studies, we used the TP1 scans (a total of five scans) only for training and validation. The remaining 16 scans from follow‐up time‐points (TP2, TP3, TP4, and TP5) were used to evaluate the model performance. UNet‐BaseB was trained with a fivefold cross‐validation strategy. Each fold has four training and one validation scans, where in each fold, one scan was set aside for validation and the rest were used to train the model. For example, when Subject 1 was selected for validation, the training dataset consisted of TP1 scans from Subjects 2–5. For each fold, the model performance was evaluated in terms of Dice after each epoch on the validation scan, and the model parameters with the highest Dice were saved. This model weight was used to predict lesions in 16 test scans for each fold.

Traditional Transfer Learning

We fine‐tuned UNet‐BaseA to Dataset B using a general transfer learning method using the TP1 scans (UNet‐TraditionalTL). We used fivefold cross‐validation where the TP1 scans from Dataset B were split into four training and one validation dataset. The same loss function and hyper parameter settings were used in each cross‐validation fold. The model parameters with the highest Dice coefficient on the validation data were saved as the final fine‐tuned parameters. We did not freeze any weights during the transfer learning; that is, all weights were modifiable.

Proposed Subject‐Wise Transfer Learning

Unlike UNet‐TraditionalTL where the base model (UNet‐BaseA) was transfer learned using the TP1 scans from the entire Dataset B, we fine‐tuned UNet‐BaseA using the TP1 scan from each subject in Dataset B and then applied the adapted model to the respective subject's follow‐up scans without cross‐validation scheme. In the case of SSTL, we fine‐tuned UNet‐BaseA using only a single TP1 scan. Therefore, for the five subjects in Dataset B, five different trainings were performed to obtain individualized transfer learning models. In the case of MSTL, along with the TP1 scan of Dataset B, scans from Dataset A were used during training. The purpose of including previous datasets was to reduce overfitting during fine‐tuning. Thus, MSTL also generated customized transfer learning models but in combination with data from Dataset A, which was in the training dataset; the training subset consisted of 50 cases from 49 subjects. As in SSTL, MSTL also used the first scan (TP1) for training. The same loss function and hyperparameter setting as UNet‐BaseA training were used in both SSTL and MSTL. During the fine‐tuning, the same TP1 scan was used for both training and validation. Dice metric was used to determine the optimal parameters. During the training, the weights were saved as intermediate weights when the validation Dice improved. These weights were then applied to the follow‐up scans as testing in “Testing of the trained models” to evaluate its effect on the follow‐up analyses.

Evaluation

Qualitative Evaluation

First, we apply the proposed SSTL in a subset of serial MRIs from Dataset C. In patients with new and stable lesions, segmentation masks were manually corrected to show that the manual correction on TP1 can improve segmentation at follow‐up time‐points. Manual correction was similar to manual segmentation, except that we started from the existing automatic lesion mask, and lesions with subtle errors at the border (two voxels) were not corrected. Completely missed or incorrect lesions were added and removed. We report qualitative visual results from images with various lesion activities (stable and new lesions).

Quantitative Evaluation Parameters

The proposed methods were evaluated based on the following quantitative parameters. The overall segmentation accuracy between manual and automatic segmentation is commonly measured using Dice coefficient as below:

where TP and FP denote the number of voxels correctly (true positive, TP) and incorrectly (false positive, FP) classified as lesion voxels, respectively; FN denotes the number of non‐lesion voxels incorrectly classified as lesion voxels (false negative, FN). The sensitivity of the proposed method is measured at a voxel basis as follows:

To measure the overall volumetric measure, the absolute volumetric difference AVD is calculated as follows:

where V t denotes the true volume for each scan, V p denotes the predicted volume, and | | denotes the absolute value. Dice and sensitivity metrics range from 0 (worst) to 1 (best), whereas AVD ranges from 0 (best) to ∞ (worst). Finally, we calculated lesion‐wise sensitivity and false discovery rate (FDR) among the segmented objects with a minimal size of 10 voxels and allowed for any overlap to be TPs.

Testing of the Trained Models

To assess the performance and compare the training strategies, we used 16 follow‐up scans (TP2, TP3, TP4, and TP5) from Dataset B as the test data. UNet‐BaseA was applied to all 16 test scans, and lesion segmentations were obtained without any transfer learning. UNet‐BaseB generated five trained weights from fivefold cross‐validation and were applied to each of the test scans. The evaluation metrics were averaged over the five predictions. Similarly, UNet‐TraditionalTL generated five fine‐tuned models from five cross‐validation folds, and the evaluation metrics were averaged. SSTL and MSTL also generated five fine‐tuned models, and these models were applied to the follow‐up scans from the respective subjects. Apart from these final fine‐tuned models, the intermediate fine‐tuned models were also applied to the test scans to evaluate the change in segmentation performance along with the fine‐tuning process. These evaluations will depict the changes in the segmentation performance of later time‐points as the model learns the image features of each subject from TP1.

Statistical Analysis Using Dataset C

We compared lesion volume over time between progressors (i.e., CDP) and nonprogressors and evaluated statistical power using 3210 scans from Dataset C. We independently applied the UNet‐BaseA for all of the scans in Dataset C. Then using the first segmentation mask of each subject, SSTL training was performed and fine‐tuned model was applied for other time‐points. The lesion volumes from follow‐up time‐points were used in this statistical analysis.

To quantify the extent of statistical benefit in SSTL, we applied the LME model. The lesion volume was modeled in a separate LME model for UNet‐BaseA and SSTL. LME's fixed effects were follow‐up duration (years), CDP status, and an interaction effect of CDP and follow‐up interval. The random effect was the subject. The equation of LME is as follows:

where L V i j is the lesion volume at follow‐up time‐point i for subject j , in milliliters; Interval is the follow‐up interval in years; CDP is the binary CDP status of MS patients; [ Interval × CDP ] is the interaction between the Interval and CDP status; β 0 is the intercept, β 1 is the coefficient for CDP , β 2 is the coefficient for Interval , β 3 is the coefficient for interaction effect ( Interval × CDP ); γ j random effect coefficient for subject j ; and ε i j is the error for time‐point i for subject j .

We compared the t values for β 3 between SSTL and UNet‐BaseA where t values are the ratio of estimate and its standard error (SE) and where a larger t value represents increased statistical power.

Results

We applied the SSTL in our longitudinal Dataset C, and two example subjects are presented in Figure  3 . The first column represents the TP1 scan, and the second column represents the TP2 scan within each subject. In each subject, the first rows represent FLAIR images without mask overlay; the second rows represent UNet‐BaseA segmentation overlay; the third rows represent manually corrected mask for TP1, and for TP2, the output of SSTL is overlaid. The yellow arrows indicate the lesions that were missed by UNet‐BaseA (C, D, I, and J). After lesions were manually added at TP1 (E and K), the SSTL model was fine‐tuned and applied at TP2 (F and L). The manually added lesions were appropriately segmented in both subjects at TP2. The blue arrow in Subject 2 is pointing to a new lesion (K) at TP2, which was successfully segmented by UNet‐SSTL despite the fact that the model fine‐tuned from the image without that lesion.

Application of proposed single‐scan transfer learning (SSTL) method for longitudinal Dataset C with similar scan parameters as Dataset A. The images from subjects are separated in boxes. The two columns are figures from Subject 1 and the next two columns are figures from Subject 2. The first column within the subjects represents the first time‐point (TP1) scan and the second column represents the second time‐point (TP2) scan. The first row represents the fluid‐attenuated inversion recovery (FLAIR) without any mask overlay; the second row represents UNet‐BaseA segmentation overlay on MRI in red, and third row represents the input and output of SSTL algorithm after manual segmentation correction at baseline. The segmented lesions are marked as red. The yellow arrows in C, D, F, I, J, and L point to the lesions detected by UNet‐SSTL in TP2 after manual corrections on TP1 but were missed by UNet‐BaseA. The blue arrow in L points to the new lesion in TP2 which is not missed by UNet‐SSTL.

Table  1 shows the model performance from base and transfer learned models. SSTL and MSTL showed the highest Dice, sensitivity, and absolute volume difference (AVD) among the analyzed methods. There was a significant improvement in Dice and sensitivity in both SSTL and MSTL compared to UNet‐TraditionalTL or base models (UNet‐BaseA and UNet‐BaseB) (all p  < 0.05, using two‐sided t ‐tests). The performance of all of the models on each of the 16 test scans are shown in Figure  4 .

TABLE 1 Mean segmentation performance of different models on test set of Dataset B. Models Dice Sensitivity AVD UNet‐BaseA 0.591 0.775 0.624 UNet‐BaseB 0.660 0.777 0.769 UNet‐TraditionalTL 0.769 0.745 0.178 UNet‐SSTL 0.792
a

0.790
a

0.152 UNet‐MSTL 0.781 0.789 0.147
a

Abbreviations: AVD, absolute volume difference; MSTL, multi‐scan transfer learning; SSTL, single‐scan transfer learning; TL, transfer learning.
a
Highest performance.

Mean segmentation performance of different models on test set of Dataset B.

Abbreviations: AVD, absolute volume difference; MSTL, multi‐scan transfer learning; SSTL, single‐scan transfer learning; TL, transfer learning.

Highest performance.

Comparison of segmentation performance in terms of three quantitative parameters (Dice, sensitivity, and absolute volume difference) between base models (UNet‐BaseA and UNet‐BaseB) and transfer learning (TL) methods (UNet‐TraditionalTL, UNet‐multi‐scan transfer learning [MSTL] and UNet‐single‐scan transfer learning [SSTL]) in Dataset B. Each subject is shown by colors and time‐points by shape, whereas the boxplot shows the outliers, extreme, quartiles, and median values. Baseline time‐point (TP1) was used for training and validation, and the testing results for TP2‐TP5 are reported. TP, time‐point.

Figure  5 shows the performance measures between the proposed UNet‐MSTL and UNet‐SSTL for each individual in Dataset B with the reference measures from UNet‐BaseA. The maximum Dice improvement was 0.21 (Subject 4 TP2), and the minimum Dice improvement was 0.04 (Subject 1 TP2) in UNet‐MSTL. For UNet‐SSTL also, the maximum Dice improvement was 0.19 (Subject 4, TP2), and the minimum Dice improvement was 0.05 (Subject1, TP2). In the case of AVD, the highest improvement was 0.56 (Subject 5, TP4, UNet‐MSTL) and the minimum improvement was 0.01 (Subject 2, TP3, UNet‐SSTL). Figure  5 shows that the improvements were similar in both UNet‐SSTL and UNet‐MSTL. Also, there was no statistically significant difference ( p  > 0.05, using a two‐sided t ‐test) in performances in terms of Dice, sensitivity, and AVD.

Comparison of segmentation performance improvement in terms of three quantitative parameters (Dice, sensitivity, and absolute volume difference) between the proposed transfer learning (TL) methods (UNet‐multi‐scan transfer learning [MSTL] and UNet‐single‐scan transfer learning [SSTL]) in Dataset B. The improvements are measured compared with base model segmentation (UNet‐BaseA). Each column represents the subject, and each row represents different segmentation performance metrics. Each proposed segmentation model is coded with different colors. Baseline time‐point (TP1) was used for training and validation, and the remaining follow‐up time‐points (TP2, TP3, TP4, and TP5) were used to evaluate the model performance. FDR, false discovery rate; TP, time‐point.

Figure  6 shows the differences in segmentation performance in terms of Dice, sensitivity, and AVD during fine‐tuning of model UNet‐SSTL (top) and UNet‐MSTL (bottom) on Dataset B. Dice and sensitivity on the follow‐up scans improved as the Dice on the TP1 scans increased. The patterns of performance improvement on the follow‐up scans from UNet‐SSTL were similar to those from UNet‐MSTL. Each column in Figure  6 presents different quantitative parameters. The different time‐points (TP2, TP3, TP4, and TP5) are represented by different shapes and subjects by colors. The Dice coefficient measurements on TP1 scans using intermediate weights are presented on the x ‐axis, and the quantitative measurements of the predicted segmentations using corresponding intermediate weights are presented on the y ‐axis.

The segmentation performance in terms of three quantitative parameters (Dice, sensitivity, and absolute volume difference) due to single‐scan transfer learning (top) and multi‐scan transfer learning (bottom) are depicted for Dataset B. Each column represents different segmentation performance metrics. Each time‐point is coded with different colors. The transfer learning from base model was done using time‐point 1 (TP1) scans of each subject. The remaining 16 scans from follow‐up time‐points (TP2, TP3, TP4, and TP5) were used to evaluate the model performance.

The details of patient characteristics in the longitudinal MS dataset (Dataset C) are given in Table  2 . For LME modeling, the SSTL method showed a significance for the interaction effect: [ Interval × CDP ] with the coefficients for the intercept β
0  = 20.54 (SE = 0.94), for the interaction term β
1  = 0.12 (SE = 0.06, t  = 2.02, p  = 0.04), for the interval β
2  = 0.54 (SE = 0.07, p  < 0.001), whereas UNet‐BaseA did not show significance in the following: coefficients for the intercept β
0  = 19.16 (SE = 0.90), for the interaction term β
1  = 0.16 (SE = 0.11, t  = 1.62, p  = 0.10), and for the interval, β
2  = 0.30 (SE = 0.04, p  < 0.001).

TABLE 2 Baseline characteristics in longitudinal multiple sclerosis dataset (Dataset C). Total ( n  = 937) Progressors ( n  = 354) Nonprogressors ( n  = 583) Age, years 46.4 ± 11.3 47.2 ± 11.5 45.9 ± 11.2 Female 673 (72%) 258 (73%) 415 (71%) Disease duration, years 13.3 ± 9.9 16.8 ± 9.7 11.3 ± 9.5 Whole‐brain fraction 0.813 ± 0.028 0.809 ± 0.029 0.815 ± 0.027 Spinal cord area, cm 2
77.4 ± 9.1 76.6 ± 9.6 77.8 ± 8.7 Gray matter fractions  0.463 ± 0.018 0.463 ± 0.019 0.463 ± 0.018 Patient determined disease steps 1.8 ± 1.9 1.9 ± 1.8 1.7 ± 1.9 Walking speed test z ‐score −0.98 ± 3.41 −1.23 ± 3.75 −0.84 ± 3.19 Manual dexterity test z ‐score −0.93 ± 1.82 −1.13 ± 1.72 −0.8 ± 1.86 Processing speed test z ‐score −0.49 ± 1.2 −0.71 ± 1.21 −0.37 ± 1.18
Note : z‐scores are age‐, sex‐, and education‐corrected to normal controls [ 29 ]. All the data represent mean ± standard deviation unless otherwise indicated. Abbreviation: n , number of subject.

Baseline characteristics in longitudinal multiple sclerosis dataset (Dataset C).

Note : z‐scores are age‐, sex‐, and education‐corrected to normal controls [ 29 ]. All the data represent mean ± standard deviation unless otherwise indicated.

Abbreviation: n , number of subject.

Discussion

In this work, we proposed a subject‐wise transfer learning method for longitudinal MS datasets to improve lesion segmentation performance. In traditional transfer learning, the model parameters are fine‐tuned to the target dataset using all of its scans. In contrast, the model parameters are fine‐tuned to a single subject in the proposed method. We hypothesized that this would improve segmentation accuracy compared with traditional transfer learning, as the model learns about the subject's structural information. The quantitative results showed that the proposed transfer learning methods performed better than the traditional transfer learning in all three segmentation metrics.

We showed that the improvement of SSTL was statistically significant. For longitudinal comparisons, small lesions that did not significantly influence overall Dice were important and captured by the sensitivity measure. The average sensitivity improvement with our proposed method was 5.9%, and average AVD improvement was 17.4%. We also presented the performance for individual scans, which showed improvement over traditional transfer learning by quantitative metrics.

We proposed two methods for subject‐based transfer learning. We rationalized that MSTL would reduce overfitting of the initial model to the target subject, and thus, reductions in Dice, sensitivity, and AVD would be observed. We also expected that overfitting will be insensitive to new T2 lesions. However, we did not observe such patterns in our experiments (Figures  3 and  4 ). In this process, other scans from the pretrained dataset were used along with the baseline scan from the target subject to fine‐tune the base model. The intuition was that the information from other scans would restrict the model from overfitting toward the target subject scan. On the other hand, we hypothesized that SSTL would overfit since only the single scan from the target subject was used to fine‐tune. However, there were no statistically significant differences in Dice, sensitivity, and AVD between these methods. Therefore, the quantitative as well as visual results showed that the model did not necessarily overfit even with a single scan.

We also presented the segmentation performance change in terms of three quantitative parameters (Dice, sensitivity, and AVD) during fine‐tuning of the proposed SSTL and MSTL methods in Figure  6 . We see that the performance measures improved for later time‐point as Dice for TP1 scans increased; as the Train Dice from TP1 increased in x ‐axis, there was a clear overall trend for the follow‐up Dice ( y ‐axis) to increase. This also suggested that the proposed pipelines were not overfitting to TP1 scans, but instead, likely learned subject‐specific image features and improved the segmentation for later time‐points.

For the longitudinal Dataset C, we compared the lesion volume between progressors (CDP) and nonprogressors using UNet‐BaseA and the proposed SSTL method. As expected, progressors had greater lesion volume increases compared to nonprogressors in both models. The p values reached significance ( p  = 0.04) with the proposed method, whereas the basic UNet‐BaseA method did not reach ( p  = 0.10). Therefore, these results suggest that the proposed SSTL increases statistical power for the measurement of longitudinal lesion volume. This result may have significant implications when applying automated lesion analysis for research applications and potentially in clinical practice.

In clinical trials and other studies where accurate lesion masks are needed from a large population, proposed strategies such as SSTL or MSTL may produce accurate lesion masks from the follow‐up scans (Figure  3 ). The method may be used with manual correction of baseline lesion masks, but the proposed strategies are not applicable to cross‐sectional studies where there are no labels. The proposed SSTL and MSTL improved the performance metrics, but the improvement varied across the datasets (Figures  5 and  6 ), which could be attributed to the image quality, manual segmentation quality, image registration quality, baseline lesion volume, and change in lesion volumes. However, we could not associate these potential factors with the variance.

Currently, the application of deep learning in radiology is still in its infancy. Though there is high potential for building deep learning‐based pipelines to help radiologists and clinicians in diagnosis and prognosis, its real‐world deployment is currently limited. The main reason for this is a lack of generalized deep learning models for multi‐center applications. Large annotated training data [ 27 , 28 ] that fully represent the overall population can create generalized models, but the annotation of large volumes is time‐consuming and costly. Traditional transfer learning, where the model is fine‐tuned to a small volume of datasets from different centers, may cause overfitting problems. Other ways of improving automatic segmentation methods in terms of modified architecture, learning rate, and contrastive learning may still require manual segmentation verification and corrections. In our work, instead of building a single generalizable model using a large training dataset, we addressed the issues by fine‐tuning each subject and learning each subject's features. The proposed method can reduce the manual corrections to the baseline images and eliminate corrections for the later time‐points.

Though we did not find overfitting in the test subjects, based on false positive and false negative lesions, testing on even larger datasets (> 1000 subjects) may be needed. Another limitation of this study is that the methods are only applicable to longitudinal MS data. The effectiveness of other longitudinal datasets where scans may contain large volumes of labels (e.g., bone, muscle segmentation from MRI) may need to be tested. Similarly, cross‐sectional data or data without labels, and therefore the requirement for expert reviewers remains.

In conclusion, we have demonstrated two subject‐based transfer learning frameworks to segment lesions in MS. To our knowledge, the proposed method is the first fully automatic subject‐wise transfer learning methodology for a longitudinal dataset. The proposed methodology shows significant improvement in lesion segmentation compared to traditional transfer learning methods and supervised deep learning methods without transfer learning. Our proposed methodology is flexible and has the potential to be directly applied to any deep learning architectures or longitudinal data segmentation applications.

Conflicts of Interest

Sibaji Gaj is a current employee of Qualcomm. Daniel Ontaneda received research support from the National Institutes of Health, National Multiple Sclerosis Society, Patient Centered Outcomes Research Institute, Race to Erase MS Foundation, Genentech, Bristol Myers Squibb, and Novartis and consulting fees from Bristol Myers Squibb, Genentech/Roche, Novartis, and Contineum Therapeutics. Kunio Nakamura has received personal fees for licensing from Biogen, for scientific advisory meetings from INmune Bio; received research support from the National Institutes of Health (NeuroNext, National Institute of Neurological Disorders and Stroke, National Institute of Arthritis and Musculoskeletal and Skin Diseases), Department of Defense, Patient‐Centered Outcomes Research Institute (PCORI), National Multiple Sclerosis Society, Sanofi Genzyme, Novartis, Biogen, Synaptogenix, and Bristol Myers Squibb.

References