Introduction
Multiple sclerosis (MS) falls within the spectrum of central nervous system (CNS) demyelinating disease and is associated with both acute and progressive clinical courses, 1 , 2 resulting in varying degrees of neurological impairment. 3 , 4 Individuals asymptomatic for MS may also be recognized with typical T2-hyperintensities within the CNS found incidentally when an MRI study is performed for other reasons. 5 - 7 Observed intra-axial and intramedullary high-signal anomalies representing multi-focal regions of autoimmune related inflammation due to misrecognition of normal tissue by the immune system on magnetic resonance imaging (MRI) studies are critical to the diagnosis of MS. 1 Such MRI findings are also meaningful for clinical surveillance in the context of disease modifying therapy treatment or more conservative approaches to care.
High-signal anomalies associated with MS are easily recognized on MRI due to the remarkable sensitivity of the technology. Since inception, images have been provided in grayscale, displaying a spectrum of shades of gray that represent signal intensities from normal anatomical structures along with the pathological effects of disease. Observed contrast differences in signal intensities enable differentiation between different tissue types but also may indicate the advancement of disease. The transformation of grayscale MRI images to color may provide further clarity in rapidly identifying changes associated with disease advancement. Reviewing full colorized image data may also readily provide additional details that are less apparent in grayscale images.
Both community and academic neurologists frequently order MRI studies to better understand symptoms described by patients. When results are discussed, the images are routinely displayed and explained to the patient. To many, the provided images appear intimidating, not easily intuitive, and are difficult to understand. The application of colorized MRI data may serve as an aesthetic improvement, creating an experience that can be more engaging and visually appealing. Data in this format may also be of greater benefit to clinicians that have less experience in reviewing MRI data.
In this report, we describe the clinical experience of a person with MS and transform grayscale MRI data to color using generative artificial intelligence (AI) to determine the potential value of this technique in better understanding the imaging features of MS. A comparison of the observed T2-hyperintensities in color to acquired quantitative MRI data are also provided to assess the effectiveness of the colorization process.
Methods
Clinical experience description
A 31-year-old, right-handed, White man was seen in consultation at The University of Texas Southwestern Medical Center in Dallas, Texas, following complaints of headaches that began after head trauma related to military service. His headache was accompanied with nausea and vomiting, persisting for a few days. His history was negative for prior events of vision loss or focal motor/sensory complaints. A comprehensive neurological examination was performed and found to be normal. A brain MRI study was performed that revealed features highly suggestive of inflammatory CNS demyelination based on the size, number, morphology and spatial distribution of T2-hyperintensities ( Figure 1 ). A dedicated MRI study of the cervical spinal cord was unremarkable for intramedullary lesions. Cerebrospinal fluid analysis revealed the presence of >2 unique oligoclonal bands. Figure 1. (A) Grayscale axial 3.0 Tesla fluid attenuated inversion recovery image of the brain from January 2014 demonstrating T2-hyperintense foci with spatial distribution characteristic for multiple sclerosis. (B) Transformation of the grayscale image to color using generative artificial intelligence. Note the color differences in select lesions highlighted with some appearing more yellow (yellow arrows) and others white (white arrows).
The patient opted to participate in the first randomized clinical trial aimed at preventing or delaying the onset of a first clinical event related to MS. 8 Within the study, he was randomized to the active treatment arm, dimethyl fumarate (Tecfidera®) and remained symptom free. However, following the 96-week study, he developed a new spinal cord lesion that was associated with long tract motor and sensory symptoms despite full compliance with dimethyl fumarate. Given the evolution of symptoms resulting from inflammatory CNS demyelination and prior evidence of new lesion development on MRI, he fulfilled the diagnosis of relapsing-remitting MS. Following acute management of symptoms that resulted in full recovery, his disease modifying therapy was switched to ocrelizumab (Ocrevus®).
Artificial intelligence colorization technique
A deep learning network, trained on a diverse image dataset was created. The preprocessing pipeline converted the input of red, green, and blue (RGB) images into grayscale while maintaining the RGB channel structure. During training, the network learned the mapping between grayscale and full-color version, with the loss function calculated by comparing the network output against the original RGB values from the training images. Grayscale Digital Imaging and Communications in Medicine files were then converted into a feature representation via a convolutional neural network. Simultaneously, an accompanying text prompt was transformed into a feature vector using natural language processing techniques. The image and text feature vectors were subsequently entered into a deep generative network, which was trained for function approximation from the MRI grayscale image and text features to generate a colorized version of the image.
Magnetic resonance imaging data
In January 2014, a 3.0 Tesla (T) brain MRI was performed containing a 2-dimensional (2D) fluid attenuated inversion recovery (FLAIR) sequence, gadolinium-enhanced 3-dimensional (3D) T1-weighted gradient echo sequence, and a 2D T2-weighted sequence. For comparison, images acquired from a repeat 3.0 T MRI performed more than 10 years later in June 2024 were used. The following sequences were acquired: i) 3D T2-weighted sequence, ii) 3D FLAIR sequence, and iii) a non-enhanced 3D T1-weighted gradient echo sequence. Quantitative 3D data using an interleaved Look-Locker acquisition sequence with T2 preparation pulse was also acquired to obtain quantitative T1, T1, and proton density values from regions of interest (SyntheticMR, Linköping, Sweden).
Results
The transformation of grayscale MRI FLAIR images to color via generative AI revealed heterogeneity in the T2-hyperintensities present with some of the lesions having a more yellow appearance and others appearing more white in color ( Figure 1 ). Data from other accompanying sequences (T1-weighted and T2-weighted sequences) at either time point did not appear to provide a clear explanation for the observed variations in lesion color ( Figure 2 ). Over a time period of 10 years, and despite treatment with a disease modifying therapy, radiological advancement was observed with the interval development of new T2-hyperintense lesions present along with a reduction in total brain volume. Previously identified lesions from the first MRI study ( Figure 1 ) with distinct colors appeared to have a similar appearance in 2014, although enlargement of some of the existing lesions were present. For those lesions having a more yellow appearance, lower quantitative R1 (longitudinal relaxation rate represented by 1/T1 relaxation time) and R2 (transverse relaxation rate represented by 1/T2 relaxation time) and higher proton density (measure of the concentration of hydrogen protons) values were identified ( Figure 3 ). Figure 2. (A) Color transformed axial 3.0 Tesla (T) fluid attenuated inversion recovery (FLAIR) image of the brain with corresponding T2-weighted and post-contrast T1-weighted images at the same level from January 2014. (B) Color transformed axial 3.0 T FLAIR image with corresponding T2-weighted and T1-weighted images at the same level from June 2024. Note the changes in lesion number and size along with the generalized reduction in brain volume. Figure 3. Color transformed axial 3.0 Tesla fluid attenuated inversion recovery image of the brain from June 2024 with corresponding quantitative R1, R2, and proton density mean and standard deviation values from selected lesions. Note the differences in quantitative values when comparing lesions appearing more yellow than white.
Discussion
The first photograph in color was produced by James Clerk Maxwell in 1861, a Scottish physicist, using red, blue, and yellow filters. A colored image of a tartan ribbon was formed after combining the individual images in red, green, and blue-violet into one color composite. 9 Beyond pursuing his interest in trichromatic theory, more simply, the colored image provided greater realism and detail when compared to black and white.
In this report, we demonstrated the potential value of transforming grayscale MRI images into color, visually enhancing the anatomical data and possibly providing more information that may inform on the impact of disease related to MS within the CNS. Colorizing MRI data appears to offer more intuitive data to the observer. In addition, the application of generative AI color rendering techniques may enable better appreciation of findings not apparent in grayscale. When correlated with quantitative metrics, the more yellow-colored lesions appeared to have lower R1 and R2 values and higher proton density values when compared to lesions that were white. Higher T1 relaxation time values are observed within persistent T1 “black holes” on MRI, given the reduction in tissue integrity. 10 Collectively, our observation of higher R1 and R2 findings and higher proton density values, representing a higher water content, suggests that variations in color may better allow for the appreciation of greater demyelination and axonal compromise within lesions. Lesions related to MS are well known to evolve in shape, size, and location on MRI 11 - 13 and the degree of tissue compromise within the selected lesions studied here also appears to vary widely. As generative AI methodology continues to improve at a rapid pace and given the increasing prevalence of use within healthcare, advancements in the near immediate analysis of existing clinical MRI data may be possible.
Grayscale images are ideal in demonstrating contrast, enabling the differentiation of tissue types or complexities within lesions. However, grayscale values within voxels on MRI are only associated with a single variable, intensity. The visual evaluation of data in grayscale has been shown to impact our perceptions of size estimation as well as perceived changes. 14 - 16 These limitations in our visual system is the result of Mach bands where a perceived increase in contrast is observed at boundary regions of contrasting shades 17 and result from visual processing of lateral inhibitory optic neurons that are involved in enhancing contrast and edge detection. 18 On the other hand, color allows for the inclusion of three variables: hue, lightness (intensity), and saturation. 19 As a result, the visual system processes hue and saturation data, reducing the Mach band effect. Intuitively, these additional variables may change the overall experience by viewers, improving the appreciation of imaging data in color and potentially offering additional details not readily apparent when viewing conventional grayscale data. The use of color in comparison to use of black and white in multimedia has also been associated with better attention and memory performance 20 and object recognition. 21
Previous work aimed at transforming medical imaging data into color has been limited. One of the earliest attempts aimed to transform MRI data to better study the female pelvis. 22 Color-coded MRI multiparametric maps have also been studied for the diagnosis of prostate cancer. 23 The application of automated color-coding on longitudinal MRI imaging data has been studied in both MS 24 , 25 and brain metastases 26 to more easily recognize interval radiological changes. In these efforts, color is applied to more clearly emphasize expansion or reduction of the region of interest. Here, an example of the full transformation from grayscale to color is provided through generative AI. Although many AI platforms exist to accomplish such tasks, each is unique, yielding a spectrum of outputs, with the vast majority lacking biological meaning. Being able to generate the ideal hues, intensities, and saturation that hold scientific significance is the fundamental goal.
Conclusion
The application of color in neuroimaging may provide greater contextual clarity in comparison to current grayscale images for those without reduced sensitivity to red, green, and blue light. With advancing AI methods and capabilities along with the additional data that color provides in comparison to grayscale, perhaps new insights into the biology of disease may be recognized. More studies are needed not only to clarify the most ideal augmented intelligence methodologies for color transformation but also our understanding of how the software systems accomplish this task. Modifying what we measure in people with chronic conditions and how we present the data may be of greater value than conventional approaches typically used in the study, education, and care of people with MS and other neurological conditions.
Acknowledgements
We gratefully acknowledge the contribution of data for the person highlighted in this report. His generosity has been invaluable in the advancement of our research and development of new innovations within our laboratory.
Author contributions: Darin T. Okuda: Was responsible for conceptualization; Data curation; Formal analysis; Methodology; Project administration; Preparation of the original draft.
Christine Lebrun-Frenay: Was responsible for Conceptualization; Formal analysis; Methodology; Interpretation of the Study Results; Review and editing of the manuscript.
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: D.T.O. received personal compensation for consulting and advisory services from Eisai, EMD Serono, Genentech, Genzyme/Sanofi, Immunic Therapeutics, Moderna, and RVL Pharmaceuticals, Inc., Zenas BioPharma and research support from Biogen Inc., EMD Serono/Merck, and Novartis. D.T.O. has issued national and international patents along with pending patents related to the enclosed work as well as other developed technologies. D.T.O. received royalties for intellectual property licensed by The Board of Regents of The University of Texas System. D.T.O. is the Founder of Revert Health Inc. C.L.-F. reports no disclosures.
Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.
Ethical statement
Ethical approval
The study was approved by The University of Texas Southwestern Medical Center. Institutional Review Board and have, therefore, been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and later amendments.
Consent for publication
Written consent for publication from the individual described in this report was acquired.
ORCID iDs
Darin T. Okuda https://orcid.org/0000-0002-6499-1523
Christine Lebrun-Frenay https://orcid.org/0000-0002-3713-2416
References
1 Thompson AJ Banwell BL Barkhof F , et al. Diagnosis of multiple sclerosis: 2017 revisions of the McDonald criteria . Lancet Neurol . 2018 ; 17 : 162 - 173 . doi: 10.1016/S1474-4422(17)30470-2 29275977
2 Hua LH Donlon SL Sobhanian MJ Portner SM Okuda DT . Thoracic spinal cord lesions are influenced by the degree of cervical spine involvement in multiple sclerosis . Spinal Cord . 2015 ; 53 : 520 - 525 . doi: 10.1038/sc.2014.238 25582716
3 Lebrun-Frenay C Kantarci O Siva A , et al. Radiologically isolated syndrome . Lancet Neurol . 2023 ; 22 : 1075 - 1086 . doi: 10.1016/S1474-4422(23)00281-8 37839432
4 Okuda DT Lebrun-Frénay C . Radiologically isolated syndrome in the spectrum of multiple sclerosis . Mult Scler . 2024 ; 30 : 630 - 636 . doi: 10.1177/13524585241245306 38619142 PMC11071642
5 Okuda DT Mowry EM Beheshtian A , et al. Incidental MRI anomalies suggestive of multiple sclerosis: the radiologically isolated syndrome . Neurology . 2009 ; 72 : 800 - 805 . doi: 10.1212/01.wnl.0000335764.14513.1a 19073949
6 Okuda DT . Unanticipated demyelinating pathology of the CNS . Nat Rev Neurol . 2009 ; 5 : 591 - 597 . doi: 10.1038/nrneurol.2009.157 19806149
7 Lebrun-Frenay C Okuda DT Siva A , et al. The radiologically isolated syndrome: revised diagnostic criteria . Brain . 2023 ; 146 : 3431 - 3443 . doi: 10.1093/brain/awad073 36864688 PMC11004931
8 Okuda DT Kantarci O Lebrun-Frenay C , et al. Dimethyl fumarate delays multiple sclerosis in radiologically isolated syndrome . Ann Neurol . 2023 ; 93 : 604 - 614 . doi: 10.1002/ana.26555 36401339
9 Longair MS . Maxwell and the science of colour . Philos Trans A Math Phys Eng Sci . 2008 ; 366 : 1685 - 1696 . doi: 10.1098/rsta.2007.2178 18222905
10 Thaler C Faizy T Sedlacik J , et al. T1- thresholds in black holes increase clinical-radiological correlation in multiple sclerosis patients . PLoS One . 2015 ; 10 : e0144693 . doi: 10.1371/journal.pone.0144693 26659852 PMC4676682
11 Okuda DT Moog TM McCreary M , et al. Utility of shape evolution and displacement in the classification of chronic multiple sclerosis lesions . Sci Rep . 2020 ; 10 : 19560 . doi: 10.1038/s41598-020-76420-8 33177565 PMC7658967
12 Elliott C Wolinsky JS Hauser SL , et al. Slowly expanding/evolving lesions as a magnetic resonance imaging marker of chronic active multiple sclerosis lesions . Mult Scler . 2019 ; 25 : 1915 - 1925 . doi: 10.1177/1352458518814117 30566027 PMC6876256
13 Moog TM McCreary M Wilson A , et al. Direction and magnitude of displacement differ between slowly expanding and non-expanding multiple sclerosis lesions as compared to small vessel disease . J Neurol . 2022 ; 269 : 4459 - 4468 . doi: 10.1007/s00415-022-11089-9 35380254
14 Huynh TN Johnson T Poder L Joe BN Webb EM Coakley FV . T1 pseudohyperintensity on fat-suppressed magnetic resonance imaging: a potential diagnostic pitfall . J Comput Assist Tomogr . 2011 ; 35 : 459 - 461 . doi: 10.1097/RCT.0b013e31822227c3 21765301 PMC3141817
15 Adelson EH . Perceptual organization and the judgment of brightness . Science . 1993 ; 262 : 2042 - 2044 . doi: 10.1126/science.8266102 8266102
16 Fiedler A Moore CM . Illumination frame of reference in the object-reviewing paradigm: a case of luminance and lightness . J Exp Psychol Hum Percept Perform . 2015 ; 41 : 1709 - 1717 . doi: 10.1037/xhp0000123 26280265 PMC4666744
17 Ratliff F Milkman N Rennert N . Attenuation of Mach bands by adjacent stimuli . Proc Natl Acad Sci U S A . 1983 ; 80 : 4554 - 4558 . doi: 10.1073/pnas.80.14.4554 6576350 PMC384078
18 Lotto RB Williams SM Purves D . Mach bands as empirically derived associations . Proc Natl Acad Sci U S A . 1999 ; 96 : 5245 - 5250 . doi: 10.1073/pnas.96.9.5245 10220451 PMC21849
19 Farrell EJ . Visual interpretation of complex data . IBM Syst J . 1987 ; 26 : 174 - 200 . doi: 10.1147/sj.262.0174
20 Farley FH Grant AP . Arousal and cognition - memory for color versus black and white multimedia presentation . J Psychol . 1976 ; 94 : 147 - 150 . doi: 10.1080/00223980.1976.9921410
21 Bramao I Reis A Petersson KM Faísca L . The role of color information on object recognition: a review and meta-analysis . Acta Psychol . 2011 ; 138 : 244 - 253 . doi: 10.1016/j.actpsy.2011.06.010 21803315
22 Brown HK Hazelton TR Fiorica JV Parsons AK Clarke LP Silbiger ML . Composite and classified color display in mr imaging of the female pelvis . Magn Reson Imaging . 1992 ; 10 : 143 - 154 . doi: 10.1016/0730-725x(92)90384-C 1545674
23 Kather JN Weidner A Attenberger U , et al. Color-coded visualization of magnetic resonance imaging multiparametric maps . Sci Rep . 2017 ; 7 : 41107 . doi: 10.1038/srep41107 28112222 PMC5255548
24 Zopfs D Laukamp KR Paquet S , et al. Follow-up MRI in multiple sclerosis patients: automated co-registration and lesion color-coding improves diagnostic accuracy and reduces reading time . Eur Radiol . 2019 ; 29 : 7047 - 7054 . doi: 10.1007/s00330-019-06273-x 31201526
25 Dahan A Pereira R Malpas CB Kalincik T Gaillard F . PACS integration of semiautomated imaging software improves day-to-day MS disease activity detection . Am J Neuroradiol . 2019 ; 40 : 1624 - 1629 . doi: 10.3174/ajnr.A6195 31515214 PMC7028546
26 Zopfs D Laukamp K Reimer R , et al. Automated color-coding of lesion changes in contrast-enhanced 3D T1-weighted sequences for MRI follow-up of brain metastases . Am J Neuroradiol . 2022 ; 43 : 188 - 194 . doi: 10.3174/ajnr.A7380 34992128 PMC8985679