Journal of medical imaging (Bellingham, Wash.) · 2026
Synthesizing vocal tract magnetic resonance imaging sequences with phoneme-aware diffusion models.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Perez-Toro Paula Andrea, Arias-Vergara Tomas, Buess Lukas, Hutter Jana, Woo Jonghye, Maier Andreas
Аннотация
PURPOSE: Real-time speech MRI offers critical insights into articulatory dynamics for diagnostics, therapy, and speech science, but direct speech-to-MRI mapping remains highly challenging. We present a diffusion-based framework that transforms spoken language into dynamic vocal tract MRI sequences with phonemic-aware attention. This improves phoneme-level fidelity and introduces the speech-to-image phonemic fidelity score (SIPFS), a new metric to assess articulatory precision.
APPROACH: We trained audio-conditioned diffusion models with phonemic-aware attention and contrastive learning on the USC 75-Speaker Speech MRI Database. Input consisted of sagittal-view MRI sequences synchronized with speech. Three pre-trained audio encoders (Wav2Vec, WavLM, and Whisper) were used to provide embeddings. Performance was evaluated on unseen speakers and unseen speech segments using standard image/video metrics (FID, FVD, SSIM, and PSNR) and the proposed SIPFS.
RESULTS: Phonemic-aware attention consistently improved quality and phoneme-level precision. SemDiff produced high-quality static images, whereas SD3D excelled in dynamic video synthesis. SIPFS revealed up to 15% higher phonemic accuracy compared with unconditioned models. Excluding articulatory-region ROI reduced fidelity of vocal tract motion, highlighting its importance. The models generalized across new speakers and missing segments, demonstrating robustness.
CONCLUSIONS: Our framework demonstrates the feasibility of synthesizing accurate, phoneme-sensitive vocal tract MRI from speech using diffusion models. By improving image fidelity and introducing SIPFS, this approach supports clinical applications in diagnostics, speech therapy, and articulatory research. Future work will focus on expanding datasets, optimizing computational efficiency, and refining evaluation metrics to further enhance clinical integration.
APPROACH: We trained audio-conditioned diffusion models with phonemic-aware attention and contrastive learning on the USC 75-Speaker Speech MRI Database. Input consisted of sagittal-view MRI sequences synchronized with speech. Three pre-trained audio encoders (Wav2Vec, WavLM, and Whisper) were used to provide embeddings. Performance was evaluated on unseen speakers and unseen speech segments using standard image/video metrics (FID, FVD, SSIM, and PSNR) and the proposed SIPFS.
RESULTS: Phonemic-aware attention consistently improved quality and phoneme-level precision. SemDiff produced high-quality static images, whereas SD3D excelled in dynamic video synthesis. SIPFS revealed up to 15% higher phonemic accuracy compared with unconditioned models. Excluding articulatory-region ROI reduced fidelity of vocal tract motion, highlighting its importance. The models generalized across new speakers and missing segments, demonstrating robustness.
CONCLUSIONS: Our framework demonstrates the feasibility of synthesizing accurate, phoneme-sensitive vocal tract MRI from speech using diffusion models. By improving image fidelity and introducing SIPFS, this approach supports clinical applications in diagnostics, speech therapy, and articulatory research. Future work will focus on expanding datasets, optimizing computational efficiency, and refining evaluation metrics to further enhance clinical integration.