Cross-Domain Chest Radiograph Report Generation Under Dataset Shift: Bidirectional Audit Study.

BACKGROUND: Cross-domain chest radiograph report generation may produce formally valid reports that are nevertheless repetitive, weakly case-specific, or missing clinically relevant findings. These failures can be missed when evaluation relies on a single language-overlap or clinical-label score.

OBJECTIVE: This study aimed to evaluate cross-domain report generation with a bidirectional layered audit, characterize direction-specific and end point-specific failure profiles under a composite training-side control and 3 decoding policies, and explore whether selected audit dimensions remained informative for a contemporary medical vision language model.

METHODS: A fixed R2Gen-style system was evaluated bidirectionally on 575 IU X-ray and 401 MIMIC-CXR (Medical Information Mart for Intensive Care Chest X-Ray) target records. The audit included surface validity, complete-report recurrence and concentration, a predefined prefix end point, lexical diversity, official Stanford CheXpert/NegBio labels, paired cluster-bootstrap inference, and blinded review by 2 medical trainees. The primary human audit included 210 reports from 70 randomly sampled cases, with blinded adjudication of unresolved primary end point disagreements. A 60-report targeted panel was used for failure-mode analysis. Separately, a fixed MedGemma 1.5 4B configuration generated reports for all 976 records.

RESULTS: In MIMIC-CXR-to-IU X-ray transfer, out of 575 reports, ON (protocol-defined training-side control enabled)+B4 (beam search with a beam size of 4) reduced the predefined recurrent prefix from 575 (100%) to 8 (1.4%; Holm-adjusted P=.002). However, 569 (99.0%) ON+B4 outputs still belonged to exact duplicate groups. In IU X-ray-to-MIMIC-CXR transfer, the 14-category CheXpert/NegBio micro-F1 was higher for ON+B4 than for OFF (protocol-defined training-side control disabled)+B4 (0.1687 vs 0.0744; Holm-adjusted P=.01). This apparent advantage was not broad-based: after cardiomegaly was excluded in a post hoc descriptive sensitivity analysis, the difference changed from +0.0944 to -0.0610. Text-level tracing showed that the original contrast was mainly driven by a labeler response to recurrent, semantically normal heart-size wording. Across the primary human-audit cells, major omissions occurred in 12 to 15 out of 35 reports, and major unsupported findings occurred in 10 to 16 of 35 reports, with no consistent configuration ordering observed. The exploratory MedGemma micro-F1 was 0.4722 on IU X-ray and 0.4340 on MIMIC-CXR, but 381 out of 575 (66.3%) IU outputs belonged to exact duplicate groups.

CONCLUSIONS: No configuration performed consistently well across end points, and no single metric captured all forms of cross-domain degradation. The MedGemma stress test suggested that recurrence and label-traceability checks remain useful across architectures, although it was not a matched performance comparison. Cross-domain report generators should undergo bidirectional evaluation, final-text verification, and image-grounded human review before external validation or clinical deployment.