Identifying Public Response Topics to the Centers for Disease Control and Prevention's COVID-19 Communications on Social Media: Infoveillance Study Using Large Language Model-Based Rephrasing.

BACKGROUND: Public health agencies increasingly use social media platforms such as Twitter (subsequently rebranded X, X Corp) to monitor public responses to official health communications and support timely decision-making during health crises. Public replies to communications from official agencies provide valuable insights into population-level discourse and engagement with health policies. However, analyzing large-scale short-text responses remains challenging because social media replies are often brief, informal, ambiguous, and linguistically fragmented. Topic modeling offers a scalable approach for identifying thematic patterns in such data, but its performance is often limited when applied to social media short texts. Recent advances in large language models (LLMs) provide new opportunities to improve text normalization before topic modeling, but their value and limitations for public health infoveillance remain underexplored.

OBJECTIVE: This study develops and evaluates TM-Rephrase, a model-agnostic LLM-based rephrasing framework designed to improve the performance of topic models for short public health social media texts while ensuring rephrasing preserves the original semantic meaning, stance or intent, and tone.

METHODS: We analyzed 25,027 public replies to official Centers for Disease Control and Prevention tweets on Twitter collected between May 2020 and November 2022. TM-Rephrase transformed informal and noisy short texts into more standardized expressions using 2 prompt-guided schemes: general rephrasing and colloquial-to-formal rephrasing. Original and rephrased texts were analyzed using multiple topic models, with rephrasing generated by Gemini 2.5 Flash, GPT‑4o mini, and Mistral-7B-Instruct. Topic quality was evaluated using coherence, uniqueness, redundancy, and diversity. We also conducted an expert semantic-fidelity validation study in which 4 expert raters evaluated whether rephrased texts preserved the original meaning, stance or intent, and tone.

RESULTS: TM-Rephrase generally improved topic quality across models, rephrasing schemes, and LLM backbones. For latent Dirichlet allocation, Cv coherence increased from 0.3094 without rephrasing to 0.5004 with colloquial-to-formal rephrasing. For BERTopic (BERT-based topic modeling), Cv coherence improved from 0.4078 to 0.4734. Diversity-related metrics also improved in most rephrasing settings. Additional checks using CNPMI, CUCI, and UMASS generally supported the main improvement pattern. Sensitivity analysis across different topic numbers further suggested that the coherence gains were not limited to the main setting. Expert validation showed favorable semantic preservation overall, with a mean rating of 3.90 (SD 1.01) and median of 4.00 (IQR 3.00-5.00) of 5.00, and acceptable interrater reliability, intraclass correlation coefficient =0.736. General rephrasing received stronger semantic-fidelity ratings than colloquial-to-formal rephrasing, suggesting that stronger formalization may improve topic readability while introducing greater risk of changes to the original meaning, stance or intent, and tone.

CONCLUSIONS: TM-Rephrase can enhance the coherence, distinctiveness, and interpretability of topic modeling outputs for short, noisy public health social media data. However, it is a text-normalization preprocessing strategy for topic enhancement rather than a fully neutral replacement for original public discourse.