Artificial intelligence for tumor board simulation among HPV positive oropharynx cancer patients.

OBJECTIVE

To compare treatment recommendations generated by three large language models (LLMs): ChatGPT-4o, DeepSeek-R1, and a custom retrieval-augmented generation (RAG) DeepSeek-R1 model for HPV-associated tonsillar oropharyngeal squamous cell carcinoma (OPSCC) tumor board cases.

STUDY

DESIGN

Retrospective cross-sectional study.

SETTING

Tertiary hospital.

METHODS

Fifty cases of HPV-associated tonsillar OPSCC presented at the University of Utah's tumor board were input into ChatGPT, DeepSeek, and RAG-DeepSeek using structured prompts. Five head and neck surgeons rated the recommendations based on concordance with the actual tumor board decision. The graders used a modified global quality scale (GQS) and a personal preference score to evaluate responses.

RESULTS

RAG-DeepSeek demonstrated the highest overall concordance and was rated as perfectly or mostly concordant in 64.8% of cases, outperforming both DeepSeek (43.2%) and ChatGPT (38.0%). DeepSeek provided the highest scored GQS (3.57) responses, followed by ChatGPT (3.27) and RAG-DeepSeek (3.13). Surgeons chose DeepSeek as the preferred response in 40.8% of all cases, RAG-DeepSeek's in 37.6% and ChatGPT in 21.6%. We found that RAG-DeepSeek performed best where surgery was recommended by the tumor board while DeepSeek most often scored highest in cases recommending chemoradiation.

CONCLUSION

Both DeepSeek models significantly outperformed ChatGPT in concordance and surgeon preference, but RAG-DeepSeek scored lower in quality compared to DeepSeek alone. However, since no model achieved total concordance or high quality across all cases, further model tuning is needed and artificial intelligence should remain a potential auxiliary tool for now.