Comparing the performance of ChatGPT and ERNIE Bot in answering questions regarding liver cancer interventional radiology in Chinese and English contexts: A comparative study.

Introduction

Liver cancer is among the most common malignancies globally. According to the 2022 GLOBOCAN database, it accounts for over one million deaths worldwide, making it the 3rd leading cause of cancer mortality after lung and colorectal cancers. Additionally, liver cancer is the sixth most frequently diagnosed cancer worldwide, representing a substantial disease burden and presenting significant challenges in treatment. 1 Among various treatment modalities for liver cancer—such as surgical resection, radiotherapy, and chemotherapy—interventional therapy holds a crucial role, offering unique advantages over other approaches. Transcatheter arterial chemoembolization (TACE) and hepatic artery infusion chemotherapy (HAIC) are two widely utilized interventional techniques that contribute significantly to the clinical management of liver cancer. These therapies directly target the tumor while minimizing damage to surrounding healthy tissue, potentially prolonging survival and improving patients’ quality of life. The unique benefits of interventional therapies underscore their importance within the comprehensive liver cancer treatment landscape.

However, due to the complexity and specialized nature of medical knowledge, it is often difficult for patients and their caregivers to accurately understand and grasp the issues related to HAIC and TACE. The accuracy and reliability of online content vary widely. Recently, with rapid advancements in artificial intelligence (AI), tools like ChatGPT, developed by OpenAI (San Francisco, USA), and ERNIE Bot (Enhanced Representation through Knowledge Integration), created by Baidu (Beijing, China) have offered new solutions to address these challenges. These models have powerful natural language processing capabilities and can generate coherent, logically consistent responses covering a wide range of knowledge domains. Studies have demonstrated ChatGPT's potential in medical contexts: in medical education, it has proven capable of answering exam-level questions, supporting learners as an interactive educational tool 2 , 3 ; in clinical practice, ChatGPT has shown promise in enhancing the logic of clinical decision support systems, providing valuable recommendations that aid clinicians. 4 Additionally, recent studies have highlighted its potential to provide medical information tailored to specific patient groups. For instance, Gravina et al. 5 investigated its effectiveness in addressing questions related to inflammatory bowel disease (IBD), while Yeo et al. 6 explored its application in responding to queries about liver disease; in health management, ChatGPT serves as a virtual health coach for chronic disease management, offering guidance to patients to enhance health literacy and encourage positive behavior change. 7 However, most studies assessing ChatGPT's role in medicine have been conducted in English-speaking contexts, with limited exploration of its potential in other languages, such as Chinese, which is among the most widely spoken globally. Similar to ChatGPT, ERNIE Bot is a new generation of knowledge-enhanced big language models from Baidu in China. Nevertheless, few studies have reported its capabilities in medical consulting.

This study aims to evaluate the abilities of ChatGPT and ERNIE Bot to address questions related to TACE and HAIC interventions for liver cancer in both Chinese and English contexts, by examining the accuracy and comprehensiveness of each model in responding to specialized queries. Through this research, we hope to elucidate the potential value and limitations of these LLMs in the healthcare area and determine if they can be an effective tool for providing quality treatment information and education.

Materials and methods

Question design

The researchers initially developed a set of questions for TACE and HAIC by drawing from existing clinical treatment guidelines and addressing real-world challenges patients encounter in clinical practice. 8 , 9 In a subsequent screening phase, the research team carefully removed redundant, ambiguous, or overly subjective questions, ultimately refining a set of relevant and practical questions ( Figure 1 ). This final question set encompassed foundational knowledge, patient education, treatment, and care. This approach ensured a thorough assessment and enhanced the practical applicability of the findings from various perspectives.

Assessment process

We entered questions into the May 2024 version of ChatGPT 4.0 and ERNIE Bot 4.0. A question was entered in English and Chinese, both done independently using the “New Chat” function. Example prompts included: “Please answer the following questions about liver cancer.” To evaluate the accuracy and comprehensiveness of the models’ answers regarding liver cancer interventional radiology, an expert panel consisting of 10 professionals with extensive clinical experience in liver cancer interventions was assembled (years of interventional radiology experience: 5–10: 4/10, 40%; 10–20: 4/10, 40%; > 20: 2/10, 20%). Each panel member independently rated the responses from both models using a five-point Likert scale. Differences in ratings were addressed through moderated discussions among the panel members. The final score for each response was then determined by a seasoned clinical expert, who has over 20 years of interventional radiology experience, ensuring a consistent and rigorous evaluation process.

Evaluation of classification

Statistical analysis

Count data were summarized using frequencies and percentages, and statistical analyses were conducted using the Wilcoxon signed-rank test to compare the accuracy and comprehensiveness of responses between ChatGPT and ERNIE Bot. All statistical analyses were performed with SPSS version 25.0, with a two-sided P -value of less than .05 indicating statistical significance. Figures were generated using GraphPad Prism version 9.5. As this study did not involve patient data or sensitive personal information, ethics committee approval was not required ( Table 1 ).

Results

A comparative analysis of ERNIE Bot and ChatGPT across Chinese and English contexts

Based on the data illustrated in Figure 2 and Table 2 , both models demonstrate high performance across 38 core questions related to TACE and HAIC in both Chinese and English contexts, with over half of the responses receiving the top score of 5. However, it was revealed that ERNIE Bot outperformed ChatGPT in the Chinese context regarding overall response quality ( Table 2 , P = 0.001). In contrast, in the English context, ChatGPT surpasses ERNIE Bot ( Table 2 , P = 0.026). Detailed information is listed in the Supplemental material .

Subgroup analysis in the Chinese context

In an in-depth comparative analysis within the Chinese context, we identified notable differences in the performance of ERNIE Bot and ChatGPT in medical information Q&A ( Table 3 and Figure 3 ). Specifically, when we refined our evaluation to include distinct dimensions—such as fundamental knowledge, patient education, treatment and care—and developed targeted questions for each category, ERNIE Bot demonstrated a clear advantage over ChatGPT. It was demonstrated that ERNIE Bot's responses were of significantly higher quality than those of ChatGPT (fundamental knowledge, P = 0.023; treatment and care, P = 0.021, Table 3 ). In contrast, in the area of patient education, the differences were not significant.

Subgroup analyses in the English context

As illustrated in Figure 4 and Table 4 , compared to the Chinese context, the overall performance of ChatGPT is superior to that of ERNIE Bot in the English context (fundamental knowledge, P = 0.036; treatment and care, P = 0.015, Table 4 ). In the category of patient education, both ChatGPT and ERNIE Bot performed well with a non-significant difference.

LLMs’ performance varied by language in some questions

The results revealed a clear divergence in the performance of ERNIE Bot and ChatGPT based on different language contexts. For example, in the Chinese language context, when ERNIE Bot answered the question “What are the advantages and disadvantages of HAIC compared to TACE?,” the rating was 5. However, the statement “Most patients can complete TACE under outpatient conditions” in ChatGPT's answer is controversial (see Supplemental material ), and the rating is 2. When ERNIE Bot answered the question “How soon can patients eat after TACE and what are the dietary requirements?,” the information was correct and complete, while ChatGPT's answer of “immediately after surgery and several hours after surgery” was not clear. It is generally recommended to fast for 4–6 h after TACE surgery, with a rating of 4.

For instance, in the English language context, when ChatGPT answered the question “What are the common chemotherapy regimens for HAIC?,” the information was correct and complete, and the rating was 5. However, the statement “It's important to note that HAIC is typically reserved for patients with liver-dominant metastatic disease who have failed or are not candidates for systemic chemotherapy or other local therapies. HAIC is generally considered a more aggressive approach and is associated with higher risks of complications compared to systemic chemotherapy. Therefore, it is typically only offered at specialized centers with experience in this type of treatment.” This passage generated by ERNIE Bot was disputed. Because HAIC is an invasive treatment (requiring a catheter to be implanted into the hepatic artery), it carries with it many catheter-related complications and local liver risks, such as catheter-related infections and worsening liver function. The side effects of systemic chemotherapy were more likely to be systemic toxicity, such as bone marrow suppression, gastrointestinal reaction, systemic immunity decline, and organ toxicity. Therefore, it will create a misunderstanding among patients that HAIC will lead to more serious systemic problems. The rating is 2. When ChatGPT answered the question “What are the adverse effects and complications after TACE? “, the information was correct and complete, while ERNIE Bot did not mention liver abscess or upper gastrointestinal bleeding in his answer to postoperative complications. And point 6 of the answer, “Death” is disputed as a result, not a complication, so it is rated 2.

Discussion

LLMs are generating considerable interest in the medical field. In medical diagnosis, ChatGPT has demonstrated accuracy and effectiveness comparable to that of professional rheumatologists when diagnosing inflammatory rheumatic diseases. Although it may occasionally fall short of clinical practitioners in specific details, ChatGPT can still provide correct differential diagnoses across a wide range of cases. 10 ChatGPT has also shown promise in patient education; for example, Campbell et al. 11 found that ChatGPT could appropriately answer most questions about thyroid nodules, regardless of prompt type. Additionally, ChatGPT achieved a 95% response accuracy rate when serving as an educational resource for patients with multiple myeloma, underscoring its potential as a reliable tool in patient education. 12 Despite these advances, there remains limited research evaluating the quality of LLMs’ responses to questions about TACE and HAIC.

This study provides an in-depth analysis of the performance of ERNIE Bot and ChatGPT in answering questions related to TACE and HAIC in both English and Chinese contexts, highlighting the potential of AI in interventional medicine. Notably, this research introduces ERNIE Bot, an AI language model, into the medical field for the first time and directly compares it with ChatGPT. In the Chinese context, ERNIE Bot demonstrates superior language comprehension and processing capabilities, delivering more accurate and comprehensive responses than ChatGPT. This advantage can largely be attributed to its training on Chinese-specific datasets and real-time updated databases, enabling it to provide more precise information. Such capabilities make ERNIE Bot a valuable tool for Chinese-speaking healthcare environments, where accurate comprehension of complex medical terminology is critical. Its excellent performance not only reflects the benefits of specific language model training but also indicates that the use of large language models in line with the national culture may better meet the needs of users. In contrast, ChatGPT demonstrated robust language generation capabilities in English contexts, consistent with its reputation as a versatile and widely applicable AI model. This performance highlights its potential as a tool for global healthcare communication and patient education in multilingual settings. However, ChatGPT's relatively weaker performance in Chinese underscores the challenges faced by general-purpose models when applied to linguistically and culturally diverse healthcare environments. Consistent with existing studies, 13 – 19 our data also highlight the superior capabilities of ERNIE Bot and ChatGPT for patient education. These findings underscore the significant potential and value of AI in medical education. Accessible and accurate medical information is crucial for patients, as it helps them better understand their conditions and treatment options and increases their confidence in shared decision-making with healthcare providers, ultimately improving treatment outcomes. Research shows 20 that inadequate disease knowledge is closely associated with increased healthcare service utilization and costs. The integration of advanced LLMs to deliver personalized and precise educational support represents an innovative and efficient strategy for optimizing medical resources and enhancing the accessibility and convenience of healthcare services. Furthermore, this study confirms that patients selecting appropriate large language models in different contexts can achieve more accurate acquisition of medical information.

A detailed analysis of specific questions revealed that, despite the overall high performance of both large language models, some omissions and shortcomings remain in their responses. For instance, in the Chinese context, in response to “What are the adverse reactions and complications after TACE?” ERNIE Bot overlooked allergic reactions, whereas ChatGPT did not address these reactions or potential complications like ectopic embolization.

Additionally, in the English context, in answering “What are the indications and contraindications for HAIC treatment?” ChatGPT's list of contraindications omitted coagulation disorders and renal insufficiency. When responding to “What is HAIC (hepatic arterial infusion chemotherapy)?” ERNIE Bot did not mention the radial artery approach. In addressing “What are the advantages and disadvantages of HAIC compared to TACE?” ERNIE Bot stated that “Since HAIC does not use embolic agents, its effects may not be as durable as TACE” and that “HAIC is more invasive compared to TACE” which are contentious assertions. These omissions and inaccuracies can be attributed to several factors. First, ChatGPT's knowledge database was last updated in April 2023, potentially limiting its ability to provide the most current medical information. Additionally, both models may face limitations in training data, algorithms, and computational resources, leading to incomplete or inaccurate responses on specific medical topics. Lastly, the inherently complex and nuanced nature of medical practice demands comprehensive consideration of multiple factors, which general language models may not fully capture, resulting in gaps in their answers. Despite AI's significant potential, these limitations highlight the need for further optimization. To ensure the comprehensiveness and accuracy of medical information, it is crucial to combine AI with human review in practical applications. This integrated approach can foster substantial advancements in medical knowledge and health consultation, ultimately providing patients with more precise and comprehensive support.

Our findings highlight that while both models show promise, their unique strengths highlight some important considerations for developing and applying AI tools in clinical practice. First, the observed differences between ERNIE Bot and ChatGPT highlight the necessity of localized training to improve AI performance in specific regions and languages. Second, while ERNIE Bot performs well in a single language environment, combining its language-specific expertise with ChatGPT's multilingual versatility can better meet different clinical needs. Finally, while the strengths of both models in language generation and information synthesis underscore their potential as complementary tools for clinical education and decision making, careful human supervision is still needed.

Limitations

This study has certain limitations. Notably, it has yet to evaluate the comprehensibility of ERNIE Bot and ChatGPT for patients without a medical background. Such patients often require clear, concise, and easily understandable explanations and medical advice, a factor not fully addressed in the current research. Future studies should thus more rigorously assess the user-friendliness and practical effectiveness of these tools for patient populations with limited medical knowledge. This will be essential for enhancing the accessibility and utility of health information and advice offered by AI-driven platforms.

Conclusion

In conclusion, this study shows that ERNIE Bot and ChatGPT perform well in solving TACE and HAIC problems, highlighting the potential of AI in healthcare. In the Chinese context, ERNIE Bot outperforms ChatGPT, providing timely and accurate information to Chinese-speaking users. In the English context, ChatGPT outperforms and provides accurate and fluent healthcare information for English users. Therefore, choosing a suitable large language model is important for patients to get more accurate treatment.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076251315511 - Supplemental material for Comparing the performance of ChatGPT and ERNIE Bot in answering questions regarding liver cancer interventional radiology in Chinese and English contexts: A comparative study

Supplemental material, sj-docx-1-dhj-10.1177_20552076251315511 for Comparing the performance of ChatGPT and ERNIE Bot in answering questions regarding liver cancer interventional radiology in Chinese and English contexts: A comparative study by Xue-ting Yuan, Chen-ye Shao, Zhen-zhen Zhang and Duo Qian in DIGITAL HEALTH

Acknowledgments

The authors acknowledge all reviewers.

Authorship: All authors meet the ICMJE authorship criteria.

Contributorship: XTY contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing. CYS contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing. DQ contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing. ZZZ contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing. XTY contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing. CYS contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing. DQ contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing. XTY contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing. CYS contributed to planning, designing writing, conception, data analysis, write-up, reference writing, and manuscript writing.

Declaration of conflicting interest: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval: There are no human participants in this article and informed consent is not required.

Funding: The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by Postgraduate Research & Practice Innovation Program of Jiangsu Province (SJCX24_1817).

Guarantor: All four authors take responsibility for the manuscript. All authors take responsibility for any liabilities regarding this case.

ORCID iDs: Xue-ting Yuan https://orcid.org/0009-0004-1073-1541

Chen-ye Shao https://orcid.org/0009-0006-7288-816X

Zhen-zhen Zhang https://orcid.org/0009-0002-6198-9310

Duo Qian https://orcid.org/0000-0003-2324-7535

Supplemental material: Supplemental material for this article is available online.

References

1 Bray F Laversanne M Sung H , et al. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries . CA Cancer J Clin 2024 ; 74 : 229 – 263 . 38572751 10.3322/caac.21834

2 Gilson A Safranek CW Huang T , et al. How does ChatGPT perform on the United States medical licensing examination (USMLE)? the implications of large language models for medical education and knowledge assessment . JMIR Med Educ 2023 ; 9 : e45312 . 10.2196/45312 PMC9947764 36753318

3 Gravina AG Pellegrino R Palladino G , et al. Charting new AI education in gastroenterology: cross-sectional evaluation of ChatGPT and perplexity AI in medical residency exam . Dig Liver Dis 2024 ; 56 : 1304 – 1311 . 38503659 10.1016/j.dld.2024.02.019

4 Liu S Wright AP Patterson BL , et al. Using AI-generated suggestions from ChatGPT to optimize clinical decision support . J Am Med Inform Assoc 2023 ; 30 : 1237 – 1245 . 37087108 10.1093/jamia/ocad072 PMC10280357

5 Gravina AG Pellegrino R Cipullo M , et al. May ChatGPT be a tool producing medical information for common inflammatory bowel disease patients’ questions? An evidence-controlled analysis . World J Gastroenterol 2024 ; 30 : 17 – 33 . 38293321 10.3748/wjg.v30.i1.17 PMC10823903

6 Yeo YH Samaan JS Ng WH , et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma . Clin Mol Hepatol 2023 ; 29 : 721 – 732 . 36946005 10.3350/cmh.2023.0089 PMC10366809

7 Al-Anezi FM . Exploring the use of ChatGPT as a virtual health coach for chronic disease management . Learn Health Syst 2024 ; 8 : e10406 . 10.1002/lrh2.10406 PMC11257053 39036525

8 Gaojun T . Chinese clinical practice guidelines for transarterial chemoembolization of hepatocellular carcinoma (2023 edition) . Natl Med J China Epub ahead of print 2023 ; 103 : 2674 – 2694 . 10.3760/cma.j.cn112137-20230630-01114 37675541

9 Zhou J Sun H Wang Z , et al. Guidelines for the diagnosis and treatment of primary liver cancer (2022 edition) . Liver Cancer 2023 ; 12 : 405 – 444 . 37901768 10.1159/000530495 PMC10601883

10 Krusche M Callhoff J Knitza J , et al. Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4 . Rheumatol Int 2023 ; 44 : 303 – 306 . 37742280 10.1007/s00296-023-05464-6 PMC10796566

11 Campbell DJ Estephan LE Sina EM , et al. Evaluating ChatGPT responses on thyroid nodules for patient education . Thyroid® 2024 ; 34 : 371 – 377 . 38010917 10.1089/thy.2023.0491

12 Saba L Fu C Khouri J , et al. Evaluating ChatGPT as an educational resource for patients with multiple myeloma: a preliminary investigation . Am J Hematol 2024 ; 99 : 1205 – 1207 . 38602288 10.1002/ajh.27318

13 Shao C Li H Liu X , et al. Appropriateness and comprehensiveness of using ChatGPT for perioperative patient education in thoracic surgery in different language contexts: survey study . Interact J Med Res 2023 ; 12 : e46900 . 10.2196/46900 PMC10463083 37578819

14 Krittanawong C Virk HUH Kaplin SL , et al. Assessing the potential of ChatGPT for patient education in cardiac catheterization care . JACC Cardiovasc Interv 2023 ; 16 : 1551 – 1552 . 37380244 10.1016/j.jcin.2023.04.042

15 Krittanawong C Rodriguez M Kaplin S , et al. Assessing the potential of ChatGPT for patient education in the cardiology clinic . Prog Cardiovasc Dis 2023 ; 81 : 109 – 110 . 37832625 10.1016/j.pcad.2023.10.002

16 Grünebaum A Chervenak J Pollet SL , et al. The exciting potential for ChatGPT in obstetrics and gynecology . Am J Obstet Gynecol 2023 ; 228 : 696 – 705 . 36924907 10.1016/j.ajog.2023.03.009

17 Gabriel J Shafik L Alanbuki A , et al. The utility of the ChatGPT artificial intelligence tool for patient education and enquiry in robotic radical prostatectomy . Int Urol Nephrol 2023 ; 55 : 2717 – 2732 . 37528247 10.1007/s11255-023-03729-4

18 Yapar D Demir Avcı Y Tokur Sonuvar E , et al. ChatGPT’s potential to support home care for patients in the early period after orthopedic interventions and enhance public health . Jt Dis Relat Surg 2024 ; 35 : 169 – 176 . 38108178 10.52312/jdrs.2023.1402 PMC10746912

19 Moons P Van Bulck L . ChatGPT: can artificial intelligence language models be of value for cardiovascular nurses and allied health professionals . Eur J Cardiovasc Nurs 2023 ; 22 : e55 – e59 . 10.1093/eurjcn/zvad022 36752788

20 Valery PC Bernardes CM Hayward KL , et al. Poor disease knowledge is associated with higher healthcare service use and costs among patients with cirrhosis: an exploratory study . BMC Gastroenterol 2022 ; 22 : 340 . 35836105 10.1186/s12876-022-02407-6 PMC9284723