A comparison of customized GPTs and GPT-4o for self-medication knowledge: A feasibility study using the Japanese registered salesperson examination.

OBJECTIVE: This feasibility study aimed to assess whether minimally customized large language models (LLMs) can achieve knowledge levels relevant for self-medication support by evaluating their performance on the Japanese Registered Salesperson Examination. Registered salespersons provide consumers with guidance on appropriate selection and safe use of over-the-counter drugs in Japan. We explored whether LLMs customized through simple reference materials upload, without requiring programming skills or advanced prompt engineering, could achieve comparable knowledge levels.

METHODS: We used the 2024 Registered Salesperson Examination in Japan as a benchmark dataset, comprising 838 text-only multiple-choice questions across five domains. We created customized GPTs by uploading the official guide for examination question development as a PDF file to GPT-4o, without additional prompts or instructions. Both GPT-4o and the customized GPTs were evaluated using a zero-shot approach with web-browsing capabilities disabled.

RESULTS: The overall accuracy was 78.40% for GPT-4o and 92.36% for the GPTs, with this difference being statistically significant (p < 0.001). The GPTs significantly outperformed GPT-4o across all five domains (p < 0.05).

CONCLUSION: This feasibility study demonstrates that minimal customization through reference materials upload can achieve performance improvements on knowledge-based assessments. However, examination performance represents only an indirect indicator of knowledge for self-medication support. Future research should evaluate LLMs performance in real-world case scenarios and assess practical utility and safety before implementation in consumer self-medication settings.