Evaluating the First CE-Marked LLM-Based Chatbot for Evidence-Based Neurology.

BACKGROUND

Large language models (LLMs) are entering clinical workflows for information retrieval and decision support, but concerns persist about factual accuracy and traceability to evidence. Prof. Valmed is the first CE-marked LLM-based tool for medical information retrieval in Europe, raising the question whether certification aligns with reliable performance in neurology. This study aims to assess the answer accuracy of Prof. Valmed on a neurology benchmark and compare its performance with previously evaluated commercial LLM tools.

METHODS

Prof. Valmed (V2.0.1_2.0.0) was tested on a 130-item benchmark derived from American Academy of Neurology guidelines (65 case-based, 65 knowledge-based). Each question was prompted four times. Two raters scored responses as correct, inaccurate, wrong, or refused; disagreements were adjudicated. Modal ratings per question were used for analysis. Comparative performance against 16 LLMs or configurations previously assessed using the same protocol was evaluated.

RESULTS

Prof. Valmed achieved 76.2% correct answers, 14.6% inaccurate, 6.9% wrong, and 2.3% refused. In pairwise comparisons, Prof. Valmed performed significantly better than 8 models, comparable to several retrieval- or whitelist-enabled tools, and significantly worse than one reasoning model with whitelisting of neurology sources. Overall, its performance was within the range of standard commercial tools.

CONCLUSION

The CE-marked product demonstrated accuracy comparable to contemporary LLMs but did not outperform reasoning-enabled systems. CE marking ensures regulatory conformity rather than inherent performance advantage, and the certification process may limit the integration of rapidly evolving model architectures into approved systems. Future evaluations should examine reasoning quality, source use, and update feasibility across medical domains.