Generative AI in clinical practice: how should rheumatologists evaluate and monitor emerging tools?

Generative artificial intelligence (AI) is increasingly being used in rheumatology practice, to assist with tasks such as documentation, record summarization, and clinical decision support. Rheumatologists need a practical approach to determining what evidence and oversight each application requires. This narrative review proposes a framework based on two dimensions: contextual complexity and influence on patient care. Greater contextual complexity can make errors harder to detect, whereas greater influence on care increases the potential consequences of an undetected error. We examine evidence for workflow benefits and clinical effectiveness within this framework and consider safety, equity, privacy, governance, and implementation. Existing evidence on the benefits, accuracy and reliability of AI applications remains mixed. For example, studies of ambient AI scribes suggest reduced documentation burden, but evidence of improved patient outcomes from applications supporting clinical decisions is more limited. For applications with limited clinical influence, evaluation has initially focused on workflow, usability, and documentation quality. Tools that synthesize longitudinal information require additional assessment of completeness and source traceability. Those influencing diagnosis or treatment require prospective evaluation of clinical benefit and safety. Relevant vendor evidence should be assessed alongside proportionate local testing and ongoing monitoring. Safeguards should address risks such as hallucinations, automation bias, and propagating documentation errors, with attention to performance across patient groups. Changes in functionality or clinical use should prompt reassessment. The framework provides a structured basis for adoption decisions and identifies priorities for research on patient outcomes and feasible oversight in routine rheumatology care.