Performance of Large Language Models vs Conventional Machine Learning for Predicting Clinical Outcomes With Limited Data: Comparative Study.

BACKGROUND

Machine learning (ML) can be used to predict clinical outcomes. Training predictive models typically requires data for hundreds or thousands of patients. Lowering this requirement to a few tens of patients would enable new applications in clinical trials (eg, optimizing the design of a phase III trial by training a predictive model on phase II data and applying it to synthetic phase III patients) or in clinical decision support systems (for rare diseases or narrow indications). Large language models (LLMs) have recently been shown to outperform conventional ML algorithms for predictions on tabular data when the train dataset is small.

OBJECTIVE

This study aims to investigate the advantage of LLMs compared with conventional ML for predicting clinical outcomes by applying state-of-the-art models to recently published clinical datasets.

METHODS

We compared 2 LLMs, 1 proprietary (from OpenAI) and 1 open source (from the Meta Llama family), with conventional ML classification algorithms to predict clinical outcomes using 3 recently published clinical datasets spanning distinct conditions (sepsis, gastric cancer, and acute leukemia). Datasets were chosen such that their publication date was after the LLM knowledge cutoff date to ensure that models were never exposed to these data during pretraining. Datasets were sampled to vary the training size.

RESULTS

On average, the 2 tested LLMs perform better than conventional ML for training sizes below 50 patients, using the receiver operating characteristic area under the curve, the F1-score, the average precision, and the balanced accuracy as metrics. Contextual information was found to be key to this advantage.

CONCLUSIONS

These preliminary results may be further optimized. They already show the potential of LLM-based ML to enable new clinical use cases when data are available for only a few tens of patients.