Searching the Literature by Meaning: Benchmarking Free and Commercial Word Embedding Models on Social Work Text.

PURPOSE: Locating relevant studies is the first step of evidence-based practice, yet most searching relies on keyword matching. Artificial intelligence (AI) tools called embedding models search by meaning, but the best-known options are paid commercial services. The study asked which free embedding models best search the social work literature, whether they match the commercial standard, and whether rerankers are needed.

MATERIALS AND

METHODS

Twelve free embedding models and two commercial OpenAI models were tested on 64,956 social work records (1989-2025) using 150 curated queries. Two frontier AI judges, from families unrelated to every tool evaluated, made 50,328 blind head-to-head comparisons (nDCG@10). A judge-free known-item test (496 queries) and a blind 120-pair expert human instrument provided validation.

RESULTS

Free tools matched or beat the commercial standard. Two free models outperformed the paid flagship; a free 300-million-parameter model beat the paid default, essentially tied for first at finding specific papers (91.9%), and with a reranker was the best configuration overall (.846). Keyword search trailed every embedding model (.604 vs. .680-.842). Rerankers rescued weak models but added nothing to the strongest. Judges agreed on 85.5% of comparisons, and committee-to-rater agreement (69-76%) matched or exceeded rater-to-rater agreement (69-71%).

DISCUSSION: Score differences among leading models are too small to change what a searcher sees; tool choice should rest on size, speed, cost, and privacy.

CONCLUSION

High-quality, meaning-based search of the social work literature is achievable with free tools on an ordinary computer: no subscription, no queries sent to an outside company.