Journal of evidence-based social work (2019) · 2026
Searching the Literature by Meaning: Benchmarking Free and Commercial Word Embedding Models on Social Work Text.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Perron Brian E, Wang Miao, Deng Nanyi, Ahn Eunhye
Аннотация
PURPOSE: Locating relevant studies is the first step of evidence-based practice, yet most searching relies on keyword matching. Artificial intelligence (AI) tools called embedding models search by meaning, but the best-known options are paid commercial services. The study asked which free embedding models best search the social work literature, whether they match the commercial standard, and whether rerankers are needed.
MATERIALS AND METHODS: Twelve free embedding models and two commercial OpenAI models were tested on 64,956 social work records (1989-2025) using 150 curated queries. Two frontier AI judges, from families unrelated to every tool evaluated, made 50,328 blind head-to-head comparisons (nDCG@10). A judge-free known-item test (496 queries) and a blind 120-pair expert human instrument provided validation.
RESULTS: Free tools matched or beat the commercial standard. Two free models outperformed the paid flagship; a free 300-million-parameter model beat the paid default, essentially tied for first at finding specific papers (91.9%), and with a reranker was the best configuration overall (.846). Keyword search trailed every embedding model (.604 vs. .680-.842). Rerankers rescued weak models but added nothing to the strongest. Judges agreed on 85.5% of comparisons, and committee-to-rater agreement (69-76%) matched or exceeded rater-to-rater agreement (69-71%).
DISCUSSION: Score differences among leading models are too small to change what a searcher sees; tool choice should rest on size, speed, cost, and privacy.
CONCLUSION: High-quality, meaning-based search of the social work literature is achievable with free tools on an ordinary computer: no subscription, no queries sent to an outside company.
MATERIALS AND METHODS: Twelve free embedding models and two commercial OpenAI models were tested on 64,956 social work records (1989-2025) using 150 curated queries. Two frontier AI judges, from families unrelated to every tool evaluated, made 50,328 blind head-to-head comparisons (nDCG@10). A judge-free known-item test (496 queries) and a blind 120-pair expert human instrument provided validation.
RESULTS: Free tools matched or beat the commercial standard. Two free models outperformed the paid flagship; a free 300-million-parameter model beat the paid default, essentially tied for first at finding specific papers (91.9%), and with a reranker was the best configuration overall (.846). Keyword search trailed every embedding model (.604 vs. .680-.842). Rerankers rescued weak models but added nothing to the strongest. Judges agreed on 85.5% of comparisons, and committee-to-rater agreement (69-76%) matched or exceeded rater-to-rater agreement (69-71%).
DISCUSSION: Score differences among leading models are too small to change what a searcher sees; tool choice should rest on size, speed, cost, and privacy.
CONCLUSION: High-quality, meaning-based search of the social work literature is achievable with free tools on an ordinary computer: no subscription, no queries sent to an outside company.