Sensitivity and Precision of Search Strategies Built Using a Text-Mining Word Frequency Tool (PubReMiner) Compared to Current Best Practice for Building Search Strategies: A Study Within a Review (SWAR).

Abstract

<h4>Objective</h4>PubReMiner is a text-mining tool that analyses a seed set of citations to assess word frequency in titles, abstracts, and Medical Subject Headings (MeSH). This study aimed to determine the sensitivity and precision of search strategies developed using the PubReMiner tool compared to conventional search strategies developed by a librarian at our institution.<h4>Methods</h4>Twelve consecutive reviews conducted at our center were included from September 2023 to January 2025. These reviews included various types of evidence synthesis, including rapid reviews and systematic reviews, covering a variety of topics. One librarian developed a comprehensive search strategy, which included a conventional MEDLINE search for each review. Separately, two librarians independently developed MEDLINE search strategies using PubReMiner-generated word frequency tables (PubReMiner 1 and PubReMiner 2). All search strategies were constructed by experienced librarians using predefined work instructions. Primary outcomes were sensitivity and precision. Secondary outcomes included the number needed to read, the number of unique references retrieved, and the time taken to construct each strategy.<h4>Results</h4>Sensitivity of PubReMiner strategies was generally lower than that of conventional strategies; however, in one review, PubReMiner achieved a higher sensitivity (83.87%) than the conventional strategy (58.06%). Only the sensitivity outcome showed a statistically significant difference between search methods (Friedman test <i>p</i> = 0.0065). No statistically significant difference in precision between the searches was identified. PubReMiner strategies were typically faster to construct but yielded inconsistent performance across reviews and between librarians.<h4>Conclusion</h4>While PubReMiner offers efficiency advantages, its inconsistent performance in retrieving relevant studies suggests that it should not replace conventional search strategies. The study illustrates the value of multi-review SWARs in producing evidence that informs evidence synthesis practices.