Dataset: Diagnostic Evaluation of Wildcard Processing in Web of Science Smart Search

Published: 11 March 2026| Version 1 | DOI: 10.17632/7z9fbs8wmn.1
Contributors:
Weizhi Yang,

Description

This dataset presents a diagnostic evaluation of wildcard processing in the Web of Science (WoS) Smart Search interface. The study tests whether “smart” search tools reliably execute fundamental retrieval functions – specifically, right truncation (*), internal masking (?), left truncation (*), and their combinations. Data collection: 65 structured test points across seven dimensions (Right Truncation, Internal Masking, Left Truncation, Combined Wildcards, Special Characters, Case Variants, Edge Cases). Each test records: user input, Smart Search parsed query, Smart Search result count, Advanced Search benchmark query, and benchmark result count. All searches were performed on the same day on the WoS Core Collection without date restrictions. Key findings reveal systematic failures: Right truncation (*) ignored in simple queries: cell* misses 8.1% of relevant documents. For irregular words (mouse*), opaque thesaurus expansion adds mice, causing over‑retrieval (+94.8%) with no user transparency. Left truncation (*) completely ignored: *omics retrieves only 7.2% of the benchmark (51,164 vs. 710,319), with results skewed toward the rare standalone “omics” rather than intended prefixed terms (genomics, proteomics). Internal masking (?) replaced by space: c?ll becomes meaningless c AND ll fragment search. Multi‑word phrases with truncation (STEM educat*) suffer “stem‑chopping”: 20 vs. 7,623 results. Quoted wildcards (“3D print*”) have asterisk replaced by space, missing >98% of intended records. Complex combinations (CRISPR*Cas*) parsed as CRISPR Cas, causing massive over‑retrieval (11,553 vs. 260) – 97% of results contain separate words, not exact CRISPR‑Cas* variants. Short stems (5G*, AI*) executed despite Advanced Search invalidity, but wildcard ignored and results polluted with noise (84% of AI* results irrelevant due to language mismatches). Conditional activation: Wildcards processed correctly in Boolean queries (cell* AND growth matches benchmark), revealing an undocumented dual‑parser architecture. Depth analysis for eight critical cases includes manual coding of sampled records, confirming mechanisms behind quantitative differences. Dataset contents: 1_Methodology_Overview.xlsx – Core dimensions, test stem selection, variation matrix, defect categorisation. 2_Test_Results.xlsx – Complete raw data for all 65 test points. 3_Depth_Analysis.xlsx – Detailed case studies with sampled records and coding. Implications: Data inform information literacy instruction, reference services, and database evaluation by revealing hidden limitations of “smart” search tools.

Files

Steps to reproduce

This dataset was generated through a diagnostic evaluation of wildcard processing in Web of Science (WoS) Smart Search, using WoS Advanced Search as the benchmark. The methodology follows a structured framework of seven core dimensions (right truncation *, internal masking ?, left truncation *, combined wildcards, special characters, case variants, edge cases) with 65 test points. 1. Framework design Select core stems from multiple disciplines (e.g., cell, mouse, STEM education) representing diverse morphological features. For each stem, generate query variations covering different wildcard positions, quotation, Boolean operators, and edge cases. 2. Data collection Platform: WoS Core Collection (accessed via institutional subscription). Date: All searches performed on a single day (March 2026) to minimise database update effects. Temporal coverage: No date restrictions (full Core Collection). Procedure: For each test query, enter it into Smart Search; record the parsed query (from “See how we processed your query”) and result count. Immediately execute the corresponding Advanced Search benchmark query; record its result count and any error messages. All results are recorded in 2_D1_Wildcard_Test_Results.xlsx with fields: Test ID, User Input, Smart Search Parsed Query, Smart Search Result Count, Advanced Search Benchmark Query, Advanced Search Result Count, Notes. 3. Depth analysis Select critical cases with extreme differences (e.g., mouse*, STEM educat*, CRISPR*Cas*, AI*). For each, apply filters (e.g., year, WoS category) to obtain manageable subsets. Export record metadata (title, abstract, keywords) for relevant result sets (e.g., Smart Search results, benchmark results, difference sets). Randomly sample 50–100 records per set; manually code for word forms, relevance, and retrieval mechanism. Results are in 3_D1_Wildcard_Depth_Analysis.xlsx. 4. Quality control All searches for a given dimension performed consecutively to minimise temporal bias. For queries invalid in Advanced Search (e.g., short stems like 5G*), errors are noted. Depth analysis coding performed by two researchers independently; disagreements resolved through discussion. 5. Software Web browser for WoS access; Microsoft Excel for data recording. The full methodological framework is detailed in 1_D1_Wildcard_Methodology_Overview.xlsx. This dataset can be reproduced by following the same steps and query list on any date, though absolute result counts may vary slightly due to database updates.

Institutions

Categories

Computer Science Applications, Information Retrieval, Data Science, Library and Information Science, Scholarly Communication

Funders

Licence