Diagnostic Evaluation Dataset for Web of Science Smart Search vs. Advanced Search: A Controlled, Tiered Comparative Analysis with Query Parsing and Relevance Judgment Data

Published: 2 February 2026| Version 1 | DOI: 10.17632/5bkxpvnktt.1
Contributors:
Weizhi Yang,

Description

This dataset contains the complete evidence for a diagnostic evaluation of Web of Science (WoS) "Smart Search". The study hypothesized that Smart Search lacks the semantic understanding and precise control needed to substitute for the traditional Advanced Search in rigorous academic work. Data Content & Structure: The data is organized via a Three-Tier Model of Retrieval Intelligence: Tier 1 (Lexical): Tests on keyword processing, wildcards (*), spelling correction, and cross-lingual mapping. Tier 2 (Pattern): Tests on Boolean logic recognition (case-sensitive AND/OR/NOT), field recognition (author:), and complex query execution. Tier 3 (Semantic): Natural language query results and manual relevance judgments for two case studies (using prototyping... and review of...), demonstrating semantic failure. Key Findings (Data-Driven): Wildcards fail completely (e.g., cell* is run as cell). Boolean logic is brittle (only uppercase AND/OR/NOT recognized). Query expansion distorts intent (e.g., adding 90 unrelated documents to a precise Boolean query). Semantic understanding collapses: Natural language is reduced to a "bag-of-words AND" strategy. Manual assessment shows high rates of thematic deviation (36%) and strategy failure (4% precision). Data Collection Method: Controlled, comparative experiment. Each Smart Search query was benchmarked against an equivalent, precisely formulated query in WoS Advanced Search. For Tier 3, random samples (n=50) were assessed by two independent coders. All searches were limited to the WoS Core Collection (pre-2025 publications). Reuse & Interpretation: Data supports the associated paper's findings and is reusable for: Verification & Replication: Repeat tests on WoS or apply the framework to other databases. Methodology Template: The Three-Tier Model offers a structured approach for evaluating "smart" search interfaces. Information Literacy: Demonstrates practical limits of automated search tools. Note: Result counts are a snapshot; behavioral patterns (e.g., wildcard failure) are stable design features.

Files

Steps to reproduce

This dataset was generated through a controlled, comparative experiment on the Web of Science (WoS) platform. Follow these steps to reproduce the data collection: Define Test Suite: Design search queries aligned with a Three-Tier Model: Tier 1 (single/composite keywords, wildcards, typos), Tier 2 (Boolean logic, field prefixes), and Tier 3 (natural language descriptions, disambiguation, and strategy-testing phrases). Set Constants: Restrict all searches to the Web of Science Core Collection. Apply a publication date filter to exclude documents from 2025 onward. Execute Smart Search: For each query, enter the exact text into the WoS Smart Search bar (default interface). Record the system-parsed query from "See how we processed your query" and the total result count. Create Benchmark: For each query, construct a semantically equivalent, precise query using the WoS Advanced Search interface. Use explicit field tags (e.g., TS=, DT=) and Boolean operators. Execute and record the result count. Conduct Qualitative Analysis (Tier 3): For the two key queries (using prototyping... and review of...), take a random sample (n=50) from the Smart Search results. Two independent researchers assess each sampled document's title, abstract, and keywords against pre-defined relevance frameworks (provided in methodology files). Calculate inter-rater reliability, resolve discrepancies via discussion, and finalize consensus classifications. Data Compilation: Compile all parsed queries, result counts, and qualitative judgments into structured tables (as provided in the Excel files). Key Protocol: Execute Smart Search and its Advanced Search benchmark in immediate succession to minimize database update effects. No special software was used beyond the WoS web interface and standard spreadsheet tools for recording.

Institutions

Categories

Computer Science, Information Science, Education, Information Retrieval, Library and Information Science, Human-Computer Interaction, Science, Technology, Engineering and Mathematics

Funders

Licence