2026-10-16T00:51:17Z https://data.mendeley.com/oai

oai:data.mendeley.com/2xfm6z2p5v.1 2026-06-25T20:21:13Z

Kuzela, Tomas Replication package for "Vibe Testing for REST APIs: A Context-Taxonomic Benchmark of LLM-Generated Test Suites" Mendeley Data This is the complete replication package for the article "Vibe Testing for REST APIs: A Context-Taxonomic Benchmark of LLM-Generated Test Suites". Vibe Testing is an execution-guided workflow in which a large language model (LLM) synthesizes, runs, diagnoses, and repairs executable REST API test suites in a controlled feedback loop. The study quantifies how the type and amount of supplied context shape both the validity and the defect-detection power of the generated tests, using a five-level context taxonomy (L0: OpenAPI specification only; L1: + business documentation; L2: + source code; L3: + database schema; L4: + reference tests) on a purpose-built, contamination-controlled 50-endpoint Bookstore API. Package contents (folders 01-10): - 01_framework: the Vibe Testing pipeline (context assembly, planning, generation, validation, metrics, repair) and all experiment configurations. - 02_target_api: the 50-endpoint Bookstore REST API under test (FastAPI + SQLite). - 03_context_artifacts: the five context-level input artifacts. - 04_context_axis: balanced experiment (5 cloud LLMs x 5 levels x 5 runs = 125 runs) with per-run metrics and all 125 generated suites. - 05_temperature_axis: temperature sweeps (7 models x 6 temperatures at L1). - 06_openweight_context_matrix: auxiliary open-weight model results. - 07_mutation_testing: 13 hand-curated crud.py mutants and per-suite kill results for all 125 cloud suites. - 08_instruction_following: instruction-following-compliance probe results. - 09_statistics: nonparametric analyses (Friedman, Kendall's W, Wilcoxon, Mann-Whitney, Cliff's delta, bootstrap CIs). - 10_figures: the eight published figures and their generator scripts. MANIFEST.sha256 lists every file with a checksum. All framework and analysis code is in English; the target Bookstore API models a Czech-domain bookstore, so its documentation and a few domain strings are preserved verbatim in the original language as the exact inputs supplied to the models, which does not affect the measured results. Licensing: software components are under the MIT License (LICENSE); data, results, and figures are under CC BY 4.0 (DATA-LICENSE.txt). Citation metadata is in CITATION.cff. Software Testing Chuděj, Martin Burdík, Martin Dataset doi:10.17632/2xfm6z2p5v.1 oai:data.mendeley.com/2xfm6z2p5v.1 MIT License http://opensource.org/licenses/MIT https://data.mendeley.com/datasets/2xfm6z2p5v 2026-06-25T20:21:13Z