Replication package for "Vibe Testing for REST APIs: A Context-Taxonomic Benchmark of LLM-Generated Test Suites"
Description
This is the complete replication package for the article "Vibe Testing for REST APIs: A Context-Taxonomic Benchmark of LLM-Generated Test Suites". Vibe Testing is an execution-guided workflow in which a large language model (LLM) synthesizes, runs, diagnoses, and repairs executable REST API test suites in a controlled feedback loop. The study quantifies how the type and amount of supplied context shape both the validity and the defect-detection power of the generated tests, using a five-level context taxonomy (L0: OpenAPI specification only; L1: + business documentation; L2: + source code; L3: + database schema; L4: + reference tests) on a purpose-built, contamination-controlled 50-endpoint Bookstore API. Package contents (folders 01-10): - 01_framework: the Vibe Testing pipeline (context assembly, planning, generation, validation, metrics, repair) and all experiment configurations. - 02_target_api: the 50-endpoint Bookstore REST API under test (FastAPI + SQLite). - 03_context_artifacts: the five context-level input artifacts. - 04_context_axis: balanced experiment (5 cloud LLMs x 5 levels x 5 runs = 125 runs) with per-run metrics and all 125 generated suites. - 05_temperature_axis: temperature sweeps (7 models x 6 temperatures at L1). - 06_openweight_context_matrix: auxiliary open-weight model results. - 07_mutation_testing: 13 hand-curated crud.py mutants and per-suite kill results for all 125 cloud suites. - 08_instruction_following: instruction-following-compliance probe results. - 09_statistics: nonparametric analyses (Friedman, Kendall's W, Wilcoxon, Mann-Whitney, Cliff's delta, bootstrap CIs). - 10_figures: the eight published figures and their generator scripts. MANIFEST.sha256 lists every file with a checksum. All framework and analysis code is in English; the target Bookstore API models a Czech-domain bookstore, so its documentation and a few domain strings are preserved verbatim in the original language as the exact inputs supplied to the models, which does not affect the measured results. Licensing: software components are under the MIT License (LICENSE); data, results, and figures are under CC BY 4.0 (DATA-LICENSE.txt). Citation metadata is in CITATION.cff.
Files
Steps to reproduce
1. Environment: Python 3.11+, then `pip install -r 01_framework/requirements.txt`. 2. Start the target API: `cd 02_target_api && docker compose up -d` (or `uvicorn app.main:app --port 8000`). The SQLite database is created automatically. 3. API keys: copy `01_framework/.env.template` to `.env` and fill in the keys for the providers you want to run (open-weight models require an Ollama host, set in the YAML config). 4. Generate test suites: `python main.py --config configs/<experiment>.yaml`. Each run writes the generated suite + log and appends per-run metrics to results/. 5. Reproduce analyses: the scripts in folders 04/07/08/09/10 recompute the aggregations, mutation matrix, IFC scores, statistics, and figures from the archived JSON results. Note: archived analysis scripts use the original working-tree paths; see README.md ("Path note") for the mapping to this package's folders. Verify integrity against MANIFEST.sha256.