Single and Multi-Agent LLM Orchestration Benchmark 600 Execution Records on Consumer Hardware
Description
This dataset contains the complete experimental record of a controlled comparison of single-agent and multi-agent orchestration of a small open-weight LLM running locally on consumer laptops. It includes 600 execution records from 132 complexity-stratified tasks run under four conditions, the benchmark, the 50-document corpus the agents searched, all scored outputs, all statistics reported in the accompanying article, and each machine's original result archive. All four conditions share the same model (Llama 3.1 8B, Q4_0, served via Ollama; temperature 0.0, seed 42), tools, tasks and decoding settings: (A) single-agent ReAct in LangGraph; (B) Best-of-N single agent matched to Condition D on tokens and LLM calls; (C) a Planner–Researcher–Writer pipeline in LangGraph; (D) the same pipeline in CrewAI. D − B isolates collaboration at equal compute, C − A architecture overhead, and D − C framework overhead. Tasks are divided into three tiers (44 each) by a pre-registered structural complexity score. Execution was distributed across four Windows laptops (16 GB RAM; two with MX550 2 GB, two with RTX 3050 6 GB), sharded by task so every paired contrast stays within one machine. Six calibration tasks ran on every machine. The configuration hash e4da2c9abab2 is recorded in every record. Folders: 01_benchmark (tasks, corpus, calibration set); 02_merged_raw_records (150 JSONL records per condition, with tokens, calls, tool calls, latency, memory, answers, failures and per-role handoffs); 03_evaluated (all 600 scored records; gold-fact coverage is the primary quality measure); 04_statistics (paired tests, overhead decomposition, mechanism and robustness analyses); 05_per_machine_original (untouched archives). MANIFEST.json gives SHA-256 checksums; DATA_DICTIONARY.md defines every field. Key results: mean quality A 0.194, B 0.241, C 0.281, D 0.280; the D − B effect moves from −0.164 (tier 1) to +0.810 (tier 3), largely through fewer failures rather than better answers. Notes: Condition D made no tool calls; tools were not passed to its agents during collection, and a later diagnostic with tools bound also produced none. Absolute timing and VRAM must be stratified by machine class. Scores inside the per-machine archives predate a scoring correction and read zero; 03_evaluated is authoritative. Code: github.com/Aadithya2222/Research_paper. Licence: CC BY 4.0.
Files
Steps to reproduce
Clone the code repository at https://github.com/Aadithya2222/Research_paper and install the dependencies listed there. Place the contents of 02_merged_raw_records in results/raw/ and run the evaluation, analysis and figure scripts in order (evaluate.py, analyze.py, make_figures.py). Verify that the configuration hash printed by the repository matches e4da2c9abab2 and that the file checksums match MANIFEST.json. The article's statistics are those in 04_statistics.