Data and code for: A Multi-Criteria Framework for LLM-Based Manuscript Assessment: System Design and Venue-Level Validation
Description
Per-paper scores of 180 published computer science and artificial intelligence papers (six journals, two CiteScore tiers) produced by a 13-pillar LLM assessment pipeline (Gemini 3.1 Pro Preview), together with the additional runs made during peer review: a holistic single-prompt baseline (180 papers), a test-retest study (30 papers x 3 runs), and re-runs of six papers. Also included: surface features and a venue-cue audit of the extracted texts, publication dates, the Stage-1 editorial recommendations and output structure, the unchanged scoring code, the verbatim prompts, a simplified Colab notebook, and Python scripts (with a Windows launcher) that reproduce every statistic and result figure of the article. Full texts of the papers and the LLM reports are not included (publisher copyright); the papers can be retrieved through the DOIs. See README.md.
Files
Steps to reproduce
No API key is needed to reproduce the statistics. Install Python 3.9 or newer, then run run_analysis.bat (Windows) or "bash run_analysis.sh" (Linux/macOS). The launcher checks and installs pandas, numpy, scipy, scikit-learn, matplotlib and openpyxl and runs analysis/paper_stats.py, analysis/revision_analysis.py and analysis/retest_analysis.py. Outputs are written to results/. The LLM scoring code is in pipeline/ (it needs the article PDFs and a Gemini API key). See README.md for details.
Institutions
- Sepuluh Nopember Institute of TechnologyEast Java, Surabaya