Reproducibility Artifacts for Evaluating and Validating Longitudinal LLM-Based Situation Assessment for OT Security Operations
Description
This dataset contains the reproducibility artifacts associated with the article “Evaluating and Validating Longitudinal LLM-Based Situation Assessment for OT Security Operations.” It includes frozen machine-readable case views, manifests, retained model outputs including invalid responses, scoring outputs, statistical diagnostics, validation scripts, analysis scripts, and figure-generation code. Raw operational packet captures, credentials, private infrastructure configuration, and third-party SWaT and HAI source datasets are excluded for security, privacy, and licensing reasons. Public SWaT and HAI data remain available under their respective providers’ access conditions.
Files
Steps to reproduce
Download and extract the archive while preserving its directory structure. Use Python 3.12 and create an isolated environment. Install the required packages: NumPy, SciPy, scikit-learn, Matplotlib, joblib, and PyArrow. Verify file integrity using REPRODUCIBILITY-ARCHIVE-MANIFEST.json. The frozen longitudinal views, ablation variants, deterministic reference outputs, retained LLM responses, validation results, scoring artifacts, and statistical diagnostics are provided in the archive. Detailed commands are documented in platform/traffic-analysis/research/situation_v24/V24-REPRODUCE.md and platform/traffic-analysis/research/situation_v25/REPRODUCE.md. From platform/traffic-analysis/research, reconstruct the deterministic artifacts using: python -m situation_v24.views python -m situation_v24.ablations python -m situation_v24.template_run python -m situation_v25.holdout Retained model outputs can be rescored without API credentials by following the scoring commands in the two reproduction guides. Statistical diagnostics and publication figures can subsequently be regenerated using the scripts under docs/papers/scripts/. The automated test suite can be executed with Python’s unittest module. Re-executing the original LLM inference calls is optional and requires compatible model endpoints and credentials. Exact weight-level reproduction of the original LLM calls is not claimed because the original model-weight, container, and chat-template digests were not retained. The archived prompts, model-visible inputs, raw responses, validated responses, and scores support deterministic auditing and rescoring of the reported experiments. Public source datasets such as SWaT and HAI must be obtained separately from their respective providers under their original access conditions.