LLM-generated Candidate Detection Rules: YAML outputs for 123 exploits
Description
This dataset presents LLM-generated candidate detection rules in YAML format, produced by testing two commercial Large Language Models (LLMs): GPT-4o and Claude 3.5 Sonnet. The dataset is organized hierarchically, with directories for each model. The model directory contains six parameter variation folders named according to their 'temperature_top_p' configuration (e.g., 0.5_1, 0_0, 0_1, etc.), where the first value represents temperature and the second represents top_p. Each parameter folder contains YAML text files representing the model's output when tasked with generating candidate detection rules for 123 different exploits. The filenames correspond to exploit IDs from ExploitDB or PacketStorm, enabling direct reference to the original exploit samples.
Files
Steps to reproduce
The models were accessed via their respective API interfaces in December 2024, with parameters indicated by the directory names (temperature_top_p). Each model received identical system messages positioning them as part of a cybersecurity expert team making unanimous decisions. User messages instructed the models to simulate the exploitation process, extract corresponding HTTP requests, and return them in YAML format. For the Claude model, output prefilling was implemented to minimize unsolicited commentary in the responses. All outputs were systematically collected and organised according to the directory structure: commercial > model name > parameter settings > exploit ID.
Institutions
- NASK Instytut Badawczy