Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study
Description
In this paper, we present the first measurement-driven study on the reliability of LLM-based expert systems applied to CTI tasks. We propose an automated framework, HalluVision, which generates LLM outputs, extracts malware-related entities using a fine-tuned Named Entity Recognition (NER) model, and evaluates their factual consistency using multiple similarity metrics. Our analysis, based on 4,940 real-world security articles, includes both quantitative measurements and qualitative case studies, offering a comprehensive evaluation of hallucination risks in this high-stakes application domain.
Files
Steps to reproduce
Dataset for Evaluating LLM-based Named Entity Recognition in Cybersecurity This dataset was created to evaluate the performance of Large Language Models (LLMs) on Named Entity Recognition (NER) tasks within the cybersecurity domain. It includes raw text data from Open-Source Intelligence (OSINT) sources, a ground truth dataset generated by a standard NER tool, and corresponding outputs from an LLM. [ground_truth_ner.json] This file contains the ground truth dataset for Named Entity Recognition (NER). The entities were extracted from articles crawled from various OSINT sources (e.g., Threatpost, Wikipedia) using the CyNER tool. It serves as the benchmark for evaluating the LLM's performance. [llm_results_*.json] These files contain the NER outputs generated by a Large Language Model (LLM), processed through the HalluVision framework. Each file represents a different experiment or configuration and is intended for direct comparison against the entities in ground_truth_ner.json. [scraped_articles.txt] This file is a compilation of the complete, original text from all articles scraped from the OSINT sources. This raw text serves as the source material for both the ground truth and the LLM-based NER extraction. [threatpost_urls.txt] This file lists the URLs of all articles crawled from the 'Malware' section of the Threatpost website, which was one of the primary sources for this dataset. [wiki_malware_list.txt] This file contains a curated list of malware-related terms and entities sourced from Wikipedia. This list was utilized to guide the data collection and entity identification process.
Institutions
- Kwangwoon University