EnergyFETool: A Tool-Calling Dataset for LLM-Based Feature Engineering in Energy Systems

Description

EnergyFETool is an LLM-generated dataset designed to enable large language models to acquire engineer-level tool-calling capabilities for feature engineering in the energy domain through a two-stage fine-tuning process. The public release of the EnergyFETool dataset aims to promote the automation of feature-engineering tasks across various forecasting scenarios in the energy domain, thereby reducing reliance on experienced engineers and significantly lowering both human labor costs and time expenditure. All data generation, fine-tuning, and validation procedures are implemented through reproducible, open-source Python workflows, and are accompanied by accuracy evaluations to verify the effectiveness of the dataset

Steps to reproduce

1. Data Generation: Scenario descriptions and their corresponding feature contents are first generated using the TXT template files provided in prompt_examples.rar. These outputs are then manually annotated and filtered. Next, the initial feature-engineering operation pairs are constructed according to the predefined format. The synthetic_sets.py script is subsequently used to generate the training data, resulting in the SFT_FE.jsonl file. 2. LLM Fine-Tuning: The fine-tuning process consists of two stages. In the first stage, the SFT_FE.jsonl file is used to perform supervised fine-tuning with the lora_finetune_sft.py script, allowing the LLM to learn multiple feature-engineering tool calls within a single workflow. After the SFT-fine-tuned LLM is obtained, the Data_pairs_GRPO.py script is applied to generate response pairs, which are stored in SFT_output_pairs.jsonl. Larger teacher models, including the open-source Qwen2.5-72B-Instruct model and the closed-source GPT-4o and GPT-5.1 models, are used to identify the preferred and discarded responses in each pair. These preference annotations are then organized into the GRPO_FE.jsonl file. In the second stage, GRPO_FE.jsonl is used for GRPO fine-tuning, which is implemented through the lora_finetune_grpo.py script. During fine-tuning, a nohup.out file is generated, which records the gradient descent process of the base LLM throughout training. 3. Evaluation: Feature-engineering operations are executed using EnergyFETool.py, which internally invokes both FE.py and tool_sets.py to perform the corresponding feature-engineering procedures.

Institutions

• UNSW Sydney

  New South Wales, Sydney
• Hohai University

  Jiangsu, Nanjing

Categories

Licence