A Comprehensive Multilingual Dataset of Farmers' Queries and Expert Advisories for LLM and RAG-Driven Agricultural Chatbots

Description

This dataset, curated under the master file A Comprehensive Multilingual Dataset of Farmers' Queries and Expert Advisories for LLM and RAG-Driven Agricultural Chatbots, serves as a high-quality, domain-specific multilingual Question-Answering (QA) repository. It is explicitly designed for benchmarking, fine-tuning, and evaluating Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) frameworks within the smart agriculture domain. The primary goal of this data is to accelerate research in building intelligent, web-based, and offline-capable agricultural chatbots (Agribots) that diagnose crop and fish health issues, recommend fertilization strategies, and provide expert advisory directly to farmers without relying on continuous internet connectivity. ​The dataset contains a total of 29,982 high-token expert-verified rows. To facilitate balanced cross-lingual NLP research, the data is perfectly synchronized and balanced across three major languages with exactly 9,994 rows per language. These languages are English (en) for global agricultural standards, Bengali (bn) for regional South-Asian agricultural contexts, and Chinese (zh) for East-Asian crop management and advanced farming query patterns. ​Each file contains 7 distinct attributes: id (an integer unique identifier for each query-response pair), question (real-world textual queries from farmers), answer (expert-approved actionable diagnostic advisories and solutions), category (thematic classification like Crops, General Agriculture, or Government Scheme/Finance), language (ISO codes 'en', 'bn', or 'zh'), q_length (character length of the question), and a_length (character length of the answer). This dataset is ideal for instruction-tuning open-source LLMs like LLaMA, Mistral, or Qwen, benchmarking RAG pipelines using vector databases like ChromaDB or Pinecone, and training lightweight models for edge deployment in remote areas with low-bandwidth infrastructure.

Steps to reproduce

To ensure rigorous machine learning evaluation pipelines, the master dataset datast.xlsx has been partitioned into standard Train, Validation, and Test subsets using a 70:15:15 ratio. A stratified sampling technique based on the language attribute was strictly enforced during the split. This guarantees that all three languages maintain an exact, mathematically equal representation across all subsets, preventing any data bias during model training. ​The subset statistics are distributed as follows: the Train Dataset contains 70% of the data (20,987 rows) to serve as the model optimization or vector store base, the Validation Dataset contains 15% (4,497 rows) for hyperparameter tuning and prompt engineering evaluation, and the Test Dataset contains 15% (4,498 rows) for final model benchmarking and RAG retrieval accuracy testing. ​Researchers can perfectly reproduce the exact dataset split using Python and the scikit-learn library with the following automated script: ​import pandas as pd from sklearn.model_selection import train_test_split ​1. Load the master dataset ​df = pd.read_excel('datast.xlsx') ​2. Apply Stratified Split to preserve exact language ratios ​train_df, temp_df = train_test_split(df, test_size=0.30, random_state=42, stratify=df['language']) val_df, test_df = train_test_split(temp_df, test_size=0.50, random_state=42, stratify=temp_df['language']) ​3. Export subsets for machine learning workflows ​train_df.to_excel('train_dataset.xlsx', index=False) val_df.to_excel('val_dataset.xlsx', index=False) test_df.to_excel('test_dataset.xlsx', index=False) ​print("Subsets successfully reproduced with exact 70-15-15 distribution.")

Institutions

• International Islamic University Chittagong

  Chittagong, Chittagong
• National University Bangladesh

  Dhaka Division, Dhaka

Categories

Licence