BanglaDual: A 1.24-Million Pair Multi-Source Dataset for Banglish Transliteration and Translation
Description
This dataset provides a large-scale, unified resource for Banglish (Romanized Bengali) natural language processing. It contains 1,122,189 Banglish-to-Bangla transliteration pairs and 119,727 Banglish-to-English translation pairs, totaling over 1.24 million aligned texts. The data is aggregated from six distinct public sources for transliteration and four for translation, encompassing diverse domains such as general Wikipedia text (85.34%), emotion detection (7.12%), hate speech (3.33%), academic transliteration benchmarks, and regional dialects. A key feature of this dataset is its unified schema, which resolves the fragmented nature of previous Banglish resources. All textual data underwent rigorous standardization, including Unicode NFKC normalization to handle Bengali character variations, cross-source deduplication, and the removal of missing values. Furthermore, every entry is tagged with its original source, allowing researchers to filter data by domain. To support immediate model development, the dataset includes a pre-computed 80/10/10 train/validation/test split (seed = 42). This dataset is roughly 30 times larger than previously available benchmarks like BanglaTLit and is the first to support both transliteration and translation tasks from a single format. It aims to support researchers addressing the challenges of code-mixed Bengali text in low-resource environments. Repository File Structure:To facilitate ease of use, the repository is organized into five directories containing the data files in Apache Parquet format (with Snappy compression for efficiency) and standard CSVs for quick inspection. š Task-Specific-Datasets/banglish_bangla_transliteration.parquet: Contains 1,122,189 cleaned and deduplicated transliteration pairs. banglish_english_translation.parquet: Contains 119,727 translation pairs, including 470 manually translated by the authors. š Merged-Dataset/banglish_merged_dual_task.parquet: Contains 1,122,998 rows produced by an outer join of the Bangla and English datasets, supporting multi-task learning. š Model-Splits/train.parquet: Training set partition (898,024 rows). validation.parquet: Validation set partition (112,253 rows). test.parquet: Testing set partition (112,254 rows). š Sample-Data/samples_bangla.csv: 60 randomly sampled transliteration records for quick inspection. samples_english.csv: 40 randomly sampled translation records for quick inspection. sample_merged_data.csv: A representative sample showcasing the outer-joined dual-task format structure. š Processing-Scripts/banglish_to_bangla_pipeline.py: Code to process and merge the six Bangla sources. banglish_to_english_pipeline.py: Code to process the four English sources. merge_datasets.py: Code to perform the outer join and generate the 80/10/10 splits.
Files
Steps to reproduce
The dataset was constructed through a rigorous 10-step pipeline: Data Acquisition: Raw data was downloaded from six publicly available datasets for Bangla transliteration and four for English translation. Original formats included CSV, XLSX, and Parquet. Data Loading: The source files were ingested using the Python Pandas library. Column Standardization: Heterogeneous column names across sources were standardized into a unified schema: Banglish (input text), Bangla or English (target text), and Source. Unicode Normalization: NFKC (Normalization Form Compatibility Composition) normalization was applied to all text fields. This ensures the canonical representation of Bengali combining characters (e.g., vowel signs) and is critical for accurate deduplication. Missing Value Removal: Any rows containing empty or NaN values in the input or target columns were filtered out. Note: 470 English translation pairs and some missing values were manually filled by the authors prior to this step. Deduplication: Exact-pair deduplication was executed based on the combination of the Banglish input and its corresponding target, retaining only the first occurrence. This removed over 25,000 redundant pairs. Source Attribution: Each row was tagged with its origin dataset name to maintain traceability and enable domain-specific filtering. Data Export: The cleaned, task-specific datasets were saved as Apache Parquet files using Snappy compression to preserve data types and reduce storage size. Dual-Task Merging: An outer join was performed on the Banglish column to create a unified dataset supporting both transliteration and translation. Data Splitting: The merged dataset was partitioned into train (80%), validation (10%), and test (10%) sets using a fixed random seed (seed = 42) to ensure reproducibility.
Institutions
- American International University-BangladeshDhaka Division, Dhaka