Bangla Regional Dialect-to-Standard Bangla Parallel Corpus for Dialect Bias Evaluation
Description
This dataset is a balanced four-dialect parallel corpus for Bangla dialect normalization and dialect-bias evaluation. It contains 15,376 dialect-to-standard Bangla sentence pairs representing 3,844 shared meanings across four regional varieties: Barishali, Chittagonian, Sylheti, and Noakhali. Each dialect sentence is aligned with the same Standard Bangla reference, enabling controlled comparison of translation and normalization performance across dialects. The corpus was compiled by merging four publicly available resources: Vashantor (approximately 48%), ONUBAD (23%), BanglaDial (17%), and the Bangla Dialect Dataset (11%). To address gaps in aligned Noakhali data, 1,807 sentences were manually translated by native Noakhali speakers and independently checked by a second speaker from the same region. This annotation effort enabled complete four-way parallel coverage. The dataset is provided with training, validation, and test splits containing 12,304, 1,540, and 1,532 pairs, respectively. Because multiple dialect versions correspond to the same underlying meaning, the dataset follows a meaning-level splitting protocol. Meaning groups are assigned to splits first and expanded into dialect-to-standard pairs afterward, preventing direct meaning overlap between splits. As an additional diagnostic, pair-level analysis shows that 99.8% of test targets are represented in the training data through a sibling dialect. This property supports research on cross-dialect generalization, transfer learning, dialect bias, and low-resource Bangla neural machine translation.
Files
Steps to reproduce
1. Obtain the four source resources: Vashantor, ONUBAD, BanglaDial, and the Bangla Dialect Dataset. Retain their original citations and CC BY 4.0 license information. 2. Extract the Standard Bangla sentences and the corresponding regional dialect versions. Select records that share the same underlying meaning and retain the four target dialects: Barishali, Chittagonian, Sylheti, and Noakhali. 3. To complete the missing Noakhali alignments, manually translate 1,807 Standard Bangla sentences into Noakhali. Each translation is produced by a native Noakhali speaker and checked by a second native speaker from the same region. 4. Normalize file structure and text encoding while preserving the original dialect wording. Store each example as a dialect-to-standard pair with the fields: input_text, target_text, and dialect_label. 5. Combine the aligned records into a four-way parallel corpus containing 3,844 shared meanings and 15,376 dialect-to-standard pairs. 6. Split the data at the meaning level rather than at the individual pair level. First assign complete meaning groups to the training, validation, or test partition; then expand each assigned meaning into its available dialect pairs. This prevents different dialect versions of the same meaning from being placed in different partitions. 7. Export the final files as: - train.csv: 12,304 pairs - val.csv: 1,540 pairs - test.csv: 1,532 pairs 8. For evaluation of cross-dialect overlap, compare the Standard Bangla targets in the test set with the targets associated with sibling dialects in the training set. This analysis shows that 99.8% of test targets are reachable through a sibling dialect in training.
Institutions
- Bangladesh University of ProfessionalsDhaka Division, Dhaka