Dwadash: A Large-scale Benchmark Dataset for Multi-Dialectal Translation Across Twelve Bengali Variations

Description

This dataset is a comprehensive parallel corpus for poly-dialectal neural machine translation (NMT) between Standard Bangla and its regional dialects. It covers 12 linguistic variations: Standard Bengali, Sylheti, Chittagonian, Barisal, Mymensingh, Noakhali, Rangpur, Rajshahi, Kishoreganj, Narail, Narsingdi, and Tangail. The corpus integrates existing publicly available and manually curated resources, including Ancholik-NER, Anubhuti, BanglaDial, BhasaBodh, ChatgaiyyaAlap, ONUBAD, Vashantor, and a newly developed Novel Manual Corpus. The resulting corpus contains 20,414 Standard Bangla source sentences, with dialect-specific parallel data distributed across Sylheti (10,883), Chittagonian (15,029), Barisal (5,251), Mymensingh (9,193), Noakhali (5,981), Rangpur (1,157), Rajshahi (891), Kishoreganj (502), Narail (502), Narsingdi (502), and Tangail (502). The Novel Manual Corpus contributes 2,510 additional Standard Bangla sentences, covering five dialects—Rangpur, Kishoreganj, Narail, Narsingdi, and Tangail—with 502 sentences for each dialect. This expansion enables the dataset to include dialects that are substantially underrepresented in existing Bengali dialect corpora. Corpus Composition: Source Corpus | SCB (Standard Bangla) | Syl (Sylheti) | Cht (Chittagonian) | Bar (Barisal) | Mym (Mymensingh) | Noa (Noakhali) | Ran (Rangpur) | Raj (Rajshahi) | Kis (Kishoreganj) | Nar (Narail) | Ndi (Narsingdi) | Tan (Tangail) Ancholik-NER | 3,481 | 3,481 | 3,481 | 3,481 | 3,481 | 3,481 | 0 | 0 | 0 | 0 | 0 | 0 Anubhuti | 2,500 | 2,500 | 2,500 | 0 | 2,500 | 0 | 0 | 0 | 0 | 0 | 0 | 0 BanglaDial | 3,452 | 442 | 577 | 790 | 712 | 0 | 655 | 891 | 0 | 0 | 0 | 0 BhasaBodh | 980 | 980 | 980 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 ChatgaiyyaAlap | 4,011 | 0 | 4,011 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 ONUBAD | 980 | 980 | 980 | 980 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 Vashantor | 2,500 | 2,500 | 2,500 | 0 | 2,500 | 2,500 | 0 | 0 | 0 | 0 | 0 | 0 Novel Manual Corpus | 2,510 | 0 | 0 | 0 | 0 | 0 | 502 | 0 | 502 | 502 | 502 | 502 Total | 20,414 | 10,883 | 15,029 | 5,251 | 9,193 | 5,981 | 1,157 | 891 | 502 | 502 | 502 | 502 Abbreviations: SCB = Standard Bangla Syl = Sylheti Cht = Chittagonian Bar = Barisal Mym = Mymensingh Noa = Noakhali Ran = Rangpur Raj = Rajshahi Kis = Kishoreganj Nar = Narail Ndi = Narsingdi Tan = Tangail

Institutions

• Sylhet Engineering College

  Sylhet Division, Sylhet

Categories

Related Links

Licence