BanglaBias: A Bengali Counterfactual-Pair Dataset for Social Bias Evaluation
Description
BanglaBias is a manually curated dataset for evaluating social bias in pretrained Bangla language models. The dataset contains 1,626 minimal sentence pairs covering nine sociocultural bias categories relevant to the Bangla-speaking context: gender, religion, ageism, socioeconomic status, occupation, disability, physical appearance, region, and ethnicity. Each pair consists of a stereotypical and a counter-stereotypical sentence that differs primarily in selected demographic or socially relevant identifiers while preserving the surrounding context. The dataset was constructed from Bangla textual sources including Wikipedia, literature, blogs, and social media. The dataset can be used for systematic evaluation of social bias, stereotype detection, fairness assessment, and benchmarking of Bangla language models.
Files
Steps to reproduce
The BanglaBias dataset was constructed through a systematic manual process. Bangla textual contexts were first collected from diverse sources, including Wikipedia, literature, blogs, and social media. Contexts containing potentially stereotypical associations involving social or demographic groups were then identified and categorized into nine sociocultural bias categories: gender, religion, ageism, socioeconomic status, occupation, disability, physical appearance, region, and ethnicity. Selected contexts were transformed into counterfactual sentence pairs by modifying the relevant demographic or social identifiers while preserving the surrounding sentence context. For each context, stereotypical and counter-stereotypical variants were created and manually reviewed for linguistic naturalness, consistency, and category relevance. The finalized sentence pairs were compiled into the BanglaBias dataset, containing 1,626 minimal pairs.
Institutions
- Shahjalal University of Science and TechnologySylhet Division, Sylhet