BanglaEmotion: A Benchmark Dataset for Bangla Textual Emotion Analysis
We present a manually annotated Bangla Emotion corpus, which incorporates the diversity of fine-grained emotion expressions in social-media text. We tried to consider more fine-grained emotion labels such as Sadness, Happiness, Disgust, Surprise, Fear and Anger - which are, according to Paul Ekman (1999), the six basic emotion categories. For this task, we collected a large amount of raw text data from the user’s comments on two different Facebook groups (Ekattor TV and Airport Magistrates) and from the public post of a popular blogger and activist Dr. Imran H Sarker. These comments are mostly reactions to ongoing socio-political issues and towards the economic success and failure of Bangladesh. We scrape a total of 32923 comments from the three sources aforementioned above. Out of these, a total of 6314 comments were annotated into the six categories. The distribution of the annotated corpus is as follows: sad = 1341 happy = 1908 disgust = 703 surprise = 562 fear = 384 angry = 1416 We have also provided a balanced set from the above data and split the dataset into training and test set of equal ratio. We considered a proportion of 5:1 for training and evaluation purpose. More information on the dataset and the experiments on it could be found in our paper (related links below).
Steps to reproduce
In order to develop a Bangla Emotion corpus, the initial goal was to annotate user comments into six categories (anger, disgust, fear, joy, sadness and surprise). The idea was to read through all those comments and try to capture the intent of the author so that an appropriate label could be tagged to that comment out of the six categories. We decided to focus our attention on comments related to the socio-political phenomenon for two main reasons. First, these kinds of comments from social network sites typically contain a high load of emotional content, as they describe major national events, and are written in a style meant to attract the attention of the readers. Second, the structure of those comments was appropriate for our goal of conducting document-level annotations of emotions. Most of the comments are bigger than one sentence in length. The dataset was labelled by five annotators autonomously. The annotators were undergraduate students from a computer science background who were interested to get involved in NLP research. For each comment, the appropriate emotion label was assigned based on the presence or absence of words or phrases attached to emotional content. The underlying feelings invoked by those comments were also considered as well. As we have collected a huge number of comments, not all the comments were annotated mainly because we are focusing on document level annotation. For this, annotations were focused on only those comments that are at least of one sentence with a word length of 5. We tried to ignore the comments that are below the above threshold value in most of the time if the comment does not include too many informative words. In the case where more than one emotions were plausible, the annotations were done by following the ‘first intuition’. We also tried to clean up the data a little bit because those comments contained some uninformative symbols and links which are not relevant to the task of emotion analysis. For inter-annotation agreement, we considered the label given by at least three out of the five annotators (an agreement of 60%).