Code-Mixed Indic Languages with Emoticons for Sarcasm Detection
Description
This dataset consists of code-mixed multilingual text data designed for sentiment analysis research. It captures naturally occurring code-mixed patterns combining English with ten Indian languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, Telugu and Urdu. The dataset aims to support studies in multilingual NLP, sentiment classification, and language processing for real-world social media and conversational data. Dataset Description The dataset contains the following attributes: • Text: The original code-mixed text sample. • Sentiment: The corresponding sentiment label (positive, negative, or neutral). • Translated_text: English translation of the original text. • Cleaned_text: Text after preprocessing, including lowercasing, punctuation and stopword removal, and normalization. • Tokens: Tokenized representation of the cleaned text. Preprocessing involved cleaning (removal of punctuation, URLs, and emojis), normalization of repeated characters, language-specific stopword removal, translation to English, and token formation for downstream NLP tasks.
Files
Steps to reproduce
1. **Data Collection:** Gather social media posts and conversational text containing natural code-mixed usage between English and the target Indic languages (Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, Telugu, and Urdu). 2. **Filtering:** Retain posts that contain at least one Indic language token mixed with English, discarding monolingual samples. 3. **Annotation:** Manually label each sample with sentiment categories — *positive*, *negative*, or *neutral* — by bilingual annotators familiar with code-mixing patterns. 4. **Cleaning:** Remove URLs, mentions, hashtags, punctuation, and emojis; normalize elongated characters (e.g., “goooood” → “good”); and lowercase all text. 5. **Stopword Removal:** Apply language-specific stopword lists for each Indic language to remove functional words. 6. **Translation:** Generate English translations for each code-mixed text using a multilingual translation API (e.g., Google Translate or IndicTrans2). 7. **Tokenization:** Split cleaned text into word-level tokens using an Indic NLP tokenizer or multilingual tokenizer (e.g., spaCy, IndicNLP Library). 8. **File Preparation:** Compile the processed data into a CSV file with columns — *Text*, *Sentiment*, *Translated_text*, *Cleaned_text*, and *Tokens*. 9. **Reproducibility:** Researchers can replicate preprocessing using the following minimal setup: ```bash pip install pandas indic-nlp-library nltk googletrans==4.0.0rc1 python preprocess.py --input raw_data.csv --output processed_dataset.csv ``` 10. **Usage:** Load the dataset for sentiment analysis or code-mixed language modeling using: ```python import pandas as pd df = pd.read_csv("code_mixed_sentiment.csv") print(df.head()) ```