Advancing Bengali Next-Word Prediction: A Multi-Domain Text Dataset for Language Modelling and Predictive Text
Description
This is a comprehensive collection of Bengali texts drawn from multiple areas of discourse to support research into Bengali language modeling and next-word prediction. The data was compiled in Summer 2025 from 32 publically available sources which contain a wide variety of writing styles and topics (i.e., national news portals; Wikipedia; Class IX-X textbooks; religious and ethical writings; literature and poetry; blogs; publicly visible social media text; and a small number of clearly labeled examples generated by an artificial intelligence). All data was stored in UTF-8 Bengali Unicode text format and grouped at the source level to enable transparent provenance tracking, domain-based testing, and experimental replication. With respect to size, this combined corpus contained 503,257 tokens and 84,788 unique word forms before preprocessing and 498,023 tokens and 62,009 unique word forms after preprocessing, providing a large body of data for training, benchmarking, and comparative analysis of both statistical and neural methods for Bengali next-word prediction.
Files
Steps to reproduce
1. Create an environment: Set-up a python environment (local Google colab etc.) and install the core packages: numpy, pandas, re (native) and unicodedata (native). If you plan on reproducing source extraction/acquisition processes, also download and install: requests, beautifulsoup4, pymupdf (with fitz), and pytesseract (with Tesseract OCR engine). Packages such as nltk and scikit-learn are discretionary and only needed for any type of descriptive analysis/plots. 2. Obtain the source data: Obtain and validate the deposited files. Verify the text files were saved as UTF-8 Bengali Unicode. Open the raw/merged corpus files and verify the main text fields contain correct Bangla text; if available, verify the source/domain labels are readable to allow for filtering based on source/domain. 3. Perform integrity checks: Delete the non-analytical columns (i.e., index/column IDs); delete any rows containing missing/empty primary text; delete duplicate rows identified both between and across source domains. Save the cleaned staging table as the validated merged corpus. 4. Re-generate the cleaned/preprocessed corpus: Reapply the previously defined cleaning steps to generate the final corpus layer: Unicode normalization, whitespace standardization, removal of non-Bangla/Latin script and noise symbols, punctuation standardization, and controlled digit treatment. Export the generated layer as the final cleaned corpus file. 5. Re-generate the token output: Performed whitespace-based word tokenization without compound segmentation and stored the resulting tokens in a new column/structure consistent with the originally deposited format. 6. Re-generate the next-word prediction pairs (N-gram): Created overlapping 4-grams by applying a sliding window: the first three tokens are the context and the fourth token is the predicted next word. Stored the created context-target pairs in the same format as the originally deposited modelling file (i.e., spreadsheet/CSV).
Institutions
- Daffodil International University