Bangladesh in Transition: A Large-Scale Dataset of 55,000+ English News Articles from Five Major National Newspapers (2024–2025)
Description
This dataset comprises 55,368 English-language news articles from five major Bangladeshi newspapers (The Daily Star, Bangladesh Pratidin, Prothom Alo, The Bangladesh Today, Daily Sun) that include the political transition period of June 2024 to June 2025. Harvested with a custom Scrapy spider utilizing a hybrid extraction method (JSON-LD/HTML), the corpus tackles the shortage of South Asian English data for NLP and LLM training. Research Motivation & Findings The collection offers empirical investigation of media narratives and linguistic alterations amid political turbulence. Analysis finds large temporal changes, with publishing peaks in November 2024 (2,044 articles) and May 2025 (2,113 articles) corresponding with major political events. Dominant themes include "Government," "Adviser," "Yunus," and "Reform." Data Characteristics Volume: 55,368 articles (Avg. length: 337 words). Top Sources: The Daily Star (31,292) and Bangladesh Pratidin (18,375). Methodology: High-fidelity metadata extraction (authors, timestamps) via API/JSON-LD parsing. Files & Usage The data is provided in two distinct states to support diverse research needs: Raw Data Component (Archive State) Files: news_articles.db (SQLite), news_articles_raw.csv, news_articles_raw.xlsx Content: Original 55,368 records including web artifacts and potential duplicates. Usage: Source verification, reproducibility studies, and duplicate detection research. Processed Data Component (NLP-Ready State) Files: news_articles_clean.csv, news_articles_clean.xlsx Content: 54,347 unique records (853 duplicates removed, nulls filtered). Usage: Optimized for Named Entity Recognition (NER), topic modeling, summarization, and sentiment analysis.
Files
Steps to reproduce
Spider Development: A custom Scrapy spider was written in Python to target the archive portions of five major Bangladeshi newspapers. Configuration: The spider was configured with a robust extraction logic: Primary: Parsed JSON-LD structured data for high-fidelity metadata. Secondary: Used XPath/CSS selectors as a fallback for HTML parsing to pull material straight from the DOM. Data Collection: The spider crawled the target websites during the period of June 2024 to June 2025, utilizing a local SQLite database for real-time deduplication and rate limitation (0.5s–2.0s delay) to respect server load. Preprocessing: Raw Export: The scraped data was exported to a raw CSV and SQLite database. Cleaning: A Python script deleted duplicates (based on URL/headline) and filtered out records with null text fields. Normalization: Dates were standardized to ISO 8601 format, and content was stripped of HTML tags and excessive whitespace. Final Output: The processed dataset was saved as a clean CSV file (news_articles_clean.csv) containing 54,347 unique records.
Institutions
- American International University Bangladesh