Substack Articles & Newsletter Metadata Dataset
Description
This dataset presents a clean, large-scale collection of 3,513,881 unique articles and newsletter metadata scraped ethically from public Substack publication domains. It is specifically designed for Natural Language Processing (NLP), Computational Linguistics, Content Strategy Analytics, and Topic Modeling. Key Features & Value: 1. Complete Cleanliness: Includes 3.5M+ records with HTML entities fully decoded (removing encoded quotes, apostrophes, and special character artifacts). 2. Deduplicated Integrity: Filtered and deduplicated by unique article URLs to ensure zero duplicate entries. 3. Feature-Rich Metadata: Contains title character lengths, title word counts, subtitle presence flags, cleaned domain names, and standardized ISO publication timestamps (datetime64[us]). 4. Automated Topic Categorization: Posts are systematically classified into 10 key domain categories (Finance & Crypto, Technology & AI, Health & Wellness, Business & Startups, Politics & News, Sports, Culture/Arts/Literature, Food & Travel, Science & Environment, Personal Essays & Stories) using a high-throughput deterministic regex algorithm. 5. Optimized Storage: Encoded using Apache Parquet (.parquet), allowing low-memory footprints (~255 MB in RAM) and fast parallel processing in Python, R, and DuckDB.
Files
Steps to reproduce
1. Data Scraping: Publicly available article URLs, post titles, domain information, and metadata were ethically collected directly from public Substack publication RSS feeds and site indices using Python. 2. Schema Normalization: Aggregated raw data files into Apache Parquet format. Stripped PyArrow schema metadata conflicts to ensure strict type casting across large text strings and timestamps. 3. Text Cleaning: Executed automated entity replacements on post titles and subtitles to remove HTML character codes (e.g., “, ”, ’, &, "). 4. Deduplication: Applied SQL window functions via DuckDB to deduplicate entries by unique article_url, retaining the most recent post entry based on publish_date. 5. Feature Engineering: Extracted title_char_length, title_word_count, and subtitle existence (has_subtitle). Applied regular expression rule-sets across text blocks to assign standard thematic categories.
Institutions
- University of Frontier Technology, BangladeshDhaka Division, Gazipur