2026-10-15T10:40:40Z https://data.mendeley.com/oai

oai:data.mendeley.com/jmrhxb5666.1 2025-10-27T01:42:09Z

Manzoor, Faraz Pairwise Duplicate News Dataset Mendeley Data The Pairwise Duplicate News Dataset consists of 10,836 article pairs (5,418 duplicate and 5,418 non-duplicate) collected from seven major Pakistani English news outlets, DAWN, The Express Tribune, GEO News, The News International, The Nation, 24 News, and 92 News HD. The dataset was created from a larger corpus of 54,929 news articles covering eight categories: National, World, Business, Sports, Entertainment, Crime, Technology, and Health. The dataset was constructed using Sentence-BERT embeddings and FAISS cosine similarity search, with a similarity threshold of 0.85 for identifying duplicate articles. Each duplicate cluster represents a single real-world news event, and one pair was sampled per cluster. Non-duplicate pairs were formed by randomly pairing articles from different clusters and stratified into three similarity levels: easy (0.2-0.3), medium (0.3-0.5), and hard (0.5-0.7) to ensure balanced evaluation difficulty. Computer Science Artificial Intelligence Journalism Zahid, Shumaila Farooq, Uzma Qureshi, Junaid Nasir Abid, Adnan Dataset doi:10.17632/jmrhxb5666.1 oai:data.mendeley.com/jmrhxb5666.1 Creative Commons Attribution 4.0 International http://creativecommons.org/licenses/by/4.0 https://data.mendeley.com/datasets/jmrhxb5666 2025-10-27T01:42:09Z