PolyglotFakeFacts-ITPS: Reproducibility Package for Multilingual Information Threat Prioritization
Description
This repository provides the reproducibility package associated with the experimental evaluation of the Information Threat Priority Score (ITPS), an interpretable framework for multilingual information-threat prioritization. The package is based on a controlled subset of the PolyglotFakeFacts V2 dataset and contains 2,000 news articles (1,000 Fake and 1,000 Real) with identical language distributions across classes. The subset was constructed using a deterministic language-balanced sampling procedure with random seed 20260819. The experimental ITPS evaluated in this study combines three components: leakage-controlled authenticity risk (A), embedding-derived security relevance (S), and cross-lingual narrative co-presence (P). Security relevance and cross-lingual semantic similarity were computed using the sentence-transformers/paraphrase-multilingual-mpnet-base-v2 multilingual embedding model. The package includes the experimental subset, the selection and provenance manifest, the executed experiment notebook, leakage and robustness audits, precomputed out-of-fold authenticity probabilities, final article-level A/S/P/ITPS results, ranking-reversal analyses, sensitivity analyses, and documentation of the experimental environment and methodology. The authenticity component was evaluated using source/domain-disjoint and near-duplicate-aware five-fold cross-validation with boilerplate/source-fingerprint stripping. The final leakage-controlled authenticity probabilities used by the ITPS experiment achieved ROC-AUC = 0.943520. This package is intended to support transparency, verification, and reproducibility of the reported ITPS experiment. It is a derived experimental research artifact and does not replace or constitute a new version of the parent PolyglotFakeFacts dataset. Parent dataset: Ciobanu, Alexandru (2026), “PolyglotFakeFacts: A Multilingual Dataset of Fake and Real News across Politics, Security, and Social Domains,” Mendeley Data, V2, DOI: 10.17632/gff8bmr4ff.2.
Files
Steps to reproduce
Steps to reproduce Download and extract the complete reproducibility package. Use PolyglotFakeFacts_ITPS_Fake_1000.xlsx and PolyglotFakeFacts_ITPS_Real_1000.xlsx as the experimental input subset. The selection and row-level provenance of the 2,000 observations are documented in PolyglotFakeFacts_ITPS_Selection_Manifest.xlsx. The subset was generated deterministically using random seed 20260819. Install the Python dependencies specified in requirements.txt. The reference experiment was executed under Python 3.12.13 with PyTorch 2.11.0+cu128, sentence-transformers 5.7.0, scikit-learn 1.6.1, and SciPy 1.16.3. Open and execute ITPS_Reproducibility_Experiment.ipynb sequentially. A CUDA-capable GPU is recommended for embedding inference. The notebook loads the fixed multilingual embedding checkpoint sentence-transformers/paraphrase-multilingual-mpnet-base-v2 and aligns the precomputed leakage-controlled out-of-fold authenticity probabilities (A) to the 2,000 observations using URL as the unique join key. Execute the notebook cells sequentially to compute security-related semantic relevance (S), cross-lingual narrative co-presence (P), the equal-weight three-component score ITPS_3 = (A + S + P) / 3, ranking comparisons, weight sensitivity, P-threshold sensitivity, component correlations, and translation sensitivity. Compare the reproduced outputs with RRIA_ITPS_Final_Experiment.xlsx, RRIA_ITPS_Ranking_Reversals.xlsx, and RRIA_ITPS_Final_Experiment_Report.md. The construction and robustness evaluation of the authenticity component A are documented separately in RRIA_ITPS_Round2_Audit_Authenticity.xlsx, RRIA_ITPS_Round2_M1_M2_Robustness.md, RRIA_ITPS_Round2_Translation_Boilerplate_Audit.md, and AUTHENTICITY_A_REPRODUCIBILITY_NOTE.md. The deposited notebook imports the final precomputed A probabilities rather than recomputing the authenticity classifier from raw text. Reference checks include: N = 2,000 observations; authenticity ROC-AUC = 0.943520; Spearman ρ(A, ITPS_3) = 0.919745; and authenticity-vs-ITPS Top-100 overlap = 48/100.