DataGuard
Description
The DataGuard dataset links three artifacts that are usually studied separately: structured Google Play Data Safety declarations, privacy-policy text, and human expert judgments about whether the former is supported by the latter. CONTENTS - apps.csv: 2,400 Android apps balanced across 12 Google Play categories (200 each), with package name, developer, download tier, Data Safety URL, privacy-policy URL and host. - app_texts.csv: the Data Safety content and privacy-policy text for those 2,400 apps. - audit_judgments.csv: 1,506 human audit records over 1,214 unique apps by seven annotators, each with up to four verdicts (correctness and completeness, for data sharing and data collection). - evidence_excerpts_long.csv: 4,622 policy excerpts annotators selected as evidence. - rationales_long.csv: 4,866 short written rationales linked to verdicts. - annotators_anonymized.csv, category_lookup.csv, data_safety_corpus_3600.csv (the wider sampling frame). - metadata/: codebook, field dictionary, summary statistics, SHA-256 manifest. KEY FINDINGS IN THE DATA - 71.6% of audited apps carry at least one disclosure defect; per-dimension rates run 35.5-42.5%. - Agreement between independent annotators is low: 52-61% raw agreement, Cohen's kappa between -0.066 and 0.156, with 222 of 289 repeatedly judged apps drawing a conflicting verdict. THREE THINGS TO KNOW BEFORE REUSE 1. privacy_policy_content is NOT raw policy text. The retrieved policy was condensed by GPT-4 into "Data Share:" and "Data Collect:" sections, at an unfixed decoding temperature, and that condensation is what annotators reviewed. Do not reuse it as a verbatim policy corpus; re-fetch from privacy_policy_url if original wording is needed. 2. The labels are not a gold standard. Given the agreement figures above, treat them as a sample from a wide distribution of expert opinion. The repeated judgments and annotator_id are released so this uncertainty can be modelled. 3. The direction of Incomplete is unresolved. "Incorrect" means the policy mentions a practice the Data Safety declaration omits, and that direction is empirically verified. The same test is inconclusive for "Incomplete", which should be treated as noisier. The audit judgments were produced by the authors of this dataset, working as domain experts on publicly available documents. No third-party participants were recruited and no personal data of app users was collected. These are research annotations, not legal determinations of compliance. A recorded defect documents a mismatch between two disclosure artifacts; it is not a measurement of app runtime behaviour and not an allegation of wrongdoing.
Files
Institutions
- RMIT VietnamHo Chi Minh, Ho Chi Minh City
- FPT UniversityHanoi, Hanoi