Indonesian legislation as data: 90,588 norm units extracted from 378 official regulatory PDFs (1945–2024) with resolvable provenance records
Description
A corpus of 90,588 norm units segmented from 378 official Indonesian regulations (1945–2024), with page- and line-level provenance into the distributed text files. The dataset contains: (1) units.csv — the main table, one row per norm unit, with structural markers (BAB, Paragraf, Pasal, ayat), original and normalised text, IndoBERT token statistics, and a quality flag; (2) documents.csv — the document inventory, with the regulation identity read from each document's title block, the values found in the file name, and a two-axis classification of legal force under Law 12 of 2011 (ladder position under Article 7(1), or delegated force under Article 8); (3) texts.zip — 378 page-delimited plain-text files that the page and line references in units.csv resolve into; (4) scripts.zip — the Python scripts used to build the corpus and compute the quality figures; and (5) checksums.md5 and README.md. Every unit resolves to a document, page, and line range in the distributed text files, so any unit can be verified against its source. The data can be reused for legal NLP tasks such as segmentation, classification, and retrieval, for studies of Indonesian delegated legislation, and as a starting point for converting Indonesian legislation into structured formats such as Akoma Ntoso. Source PDFs are official public documents from Indonesian government legal-documentation portals (JDIH) and are not redistributed here
Files
Steps to reproduce
The corpus was built with a six-stage pipeline; all scripts are included in scripts.zip. (1) Acquisition: 378 official regulatory PDFs were collected from Indonesian government legal-documentation portals (JDIH) and stored as immutable source files, organised in folders by regulation type. (2) Text extraction: each PDF was converted to a page-delimited plain-text file using the embedded text layer (pypdf). Page boundaries are explicit and every non-empty line carries a stable ordinal within its page. These files are distributed in texts.zip and are what all later stages read. (3) Page furniture removal: repository URLs, gazette headers, and folio marks were removed by logged line-level pattern rules before segmentation; phrases that also occur inside legitimate citations were flagged rather than removed. (4) Hierarchical segmentation: cleaned lines were segmented into norm units along the drafting hierarchy (BAB, Paragraf, Pasal, ayat) by carrying a heading state across lines and closing a unit at sentence terminators. (5) Identity and hierarchy: the type, number, and year of each regulation were read from the document's own title block, cut at the first recital marker (Menimbang / Mengingat / Memutuskan / Menetapkan), and each document was classified on two axes of legal force under Law 12 of 2011 (ladder position under Article 7(1), or delegated force under Article 8). Results are in documents.csv. (6) Audit: token statistics were computed with the IndoBERT tokeniser and a quality flag was assigned per unit (units.csv). Checksums of all data files are in checksums.md5; run the audit scripts against units.csv and texts/ to recompute the reported figures.
Institutions
- Binus UniversityJakarta, Jakarta