Self-Improving AI Systems: A Combined Publications and Patents Corpus (1995-2026)
Description
This dataset brings together research and patent records on self-improving AI systems: artificial intelligence that improves its own capability, architecture, training data, or learning procedure with limited human involvement. It holds 41,875 records from 1995 through July 2026, the final year being partial and flagged as such. Of these, 41,348 are publications from OpenAlex and 527 are US patent families drawn from Google Patents Public Data, covering 899 underlying patent documents. Both sources share one schema for identifiers, authors, citations, subfields, and retrieval details. Records were found using a fixed list of search phrases, and each one is marked with a tier showing how strong that match was: tier 1 for a title match, tier 2 for a broader match, and tier 3 for the weakest match. Weak matches are kept and labelled rather than removed, so users can see and judge the boundary themselves. Precision was measured, not assumed. Two people independently reviewed 100 records per tier, and their agreement is reported as a range rather than one number, since the reviewers did not always agree on whether papers that merely apply an existing technique should count. Tier 1 precision falls between about 56% and 98% depending on that judgment call, and the overall corpus falls between about 52% and 81%. Before this review, 322 records that were not actual research, such as referee reports and front matter, plus 12,699 records pulled in by two overly broad search terms, were removed and documented. Patents are grouped by invention family instead of counted as raw filings. About 7% of publications come from open repositories with no editorial review; these are flagged, not deleted. The dataset includes the data in CSV and JSONL formats, a field dictionary, full documentation of how it was built, checksums, the raw patent data, charts and tables, and the notebook used to create it all. It suits studies of research trends, patent analysis, and testing methods for judging research scope.
Files
Steps to reproduce
The corpus was built using Python 3.10 or later, with pandas, numpy, requests, matplotlib, and google cloud bigquery. A real contact email is required for OpenAlex and Crossref, and a Google Cloud project with BigQuery billing must be set up in advance. The process begins by defining the scope: a fixed list of search phrases marking what counts as self improving AI, plus terms for review that helped catch false matches. Before collecting anything, the size and cost of every planned search is estimated; any phrase returning more than 25,000 results is skipped and recorded rather than pulled in full. Publications are then collected from OpenAlex across several passes covering titles and abstracts, saved progressively so the process can resume if interrupted. Missing venue information is filled in through Crossref, records that are not actual research, such as referee reports and front matter, are removed, and the search phrase behind each result is recorded for traceability. Patent data is collected from Google Patents through BigQuery using the same scope, checked to confirm the boundary works as intended, then merged with the publication data into one shared format. Two overly broad search phrases are removed, publications are deduplicated, and patents are grouped by invention family rather than by individual filing. A quality check follows, then a random sample of 100 records per confidence tier is drawn for manual review, labelled independently by two people using a written boundary rule. Those labels are scored for agreement, producing a precision range for each tier rather than a single number, since the two reviewers did not always agree. Charts, tables, field definitions, and full build documentation are generated directly from the finished data, checksums are computed for every file, and everything is packaged together for deposit.
Institutions
- North South UniversityDhaka Division, Dhaka