Berna R7 v0.1.0 — Documentation and Code for a 1.06B Multilingual Language Model with Modular Cell-Based Architecture
Description
Berna R7 is a 1.06B-parameter multilingual language model trained from scratch on English, Mathematics, and Code (2.66B tokens). It is the first implementation of the DNA-Kernel Plexus architecture proposed in Berna R5 (DOI: 10.5281/zenodo.23015308). This dataset (v0.1.0) contains: - Architecture documentation (ARCHITECTURE.md, DNA.md) - Cell lifecycle implementation: saturation (6D knowledge vector), DNA Kernel (100 chromosomes x 4 genes), splitting mechanism (S >= S*), dynamic plexus graph - Training pipeline: train.py, dataset.py, eval.py - Experiment scripts for H1 (saturation-triggered splitting) and H2 (bounded forgetting) - Baselines: vanilla transformer, EWC, PackNet - SQL registry with schema for model lineage and connection tokens - Paper outline - Bundled documentation PDF Status: Model weights are still training; they will be released as v0.2.0. Key facts: - Parameters: 1,056,265,728 (1.06B) - Data: 2.66B tokens - Hardware: 1x RTX 5090 (32 GB) - License: Berna Research License v1.0 Related works: - Berna R5: DOI 10.5281/zenodo.23015308 - Zenodo mirror: DOI 10.5281/zenodo.23037065 - GitHub: https://github.com/Berna-Labs/berna-r7 - HuggingFace: https://huggingface.co/BernaLabs/berna-r7-mother
Files
Steps to reproduce
Note on licensing: The dataset is released under CC BY-NC 4.0 for compatibility with this platform. The canonical license for Berna R7 is the Berna Research License v1.0 (https://github.com/Berna-Labs/berna-r7/blob/main/LICENSE), which also permits separate commercial licensing arrangements.