Berna R8: Code, Model, and H2 Experiment Results

Published: 12 October 2026| Version 1 | DOI: 10.17632/8t4mk7cw5k.1
Contributor:

Description

Berna R8 is a proof-of-concept follow-up to Berna R7 v3. It implements Bounded Forgetting — the central claim of the parent theory R5. Contents: 1. Code: 5 architectural fixes (K5, K6, K2 per-cell; Plexus threshold; DNA persistence) plus CellFreezer for Bounded Forgetting. 2. Model weights: R8 after German fine-tune. 3. H2 Experiment: 2000-step German fine-tune preserves English loss (ENG_BEFORE=4.0781, ENG_AFTER=4.0703, F_eng=0.000000). Related: - R8 Paper (Zenodo): 10.5281/zenodo.23310767 - R8 Model (HuggingFace): https://huggingface.co/BernaLabs/berna-r8 - R7 v3 (Zenodo): 10.5281/zenodo.23305282 - R5 Theory (Zenodo): 10.5281/zenodo.23015308 License: CC BY-NC 4.0 — research use only.

Files

Steps to reproduce

1. Base model: Berna R7 v3 (checkpoint step_1400, 1.2B params) Weights: https://huggingface.co/muhamedkamil/berna-r7-v3 2. Data preparation: - English: WikiText-103 test set (1.29M chars, 317K tokens) - German: Wikipedia DE (115M chars, 46M tokens) - Tokenizer: custom BPE (vocab=32000) from R7 v3 3. Baseline measurement (before fine-tuning): - Load R7 v3 with bfloat16 on NVIDIA L40S - Evaluate loss on 10 batches of 2048 tokens each - ENG_BEFORE = 4.0781, DE_BEFORE = 4.7406 4. Fine-tuning on German: - Steps: 2000 - Sequence length: 512 - Learning rate: 1e-5 (AdamW) - Batch size: 1 - Gradient clipping: 1.0 5. Final measurement (after fine-tuning): - Same evaluation protocol as step 3 - ENG_AFTER = 4.0703, DE_AFTER = 4.0797 6. Forgetting metric: F_eng = max(0, ENG_AFTER - ENG_BEFORE) = 0.000000 Bounded = (F_eng <= 1e-6) = True 7. Software: - PyTorch 2.x, CUDA 12.8 - transformers 4.45.2 - Custom architecture: modeling_berna_v2.py 8. Hardware: - NVIDIA L40S (48 GB) - Wall time: ~1 hour (fine-tune + eval) Full code: see attached files (train_r8.py + code/)

Categories

Computer Science, Artificial Intelligence, Artificial Intelligence Model

Licence