SKALDBETA

Project documentation

Data, training
and method

How Saga is built — a Polish language model in the Skald family: Saga 3B, trained from scratch on the JUPITER supercomputer, and the 15.45-billion-parameter model built from it, which we are training further.

Status from the report

Editorial snapshot: 19 September 2026 · 18:17:19 Europe/Warsaw. Reading the current published report requires JavaScript.

Job-history reading: 19 September 2026, 18:17:19 (Warsaw). Measurements from the FAKTY document: 19 September 2026, 18:17:09 (Warsaw).

The full Saga 3B training run is complete.

Recipe, settings and evaluation from the model card (README.md) and receptura-podsumowanie.json, read on 19 September 2026. This page describes the base model, which predicts how text continues. Links to source files require access to the private repository.

Project phases

Reproduction, tokenizer selection, corpus-composition comparison and full Saga 3B training are complete. Work in the Development project is complete: continued pre-training on the new corpus, building the 15.45-billion-parameter model from the base and preparing the data for Fast Lane. Since 8 October that model has been training in the Fast Lane allocation. The weights stay private. Phase statuses refresh from the published report.

  1. Complete

    Phase A · Reproduction and scaling

    Reproduction and scaling measurements are complete.

    Validation loss in the JUPITER trial: 2.9847; local reference: 2.9867. Difference −0.0020. Trial duration: 13 min 11 s.

    Largest measured scale in the report: 64 GPUs; efficiency relative to the baseline configuration: 89.5%; throughput: 9.0 million tokens/s. These measurements describe our training trials.

  2. Complete

    Phase B · Tokenizer selection

    The 64k tokenizer was selected, with a mean of 1.4299 bits per character.

    Saga uses its own byte-level BPE tokenizer, trained on Polish text, with 65,536 tokens. Bits per character allow comparison between models with different tokenizers.

  3. Complete

    Phase C · Corpus composition

    The corpus-composition comparison is complete.

    The shares of five pools were selected after six smaller ablation runs, evaluated on a separate exam. The recipe below describes the completed training file.

  4. Complete

    Phase D · Training Saga 3B

    The full Saga 3B training run is complete.

    Training ended on 14 September 2026, after 186,688 updates in six jobs on 64 GH200 GPUs. It processed 195,756,556,288 tokens, or about 195.8 billion.

    Report values and the separate technical trial

    The report’s plan fields contain 186,688 updates, 195.8 billion tokens, and an estimate of 68 hours on 64 GPUs. The time estimate is not a measurement of the completed training run.

    Separate technical trial: 13,883 tokens/s/GPU, 50.2% MFU, and 76.1 GiB of memory per GPU. Averages for the full run appear in the training settings section.

  5. Complete

    Phase E · Continued pre-training and new corpus

    Work in the Development project is complete: continued pre-training of the Saga 3B base, building the 15.45-billion-parameter model from it and preparing 15 data stages for Fast Lane.

    The Development project (EHPC-DEV-2026D09-196) has been running since 18 September 2026: first multi-node training measurements, then two stages of continued pre-training (22 September, 21.3 billion tokens). Validation loss on an unchanged set fell from 2.5105 to 2.4840. The next stage, on the unused remainder of the existing corpus, was stopped on 23 September after about 9 billion tokens because the loss did not move. Further continued pre-training therefore needs new material.

    New corpus (ready on 25 September): deduplication reduced 167.9 million documents from five pools to 151.1 million, which the 64k tokenizer turns into 105.4 billion tokens. A full scan against the evaluation texts found coverage below 1% in each of the exam's three parts, lower than in the Saga corpus. Continued pre-training ran in stages of 20 billion tokens with about 92% new material, against a required minimum of 60%. Stages 3.1–3.3 lowered validation loss from 2.4841 to 2.4704, and a learning-rate decay from the end of stage 3.3 brought it to 2.4667, the lowest value of the Saga 3B base in the project. Stages 3.4 and 3.6 in the FSDP setup did not lower the loss; the stage 3.5 mixture was used for growth trials.

    Since 27 September the Development project has been used to prepare the growth. We tested the recipe first on small models of the same architecture. Doubling the number of layers at once proved unstable at the 5.8-billion scale. Two methods that preserve the model's function were stable: inserting a copy with zeroed output after every third layer, and exactly doubling the width. This is how the 15.45-billion-parameter model (48 layers × 5120) was built directly from the 3B base on 29 September. After 6.3 billion tokens of the first training run its validation loss was 2.4944 against 2.4667 for the base. A small-model test on 1 October showed that a model grown this way catches up with the control after about 12–15% of the tokens the base was trained on, and then pulls ahead. In further training the learning rate proved decisive. Peaks of 1.5e-4 and 7e-5 made training diverge, and a rate of 4e-5 held for 6,000 updates gave a loss of about 2.52 and a rise towards the end. Small-model diagnostics on 2 October ruled out the way training is distributed across GPUs: FSDP, ZeRO-1 and activation checkpointing give the same result as plain DDP (3.4943–3.4944). Since then the model has been trained in short cycles with a warm-up and a full learning-rate decay. A 3e-5 → 3e-6 decay on the rest of the T5 mixture lowered the loss from 2.5236 to 2.4700, and two cycles on the T4 mixture (peak 1.5e-5, then 1.2e-5) to 2.4525, below the 2.4667 of the 3B base. The second of these cycles ran on 32 nodes: about 2.8 s per update against 5.45 s on 16 nodes. A third cycle, a second pass over the T6 mixture, lowered the loss to 2.4502. The prepared Polish mixtures ran out, so the T7 mixture combines a Polish repeat and new English FineWeb-Edu text half and half (ODC-By licence, 44.5 billion tokens after deduplication). After 8,000 of 9,536 updates the loss on the English validation set fell from 2.1398 to 1.8078, while on the Polish set it moved from 2.4031 to 2.4078 (the same 160 windows of each set). The Development project closed with stage T8 and the preparation of 15 mixtures for Fast Lane (T9–T23, 20.41 billion tokens each): new Polish text from HPLT 3.0 checked for novelty against the whole training history and for rights, a repeat of Polish text from earlier sets, and 15% English FineWeb-Edu.

  6. In progress

    Phase F · Growing Saga

    In progress since 8 October: training the 15.45-billion-parameter model in the Fast Lane allocation. Twelve stages (T9–T20, by 2 November) lowered the loss on the unchanged Polish validation set from 2.4872 to 2.4273. After the fix to the gradient computation on 12 November the model went through 18,000 updates on the mixture with translations: against the end of T20, the loss on unseen translations fell by 0.26 and on DynaWord texts by 0.13. A second pass over the same data gained less than the condition written down before the start required and was stopped after its first segment. A chat model fine-tuned from these weights answers in the chat window as a test version.

    Allocation EHPC-AIF-2026FL01-1001: 50,000 GPU-hours for 3 months, awarded on 25 September 2026, until 31 December 2026. We have accepted the project invitation on the EuroHPC platform; the JUPITER account has been active since 8 October and training started the same day. Plan from the proposal: first increase the depth from 36 to 72 layers (about 5.8 billion parameters), then the model width (about 15 billion). Each step is preceded by a pilot with a criterion recorded before it starts; if a step worsens the evaluation, the smaller working model stays. The 15.45-billion model was built differently from the proposal: with 48 layers and doubled width, because doubling the layers at once was unstable (see phase E). Its main training is under way in the Fast Lane allocation: stages of 20.41 billion tokens (19,468 updates on 32 nodes), each with its own warm-up and learning-rate decay, continuing from the checkpoint of the previous stage. After every stage the loss is measured on the same 160 windows of the Polish and English validation sets: after T9 it was 2.4188 (Polish) and 1.7711 (English), after T10 2.3977 and 1.7578, after T11 2.3922 and 1.7418, after T12 2.3852 and 1.7336, after T13 2.3852 and 1.7277, after T14 2.3828 and 1.7230, after T15 2.3828 and 1.7199, after T16 2.3867 and 1.7148, after T17 2.3836 and 1.7148, after T18 (29 October) 2.3797 and 1.7125, after T19 2.3813 and 1.7121, after T20 (2 November) 2.3797 and 1.7102; the loss on the Polish training validation set fell over that time from 2.4872 to 2.4273. Stages T9–T11 contained new Polish text from HPLT 3.0 (60%, 29% and 30% of the mixture) and brought the largest gains; T12–T14 consisted only of the Polish repeat and 15% English and improved the loss by mere thousandths. From T15 the mixtures therefore received new sources: T15 contained 13.9% new text from Polish PDF files in the FinePDFs set (ODC-By licence) after quality, personal-data, novelty-against-the-whole-training-history and repetition filters; T16 contained 60% and T17 34.2% new text, mostly from MADLAD-400 (ODC-By), after which the pool of new text ran out and T18–T22 consisted of repeated Polish and 15% English. FineWeb-2 v2.1 was checked and dropped, because 99.7% of its Polish text was already in the base corpus. The new sources brought no progress: after T15 the Polish measure did not change, and after T16 it rose by 0.0039. Stability: gradient divergences in T9 and T10 forced the peak learning rate down from 1e-5 to 5e-6, further ones in T13 and T14 down to 3e-6, and T15, after diverging at update 12,009, was finished from the 10,000 checkpoint with a 2e-6 peak; each time with a fresh optimizer state and unchanged data. T16 passed at 3e-6. Our supervision repeatedly stopped T17 on the gradient norm, and the stage passed only at a 3.94e-7 peak. T18 tested a 5e-6 peak with a 2,000-update warm-up; stopped at update 18,219 on the norm alone, while the validation loss was at its lowest, it was finished from the 16,000 checkpoint at 3.33e-6. That rule was too strict: since 30 October the regular norm threshold counts only when the loss worsens, and the extreme threshold is unchanged. T19 at a 5e-6 peak diverged at update 5,091 (validation loss 2.4320 → 2.4528). The warm-up therefore does not remove divergences at this peak. The second cycle of T19 at 3.33e-6 was stopped by our guard at update 3,544 (validation loss 2.4297 → 2.4408); the stage passed only at 2.22e-6 and gave 2.3813 on the 160 Polish windows, worse than T18. T20 at the same peak matched T18 in Polish (2.3797) and improved English (1.7102) — the best point reached with the old method. T21 came out worse (2.3836 and 1.7129), so we rejected it. T22 was first interrupted by a cooling failure of the nodes and then diverged three times although we lowered the peak from 2.22e-6 to 9.87e-7; we skipped the stage. For T23 we built a mixture with new sources — Polish DynaWord (including EUR-Lex, Sejm and Senate transcripts, Biblioteka Nauki and Dziennik Ustaw; CC BY, CC BY-SA, official and EU licences) and Polish PDFs from FinePDFs with a low educational score — but after the novelty and repetition filters they made up only 6.4% of the mixture. T23 started from the end of T20, was interrupted by a node failure, and after resuming diverged twice without ever measuring better than at its start; we abandoned it. To obtain more new Polish text we translated English educational texts from FineWeb-Edu (ODC-By) with the Bielik-11B-v2.3-Instruct model (Apache-2.0): after two pilots on 400 documents, a job array translated 9.65 million documents on 8–9 November, of which 8.2 billion tokens remained after filtering. T24 (46.2% new Polish text — translations, DynaWord and FinePDFs — 38.8% repeats, 15% English; 1e-6 peak) ran without stops but gave 2.3844 and 1.7133, so it did not beat T20. Our computation error: while fine-tuning the model for conversation on 12 November we compared the gradient computed in the training format bf16 with an fp32 reference. In the bottom blocks of the model (where attention logits reach 4.9e7) the bf16 gradient was rounding error — with the attention kernel used in pre-training, ten to a hundred thousand times larger than the reference — and the model has had such weights at least since the start of T9. A measurement of the optimizer state showed the effect: at the end of T20 and T24 the update step of the upper blocks was 0.2–1.3% of the learning rate instead of 15–18%. In the old stages the upper blocks therefore learned far more slowly than the configured learning rate implied. The fix: the eight bottom blocks are frozen and computed in fp32, and the remaining forty train as before. The gradient norm fell from about 2,000 to about 0.2, and in the trial training was stable even at a 1e-5 learning rate, at which it used to diverge. The trial stage (1,000 updates of the T24 mixture from the end of T20 at a constant 1e-5) lowered the loss on fixed windows of the mixture from 2.0960 to 2.0442 while the loss on the Polish validation set rose from 2.4249 to 2.4401; finishing it with a decaying learning rate (2,000 updates) gave 2.0270 and 2.4346. The model then went through the rest of the mixture in two stages (13 and 14 November, 14,998 updates in five segments; moving on to the second stage had a condition written down before the start, and it was met). Updates from 18,000 onwards were held out: they provide the windows on which we measure the loss separately for each source in the mixture. At the end, compared with the end of T20: translations −0.258, DynaWord −0.135, FinePDFs −0.055, English on the 160 windows −0.030, the existing Polish validation set +0.0077 and the 160 Polish windows +0.0115. So the model learned the new sources and English, and lost a little on the existing Polish. These measurements are computed with the bottom blocks in fp32, so they are not directly comparable with the stage figures above (by this measure the end of T20 has 2.4249 on the validation set, and 2.3723 and 1.7041 on the 160 windows). A chat model fine-tuned from these weights has been answering in the chat window since 15 November as a test version. A second pass over the same data: a trial on 15 November (1,000 updates from the point before the learning rate was lowered) gave −0.0037 on translations and −0.0023 on DynaWord on unseen windows, as much as the first pass on that segment. A full segment (3,000 updates from the end of the continuation) gave −0.0034 on translations and −0.0037 on DynaWord against its starting point, with English up by 0.0037 and the existing Polish validation set up by 0.0032. The condition written down before the start required at least −0.0045 on translations, so the chain stopped after the first segment. Downtime: while access to JUPITER was blocked (19–23 October) our supervision loops stopped, and a file-system failure interrupted T16; the chain stood still for about 66 h. Our own supervision errors stopped it for another 39 h (25–27 October, two parallel copies) and about 19 h (29–30 October, a false alarm after the Slurm controller was briefly unavailable). In November: on 2 November JSC switched off, for a software update, the login nodes on which our supervision loops ran, and on 3 November it reserved the machine for large-scale tests — the T21 evaluation and the start of T22 waited until the evening; on 6–7 November our supervision failed for 23 h to handle the stop of T23, because restarting it depended on a connection from our own computer. Fast Lane account at report build time: 48,198 of 50,000 GPU-hours.

  7. Planned

    Phase G · Evaluation and publication

    The Saga 3B weights stay in a private repository. What remains to be published is the report on the method and the recipe.

    The final checkpoint was evaluated. Model repository — restricted access ↗

Corpus recipe

The completed corpus combines five pools of Polish text after global deduplication. The shares refer to tokens in the training file.

216.8 billion

Tokens in the training file.
Separate validation file: 10 million tokens.

Training-file composition from the completed recipe
Pool and sourcesToken shareTokens in filePasses over the pool
Web: FineWeb-2 + HPLT 2.0 cleaned89.6%194.17 billion≈1.04
FinePDFs, additional quality filter8.9%19.29 billion1
Wikipedia1.1%2.34 billion4
Wikisource0.4%0.78 billion4
Wolne Lektury0.1%0.21 billion4

Shares are rounded separately to 0.1 percentage points, so they sum to 100.1%. Passes describe copies in the training file, not guaranteed exposure of every token during training. Exact values: receptura-podsumowanie.json.

Deduplication

328 million documents

328,007,101 documents were processed. 53,819,391 duplicates were removed: about 53.8 million documents and 33.1 billion tokens. A further 30 FineWeb-2 documents containing a literal end-of-text marker were rejected.

Data split

Validation before mixing

Whole unique documents were reserved for validation before the pools were mixed. Unused tails of those documents did not enter training. The validation file contains 10 million tokens.

How duplicates were removed

A 64-bit hash identified candidates, followed by verification on normalised decoded text. Sources listed earlier in the mixture definition took precedence. Deduplication also covered repetitions between sources.

Provenance and source terms

The corpus uses Polish subsets of FineWeb-2, HPLT 2.0 cleaned and FinePDFs; Wikipedia from 1 November 2023; Wikisource from 1 September 2026; and the Wolne Lektury corpus. Sources and their terms of use are documented in the model card.

Training settings

Settings for the completed run and its average measurements come from the base-model card.

Model

About 3 billion parameters

A decoder-only transformer in the GPT-2 layout: 36 layers, 20 heads, width 2,560. Context: 2,048 tokens. LayerNorm without bias, GELU, learned positions and tied input/output embeddings.

Hardware and run

64 GH200 GPUs

16 JUPITER Booster nodes, six jobs. Training finished on 14 September 2026 at update 186,688. Code based on nanoGPT, bf16 mixed precision, torch.compile and DDP.

Batch and optimisation

Each update covered 1,048,576 tokens in sequences of 2,048 tokens. AdamW optimiser: β₂ = 0.95, weight decay 0.1, and gradient clipping at 1.0.

Learning-rate schedule

Warm-up over 3,000 updates, then a constant rate of 3e-4 and linear decay to 3e-5 over the last 15% of training.

Measurements from the full run

Mean throughput: 12,329 tokens/s per GPU; mean MFU: 46.6%. Mean training loss fell from 5.32 at the start to 2.08 in the last 2,000-update window.

Source: model card — Training.

Evaluation and limits

The final checkpoint was evaluated after 186,688 updates. Lower perplexity and fewer bits per character indicate better text prediction on a given set.

Base-model evaluation from the model card
SetPerplexityBits per characterShared 13-token windows
Validation slice: 1,048,576 tokens12.06——
Encyclopedia after the data cut-off10.290.7260.80%
Web text after the cut-off11.210.6550.41%
Prose after the cut-off15.451.0140.05%

A base model

The base version of Saga 3B predicts how text continues. It is not instruction-tuned or aligned for conversation. The table describes this base model.

History and technical trials

The local Canto 33M model from August 2026 remains a reference for the training process. Its results do not describe Saga 3B. Scaling trials and full training have separate measurements; reported throughput describes our code and configurations, not the performance of JUPITER as a whole or its host.

Source: model card — Evaluation.

Publication

The Saga 3B model repository on Hugging Face remains private. The corpus recipe, the measurements and the work journal are public; the weights are not.

The Saga 3B weights stay in a private repository. What remains to be published is the report on the method and the recipe.

decisioninterface/saga-3b — restricted access ↗

The package contains 55 float32 safetensors weight files in Transformers GPT-2 format. The card documents the lossless conversion of the training export and the comparison of logits and text continuations.

Weight licence stated in the model card: Apache 2.0. The repository remains private. The weight licence does not replace the terms of the source data. Details: model card and condensed recipe.

Epos remains a future large mixture-of-experts model planned outside the current allocation.

Resources and acknowledgement

Allocation EHPC-AIF-2026PG01-1218, EuroHPC AI Factory — Playground, provides 5,000 GPU-hours. The full Saga 3B training run used this allocation on JUPITER Booster at Jülich Supercomputing Centre, with NVIDIA GH200 GPUs.

Allocation EHPC-DEV-2026D09-196 (EuroHPC Development Access), awarded on 10 September 2026, provides 14,000 GPU-hours, or 3,500 node-hours, for 12 months from the start date specified by JSC. It is accounted for separately and supports development of training across multiple nodes.

Allocation EHPC-AIF-2026FL01-1001 (EuroHPC AI Factory — Fast Lane), awarded on 25 September 2026, provides 50,000 GPU-hours on JUPITER Booster for 3 months. Training in this allocation has been running since 8 October 2026; there are no extensions. It is accounted for separately and supports growing Saga 3B into a model of about 15 billion parameters.

We acknowledge EuroHPC JU for awarding the project ID EHPC-AIF-2026PG01-1218 access to JUPITER Booster hosted by Jülich Supercomputing Centre (JSC), Germany.

Na zasobach EuroHPC (JUPITER Booster) wykonano: faza A: przebiegi wielowezlowe i pomiary skalowania; faza B: przebiegi potwierdzajace wybor slownika. Poza przydzialem EuroHPC powstaly: trening slownikow i pomiar plodnosci, rozwoj filtrow korpusu, lokalna linia odniesienia oraz kod pakujacy i raportujacy.

Translation of this historical record: EuroHPC resources (JUPITER Booster) were used for phase A runs across multiple nodes and scaling measurements, and phase B runs confirming the tokenizer choice. Tokenizer training and fertility measurements, corpus-filter development, the local reference model, and packaging and reporting code were developed outside the EuroHPC allocation.