Phase F · Growing Saga
In progress since 8 October: training the 15.45-billion-parameter model in the Fast Lane allocation. Twelve stages (T9–T20, by 2 November) lowered the loss on the unchanged Polish validation set from 2.4872 to 2.4273. After the fix to the gradient computation on 12 November the model went through 18,000 updates on the mixture with translations: against the end of T20, the loss on unseen translations fell by 0.26 and on DynaWord texts by 0.13. A second pass over the same data gained less than the condition written down before the start required and was stopped after its first segment. A chat model fine-tuned from these weights answers in the chat window as a test version.
Allocation EHPC-AIF-2026FL01-1001: 50,000 GPU-hours for 3 months, awarded on 25 September 2026, until 31 December 2026. We have accepted the project invitation on the EuroHPC platform; the JUPITER account has been active since 8 October and training started the same day. Plan from the proposal: first increase the depth from 36 to 72 layers (about 5.8 billion parameters), then the model width (about 15 billion). Each step is preceded by a pilot with a criterion recorded before it starts; if a step worsens the evaluation, the smaller working model stays. The 15.45-billion model was built differently from the proposal: with 48 layers and doubled width, because doubling the layers at once was unstable (see phase E). Its main training is under way in the Fast Lane allocation: stages of 20.41 billion tokens (19,468 updates on 32 nodes), each with its own warm-up and learning-rate decay, continuing from the checkpoint of the previous stage. After every stage the loss is measured on the same 160 windows of the Polish and English validation sets: after T9 it was 2.4188 (Polish) and 1.7711 (English), after T10 2.3977 and 1.7578, after T11 2.3922 and 1.7418, after T12 2.3852 and 1.7336, after T13 2.3852 and 1.7277, after T14 2.3828 and 1.7230, after T15 2.3828 and 1.7199, after T16 2.3867 and 1.7148, after T17 2.3836 and 1.7148, after T18 (29 October) 2.3797 and 1.7125, after T19 2.3813 and 1.7121, after T20 (2 November) 2.3797 and 1.7102; the loss on the Polish training validation set fell over that time from 2.4872 to 2.4273. Stages T9–T11 contained new Polish text from HPLT 3.0 (60%, 29% and 30% of the mixture) and brought the largest gains; T12–T14 consisted only of the Polish repeat and 15% English and improved the loss by mere thousandths. From T15 the mixtures therefore received new sources: T15 contained 13.9% new text from Polish PDF files in the FinePDFs set (ODC-By licence) after quality, personal-data, novelty-against-the-whole-training-history and repetition filters; T16 contained 60% and T17 34.2% new text, mostly from MADLAD-400 (ODC-By), after which the pool of new text ran out and T18–T22 consisted of repeated Polish and 15% English. FineWeb-2 v2.1 was checked and dropped, because 99.7% of its Polish text was already in the base corpus. The new sources brought no progress: after T15 the Polish measure did not change, and after T16 it rose by 0.0039. Stability: gradient divergences in T9 and T10 forced the peak learning rate down from 1e-5 to 5e-6, further ones in T13 and T14 down to 3e-6, and T15, after diverging at update 12,009, was finished from the 10,000 checkpoint with a 2e-6 peak; each time with a fresh optimizer state and unchanged data. T16 passed at 3e-6. Our supervision repeatedly stopped T17 on the gradient norm, and the stage passed only at a 3.94e-7 peak. T18 tested a 5e-6 peak with a 2,000-update warm-up; stopped at update 18,219 on the norm alone, while the validation loss was at its lowest, it was finished from the 16,000 checkpoint at 3.33e-6. That rule was too strict: since 30 October the regular norm threshold counts only when the loss worsens, and the extreme threshold is unchanged. T19 at a 5e-6 peak diverged at update 5,091 (validation loss 2.4320 → 2.4528). The warm-up therefore does not remove divergences at this peak. The second cycle of T19 at 3.33e-6 was stopped by our guard at update 3,544 (validation loss 2.4297 → 2.4408); the stage passed only at 2.22e-6 and gave 2.3813 on the 160 Polish windows, worse than T18. T20 at the same peak matched T18 in Polish (2.3797) and improved English (1.7102) — the best point reached with the old method. T21 came out worse (2.3836 and 1.7129), so we rejected it. T22 was first interrupted by a cooling failure of the nodes and then diverged three times although we lowered the peak from 2.22e-6 to 9.87e-7; we skipped the stage. For T23 we built a mixture with new sources — Polish DynaWord (including EUR-Lex, Sejm and Senate transcripts, Biblioteka Nauki and Dziennik Ustaw; CC BY, CC BY-SA, official and EU licences) and Polish PDFs from FinePDFs with a low educational score — but after the novelty and repetition filters they made up only 6.4% of the mixture. T23 started from the end of T20, was interrupted by a node failure, and after resuming diverged twice without ever measuring better than at its start; we abandoned it. To obtain more new Polish text we translated English educational texts from FineWeb-Edu (ODC-By) with the Bielik-11B-v2.3-Instruct model (Apache-2.0): after two pilots on 400 documents, a job array translated 9.65 million documents on 8–9 November, of which 8.2 billion tokens remained after filtering. T24 (46.2% new Polish text — translations, DynaWord and FinePDFs — 38.8% repeats, 15% English; 1e-6 peak) ran without stops but gave 2.3844 and 1.7133, so it did not beat T20. Our computation error: while fine-tuning the model for conversation on 12 November we compared the gradient computed in the training format bf16 with an fp32 reference. In the bottom blocks of the model (where attention logits reach 4.9e7) the bf16 gradient was rounding error — with the attention kernel used in pre-training, ten to a hundred thousand times larger than the reference — and the model has had such weights at least since the start of T9. A measurement of the optimizer state showed the effect: at the end of T20 and T24 the update step of the upper blocks was 0.2–1.3% of the learning rate instead of 15–18%. In the old stages the upper blocks therefore learned far more slowly than the configured learning rate implied. The fix: the eight bottom blocks are frozen and computed in fp32, and the remaining forty train as before. The gradient norm fell from about 2,000 to about 0.2, and in the trial training was stable even at a 1e-5 learning rate, at which it used to diverge. The trial stage (1,000 updates of the T24 mixture from the end of T20 at a constant 1e-5) lowered the loss on fixed windows of the mixture from 2.0960 to 2.0442 while the loss on the Polish validation set rose from 2.4249 to 2.4401; finishing it with a decaying learning rate (2,000 updates) gave 2.0270 and 2.4346. The model then went through the rest of the mixture in two stages (13 and 14 November, 14,998 updates in five segments; moving on to the second stage had a condition written down before the start, and it was met). Updates from 18,000 onwards were held out: they provide the windows on which we measure the loss separately for each source in the mixture. At the end, compared with the end of T20: translations −0.258, DynaWord −0.135, FinePDFs −0.055, English on the 160 windows −0.030, the existing Polish validation set +0.0077 and the 160 Polish windows +0.0115. So the model learned the new sources and English, and lost a little on the existing Polish. These measurements are computed with the bottom blocks in fp32, so they are not directly comparable with the stage figures above (by this measure the end of T20 has 2.4249 on the validation set, and 2.3723 and 1.7041 on the 160 windows). A chat model fine-tuned from these weights has been answering in the chat window since 15 November as a test version. A second pass over the same data: a trial on 15 November (1,000 updates from the point before the learning rate was lowered) gave −0.0037 on translations and −0.0023 on DynaWord on unseen windows, as much as the first pass on that segment. A full segment (3,000 updates from the end of the continuation) gave −0.0034 on translations and −0.0037 on DynaWord against its starting point, with English up by 0.0037 and the existing Polish validation set up by 0.0032. The condition written down before the start required at least −0.0045 on translations, so the chain stopped after the first segment. Downtime: while access to JUPITER was blocked (19–23 October) our supervision loops stopped, and a file-system failure interrupted T16; the chain stood still for about 66 h. Our own supervision errors stopped it for another 39 h (25–27 October, two parallel copies) and about 19 h (29–30 October, a false alarm after the Slurm controller was briefly unavailable). In November: on 2 November JSC switched off, for a software update, the login nodes on which our supervision loops ran, and on 3 November it reserved the machine for large-scale tests — the T21 evaluation and the start of T22 waited until the evening; on 6–7 November our supervision failed for 23 h to handle the stop of T23, because restarting it depended on a connection from our own computer. Fast Lane account at report build time: 48,198 of 50,000 GPU-hours.