Tot-360M at 1.66B tokens: 4K context expansion, our first post-trained checkpoint next, and a live demo in under a week

Hi

A lot happened since Run 2 finished at 917M tokens. We finished the entire 651.8M-token Batch 3 pack across two TPU runs, crossed 1.568 billion base tokens, and then ran a 4,096-token (4K) context-expansion and ChatML mid-training run on Dual Tesla T4 GPUs to reach 1.663 billion tokens.

Right now, one more 9-hour Dual-T4 GPU run (Run 5, pushing toward 1.80B tokens) is running on Kaggle, and right after it finishes we are running our first dedicated SFT post-training pass. Instead of waiting until all 7.2B tokens are done before ever post-training the model, we are releasing a post-trained version every 2 to 3 checkpoints. That way we get a usable chat and coding model at each milestone while base pretraining keeps climbing in between.

Here is how the runs went, how the checkpoint family is split up, and what is coming in the live demo in under a week.

The actual numbers

• Run 3 Part 1 (Voy-360M 1.43B): steps 56,058 to 87,395 on a Kaggle TPU v5e-8. Trained 512,924,016 new tokens (1,430,481,360 cumulative) at 16,567 tokens/s end-to-end with only 3 saves. Fixing the cosine learning rate bug dropped the 2,000-step average loss from 2.61 down to 1.93-2.20.
• Run 3 Part 2 (Irish-360M 1.57B): steps 87,395 to 95,816 on TPU v5e-8. Finished the remaining 137,834,928 tokens of Batch 3 (1,568,316,288 cumulative) with final 500-step average loss at 2.13.
• Run 4 / 4K Context Expansion & ChatML Mid-Training (Tot-360M 1.66B 4K): steps 95,816 to 101,591 on 2x NVIDIA Tesla T4 GPUs (32 GB total VRAM, FP16 DistributedDataParallel).

- Context window: doubled from 2,048 to 4,096 tokens (max_position_embeddings = 4096, seq_len = 4096). - Tokens trained: 94,594,500 tokens in this run, bringing the cumulative total to 1,662,910,788 tokens (1.663B). - Wall time & throughput: 6 hours 10 minutes (22,212.8 seconds) at 4,276.8 tokens/s steady-state across the two T4 GPUs (about 3.82 seconds per 4K optimizer step, with 0 FP16 gradient overflow retries).

Why we switched Run 4 and Run 5 to Dual T4 GPUs

Kaggle's TPU v5e-8 queue was backed up and our weekly 30-hour GPU quota was sitting unused. On two Tesla T4 GPUs (16 GB VRAM each), running a 4,096-token sequence length with PyTorch's fused Flash/SDPA attention used 10.41 GB of VRAM per GPU and ran at 4,276.8 tokens/s. That was fast enough to finish the entire 94.6M-token 4K pack in a single 6-hour session, and still left enough weekly GPU quota to kick off a full 9-hour Run 5 (~151M tokens of MIT code, Korean Wikidata textbooks, Open Korean Instructions, MetaMathQA, and Numina-CoT reasoning) before the weekly quota resets.

How we taught a 345M model to use all 4,096 tokens (and write HTML5 games on the go)

In earlier runs, training sequences were packed at 1,024 tokens by concatenating documents end-to-end and slicing every 1,024 tokens. That works fine for prose, but it is terrible for complete single-file programs: if a 900-token HTML5 game starts at token 600 of a sequence, the last 476 tokens get chopped off into the next sequence without the user prompt. The model learns to stop halfway through <script>.

For our 4K runs, we changed two things:

1. Sequence-aligned 4K packing: every complete single-file HTML5 <canvas> game (Tetris Pro with Next piece preview and ghost landing shadow, Snake, Breakout, Pong, Flappy Bird, 2048, and Space Invaders) starts at token 0 of a 4,096-token sequence, followed by a second complete game turn or bilingual Q&A in the second half of the same 4,096-token window. Not a single HTML file is ever cut across a sequence boundary. 2. Multi-turn 4K dialogues: we included multi-turn conversations spanning 3,200 to 3,700 tokens (for example, asking for a basic Tetris game in Turn 1 and then asking to upgrade it into Tetris Pro with wall-kick rotation, ghost shadow, and Next preview in Turn 2) alongside 88M tokens of general Korean instructions, SmolTalk ChatML, Korean Wikidata textbooks, and MIT-licensed web/Python code.

We tested whether the exported GGUF actually uses the second half of the 4K window by feeding it a 2,049-token prompt containing a full Turn 1 game and asking for a second game in Turn 2. It attended across the 2,048 boundary and generated a complete, runnable HTML5 Snake game from scratch at 35.9 tokens/s on CPU.

What our first dedicated SFT post-training pass adds next

Because Run 4 was still standard next-token mid-training (computing loss on every token in the 4,096-token window), right after the 9-hour GPU Run 5 finishes at ~1.80B tokens we are running Post-Training v1 (Masked-Loss ChatML SFT):

• Masking user & system turns (`label = -100`): loss is computed only on <|im_start|>assistant ... <|im_end|> responses so the model never tries to continue the user's question.
• Stop-token tuning: training clean termination on <|im_end|> so answers stop cleanly without rambling.
• Iris persona & 10% pretraining replay: pairing multi-turn English/Korean help and HTML5/Python coding with 10% pretraining replay so the model stays sharp without forgetting base facts.

Voy-360M, Irish-360M, and Tot-360M: our checkpoint family

To keep the base checkpoints and post-trained checkpoints clear as we release a new post-trained version every 2 to 3 runs:

• Voy-360M (405M, 917M, 1.43B): the early foundation pretraining checkpoints from Runs 1 to 3.
• Irish-360M (1.57B): the completed Batch 3 pre-4K checkpoint at step 95,816.
• Tot-360M (1.66B 4K & upcoming 1.80B SFT v1): our flagship 4,096-token instruction and post-trained tier.

All released checkpoints are open-weight (Apache-2.0) in Q4_K_M (253 MB), Q8_0 (369 MB), and FP16 (692 MB) GGUF formats on Hugging Face:

• Download Tot-360M, Irish-360M & Voy-360M GGUFs on Hugging Face

With llama.cpp, run the 4K checkpoint with a 4,096 context window and ChatML tags:

./llama-cli -m tot_360m_q8_0.gguf -c 4096 --temp 0.2 --repeat-penalty 1.05 -p "<|im_start|>user
Make a complete, playable Tetris game in a single HTML file using HTML5 Canvas and JavaScript.<|im_end|>
<|im_start|>assistant
"

Live multi-model demo coming in under a week

We are putting the finishing touches on a live browser studio hosted directly from my Raspberry Pi 4 (4 GB RAM), coming out in under a week:

• Switch between all 5 checkpoints live: compare Tot-360M (4K), Irish-360M (1.57B), and the Voy-360M checkpoints side by side. To fit comfortably inside the Pi 4's 4 GB RAM alongside my other servers, all 5 model folders live on the Pi's SD card and the server keeps only 1 model loaded in RAM at a time, swapping weights on the fly when you switch.
• 3-step Effort / Precision slider: switch between Low (Q4_K_M, 253 MB), Med (Q8_0, 369 MB), and High (FP16, 692 MB + syntax guard).
• Playable in-chat HTML5 game cards: whenever the model writes an HTML5 <canvas> game, a Play Inside Chat preview card renders right inside the message bubble (plus an optional split-screen arcade stage) so you can play the generated Tetris or Snake game immediately.

Thanks for reading, and I will follow up as soon as Run 5 and our first dedicated SFT post-trained checkpoint wrap up!