Tot-360M SFT v2 (1.87B) is live on a Raspberry Pi 4: why we switched to 100% English + Code after a '수입 OS' translation bug

Hi

Both Run 5 (1.80B 4K Base) and Tot-360M SFT v2 (1.87B Post-Trained) are finished, quantized to GGUF, and running live right now on my 4 GB Raspberry Pi 4:

• [Try Tot-360M SFT v2 Live in Your Browser (Raspberry Pi 4 Demo)](/tot-360m/demo/)
• [Download Tot-360M SFT v2 Q4_K_M.gguf (241.0 MiB)](/downloads/tot_360m_sft_v2_q4_k_m.gguf)
• [Download Tot-360M SFT v2 Q8_0.gguf (351.4 MiB)](/downloads/tot_360m_sft_v2_q8_0.gguf)

Along the way we hit one of the funniest (and most educational) dataset bugs of the entire project, which convinced me to move SFT v2 and our upcoming TPU runs to 100% English + Code. Here is what happened and what the numbers look like.

1. Run 5 (1.80B 4K Base) numbers

Before starting post-training, we finished Run 5 on Dual NVIDIA Tesla T4 GPUs at a 4,096-token context length:

• Steps: 101,591 to 109,958 (8,367 steps at 16,384 tokens/step).
• Tokens: 137,084,928 new 4K tokens (1,799,962,248 cumulative tokens).
• Loss: 2,000-step window average went from 1.381 down to 1.137 (final step loss 1.035).
• Speed & stability: 4,511.8 tokens/s across 2x Tesla T4 GPUs with 0 FP16 gradient overflow retries.

2. The "수입 OS" bug in SFT v1: why machine-translated code datasets ruin small models

For our first masked-loss ChatML post-training pass (SFT v1), we mixed English instructions and code with 16M tokens from an open Korean instruction dataset (heegyu/open-korean-instructions).

When I put SFT v1 on the live Raspberry Pi demo and typed "테트리스 만들어줘" ("Make Tetris"), it replied:

수입 OS

def get_file_info():
    """
    >>> get_file_info('테트리스.txt')
    {'파라미터': 3, '자식': 2, '자식': 1, '자식': 0, ...}

Look at that first line: `수입 OS`. In Korean, "수입" means "import" (as in importing goods into a country). Whoever built that open-source Korean instruction dataset had run English Python tutorials through an automated translator without excluding code blocks, turning import os into 수입 OS and translating dictionary keys inside Python doctests!

In a 70B model, a few thousand machine-translated code blocks might get drowned out. In a 345M-parameter model, splitting capacity across two human languages plus machine-translated Python syntax directly corrupts code generation whenever a prompt contains Korean characters.Meanwhile, when we prompted the exact same model in English, its grammar and code structure were dramatically cleaner.

3. Going 100% English + Code in SFT v2 (1.87B)

Rather than trying to patch SFT v1 on top of corrupted weights, we rewound to the clean Run 5 (step 109,958) checkpoint so the model had zero exposure to SFT v1, and built SFT v2 (`tot-360m-sft-v2`) with 100% pure English and code:

• Steps: 109,958 to 113,958 (4,000 steps of masked-loss ChatML SFT at 4,096 context length).
• Tokens: 65,520,000 post-training tokens (1,865,482,248 cumulative tokens).
• Dataset mix (100% English + Code):

- 40% HuggingFaceTB/smoltalk multi-turn English ChatML dialogues - 27.5% teknium/OpenHermes-2.5 and glaiveai/glaive-code-assistant-v2 coding & technical instructions - 15% meta-math/MetaMathQA and AI-MO/NuminaMath-CoT step-by-step math reasoning - 10% Clean English + code pretraining replay (so post-training never causes catastrophic forgetting) - 7.5% Sequence-aligned, single-file playable HTML5 Canvas games (Tetris, Snake, 2048, Breakout, Pong, Flappy Bird, Space Invaders) and Python algorithms

• Masked SFT Loss (computed strictly on `<|im_start|>assistant ... <|im_end|>` tokens): dropped from 2.6429 (first 250 steps) down to 1.1408 (final 250 steps, minimum single-step loss 0.1522) at 4,261.0 tokens/s on Dual Tesla T4 GPUs with 0 overflow retries.

Starting from Run 7 on Kaggle TPU v5e-8 (right after our currently queued Run 6 finishes its TPU queue slot), our base pretraining shards will also switch to 100% English + Code so every parameter in Tot-360M is dedicated to English reasoning and clean code generation.

4. What is running on the live Raspberry Pi 4 demo (`/tot-360m/demo/`)

You can test Tot-360M SFT v2 right now at [/tot-360m/demo/](/tot-360m/demo/) without installing anything:

• All 8 checkpoints on a single 4 GB Raspberry Pi 4: switch between Tot-360M SFT v2 (1.87B), Tot-360M SFT v1, Run 5 (1.80B), Run 4 (1.66B), Irish-360M (1.57B), and Voy-360M (1.43B / 917M / 405M). Each checkpoint runs in Q4_K_M (241.0 MiB, ~4.85 bits/weight, 63.5% smaller than the 660.15 MiB FP16 file). Only one model sits in RAM at a time (slots=2), streaming at 10.7 to 13.5 tokens/s on 4 ARM Cortex-A72 cores.
• Effort selector (`Low`, `Medium`, `High`): adjusts token budget (1,024, 2,048, or 2,560 tokens) and sampling temperature (0.35, 0.20, 0.12) instantly without reloading the model in RAM.
• Sandboxed Live Preview & Editable Code Runner: when Tot-360M writes an HTML5 game or Python/JS snippet, it opens beside the chat with a Live Preview / Edit Code toggle, an in-memory localStorage shim (so sandboxed games like 2048 and Tetris never crash with SecurityError), and a one-click Download .html button.
• Fixing a sneaky Node.js reverse-proxy ghost-slot bug: while testing the live demo, I noticed that clicking Stop or refreshing the page left the Pi showing 1/2 generating for another 70 seconds. It turned out that Node's http.request reverse proxy on port 443 was piping the SSE stream without calling proxyReq.destroy() on res.on("close"). When a browser disconnected early, llama-server kept generating the rest of the 2,000-token response into a dead pipe! Adding res.on("close", () => { if (!res.writableEnded) proxyReq.destroy(); }) made clicking Stop free the Pi slot in under 0.5 seconds.

Give [the live Raspberry Pi 4 demo](/tot-360m/demo/) a spin, or grab the Apache-2.0 GGUF files on the [Tot-360M page](/tot-360m/).