Tot-360M Second Training Run (In the Middle)

Hi, I'm Oliver. I'm writing this while the second training run for Tot-360M is actively running on Kaggle, and I'll follow up once it finishes.

The main differences in this run

• Saving every 1,000 steps instead of every 250: Checkpoints take around 90 seconds each to write to disk. Saving 4x less often means the TPU spends almost two extra hours training instead of waiting on the disk, which lets us fit roughly 200M+ more training tokens into the session.
• Direct `.gguf` export at the end: Unlike the first run, which only wrote a raw PyTorch .pt checkpoint, this notebook exports a ready-to-run .gguf file with the weights and tokenizer packed inside.
• Formula 1 in the training data: This batch adds English Wikipedia with Formula 1 articles boosted alongside FineWeb-Edu, clean code, and curated packs.

Small changes for the next run

• Save every 2,000 steps instead of every 1,000 to get even more training tokens per session.
• Might test running on a dual T4 GPU setup in addition to the TPU.

Open weights

This version of Tot-360M will be released as open weights, with many more updates to come. Thanks for reading. This post is a quick update while the run is in progress, and once training finishes I'll follow up with a detailed write-up.