2nd training run done?
Hi So I'm back with the results: 1. Trained on 600M more tokens due to being able to save less (every 1k steps instead of 250), which gained us about 200M more training tokens than last time. Crazy. 2. Loss rates were down to sub-2 at the end, which is impressive. This one also improved grammar, coding knowledge, English, a bit of Korean, and F1. 3. Wrapped up at around 9 hours, but if we don't count saving, real time is around 8 hours 20 minutes. 4. Also, this version ended with a .gguf so I can use it right away.
Download here
Further notes This version is open source, but it will not understand how to respond yet, so it will spit out strange stuff. Post-training will be done in around 3 weeks when we have 7B training tokens matching Chinchilla's rule. Looking for help—email me if you have more questions or want to join our journey.
(Next training plans) 1. Will save only 3 times to get the most training tokens, expecting 650M. 2. Will train on more Korean and code. 3. Improve loss rate.
Future plans (3mo+) Hoping to build a 1B model after this with a bigger context on top of Tot-360M. Will host as an API from my Raspberry Pi. Maybe do training on AWS if I get ads.
Thanks for reading, will follow up when training 3 starts.