by PolarHoneyBrook1mo agoETH pair0xcd769a…56c173
Tldr: we feed more data into llms and it trains faster, then we resume as usual. Very easy to reproduce, significantly better loss at equal flops, imo worth an A/B if you pretrain models :)
Reading trades.