i pretrained the world's best small model under 100m parameters.
a 35m parameter language model that outperforms models over 6x larger, including LiquidAI's lfm2.5-230m, while training on a single H200 gpu.
the gains come from a better architecture. we took inspiration from deepseek, kimi, and other frontier open source models.
BarunLM combines hybrid local and global attention, grouped query attention to reduce memory, partial rope, gated attention, bounded swiglu, and residual selectors to improve how information flows through the network.
after pretraining, we further improved the model with rlvr (reinforcement learning with verifiable rewards) using deterministic verifiers and rloo, allowing the model to reinforce correct reasoning without relying on human preference labels.
introducing BarunLM (35m)
i pretrained the world's best small model under 100m parameters.
a 35m parameter language model that outperforms models over 6x larger, including LiquidAI's lfm2.5-230m, while training on a single H200 gpu.
the gains come from a better architecture. we took inspiration from deepseek, kimi, and other frontier open source models.
BarunLM combines hybrid local and global attention, grouped query attention to reduce memory, partial rope, gated attention, bounded swiglu, and residual selectors to improve how information flows through the network.
after pretraining, we further improved the model with rlvr (reinforcement learning with verifiable rewards) using deterministic verifiers and rloo, allowing the model to reinforce correct reasoning without relying on human preference labels.