Mirendil introduces Fern, a model with up to 10M tokens of context

Behnam Neyshabur (@bneyshabur) introduced Fern, Mirendil's first model, on Monday. It is pretrained from scratch with up to 10 million tokens of context. The post drew about 1,000 likes.

The post says that is roughly 0.5 million lines of code, about the scale of Megatron-LM, or days of an agent's work held in context at once. Its first version, Fern 0.1, had a short round of post-training.

Why it mattersVery long context is a direct bet on agents that need to keep entire codebases or long work histories in view.