Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Hugging Face Blog · Publié le 1 oct. 2026

Kyle Wiggers’ post on Hugging Face reports Ai2’s own Olmo-core 3 benchmarks. It describes scaling the expert pool with roughly fixed active parameters, a preliminary eight-GPU throughput comparison with the earlier stack, and a controlled MXFP8 comparison with BF16. These are reported benchmark results from the post, not independent measurements. Lisez 3 points de vue avec leurs éléments à l’appui et les liens vers les sources.

Kyle Wiggers

Lire les points clés Ouvrir Éditeur

En un coup d’œil

Maintaining throughput while scaling expert count
The post reports that increasing the expert pool from 8 to 128 while selecting only four experts per token kept active parameters per token roughly fixed at ~3.2B; total parameter capacity grew from 4.6B to 47B with less than 5% drop in training throughput.
Lire le moment probant · Paragraphe 5
2.7× throughput gain with DDP-based MoE stack
The post reports that on eight NVIDIA B300 GPUs, Olmo-core 3’s new distributed data parallelism (DDP)-based MoE training stack achieved 52,000 tokens/sec/GPU versus 19,400 tokens/sec/GPU with the prior FSDP-based implementation — a ~2.7× throughput improvement.
Lire le moment probant · Paragraphe 11
MXFP8 improves throughput and peak active memory in a controlled B300 benchmark
The post reports that in a controlled benchmark on four NVIDIA B300 GPUs with uniform expert load, enabling MXFP8 in performance-critical parts increased end-to-end training throughput by ~21% over BF16 baseline and reduced peak active memory from 103 GiB to 95 GiB; most gains came from feed-forward computation and inter-expert data movement, not attention alone.
Lire le moment probant · Paragraphe 21

Passages clés3

Passages attribués et accompagnés du contexte nécessaire à leur vérification. Ouvrez le texte original pour vérifier la source.

Paragraphe 11

training infrastructure performance

2.7× throughput gain with DDP-based MoE stack

Extrait original

In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU with the new stack, compared with 19,400 using our earlier implementation—about 2.7× the throughput.

Kyle Wiggers·1 oct. 2026

Contexte

NVIDIA’s Megatron-Core is an established option for training large MoEs. Olmo-core 3 brings an integrated MoE training stack to the framework behind Olmo, with a redesign that improves throughput over our earlier FSDP-based implementation.

Paragraphe 5

MoE scaling efficiency

Maintaining throughput while scaling expert count

Extrait original

In one benchmark, we increased the expert pool from 8 to 128 while still selecting only four experts per token – the small units of text a language model processes – keeping the number of active parameters per token roughly fixed at about 3.2B. Total parameter capacity grew from 4.6B to 47B, while training throughput fell by less than 5%.

Kyle Wiggers·1 oct. 2026

Contexte

Olmo-core 3 is built to close that gap.

Paragraphe 21

numerical precision optimization

MXFP8 improves throughput and peak active memory in a controlled B300 benchmark

Extrait original

With MXFP8 enabled across the parts of the system where it helped most, training throughput was about 21% higher than with BF16, the higher-precision format we used as our baseline, while peak active memory fell from 103 GiB to 95 GiB. Most of the gain came from feed-forward computation and moving data between experts rather than attention alone.

Kyle Wiggers·1 oct. 2026

Contexte

We measured MXFP8’s effect on end-to-end training throughput in a controlled benchmark on four NVIDIA B300 GPUs, with work distributed uniformly across experts.

Source et méthodologie

Ces points de vue renvoient à leurs sources originales. Les reformulations sont signalées et ne sont pas des citations mot à mot.

Ouvrir la transcription ou les documents sources Signaler un problème