nafyi.

PUBLIC VIEWPOINTS

Kyle Wiggers

1 sources · 3 viewpoints · 3 topics

Content updated:

Kyle Wiggers on MoE scaling efficiency, numerical precision optimization, training infrastructure performance. Explore 3 viewpoints by topic, with evidence from 1 source.

Explore connections

Viewpoints by topic

Attributed viewpoints, ordered by source publication date. A snapshot of these conversations, not a definitive statement of someone’s beliefs.

Translations are for reading; original excerpts remain the evidence.

MoE scaling efficiency

View this topic
MoE scaling efficiency

Maintaining throughput while scaling expert count

The post reports that increasing the expert pool from 8 to 128 while selecting only four experts per token kept active parameters per token roughly fixed at ~3.2B; total parameter capacity grew from 4.6B to 47B with less than 5% drop in training throughput.

Supporting evidence

Kyle Wiggers·

Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Original excerpt

In one benchmark, we increased the expert pool from 8 to 128 while still selecting only four experts per token – the small units of text a language model processes – keeping the number of active parameters per token roughly fixed at about 3.2B. Total parameter capacity grew from 4.6B to 47B, while training throughput fell by less than 5%.
Context

Olmo-core 3 is built to close that gap.

Open original · Paragraph 5 (opens in a new tab)Original text source (opens in a new tab)
Kyle Wiggers
Share insight

training infrastructure performance

View this topic
training infrastructure performance

2.7× throughput gain with DDP-based MoE stack

The post reports that on eight NVIDIA B300 GPUs, Olmo-core 3’s new distributed data parallelism (DDP)-based MoE training stack achieved 52,000 tokens/sec/GPU versus 19,400 tokens/sec/GPU with the prior FSDP-based implementation — a ~2.7× throughput improvement.

Supporting evidence

Kyle Wiggers·

Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Original excerpt

In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU with the new stack, compared with 19,400 using our earlier implementation—about 2.7× the throughput.
Context

NVIDIA’s Megatron-Core is an established option for training large MoEs. Olmo-core 3 brings an integrated MoE training stack to the framework behind Olmo, with a redesign that improves throughput over our earlier FSDP-based implementation.

Open original · Paragraph 11 (opens in a new tab)Original text source (opens in a new tab)
Kyle Wiggers
Share insight

numerical precision optimization

View this topic
numerical precision optimization

MXFP8 improves throughput and peak active memory in a controlled B300 benchmark

The post reports that in a controlled benchmark on four NVIDIA B300 GPUs with uniform expert load, enabling MXFP8 in performance-critical parts increased end-to-end training throughput by ~21% over BF16 baseline and reduced peak active memory from 103 GiB to 95 GiB; most gains came from feed-forward computation and inter-expert data movement, not attention alone.

Supporting evidence

Kyle Wiggers·

Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Original excerpt

With MXFP8 enabled across the parts of the system where it helped most, training throughput was about 21% higher than with BF16, the higher-precision format we used as our baseline, while peak active memory fell from 103 GiB to 95 GiB. Most of the gain came from feed-forward computation and moving data between experts rather than attention alone.
Context

We measured MXFP8’s effect on end-to-end training throughput in a controlled benchmark on four NVIDIA B300 GPUs, with work distributed uniformly across experts.

Open original · Paragraph 21 (opens in a new tab)Original text source (opens in a new tab)
Kyle Wiggers
Share insight

Statements by source date1

Statements are ordered by the original source publication date; differences in wording do not establish a change of position.