nafyi.

OPINIONS EXPRIMÉES EN PUBLIC

Kyle Wiggers

1 sources · 3 points de vue · 3 sujets

Contenu mis à jour:

Kyle Wiggers sur MoE scaling efficiency, numerical precision optimization, training infrastructure performance. Explorez 3 points de vue par thème, avec des éléments tirés de 1 source.

Explorer les liens

Points de vue par sujet

Points de vue attribués, classés par date de publication de la source. Un aperçu de ces échanges, sans prétendre définir toutes les convictions de la personne.

Les traductions sont destinées à la lecture ; les extraits originaux restent la source evidence.

MoE scaling efficiency

Voir ce sujet
MoE scaling efficiency

Maintaining throughput while scaling expert count

The post reports that increasing the expert pool from 8 to 128 while selecting only four experts per token kept active parameters per token roughly fixed at ~3.2B; total parameter capacity grew from 4.6B to 47B with less than 5% drop in training throughput.

Éléments favorables

Kyle Wiggers·

Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Extrait original

In one benchmark, we increased the expert pool from 8 to 128 while still selecting only four experts per token – the small units of text a language model processes – keeping the number of active parameters per token roughly fixed at about 3.2B. Total parameter capacity grew from 4.6B to 47B, while training throughput fell by less than 5%.
Contexte

Olmo-core 3 is built to close that gap.

Ouvrir l’original · Paragraphe 5 (s’ouvre dans un nouvel onglet)Source du texte original (s’ouvre dans un nouvel onglet)
Kyle Wiggers
Partager un aperçu

training infrastructure performance

Voir ce sujet
training infrastructure performance

2.7× throughput gain with DDP-based MoE stack

The post reports that on eight NVIDIA B300 GPUs, Olmo-core 3’s new distributed data parallelism (DDP)-based MoE training stack achieved 52,000 tokens/sec/GPU versus 19,400 tokens/sec/GPU with the prior FSDP-based implementation — a ~2.7× throughput improvement.

Éléments favorables

Kyle Wiggers·

Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Extrait original

In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU with the new stack, compared with 19,400 using our earlier implementation—about 2.7× the throughput.
Contexte

NVIDIA’s Megatron-Core is an established option for training large MoEs. Olmo-core 3 brings an integrated MoE training stack to the framework behind Olmo, with a redesign that improves throughput over our earlier FSDP-based implementation.

Ouvrir l’original · Paragraphe 11 (s’ouvre dans un nouvel onglet)Source du texte original (s’ouvre dans un nouvel onglet)
Kyle Wiggers
Partager un aperçu

numerical precision optimization

Voir ce sujet
numerical precision optimization

MXFP8 improves throughput and peak active memory in a controlled B300 benchmark

The post reports that in a controlled benchmark on four NVIDIA B300 GPUs with uniform expert load, enabling MXFP8 in performance-critical parts increased end-to-end training throughput by ~21% over BF16 baseline and reduced peak active memory from 103 GiB to 95 GiB; most gains came from feed-forward computation and inter-expert data movement, not attention alone.

Éléments favorables

Kyle Wiggers·

Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Extrait original

With MXFP8 enabled across the parts of the system where it helped most, training throughput was about 21% higher than with BF16, the higher-precision format we used as our baseline, while peak active memory fell from 103 GiB to 95 GiB. Most of the gain came from feed-forward computation and moving data between experts rather than attention alone.
Contexte

We measured MXFP8’s effect on end-to-end training throughput in a controlled benchmark on four NVIDIA B300 GPUs, with work distributed uniformly across experts.

Ouvrir l’original · Paragraphe 21 (s’ouvre dans un nouvel onglet)Source du texte original (s’ouvre dans un nouvel onglet)
Kyle Wiggers
Partager un aperçu

Propos par date de source1

Les propos sont classés par date de publication de la source originale ; une différence de formulation ne prouve pas un changement de position.