nafyi.
Thèmes / MoE scaling efficiency

Point de vue reformulé

Maintaining throughput while scaling expert count

The post reports that increasing the expert pool from 8 to 128 while selecting only four experts per token kept active parameters per token roughly fixed at ~3.2B; total parameter capacity grew from 4.6B to 47B with less than 5% drop in training throughput.

Kyle Wiggers·Publié

Derrière ce point de vue

Les traductions sont destinées à la lecture ; les extraits originaux restent la source evidence.

Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Extrait original

In one benchmark, we increased the expert pool from 8 to 128 while still selecting only four experts per token – the small units of text a language model processes – keeping the number of active parameters per token roughly fixed at about 3.2B. Total parameter capacity grew from 4.6B to 47B, while training throughput fell by less than 5%.
Contexte

Olmo-core 3 is built to close that gap.

Ouvrir l’original · Paragraphe 5 (s’ouvre dans un nouvel onglet)Voir la page source