nafyi.

What Comes After Transformers?

The Data Exchange ·

Zuzanna Stamirowska calls RAG, context, and memory bottlenecks in 2023 and 2024 for what she describes as 'AI with no ceiling, if you wish.' She recalls realizing that reasoning should not just be linguistic, and says latent thinking can help solve constraint-satisfaction problems and math with 'tiny, tiny models.'. Read 3 viewpoints with supporting evidence and source links.

Ben LoricaZuzanna Stamirowska
Read key viewpointsOpen Publisher (opens in a new tab)

Understand this piece

3 key points

Synthesis

  1. RAG, context, and memory were bottlenecks in 2023–2024

    Stamirowska says the team saw RAG, context, and memory as bottlenecks in 2023 and 2024. They were not efficient or sufficient for what she describes as 'AI with no ceiling, if you wish.'

    Supporting evidence 1

    Original excerpt

    But very concretely, what prompted us to start Pathway and to start working on post-transformer architectures was that we were looking forward to reasoning becoming a thing. We have a very strong background in dealing with the dimension of time in machine learning—any sort of online learning. And we saw RAG, context, and memory, and we’re talking about 2023 and 2024, as bottlenecks. They were not efficient or sufficient to unlock AI as we would like—AI with no ceiling, if you wish.

    Zuzanna Stamirowska · Publisher transcript paragraph 7

    What Comes After Transformers? · Sep 24, 2026

    Read in source context →

    Continue exploring

    post-transformer architecture motivation →

  2. Stamirowska says reasoning shouldn't just be linguistic

    Stamirowska says, 'We realized that reasoning shouldn't just be linguistic.' She says verbalizing things in chain-of-thought is not optimal: it involves 'gluing it on top of a transformer,' and transformers are naturally ill-suited to dealing with time and learning from experience.

    Supporting evidence 1

    Original excerpt

    Specifically, we’re talking about continual learning, memory, and reasoning first. And as we moved through it, we realized that reasoning shouldn’t just be linguistic. Verbalizing things in chain of thought is not the optimal way to do reasoning. With chain of thought, we’re essentially gluing it on top of a transformer, and transformers are naturally ill-suited to dealing with time and learning from experience.

    Zuzanna Stamirowska · Publisher transcript paragraph 8

    What Comes After Transformers? · Sep 24, 2026

    Read in source context →

    Continue exploring

    reasoning modality →

  3. Latent thinking can help solve constraint-satisfaction and math problems

    Stamirowska says latent thinking can help solve constraint-satisfaction problems and math with 'tiny, tiny models.'

    Supporting evidence 1

    Original excerpt

    Latent thinking has huge benefits. It can help us solve constraint-satisfaction problems natively, like what we’re showing. We’re solving constraint-satisfaction problems and math with tiny, tiny models. And then there’s cost efficiency. There are just insane cost gains coming from latent reasoning.

    Zuzanna Stamirowska · Publisher transcript paragraph 37

    What Comes After Transformers? · Sep 24, 2026

    Read in source context →

    Continue exploring

    latent reasoning benefits →

Key moments3

Short, attributed passages with the context to verify them. The full conversation stays with its publisher.

Publisher transcript paragraph 7 (opens in a new tab)
post-transformer architecture motivation

RAG, context, and memory were bottlenecks in 2023–2024

Original excerpt

But very concretely, what prompted us to start Pathway and to start working on post-transformer architectures was that we were looking forward to reasoning becoming a thing. We have a very strong background in dealing with the dimension of time in machine learning—any sort of online learning. And we saw RAG, context, and memory, and we’re talking about 2023 and 2024, as bottlenecks. They were not efficient or sufficient to unlock AI as we would like—AI with no ceiling, if you wish.
Zuzanna Stamirowska·
Publisher transcript paragraph 8 (opens in a new tab)
reasoning modality

Stamirowska says reasoning shouldn't just be linguistic

Original excerpt

Specifically, we’re talking about continual learning, memory, and reasoning first. And as we moved through it, we realized that reasoning shouldn’t just be linguistic. Verbalizing things in chain of thought is not the optimal way to do reasoning. With chain of thought, we’re essentially gluing it on top of a transformer, and transformers are naturally ill-suited to dealing with time and learning from experience.
Zuzanna Stamirowska·
Publisher transcript paragraph 37 (opens in a new tab)
latent reasoning benefits

Latent thinking can help solve constraint-satisfaction and math problems

Original excerpt

Latent thinking has huge benefits. It can help us solve constraint-satisfaction problems natively, like what we’re showing. We’re solving constraint-satisfaction problems and math with tiny, tiny models. And then there’s cost efficiency. There are just insane cost gains coming from latent reasoning.
Zuzanna Stamirowska·

Source & methodology

These viewpoints are linked to their original sources. Paraphrases are labeled and are not verbatim quotes.

Open transcript or source material (opens in a new tab)Report an issue

Explore these viewpoints by person