Transformer 之后是什么？
但非常具体地说，促使我们创办 Pathway 并开始研究后 transformer 架构的原因是，我们期待推理成为一件真正的事。我们在处理机器学习中的时间维度方面——也就是任何形式的在线学习——拥有非常深厚的背景。而我们看到 RAG、上下文和记忆，这里说的是 2023 年和 2024 年，成为了瓶颈。它们不够高效，也不足以实现我们所期望的人工智能——如果你愿意这么说的话，就是没有天花板的人工智能。
原始摘录
But very concretely, what prompted us to start Pathway and to start working on post-transformer architectures was that we were looking forward to reasoning becoming a thing. We have a very strong background in dealing with the dimension of time in machine learning—any sort of online learning. And we saw RAG, context, and memory, and we’re talking about 2023 and 2024, as bottlenecks. They were not efficient or sufficient to unlock AI as we would like—AI with no ceiling, if you wish.