Runway Research | Introducing General World Models

Synthesis

1. 01

   Definition and scope of general world models

   A world model is an AI system that builds an internal representation of an environment to simulate future events within it. Current research is limited to toy simulated worlds (e.g., video games) or narrow domains (e.g., driving). General world models aim to represent and simulate diverse, real-world situations and interactions.

   Supporting evidence 1

   Original excerpt

   A world model is an AI system that builds an internal representation of an environment, and uses it to simulate future events within that environment. Research in world models has so far been focused on very limited and controlled settings, either in toy simulated worlds (like those of video games ) or narrow contexts (such as developing world models for driving ). The aim of general world models will be to represent and simulate a wide range of situations and interactions, like those encountered in the real world.

   Anastasis Germanidis · Paragraph 1

   Runway Research | Introducing General World Models · Dec 11, 2023

   Read in source context →

   Continue exploring

   AI research direction →
2. 02

   Gen-2 as an early, limited general world model

   Video generative systems like Gen-2 are early, limited forms of general world models: they show some understanding of physics and motion needed for realistic short video generation, but struggle with complex camera or object motions.

   Supporting evidence 1

   Original excerpt

   You can think of video generative systems such as Gen-2 as very early and limited forms of general world models. In order for Gen-2 to generate realistic short videos, it has developed some understanding of physics and motion. However, it’s still very limited in its capabilities, struggling with complex camera or object motions, among other things.

   Anastasis Germanidis · Paragraph 2

   Runway Research | Introducing General World Models · Dec 11, 2023

   Read in source context →

   Continue exploring

   AI capability assessment →
3. 03

   Core technical challenges for general world models

   Building general world models requires solving several open research challenges: generating consistent environmental maps; enabling navigation and interaction within those environments; capturing not only world dynamics but also the dynamics of inhabitants—especially realistic models of human behavior.

   Supporting evidence 1

   Original excerpt

   To build general world models, there are several open research challenges that we’re working on. For one, those models will need to generate consistent maps of the environment, and the ability to navigate and interact in those environments. They need to capture not just the dynamics of the world, but the dynamics of its inhabitants, which involves also building realistic models of human behavior.

   Anastasis Germanidis · Paragraph 3

   Runway Research | Introducing General World Models · Dec 11, 2023

   Read in source context →

   Continue exploring

   AI research challenges →