The Data Exchange ·
Richard Ho describes OpenAI’s chip work as an effort to lower inference and compute costs. He explains the memory-compute locality built into the architecture and how one device can operate along the Pareto curve between high throughput and low latency. Read 3 viewpoints with supporting evidence and source links.
Lines show the reading structure. Select an idea to read its explanation and evidence.
Synthesis
The mission was to lower inference and compute costs—making intelligence faster and cheaper for users.
Original excerpt
They wanted to look really hard at what we could do to lower the cost of inference and lower the cost of compute.
Richard Ho · Publisher transcript paragraph 10
I think this project really kicked off when Sam and Greg decided that infrastructure was going to be a key differentiator and one of the key drivers of AI and intelligence to users.
Continue exploring
AI infrastructure strategy →The only key hardwired architectural feature is tight memory-compute locality—specifically, affinity between HBM and compute cores—to improve performance, power efficiency, and reduce data movement.
Original excerpt
The one thing that is really unique about our architecture, that is kind of hardwired in, is the locality and affinity between memory and compute. That’s the key architectural difference of this device.
Richard Ho · Publisher transcript paragraph 31
Read in source context →Continue exploring
Chip architecture →Ho says a single device can operate at either end of the Pareto curve between high throughput and low latency, or at points between them. He connects this capability to lowering infrastructure costs.
Original excerpt
What we have in a single device—and I think this is really critical for our infrastructure and being able to lower the cost of the infrastructure—is a single device that can operate at both points, or at any point along that curve. We call it the Pareto curve between high throughput and low latency.
Richard Ho · Publisher transcript paragraph 43
Read in source context →Continue exploring
Hardware flexibility →Short, attributed passages with the context to verify them. The full conversation stays with its publisher.
Original excerpt
They wanted to look really hard at what we could do to lower the cost of inference and lower the cost of compute.
I think this project really kicked off when Sam and Greg decided that infrastructure was going to be a key differentiator and one of the key drivers of AI and intelligence to users.
Original excerpt
The one thing that is really unique about our architecture, that is kind of hardwired in, is the locality and affinity between memory and compute. That’s the key architectural difference of this device.
Original excerpt
What we have in a single device—and I think this is really critical for our infrastructure and being able to lower the cost of the infrastructure—is a single device that can operate at both points, or at any point along that curve. We call it the Pareto curve between high throughput and low latency.
These viewpoints are linked to their original sources. Paraphrases are labeled and are not verbatim quotes.
Open transcript or source material (opens in a new tab)Report an issueRichard Ho on AI infrastructure strategy, Chip architecture, Hardware flexibility. Explore 3 viewpoints by topic, with evidence from 1 source.