How do Language Processing Units (LPUs) achieve efficiency
My take on understanding the LPU architecture briefly.
On the heels of Nvidia’s acquisition of Groq, I have been trying to understand how Language Processing Units (LPUs) provide predictable, low-latency inference at scale. This is my attempt at explaining it simply and succinctly.
The LPU design and architecture are influenced by a number of design principles. I won’t repeat those here. I’d recommend the white-paper Groq published for it.
In essence, LPUs achieve their efficiency in inference by providing the following two major benefits:
Predictable execution flow of instructions and data.
Low-latency, fast access to compute and storage.
How do they achieve those is the more complex part.
Unlike GPUs and their tooling, LPUs provide a software-first approach to inference. When a specific model is being compiled to be run on LPUs during model initialization/deployment time, all execution and data flow paths are scheduled/defined in advanced. There are no surprises, and no use of caches and buffers to bring in uncertainty in execution times and flow. It is what they call a statically scheduled program or compiler. If you run it again and again, it chooses the same flow, the same path. And because it is statically compiled, it knows beforehand all the paths that will be taken, which allows it to eliminate resource contention (which is a big problem in GPUs), even when the memory/storage is largely shared. If nothing is blocking or waiting on another process to release access to storage or compute, everything can execute fast. What’s more impressive is that the scheduled paths compiled into the static program are all chosen via software. It does not require synchronization at the hardware level at all. The compiled program takes care of it all. Hardware synchronization leads to indeterministic execution, which introduces delays in execution flows.
The second approach LPUs use is their conveyor belt and Single Input Multiple Data (SIMD) function unit architecture. This architecture is influenced by Groq’s initial work on Tensor Streaming Processor, which was designed specifically for tensor or matrix multiplications (which are the core operations LLMs perform) — in contrast, GPUs are designed for general purpose operations, particularly favoring graphics related manipulations. Each conveyor belt carries instructions and data. SIMD function units receive instructions telling them which belt to pick input data from, which operations to perform, and where to put the output data. All of this flows in the form of streams, that don’t overlap and don’t block other streams. Given that all of this is on the same silicon (what is known as on-chip) along with ample compute and storage, and given that the execution path is schedule beforehand (statically compiled) resulting in no resource contention or blocking or indeterministic delays, you can begin to understand how efficient this mode of inference becomes.
Finally, the use of on-chip SRAM for storage, which can see transfer speeds of up to 80 TB/s (compared to the ~8 TB/s that high memory bandwidth modules on GPUs can provide), provide the high speed bandwidth needed to move data around SIMD function units. However, you have to keep in mind that unlike High Bandwidth Memory, which is a dedicated memory unit off-chip on GPUs, SRAM can store limited data. That is where the LPU architecture goes one step ahead. Not only does it perform efficiently on-chip, it can combine multiple chips together and use the same conveyor belt architecture to transfer instructions and data efficiently across multiple chips without sacrificing anything.
The result: predictable, low-latency inference at scale.