Researchers at Georgia Tech, Nvidia and Stanford have published a method that raises large language model inference throughput by letting accelerators access high-bandwidth memory (HBM) and host DRAM concurrently, rather than treating host memory as an overflow tier used only when HBM fills up.

The work targets a structural bottleneck in LLM deployment. Model weights and key-value caches have outgrown the HBM capacity of single accelerators, which forces operators to shard models across GPUs or offload tensors to host memory in sequential, blocking transfers. Both paths cost time.

What does the technique change?

The paper's core move is scheduling. Instead of staging data from host DRAM into HBM before the compute units can touch it, the system overlaps reads from both pools during inference.

That shifts host memory from a fallback store into a working tier. The GPU draws on PCIe- or CXL-attached DRAM while it continues to stream from HBM, so the compute engines stall less often waiting for a transfer to finish.

The practical consequence: models that exceed local HBM can run on fewer accelerators, or serve more requests per second on the same hardware.

Who built it, and why it matters now

Three institutions appear on the work: Georgia Tech, Stanford and Nvidia. Nvidia's involvement signals vendor interest, since the company sells the HBM-equipped GPUs — H100 with 80 GB, H200 with 141 GB, B200 above that — whose capacity limits the technique addresses.

Capacity pressure keeps rising. Frontier-class LLMs now reach hundreds of gigabytes of weights, and key-value caches grow with context length and concurrent users. Memory, not floating-point throughput, increasingly sets the serving ceiling.

Why throughput, not just capacity

Concurrency is the operative word in the title. Prior offloading schemes gain capacity but lose latency, because transfers serialize with compute. Overlapping the two memory pools recovers part of that loss.

For operators, the metric that counts is tokens per second per dollar of installed hardware. A method that lifts throughput on existing GPUs defers capital spending on additional accelerators — a direct line from a memory-scheduling change to a data-center cost figure.

The takeaway: the researchers turned host DRAM into a peer of HBM during inference, and throughput improved without new silicon.