A six-person research team led by AMD's Jiakun Yan has developed KIWI, a technology that hands communication control from the GPU to the CPU. Applied to the decode stage of the AI model GLM-5.3 as "KIWI-EP," it cut processing time per output token from about 55 milliseconds to about 39 milliseconds, compared with the conventional communication method MORI-EP. This result is for a configuration with 8 nodes each allocated to prefill and decode and a per-GPU concurrency of 1. The figures are approximate values read from Figure 8 of the paper.
On November 18, 2026, the team published the design and real-hardware test results as a preprint on arXiv (arXiv:2611.23848v1, DOI: 10.48550/arXiv.2611.23848). KIWI's defining feature is that the GPU initiates communication while the CPU handles issuing instructions to the network interface card (NIC). The data itself is transferred directly between GPU memories by the NIC, without passing through the CPU. KIWI also lets applications freely define the work delegated to the CPU, and KIWI-EP applies this mechanism to language models that use multiple expert networks.
What the GPU hands to the CPU is communication control
In GPU-to-GPU communication, having the GPU itself initiate transfers means it no longer has to interrupt computation to wait for CPU instructions. In GPUDirect Async (GDA), the widely used method in which the GPU operates the NIC directly, the GPU prepares the destination and communication commands and notifies the NIC to start sending. However, this kind of communication control involves a lot of sequential processing, where commands must run in a fixed order. The team reasoned that handing this work to CPU cores might be more efficient than leaving it to the GPU, which excels at massively parallel computation.
KIWI adopts a method called Reverse Offload. The GPU still initiates communication as before, but a proxy thread on the CPU, which receives the request from the GPU, is responsible for issuing instructions to the NIC. As Figure 1 of the paper shows, in both GDA and reverse offload the NIC transfers the data itself directly between GPU memories. The difference lies in whether the GPU or the CPU assembles the transfer commands and hands them to the NIC.
In KIWI, the application registers in advance the functions it wants to run on the CPU. On AMD GPUs, a group of threads that execute the same instruction simultaneously is called a wave, and KIWI provides a dedicated queue for each wave. The GPU simply writes a function identifier and arguments into the queue and moves on to its next task, while a CPU-side proxy thread picks up the request and runs the registered function. Because multiple waves do not have to contend for the same queue, contention is kept down when requests are concentrated.
The team also measured the time this handoff takes. Using a single Instinct MI300X, they sent a request from the GPU to the CPU with no other requests pending and measured the round-trip time until the response returned to the GPU. The median was 1.92 microseconds. This matches the baseline implementation that uses a simple memory write, meaning KIWI's queue adds almost no measurable latency.
Table III of the paper also compares the time for the GPU to write a request into the queue. For KIWI's queue alone, the registration time was 0.12 microseconds with either 1 wave or 1,024 waves. Even in the configuration that invokes a registered function on the CPU, the GPU-side registration time at 1,024 waves was only 0.16 microseconds. However, these are times for handing requests between GPU and CPU; transferring data to another node adds the time needed to instruct the NIC and for network communication.
Which method wins flips depending on token count
KIWI-EP targets the communication processing of Mixture of Experts (MoE) models, which switch between multiple expert networks. In MoE, each input token must be sent to the GPU hosting the expert responsible for it, and the computed results must then be collected and combined. Because small pieces of data are exchanged frequently among many GPUs, communication latency affects inference performance. KIWI-EP bundles sends to multiple destinations into a single request and lets notification of received counts and synchronization also run as CPU-side functions. The GPU retains tasks such as repacking data according to destination and unpacking received data.
However, simply handing requests from the GPU to the CPU can leave the load unevenly distributed among proxy threads. For example, if only the first wave of each 16-wave group issues requests, the queues used are 0, 16, 32, and so on, at regular intervals. Assigning these to four proxy threads in simple round-robin fashion would send every request to the same thread. KIWI-EP therefore adopts an assignment scheme that shifts the responsible thread for each group and also manages the range of queues that actually issue requests, thereby spreading the load.
For the communication benchmarks, the team used the same MoE layer configuration as DeepSeek-V3: a hidden dimension of 7,168 and 256 experts, with 8 experts selected per token. The data format was unified as BF16, and each communication method was given identical input. The measurement covers the total time to distribute tokens and collect computed results, using the value from the slowest participating process.
With a 2-node configuration processing 1 token per GPU, the communication times read from Figure 7 of the paper were about 111 microseconds for MORI-EP, the fastest of the comparison targets, and about 49 microseconds for KIWI-EP. KIWI-EP held communication time to less than half. But in the same 2-node configuration at 64 tokens per GPU, MORI-EP took about 174 microseconds and KIWI-EP about 274 microseconds, reversing the result.
KIWI-EP was the fastest among the comparison targets up to 16 tokens per GPU in the 2-node configuration and up to 32 tokens in the 4-node configuration. In the 8-node configuration, by contrast, KIWI-EP recorded the shortest communication time under every condition tested, from 1 to 64 tokens.
The team attributes this performance gap to differences in how data is transferred between nodes. KIWI-EP transfers data to each destination GPU individually, whereas MORI-EP first sends data to the destination node and then distributes it to each GPU within that node. As the token count grows and network bandwidth becomes more constrained, MORI-EP, which can reduce the number of inter-node transfers, gains the advantage. KIWI-EP's advantage turned out to vary with the number of tokens processed and the node configuration.
GLM-5.3 generation time cut by up to 29%
The team evaluated not only the communication component but also performance when running an actual large language model. They used GLM-5.3 in 8-bit floating-point format (FP8) with vLLM as the inference engine. Input was set to 8K tokens and output to 1K tokens, and "prefill," which processes the input in bulk, and "decode," which generates tokens one at a time, were run on separate nodes. Prefill continued to use the conventional MORI-EP, and only the decode side was switched between MORI-EP and KIWI-EP for comparison. Each measurement point used the median of at least three runs.
They evaluated three configurations, allocating 2, 4, or 8 nodes each to prefill and decode. The largest, the "8P8D" configuration, uses 16 nodes in total. Each node has eight AMD Instinct MI300X GPUs, two 56-core Intel Xeon Platinum 8480+ CPUs, and eight NVIDIA ConnectX-7 NICs supporting 400Gb/s. Each GPU and its corresponding NIC are connected to the same PCIe switch, and the CPU-side proxy threads were pinned to CPU cores in the NUMA region closest to the corresponding GPU.
In Figure 8 of the paper, KIWI-EP's time per output token (TPOT) was lower than MORI-EP's in every configuration and concurrency level tested. The largest improvement was 29%. In the 8P8D configuration with a per-GPU concurrency of 1, the values read from the figure are about 55 ms for MORI-EP and about 39 ms for KIWI-EP. Even in the same 8P8D configuration with concurrency raised to 128, the time fell from about 62.8 ms to about 51.0 ms.

Note that in vLLM, in-flight requests are swapped out one after another, so the number of requests actually running decode is lower than the configured concurrency. For this reason, the improvement in the communication component does not translate directly into the reduction in overall model generation time.
The performance gain comes not only from moving communication control to the CPU but also from the mechanism that bundles sends to multiple destinations and from proxy-thread load balancing. In the 8-node communication tests, the team added these changes step by step to examine their effects. The maximum 29% improvement measured with GLM-5.3, meanwhile, is the result of switching the entire decode-side communication method from MORI-EP to KIWI-EP.
The team has published KIWI's source code and benchmark scripts on GitHub. The performance evaluation is the authors' own measurements on an MI300X cluster, and no independent replication has been confirmed. In addition, the KIWI-EP used in the evaluation assigns four CPU cores per GPU to proxy threads. Anyone considering deployment in a different inference environment will need to weigh the added CPU core load against how much generation-time reduction they can actually achieve at their own token counts and concurrency levels.
