HAWK enables direct GPU-to-NPU KV cache transfer for phase-disaggregated LLM serving

By remapping tensor layouts directly during RDMA transfers, HAWK disaggregates GPU prefill and NPU decode to boost serving power efficiency by 1.26x and cost efficiency by 1.72x.

Primary source: arXiv:2610.29873: https://arxiv.org/abs/2610.29873

What’s new: Serving large language models efficiently requires handling two compute phases with starkly different hardware requirements: prefill is compute-bound and benefits from massive GPU parallel throughput, whereas autoregressive decoding is memory-bandwidth-bound, leaving GPU tensor cores heavily underutilized. While Neural Processing Units (NPUs) offer superior decode cost and power efficiency, pairing GPUs and NPUs in a phase-disaggregated cluster has been blocked by severe cross-device communication barriers and conflicting key-value (KV) cache tensor memory layouts. In HAWK: Heterogeneity-Aware KV Cache Transfer for Efficient GPU-NPU Phase-Disaggregated LLM Inference (arXiv:2610.29873, 21 Oct 2026), researchers Junhyeok Park, Jungeun Yoon, Kyungmin Kim, Hunsoon Lee, Shinyoung Ahn, and Kangho Kim from Georgia Tech and ETRI introduce HAWK, a cross-device transfer system that enables high-speed KV cache migration across heterogeneous accelerators.

HAWK overcomes the heterogeneity bottleneck by establishing a direct Remote Direct Memory Access (RDMA) pipeline between GPU and NPU device memories without host-staging intermediate copies. To resolve tensor format mismatches between GPU contiguous layouts and NPU tiled banking schemes, HAWK allocates layout-aware RDMA receive buffers directly on the NPU. It orchestrates scatter-gather transfers so that incoming KV tensors are rearranged into target NPU memory structures in-flight during network transport, eliminating secondary reorganization overhead. Evaluated on a production testbed combining NVIDIA A100 GPUs and Tenstorrent p150a NPUs, HAWK delivers up to 27.69 times faster heterogeneous KV transfers than host-staged baselines, matching peer GPU-to-GPU RDMA transfer latency within 0.65%. Under strict service level objectives (SLOs), the disaggregated cluster trails homogeneous GPU setups by less than 7.88% in latency while achieving up to 1.26 times higher power efficiency and 1.72 times higher cost efficiency.

Why it matters: Hyperscale AI data centers face severe power constraints and soaring accelerator capital expenditures. Decoupling prefill and decode across specialized silicon allows operators to route computationally dense prompt phases to high-FLOP GPUs while shifting prolonged decoding sequences to cost-effective, power-efficient NPUs. By erasing the cross-architecture communication penalty, HAWK makes heterogeneous accelerator clusters practical for production serving.

Caveats: The layout-aware buffering mechanism requires explicit hardware mapping definitions for each target NPU memory architecture, necessitating driver-level integration when introducing new accelerator families. Additionally, extremely high network congestion in shared multi-tenant clusters could narrow the latency margins required to meet tight time-to-first-token SLOs.