As frontier AI evolves from stateless single-turn prompts to million-token agents, disaggregated serving, and continuous reinforcement learning (RL), the primary bottleneck in AI infrastructure has shifted from raw compute to orchestrating live model state — both the contextual state held in massive key-value (KV) caches during inference and the parameter state updated and broadcast from trainers to rollout samplers during RL. Whenever this state is trapped inside a single chip's memory or stalled on network transfers, AI accelerators sit idle.
We designed TPU Sync from first principles to serve as the model-state layer across TPU clusters — built on the architectural insight that accelerator High Bandwidth Memory (HBM), host DRAM, shared storage, and the data center network (DCN) should operate as a single, co-designed memory hierarchy. An open-source, native C++ library, TPU Sync moves TPU-tiled tensors directly across this hierarchy with asynchronous DMA transfers that overlap data movement with active computation.
For inference, it decouples TPU compute from network backpressure and turns host DRAM, local SSDs, and Google Cloud Managed Lustre into an automated, multi-tier KV cache; and for RL, it exposes low-level DMA and DCN primitives that higher-level RL frameworks can use to stream updated weights from trainers to samplers.
By accessing device buffers in place rather than copying or locking them, TPU Sync consistently slashes KV transfer latency by over 90% and reduces Time to First Token (TTFT) by an average of 90% for heavyweight open-weight MoE models (235B–480B parameters) in prefix-caching and disaggregated-serving workloads, compared to running without TPU Sync.
