UALink

P. Borrill

This is the third of five papers comprising The Semantic Arrow of Time. Parts I and II identified computing's hidden semantic arrow of time, the FITO category mistake, and presented the constructive alternative: the OAE link state machine with its mandatory reflecting phase. This paper examines what happens when those principles are violated at industrial scale. Remote Direct Memory Access (RDMA) is the highest-performance data movement technology in production, deployed across Meta's 24,000-GPU clusters, Google's data centers, and Microsoft's Azure infrastructure. We argue that RDMA's completion semantics contain a category mistake: they guarantee placement (data written to a remote NIC buffer) but not commitment (data semantically integrated by the receiving application). We call this the completion fallacy. We document the fallacy through seven temporal stages of an RDMA Write operation, showing that the gap between completion signal and application semantic satisfaction can be arbitrarily large. We trace consequences through four case studies: Meta's RoCE fabric, Google's 1RMA redesign, Microsoft's DCQCN failures, and SDR-RDMA partial completions. A comparative analysis shows CXL 3.0, NVLink, and UALink each address parts of the completion fallacy but none eliminates it entirely. Only a protocol architecture with a mandatory reflecting phase can close the gap between delivery and commitment.

Rajesh Arsid

The emergence of Ultra Ethernet and UALink technologies marks a transformative advancement in AIinfrastructure networking, addressing the increasing demands of modern artificial intelligence and machinelearning workloads. Ultra Ethernet, developed through industry collaboration under the Linux Foundation,evolves traditional Ethernet technology with AI-optimized capabilities while maintaining backwardcompatibility. In parallel, UALink provides specialized accelerator-to-accelerator communication for AItraining environments. Both technologies introduce sophisticated features including Remote DirectMemory Access, packet spraying, congestion control, and advanced traffic management mechanisms. UltraEthernet focuses on enterprise-wide deployment with diverse workload support, while UALink specializesin high-density AI training clusters with direct load/store operations. Together, these complementarytechnologies enable organizations to scale their AI infrastructure effectively while maintaining reliabilityand standardization across computing environments.

Xiaoli Fang, Xuhui Liu, Chun-Zhang Chen, Liang Wang, Quan Pan, Hanming Wu

To meet the high bandwidth and low latency communication requirements between modern computer processors and accelerators or memories, for applications such as in data center processing, and "multi-GPU and CPU" in machine learning (ML), in the latter case though NVLink (2014) and the latest UALink (2024) are available, while CXL (Compute Express Link) has been developed in 2019 and continuously provides a cache consistent interconnect technology between CPU and device, as well as between devices. This article reports the custom design of CXL controller on Intel FPGA R-Tile architecture. The R-Tile supports PHY Interface for PCI Express (PIPE) direct mode, it can provide up to 16 SerDes (serializer / deserializer) channels. The PIPE SerDes mode interface is connected to the custom controller implemented in soft IP through the Embedded Multi-die Interconnect Bridge (EMIB). The CXL custom controller designed includes the Logic Physical Layer (PHY) and Medium Access Control (MAC), Data Link Layer (DLL) and Transaction Layer (TL), that includes the Transmitter (TX) and Receiver (RX). The custom controller design of this CXL device is expected to promote efficient data transmission, expand memory bandwidth and capacity, efficiently meet complex computing requirements, and significantly improve system performance, further verification and application are to be applied.

Giyong Jung, Saeid Gorgin, John Kim, Jungrae Kim

As AI models outpace the capabilities of single processors, interconnects across chips have become a critical enabler for scalable computing. These processors exchange massive amounts of data at cache-line granularity, prompting the adoption of new interconnect protocols like CXL, NVLink, and UALink, designed for high bandwidth and small payloads. However, the increasing transfer rates of these protocols heighten susceptibility to errors. While mechanisms like Cyclic Redundancy Check (CRC) and Forward Error Correction (FEC) are standard for reliable data transmission, scaling chip interconnects to multi-node configurations introduces new challenges, particularly in managing silently dropped flits in switching devices. This paper introduces Implicit Sequence Number (ISN), a novel mechanism that ensures precise flit drop detection and in-order delivery without adding header overhead. Additionally, we propose Reliability Extended Link (RXL), an extension of CXL that incorporates ISN to support scalable, reliable multi-node interconnects while maintaining compatibility with the existing flit structure. By elevating CRC to a transport-layer mechanism for end-to-end data and sequence integrity, and relying on FEC for link-layer error correction and detection, RXL delivers robust reliability and scalability without compromising bandwidth efficiency.

Sai Krishna Vemuri, Ajay Joshi, Vijay Kumar Motagi, Rohit Sindhu, Venkata Ravi Shankar Jonnalagadda, Amandeep Singh, Praveen Kumar Rukmangada, Sachin Shaw, Vishal Tanna

This paper presents a UALink-first comparative analysis of interconnects for large-scale AI training. UALink 1.0 is an open, vendor-neutral, rack-scale fabric that provides memory-semantic load/store, deterministic latency targets below one microsecond request-to-response and scaling up to 1024 accelerators per pod. We develop analytical models for latency and bandwidth-sensitive collectives common in transformer training and instantiate them under flat and hierarchical topologies. We compare UALink to NVIDIA’s NVLink generations 3–5, which represent the current proprietary state-of-the-art in per-GPU bandwidth and switch-based scaling. Results show that a flat UALink pod can reduce the step-time contribution of small-message collectives at scales of up to 1024 GPUs, while NVLink 5 maintains an advantage in bandwidth dominated regimes due to aggregate per-GPU bandwidth near 1.8 TB/s and large non-blocking pods. The figure-generation methodology directly maps tabulated parameters into closed-form equations, making assumptions explicit and reproducible.