Ryzen AI

Alejandro Rico, Satyaprakash Pareek, Javier Cabezas, David N. J. Clarke, B. Ozgul, F. Barat, Yao Fu, Stephan Münz, D. Stuart, Patrick Schlangen, P. Duarte, Sneha Date, Indrani Paul, Jian Weng, Sonal Santan, Vinod Kathail, Ashish Sirasao, J. Noguera

The AMD Ryzen 7040 series is the first x86 processor with an integrated neural processing unit (NPU). The AMD XDNA technology in the NPU of Ryzen artificial intelligence (AI) processors provides optimized compute and memory resources for the needs for AI workloads and a specialized data movement architecture that allows to minimize bandwidth requirements leading to higher performance and efficiency. Across a selection of neural network benchmarks, XDNA provides between 4.3× and 33× better performance per watt, leading to extended battery life. The AI-optimized capabilities of the Ryzen 7040 NPU enable new AI experiences that are not possible without XDNA, making it a fundamental component in today’s Ryzen-AI-powered devices and setting the foundation for an exciting roadmap toward future AI capabilities in mobile PCs.

Endri Taka, A. Roesti, Joseph Melber, Pranathi Vasireddy, Kristof Denolf, Diana Marculescu

The high computational and memory demands of modern deep learning (DL) workloads have led to the development of specialized hardware devices from cloud to edge, such as AMD's Ryzen™ AI XDNA™ NPUs. Optimizing general matrix multiplication (GEMM) algorithms for these architectures is critical for improving DL workload performance. To this end, this paper presents a common systematic methodology to optimize GEMM workloads across the two current NPU generations, namely XDNA and XDNA2. Our implementations exploit the unique architectural features of AMD's NPUs and address key performance bottlenecks at the system level. End-to-end performance evaluation across various GEMM sizes demonstrates state-of-the-art throughput of up to 6.76 TOPS (XDNA) and 38.05 TOPS (XDNA2) for 8-bit integer (int8) precision. Similarly, for brain floating-point (bf16) precision, our GEMM implementations attain up to 3.14 TOPS (XDNA) and 14.71 TOPS (XDNA2). This work provides significant insights into key performance aspects of optimizing GEMM workloads on Ryzen AI NPUs.

Nick Brown, Gabriel Rodriguez-Canal

A major challenge that the HPC community faces is how to continue delivering the performance demanded by scientific programmers, whilst meeting an increased emphasis on sustainable operations. Specialised architectures, such as FPGAs and AMD's AI Engines (AIEs), have demonstrated significant energy efficiency advantages, however a major challenge is that to most effectively program these architectures requires significant expertise and investment of time which is a major blocker. Fortran in the lingua franca of scientific computing, and in this work we seamlessly accelerate Fortran intrinsics via the AIEs in AMD's Ryzen AI CPU.

Nick Brown, Gabriel Rodriguez-Canal

A major challenge the HPC community faces is how to deliver increased performance demanded by scientific programmers, whilst addressing an increased emphasis on sustainable operations. Specialised architectures, such as FPGAs and AMD’s AI Engines (AIEs), have demonstrated significant energy efficiency advantages, however a major challenge is that substantial expertise and investment of time is required to gain best performance from this hardware which is a major blocker.Fortran in the lingua franca of scientific computing, and in this paper we explore the automatic offloading of Fortran intrinsics to the AIEs in AMD’s Ryzen AI CPU as a case study, demonstrating how the MLIR compiler ecosystem can provide performance and programmer productivity. We describe an approach that lowers the MLIR linear algebra dialect to AMD’s AIE dialects, and demonstrate that for suitable workloads the AIEs can provide significant performance advantages over the CPU without any code modifications required by the programmer.CCS Concepts• Computer systems organization → Reconfigurable computing; Data flow architectures; • Software and its engineering→Source code generation; Runtime environments; Compilers.

Rachid Karami, Rajeev Patwari, Hyoukjun Kwon, Ashish Sirasao

The integration of generative AI models, particularly large language models (LLMs), into real-time multi-model AI applications such as video conferencing and gaming is giving rise to a new class of workloads: real-time generative AI (RTGen). These workloads combine the compute intensity and dynamic execution patterns of generative models with the stringent latency and concurrency constraints of real-time inference. To meet the diverse demands of RTGen workloads, modern edge platforms increasingly adopt heterogeneous system-on-chip (SoC) architectures that integrate CPUs, GPUs, and NPUs. Despite the potential of heterogeneous SoC, the scheduling space complexity and performance implications of RTGen workloads on such platforms remain underexplored. In this work, we perform a comprehensive characterization of RTGen workloads on AMD's latest heterogeneous SoC, Ryzen AI. We construct realistic multi-model scenarios inspired by industry use cases and profile model performance across all available backends. Using this data, we evaluate five scheduling policies and their impact on both real-time metrics (e.g., deadline violation rate) and LLM performance (e.g., time-to-first-token and tokens-per-second). Our results show that scheduling decisions significantly affect workload performance (e.g., leading to a 41.7% difference in deadline violation rates on average), and highlight the need for scheduling strategies that are aware of workload dynamics and hardware heterogeneity. Our findings underscore the importance of workload-aware, dynamic heterogeneous scheduling in enabling high-performance, on-device RTGen applications.