Independent analysis estimates that NVIDIA's next-generation GPU, "Rubin," will draw between 1,800 and 2,300 W of power — comparable to the outdoor unit of a home air conditioner. Pack 72 of these into a single rack, as in the Vera Rubin NVL72, and you get up to 600 kW per rack — roughly the electricity consumption of about 500 average Japanese households, concentrated in one server rack.
What these numbers reveal is that improving AI chip performance has become synonymous with fighting heat. According to a TrendForce study released on August 17, 2026, the thermal design power (TDP) of AI chips from NVIDIA, AMD, and Google has already surpassed 1 kW per chip, and reaches several hundred kilowatts at the rack scale. Liquid-cooling adoption has climbed from 6% in 2023 to 14% in 2024 and 33% in 2025, and is projected to hit 53% in 2026 and roughly 60% in 2027 — nearly a tenfold increase in adoption over three years.
Why "boost performance now, worry about heat later" no longer works
Thermal design used to be an afterthought, handled at the very end of the design flow. In the era of monolithic single-die chips, engineers simply estimated junction temperature from a package's thermal resistance and ambient temperature, then confirmed it stayed below the ceiling (roughly 105°C for CMOS, 85°C for DRAM). As Amkor Technology Vice President Mike Kelly notes, this was a relatively straightforward calculation.
The picture changed with the arrival of the chiplet era and the spread of 3D stacking. Placing multiple dies close together — and then stacking them vertically — leaves heat with nowhere to escape. This is especially true for High Bandwidth Memory (HBM), where DRAM dies are stacked 8 or 12 layers high, and the lower layers run hotter. SK hynix introduced Mass Reflow Molded Underfill (MR-MUF) in 2019 precisely because heat had already become a non-negligible issue at the HBM2E stage. In 2023, it developed Advanced MR-MUF for 12-layer HBM3, improving heat dissipation by 1.6x over the previous generation. HBM3E achieved a further 10% improvement.
But such material-level improvements are approaching their limits. A 2025 review paper published in the journal Electronics found that beyond 12 layers of stacking, internal thermal resistance rises sharply, exceeding what conventional heat spreaders and microchannels can handle. In a 3D finite element analysis that increased DRAM layers to 4, 8, and 12, the 12-layer model's peak temperature reached 80°C — approaching DRAM's maximum allowable temperature.
STCO: making heat the starting point of design, not an afterthought
In response to this structural bottleneck, the industry is turning to STCO (System Technology Co-Optimization) — a method that simultaneously optimizes every design variable, from chiplet placement and package structure to cooling method and even clock frequency. The critical departure from convention is that thermal simulation now comes first in the design flow, not last.
"In these high-end systems, we're at the limits of what's possible. Thermal simulation has been pushed all the way forward to the prototyping stage. That's the exact opposite of the traditional design flow," said Marc Swinnen, Product Marketing Director for Synopsys's semiconductor division.
The clearest demonstration of STCO's effectiveness came from research imec presented at IEEE IEDM in December 2025. A team led by Yukai Chen simulated a 3D configuration in which four 12-layer HBM stacks were placed directly on top of a 414 W GPU. Without any countermeasures, the GPU's peak temperature reached 141.7°C. Under the same conditions, a conventional 2.5D implementation — with HBM arrayed around the GPU on an interposer — reached only 69.1°C. In other words, 3D stacking more than doubled the GPU's temperature.
The imec team combined technology-level and system-level countermeasures. At the technology level, they removed the HBM stack's base logic die to improve thermal coupling, merged adjacent stacks to create heat escape paths, and placed highly thermally conductive silicon at hotspots. At the system level, they applied double-sided cooling and halved the GPU's core frequency. Together, these measures brought peak temperature down from 141.7°C to 70.8°C — nearly matching the 2.5D implementation's 69.1°C.
However, James Myers, imec's Program Director for System Technology, acknowledges that "halving the GPU core frequency imposes a 28% workload penalty (delaying AI training steps)." Even so, he says the 3D configuration delivers higher overall performance than the 2.5D baseline thanks to its greater throughput density.
| Metric | 2.5D Implementation (Conventional) | 3D HBM-on-GPU (No Countermeasures) | 3D HBM-on-GPU (After STCO) |
|---|---|---|---|
| GPU Peak Temperature | 69.1°C | 141.7°C | 70.8°C |
| HBM Placement | Around GPU (on interposer) | Directly on GPU | Directly on GPU |
| Package Area | Larger (HBM occupies surrounding space) | Smaller | Smaller |
| GPU Frequency | Normal | Normal | Halved (28% delay) |
(Source: imec, IEEE IEDM 2025)
Samsung's zHBM: a bet on placing memory directly above the processor
While STCO represents an innovation in design methodology, zHBM — which Samsung unveiled at the FMS (Future of Memory and Storage) conference on August 4, 2026 — attempts to redefine the architecture itself. Conventional HBM sits around an AI accelerator, but zHBM vertically stacks the HBM stack directly on top of the accelerator, aiming to minimize the distance data must travel and simultaneously boost bandwidth and power efficiency.
According to Samsung's official announcement, zHBM, built using next-generation wafer bonding technology, achieves more than 10x the memory density of HBM5, 3x the energy efficiency, and a reduction in thermal resistance of over 50%. Samsung further claims that a next-generation interface system incorporating zHBM would deliver roughly 8x the "performance" of HBM5.
However, as Tom's Hardware points out, these figures remain vague. It's unclear whether "8x performance" refers to bandwidth or effective application-level performance, and HBM5 itself — the point of comparison — has not yet been finalized as a standard. Samsung also has not disclosed a timeline for commercialization. According to Bloomberg, Samsung plans to begin mass production of HBM4 in the second half of 2026, but timelines for HBM5 and zHBM remain undetermined.
CPO: avoiding heat with light, but creating a new contradiction
Another approach to the heat problem is CPO (Co-Packaged Optics). CPO converts electrical signals into optical signals for inter-chip connections. Copper wiring suffers from resistive loss and capacitive load that degrade signal quality — and generate heat in the process. Switching to light reduces propagation loss, widens bandwidth, and improves signal integrity. A 2026 review paper in Nature Electronics positions CPO as "a path toward foundational communication technology for HPC and AI."
Professor Kim Sung-dong of Seoul National University of Science and Technology told ETNews that "the AI industry is beginning to prioritize thermal management over further performance gains, and CPO and STCO are at the center of that shift."
But CPO introduces its own new thermal problems. Optical components — lasers and photonic integrated circuits in particular — are extremely temperature-sensitive. A Siemens technical blog post from February 2026 notes that because CPO places temperature-sensitive optical components near high-power ASICs, localized heat flux rises, causing wavelength drift and degraded modulation contrast. Dynamic AI workloads trigger rapid temperature swings that directly affect optical performance. CPO is thus simultaneously a technology that "reduces" heat and one that is "vulnerable" to it. How this contradiction is resolved will be a watershed moment for CPO's practical deployment.
KAIST's experiment: letting AI design the cooling of AI chips
KAIST Professor Kim Joung-ho — known as the "father of HBM" — advocates incorporating AI itself into the design process as a solution to the heat problem. His lab operates a system called the "HBM Design AI Agent," which uses reinforcement learning to automatically explore microbump and TSV (through-silicon via) placement, as well as optimal decoupling capacitor placement. In July 2026, the lab launched a platform called "OpenClaw AI Agent" and held a workshop where AI agents handled package design, simulation automation, and power delivery network analysis.
This direction is gaining academic support as well. "WarpagePINN," a paper posted to arXiv in 2025, uses a physics-informed neural network (PINN) to simultaneously predict package warpage and temperature distribution, achieving roughly 1000x faster computation than conventional finite element methods in a CTE parameter study, with a mean absolute error of just 0.2 μm. A research group at Purdue University has also developed a neural network that solves 3D steady-state heat conduction via image transformation, reporting a 1663x speedup over FEM.
Professor Kim also predicts the emergence of next-generation memory beyond HBM: High Bandwidth Flash (HBF), which stacks NAND flash, and High Bandwidth SRAM (HBS), which stacks SRAM. KAIST's TERA Lab held a seminar on HBF in February 2026 and published workload analysis for hybrid memory configurations combining GPU, HBM, and HBF. As stacking counts grow and architectures become more complex, controlling heat through human intuition alone becomes impossible. AI-driven design automation is shifting from an option to a prerequisite.
What the "standardization" of liquid cooling reveals
Returning to TrendForce's data, the sharp rise in liquid-cooling adoption shows that the heat problem is no longer a "future risk" but a "present-day constraint." Google now uses liquid cooling in more than 80% of its AI servers, and NVIDIA's Vera Rubin platform assumes a fanless, fully liquid-cooled design from the outset. Cooling targets have also expanded beyond GPUs and CPUs to network interface cards, busbars, and optical transceiver modules.
| Year | AI Chip Liquid-Cooling Adoption | Main Driving Factor |
|---|---|---|
| 2023 | 6% | Early adoption phase |
| 2024 | 14% | Spread of HBM3 generation |
| 2025 | 33% | NVIDIA Blackwell generation, TDP exceeds 1 kW |
| 2026 (projected) | 53% | Vera Rubin, Google TPU fully liquid-cooled |
| 2027 (projected) | ~60% | Standardization of rack-scale liquid cooling |
(Source: TrendForce, August 17, 2026)
Questions that remain
imec's STCO research is based on simulation, and has not yet been validated on real chips. Samsung's zHBM likewise remains at the conceptual model stage, with the specific wafer-bonding technology and manufacturing process still undisclosed. CPO reduces heat overall but introduces a new constraint in the temperature sensitivity of the optical components themselves. Design automation using PINNs and reinforcement learning shows promise, but examples of its integration into actual mass-production design flows remain limited.
As Professor Kim Sung-dong put it, the industry is shifting its priorities from "performance improvement" to "thermal management." How far this transition can sustain the pace of AI infrastructure expansion will depend on how quickly the gap between simulation and real-world hardware can be closed.