Micron Warns: AI 'Memory Wall' Is Intensifying, HBM Becomes Key Bottleneck

According to foreign media reports, Raghu Sriramaneni, an HBM design architecture researcher at Micron, pointed out in a recent presentation that AI systems are currently facing an increasingly severe structural contradiction:

AI compute performance: roughly triples every two years

HBM bandwidth growth: grows by less than double over the same period

The widening gap between compute and memory bandwidth growth means the "memory wall" is not easing — it may in fact intensify further.

Why Memory Becomes a Bottleneck for AI

AI accelerators (such as GPUs) need to continuously fetch data from memory during computation. No matter how powerful the processor is, once the data supply speed fails to keep up with computational demand, the compute units are forced to idle and wait, significantly degrading overall performance.

HBM (High Bandwidth Memory) addresses this by vertically stacking multiple DRAM chips and placing them in close proximity to the GPU, delivering much wider bandwidth than conventional DRAM. It has become the preferred memory solution for current AI accelerators. However, as AI model sizes continue to grow, HBM bandwidth growth is gradually falling behind the pace of compute scaling, bringing the memory bottleneck back into focus.

This is not merely a theoretical concern. Micron cited data from Meta's Llama 3 training paper, which showed that 17% of unexpected interruptions during training were caused by HBM-related issues — clearly demonstrating that memory reliability has become a practical obstacle to large-scale AI training.

Technical Challenge: Higher Stack Counts Bring Thermal Challenges

HBM stack counts are steadily increasing: from 4 layers in the early days, to the current mainstream of 12–16 layers, with a future direction of 20 layers and beyond. More layers mean greater bandwidth and capacity, but also a sharp rise in power density, making thermal management significantly more difficult. This has become a core engineering challenge for further HBM scaling.

Path Forward: Disruptive Process and Packaging Technologies

Micron emphasized that breaking through these bottlenecks requires the industry to adopt disruptive process and packaging technologies. Key approaches include:

Liquid cooling: to address the thermal pressure from high power density

Hybrid bonding: to increase chip-to-chip interconnect density and reduce thermal resistance

Next-generation advanced packaging: to optimize memory-compute协同 efficiency at the system level