From cHBM to 3D Stacked SRAM, AI inference drives AI accelerators toward memory architecture innovation

DIGITIMES identifies three main directions for memory development in cloud AI accelerators: processing near memory, 3D Stacked SRAM, and high-bandwidth flash. First, custom high-bandwidth memory (HBM) (cHBM) can serve as a practical implementation path for processing near memory (PNM) in HBM systems by shifting the logic base die from data transport to management and compute. Second, 3D Stacked SRAM helps relieve cost and area pressure on on-chip SRAM and brings high-reuse, latency-sensitive data closer to the compute core. Third, although high-bandwidth flash (HBF) offers a capacity advantage, it remains constrained by latency and power efficiency and is likely to remain in validation through 2028. Overall, competition in AI memory is shifting away from single-component specifications toward coordinated optimization of memory architecture and packaging, with the memory hierarchy becoming the key battleground.

Cloud AI accelerator development can be divided into three phases. From 2020 to 2024, training and inference architectures were used interchangeably. In 2025 and 2026, decode-oriented accelerators gradually took shape, and the industry moved from a single architecture to dual architectures for training and decoding. From 2027 onward, the industry enters a phase of memory architecture innovation. As key-value cache (KV cache) and long-context requirements expand, competition is shifting from simple compute scaling to improvements in memory bandwidth, capacity, and data movement efficiency. PNM, 3D Stacked SRAM, and HBF are emerging as the three main directions.

This report covers the current development progress, their pros and cons, as well as the remaining challenges for PNM, 3D Stacked SRAM, and HBF.