Micron Unveils First-Ever 256GB LPDDR5X SOCAMM2 For Huge Memory Capacity Gains

We've already reported a few times on SOCAMM2 memory, so if you're not familiar, check out one or all of these stories. The last of those links goes to a story about Micron sampling 192GB SOCAMM2 modules. Well, as you've already read in the headline, the US memory firm is now able to make 256GB SOCAMM2 modules thanks to new 32-Gigabit LPDDR5 chips, which Micron says it is the first to bring to market.

If you thought, "Zak, 32-gigabit is way smaller than 256 gigabytes," you're right! The LPDDR5X chips on the SOCAMM2 modules stack sixteen of these dies in a single package, meaning each of the four LPDRAMs on the SOCAMM2 gumstick is packing in 64GB of high-speed RAM. Multiply by four and you get the 256GB capacity we noted in the headline, the densest memory modules to date in terms of both logical capacity and physical volume.


This is a big deal because AI workloads will gladly use all of the RAM you can throw at them. These new modules enable a system with 8-channel memory to support up to 2 terabytes of total RAM per CPU. More impressively, Micron claims that the SOCAMM2 modules use just one-third the power of traditional registered memory (RDIMMs), which is a big help considering that massive mountains of memory modules end up being a major contributor to the power consumption of the mighty rack servers used for AI training and inference.


Micron makes some bold claims about the benefits of adding more memory to your AI servers. Specifically, the company claims that you can slash your time to first token (at a frankly absurd 1 million tokens context length) to just one second with 2TB of RAM. The company also says moving from 192GB modules to 256GB moduels can double performance in Spark SVM data analytics, and offer a 2.3x improvement in time to first token at 500K context length versus the 1.5TB configuration with 192GB modules. It wouldn't surprise us if many operators with systems in place will actually strip the current memory and install the new RAM for the extra capacity.

SOCAMM2 modules are already in use in some systems, like NVIDIA's Blackwell Ultra as well as the desktop workstations based on Grace Blackwell superchips like Dell's Pro Max GB300. It would be nice to see them come to the desktop as well, as they offer improved transfer rates compared to standard DIMMs, too. Standardizing the whole industry on a single type of module would improve logistics for everyone, but with the way things are going, maybe we should be breathing a sigh of relief that standard desktops use a different type of memory.