The scale of the shift is hard to overstate. Google was processing 9.7 trillion tokens a month in 2024; by mid-2025 that had risen to 480 trillion, and at this year’s I/O, Sundar Pichai put the figure at 3.2 quadrillion tokens a month. Goldman Sachs projects industry-wide token consumption will multiply another 24 times by 2030. Whatever an organization budgeted for inference last year, the volume assumption behind it is already out of date, and a growing share of that volume is input, not output: agentic workflows that plan, call tools and re-read context on every step are pushing input tokens, not generated ones, to the top of the cost stack.
Your AI budget has a serious NAND problem
The clearest sign of where this is heading came at CES 2026, when Nvidia unveiled what it first called its Inference Context Memory Storage Platform, built around its coming BlueField-4 DPU, and has since formalized as the NVIDIA CMX context memory storage platform. CMX standardizes the offload of the model’s working memory, the KV cache, onto NVMe SSDs, extending GPU memory into a shared, pod-level context tier and making it persistent across inference runs. Nvidia says the approach can deliver up to five times greater power efficiency and up to five times higher tokens-per-second throughput, and it already has more than a dozen storage partners lined up to support it, including Dell, IBM, NetApp, VAST and WEKA.
That is a significant validation point: several storage vendors were already extending GPU memory onto flash before ICMSP existed. Nvidia standardizing the approach across its ecosystem confirms this is now an architectural direction for the industry, not a niche workaround. But it also means the economics of inference are no longer determined solely by GPU pricing; they are now directly exposed to storage media pricing.
Leave a Reply