Storage Is Becoming the Limit on Enterprise AI Accelerator Output

Most enterprise AI budgets over the past two years went toward one question: how to get enough compute. Teams chased GPU allocations, negotiated infrastructure deals and stood up machine learning platforms as fast as they could. Many of them are now finding that compute was only one constraint. Storage is throttling the accelerators they worked so hard to secure, and it's showing up in utilization numbers rather than in architecture reviews.

Ronen Schwartz, vice president of products for Google Cloud Storage, recently walked through why storage has become a defining limit on enterprise AI infrastructure and what purpose-built storage for AI workloads looks like in practice. His framing is blunt. "Storage just got promoted," he said. "Instead of being at the backend as an enabler, it is moving into the front as a critical part of accelerator execution."

Idle accelerators are the expensive part

The economics are simple. Accelerators are typically an enterprise's most expensive line item, so every second they sit idle waiting for data is money spent for nothing. That waste is easy to measure, which is part of why it's starting to get attention. Schwartz pointed to a pattern he hears from customers: teams spend months fighting for TPU and GPU capacity, then find the output they need doesn't arrive. "Very often it actually goes back to not enough thought put into the data delivery path," he said.

Many organizations only identify the storage problem once it's already hurting them. GPU utilization falls below targets, training runs stretch past their schedules, and the cause turns out to be the data path. By then the cost of underinvesting in storage architecture is baked into the project, and fixing it means rework rather than a design decision.

Training, inference and agents move data differently

One of the more useful points in the conversation is that "AI" isn't a single workload. Schwartz described training, inference and agentic systems as having different data movement patterns, or what he called different "heartbeats." Treating them as one storage problem compounds gaps over time.

Training is dominated by large sequential reads, but checkpointing is where the pain often shows up. A training cluster periodically pauses to write a multi-terabyte checkpoint so that work isn't lost if something fails. Schwartz's rough cadence was about every 15 minutes. "Slow storage in that place freezes the entire GPU cluster," he said. Google says its Rapid storage can write checkpoints more than three times faster and restore them more than five times faster than standard object storage. At scale, those differences add up to real idle time.

He cited Torin, an autonomous vehicle company that ingests millions of sensor data points to train vehicle models. For a workload like that, checkpoint performance isn't an edge case. It's part of normal operations.

Inference brings a different constraint: latency. Response time affects user experience directly and shapes token economics, so storage that adds delay to retrieval or model loading shows up in the product. Agentic workloads add a third set of requirements, including shared workspaces, state that persists across agents, low-latency retrieval and support for both short-lived and long-running processes that may hibernate and wake up on demand.

Performance claims and the zonal design choice

Google's case rests on the claim that its storage stack has been running hyperscale AI workloads internally for longer than most providers have offered comparable services externally. Schwartz cited throughput figures of 15 terabytes per second on Rapid object storage and 10 terabytes per second on Google Cloud managed clusters, which he described as verified, industry-leading numbers. Those are vendor-reported figures, and they're worth checking against your own workload before they drive a purchase. What matters more is the design choice behind them.

The central idea is zonal colocation: placing data physically close to GPU and TPU clusters to shorten the distance a byte travels before it reaches an accelerator. Schwartz called this a first-to-market capability that required rethinking defaults in storage design. Traditional storage was built around multi-zone or multi-region distribution for durability and availability. For AI training, he argued, proximity to the accelerator should take priority. Google's Rapid Bucket technology reduces processing steps in the storage path, and Google says a single bucket can deliver more than 15 terabytes per second of throughput, support 20 million queries per second and maintain sub-millisecond latency.

Schwartz also argued that Google's position comes from its customer base as much as its internal experience. He said the company is working with nine of the ten leading AI frontier model companies, which gives it a large sample of demanding training and serving patterns. Whether that translates to your environment depends on your workloads, but the underlying logic is sound: a storage layer that keeps GPUs busy means faster iteration and a lower effective cost per training run.

Agents need to read unstructured data

The agent discussion surfaced a gap that many enterprise readiness plans miss. Agents need more than web browsing and database queries. Most organizations' real knowledge sits in images, video, documents and PDFs, and agents can't use that material well if it's opaque to them. Without a way to make that content semantically accessible, an agent works from a narrow and often misleading picture of what the company knows.

Google's answer is what Schwartz called a Smart Storage Initiative, anchored by a recently launched Object Enrichment capability. When a file or image lands in a bucket, Google's AI models extract its content and attach structured metadata to the object. Schwartz said this happens in seconds and at scale. The practical effect is that content which used to need manual tagging or a separate processing pipeline becomes queryable. "An agent can now interact with what used to be called unstructured data because there is actually a very good understanding of the content," he said.

The storage layer is also tied into Google's agentic data cloud and borderless lakehouse architecture, so agents can reach data kept outside Google's environment. For agent infrastructure at scale, Schwartz described dedicated Filestore agent volumes integrated with Google Kubernetes Engine and an agent sandbox environment. The stated goal is to handle the cost, response speed and data-sharing needs of running thousands or millions of agents at once.

Where this leaves platform teams

The bigger shift here is architectural. Enterprise AI stacks were usually assembled in layers: compute first, then frameworks, then data. Storage was often bought as a checkbox rather than managed as a performance variable. That sequencing is getting expensive. As GPU utilization becomes a measurable KPI and AI operations budgets face tighter scrutiny, storage is being reexamined in a way it hasn't been in years.

For platform engineers and ML infrastructure teams, the practical takeaway is that storage decisions made during the initial build will either constrain or enable performance for every workload that follows. Revisiting them after a bottleneck appears costs far more than designing for them up front. A few questions are worth asking now: how long do checkpoints take to write and restore relative to your step time, how far is your data from your accelerators, and what happens to agent state when a process hibernates?

Schwartz's argument is that hyperscale experience at the storage layer shows up as measurable differences in checkpoint speed, throughput, latency and agent responsiveness. The teams running frontier training and inference have already been forced to solve these problems. Enterprise teams are now meeting them for the first time, and the ones that treat storage as a primary tier rather than an afterthought are likely to get more out of the accelerators they've already paid for.

To learn more about how Google Cloud Storage is positioned for enterprise AI workloads, see the company's product documentation and compare the performance figures against your own training and inference profiles.