Agent workloads are reshaping AI costs, and the data layer is where they land

Most AI cost debates still start with the model: which one to pick, what it charges per million tokens, and whether a smaller one will do. That framing made sense when AI mostly meant a chatbot. An agent is a different workload. One task can fire dozens of queries, pull context from several systems, call a model multiple times and write results back. The bill and the response time are increasingly set by the plumbing around the model, not the model alone.

Agents are a different kind of load

Tim Finley, senior vice president of AI at Couchbase, framed the problem with queuing theory. As load rises, latency grows faster than traffic, and agents push load up quickly. He cited a Cisco study finding that agents generate 450% more traffic per task than humans doing the same work. Cloudflare data showed machine traffic passed human traffic on the web in June, about 18 months earlier than forecast.

"Capacity plans built for human growth curves are likely to break," Finley said, calling agents "probably the nastiest, most demanding user that you can ever imagine" for a system. That's the risk I'd flag first. Production databases were sized for people who click, wait and click again. Agents fan out into parallel tasks, need current and historical data in the same loop, and never sleep. Treat them as just more users and the data tier becomes the bottleneck, often right after a pilot succeeds and the business asks to scale it.

The model downgrade trap

Finley's sharpest comment was about cost: "Customers are actively sacrificing intelligence by downgrading models because of the bill once they hit scale." The pattern is familiar. A pilot runs on a frontier model, the results look strong, finance sees the inference bill at production volume, and the team swaps in a cheaper model that weakens the very results that justified the project.

Much of that spend is often waste: the same questions sent repeatedly, bloated prompts, and agents that rebuild context from scratch at every session. Cutting that waste is usually a better lever than cutting model quality.

Couchbase's pitch targets exactly that. Its semantic caching serves similar queries without calling the model again, and the company says it cuts large language model calls by 40% to 60%. It also claims Agent Memory reduces token usage by 33% to 45% compared with alternatives on the LoCoMo long-term conversational memory benchmark, and that vector-based context retrieval produces prompts about 70% smaller than full-context approaches. These are the company's own figures, and savings will track how repetitive a workload really is. A support bot answering the same forty questions will benefit far more than an agent doing novel research. Still, the direction holds: most organizations have more room to cut token waste than to squeeze out quality gains.

Convergence versus consolidation

Finley drew a line between consolidation and convergence. Consolidation reduces the number of tools you pay for. Convergence means every capability sits on a single copy of the data. In his words, "a converged system is greater than the sum of its parts."

His example was a phone maker that wanted to know which factories link to the most positive battery-life reviews. The usual stack involves a relational database, a key-value store, a vector database, an ETL pipeline and a warehouse. Couchbase says its vector search in Operational Insights answers that question in one SQL query, using a "cluster by" clause much like "group by," then writes the result back so agents can act on it. "There has been a fallacy for a long time with analytics that a really great insight was the finish line," Finley said. "We think it's the action."

Each hop between systems adds latency, creates a sync job that can fail, adds another access model and opens another place for data to go stale. An agent working from stale data can be confidently wrong. Fewer moving parts matter for accuracy and speed as much as they do for the invoice.

Postgres with pgvector is the default for many AI teams because it's free and familiar, and Snowflake and Databricks have both bet on Postgres. Finley, who previously worked in AWS's database business, didn't dismiss that appeal. His argument is that it strains at Fortune 2000 scale once vector counts reach the hundreds of millions. Couchbase points to deployments at LinkedIn, Amadeus and telecom software provider Mavenir. Brandon Larson, senior vice president and general manager of Cloud, AI & IMS Business Strategy, said Couchbase "has been a key component in achieving high availability in Mavenir's voice solutions for years," and that Mavenir is now building its AI voice context layer on Couchbase because of its production track record.

What to do before the next scale-up

Several of these points translate into practice regardless of vendor.

Model agent load, not user load. Rebuild capacity plans around parallel fan-out, many queries per task and sustained bursts. Load-test the data tier with simulated agent traffic before scaling a pilot. If latency climbs faster than request volume, the queuing effect is already at work.

Measure cost per task before changing models. Count near-duplicate prompts, check how much context gets resent each turn, and track how often agents rebuild state they already have. Caching and memory should come first. A cheaper model should be a quality decision, not a budget reflex.

Count the hops. Map every system and sync pipeline an agent touches on a typical task. Each hop is a latency cost, a failure point and a governance gap.

Treat caches and memory as production data. These stores hold customer context and model outputs, so retention rules, per-user isolation, access controls and audit trails belong in place from day one. A cache that returns a "similar" answer to a genuinely different question is a correctness problem, so tune and monitor similarity thresholds.

Test at your own scale. Vendor benchmarks are a starting point. Check recall and throughput at your data volumes, concurrency and vector counts. A setup that performs well at 10 million vectors may behave differently at a billion.

Keep the memory layer portable. Agent frameworks are changing fast. Pick memory services that work across LangGraph, CrewAI, LlamaIndex and others, so a framework switch doesn't mean rebuilding the most valuable part of the stack.

Where this leaves the stack

For two years the AI conversation has centered on models. The next stretch looks more likely to center on what it costs, and how reliably agents run, at scale. Much of that gets decided in the data layer. Couchbase is right that agents are a fundamentally different kind of user, and its token-economics framing matches the discussion many CIOs are now having with CFOs. The harder challenge is architectural. Teams already know the Postgres-plus-point-tools stack, and convincing them to converge their data is a bigger sell than any single benchmark. My read is that the argument will be won by whoever can show production numbers at the scale where the Postgres approach starts to strain.