Targeted DNS Builds Cut Cloudflare's Peak Memory 85% and Halve Change Delay

The interesting number here isn't the event rate. Cloudflare's Zone Builder now handles about 700 DNS record changes per second, nearly triple the 250 it saw when the team last wrote about its propagation pipeline in 2022, plus roughly 100 other events such as zone setting changes. The real problem was what some of those events forced the system to do, and how that work scaled with zone size. Some zones now hold more than 20 million records, and Cloudflare expects them to grow toward hundreds of millions.

The fix is a shift from rebuilding whole zones to rebuilding only what a change can affect. That sounds obvious, but proving a smaller build is safe turned out to be most of the engineering. Here's how the pipeline works, what changed, and what the numbers show.

What a build actually does

Each change starts a clock that measures how long it takes to turn that change into data Cloudflare's authoritative DNS servers can serve. The unit of that work is a build. A build reads the current database state, converts it into the key format used at the edge, and writes any differences to Quicksilver, Cloudflare's distributed key-value store.

The path from a customer edit to the edge runs through several stages. A DNS record change arrives through the DNS Records API or the dashboard, or another Cloudflare service modifies a zone setting. The change lands in a database, which produces a Kafka event. The DNS Kafka Aggregator combines events from several topics and forwards them to the Zone Builder. The Zone Builder reads the zone data it needs, converts it into keys, compares those keys with what Quicksilver already holds, and writes only the differences. Quicksilver then propagates them across the network.

That description is simplified. The Zone Builder doesn't copy database rows to the edge. It merges records with zone settings, applies DNS serving rules, and arranges the output so the DNS servers can read it quickly.

In 2022, per-record builds were 150 times faster than full builds on average, and 4,250 times faster on a test zone with one million records. The obvious next step was to apply the same idea to zone settings. That turned out to be the hard part.

Why full builds stopped scaling

The original Zone Builder had one useful property: it didn't need to know why a zone changed. It read the zone and its settings, read every DNS record, converted them to keys, compared those keys with stored ones, and removed stale keys while writing changed ones.

The final write is small, since it contains only changed keys. But the full build still reads, encodes, and compares every record, and it holds the whole zone in memory while doing so. For a small zone that's cheap. For the largest zones, the data doesn't fit in memory at all.

Zone settings complicate things. Universal SSL is a good example. Toggling it updates metadata and can change Certification Authority Authorization (CAA) records at the zone apex, the bare domain such as example.com. CAA records tell certificate authorities whether they may issue certificates for that domain. So a settings change isn't always confined to one name.

A full build only needs to know the zone changed. A targeted build has to answer three harder questions: which DNS names can change, whether shared metadata or the list of published zones can change, and what previous state must be deleted from Quicksilver. Answering those safely meant solving five problems: events had to say what changed, each change had to map to the smallest correct build, builds had to account for data shared across zones with the same name, the database could change between event and build, and combining work could never drop a Kafka message.

Making events say what changed

The team changed the event producer to include changed fields and settings, with previous and new values where available. Then it traced each setting through the code that generates DNS output and classified what that setting could affect.

Not every event is that specific. Some database functions only update a timestamp, which signals that "something about this zone changed." Sometimes that's harmless bookkeeping. Sometimes it means data elsewhere changed and must be re-read. The two look identical in the event stream.

Until the caller and its purpose are identified, a timestamp-only update still triggers a full build. The same goes for missing details, unknown event types, or any change that might touch records anywhere in the zone. The planner never resolves uncertainty by doing less work. That's the design principle worth noticing: correctness is the constraint, and efficiency is only pursued inside it.

Choosing the smallest safe build

Quicksilver output has two parts that matter here. Record keys hold the DNS answers for one zone and one name. A smaller shared metadata key tells the DNS servers how to use the zone, including where to find special records such as nameservers and wildcards. Some events touch only record keys. Others must update both.

With that distinction, the planner chooses among five build types:

1. Resources only: rebuild records at specific DNS names, without touching zone metadata. Ordinary A, AAAA, TXT, and CNAME changes usually take this path.
2. Resources and metadata: rebuild specific names plus shared metadata. Nameserver and wildcard changes go here.
3. Apex and metadata: rebuild the zone apex and metadata. Settings such as Universal SSL can take this path.
4. Purge: remove a zone that should no longer be published.
5. Full zone: rebuild everything when smaller work can't be proven safe.

The governing rule is strict: every key a targeted build writes must be byte-for-byte identical to what a full build would have written. If that can't be shown, the system falls back to a full build.

Cleanup of old state works the way per-record builds already did. Record events carry deleted records and the previous version of updated ones. The Zone Builder scans Quicksilver for existing keys under each rebuilt name and removes any the new build didn't produce.

There is one deliberate exception. A full build holds every database record and every existing Quicksilver key for the zone in memory at once, and the largest zones can't fit. Above a fixed record-count threshold, the Zone Builder no longer escalates to a full build. It rebuilds the apex, the metadata, and every name the queued events identify. The team accepts this tradeoff because a full build of those zones can't finish, and while it retries it holds back every other change on its Kafka partition. For the rare cases that remain, engineers can manually rebuild individual names.

When one name maps to several zones

Quicksilver stores each zone's record keys separately. It also stores one metadata key shared by every Cloudflare zone for a given domain name. That key works like a directory, telling the DNS servers which internal zone IDs they may use when answering a query.

Why would one name have several zone IDs? During onboarding, offboarding, and other lifecycle changes, multiple zone objects can briefly exist for the same public domain. The system uses a priority rule to pick a small, bounded set to publish.

Take four internal zones for example.com: 101, 202, 303, and 404. The metadata key currently lists 101, 202, and 303. Zone 404 exists but isn't selected. These numbers identify separate Cloudflare zone objects. Customers still see one domain.

A TXT change at www.example.com in Zone 202 affects only Zone 202's record keys, so the metadata key stays put. Nameserver and wildcard changes are different, because the metadata key encodes information about them. Those builds must update the zone's records and the metadata together.

At execution time, the Zone Builder re-reads the current candidate list. If it writes the metadata key, it reconstructs the whole key from that fresh view rather than patching an older copy. It also writes the triggering zone's records before the metadata that points to them, so the DNS servers never see a reference to data that doesn't exist yet.

Planning twice

An event describes the past. By the time a build runs, more events may have arrived and the database may have changed. So planning happens in two passes. The event creates a conservative work list. Immediately before execution, the Zone Builder reads a fresh snapshot: current zone settings, whether this deployment should publish the zone to its network, and which same-name zone IDs belong in the metadata key.

Each Zone Builder deployment writes to one network, which is an isolated copy of Quicksilver. A zone can be enabled on some networks and not others, so this check matters.

Fresh state can change the plan in four ways. A targeted build can become a full build if the zone gained a feature that requires complete-zone work. It can become a purge if the zone shouldn't publish to this network anymore. It can be skipped if the triggering zone no longer belongs in the metadata key. Or a metadata build can reconstruct the shared key from the current candidate set.

The fresh read uses read-only database replicas, so builds don't compete with customer writes. Kafka delivery and database replication move on separate clocks, though, so an event can arrive before its database change is visible on a replica. The DNS Kafka Aggregator holds events long enough to cover normal lag, and a final check on the database rejects the read if the replica is still behind.

One scheduler for both kinds of work

Record work and full-zone work used to run through separate schedulers. Each could batch its own work, but neither could merge with the other. The team replaced both with one unified scheduler. The planner decides what must be built. The scheduler decides when it can run.

When several events for one zone arrive before a build starts, the scheduler combines them. An A record change at www.example.com requires that name. A wildcard change at *.example.com requires that name plus shared metadata. The scheduler merges both names and the metadata update into one build.

Merging only adds work. If any event needs the complete zone, the combined build becomes a full build. A purge overrides every rebuild, since its desired end state is that the zone is no longer published.

The scheduler tracks two things separately: the DNS work to perform and the Kafka messages waiting to be acknowledged. A retry preserves both. Even when a full build no longer needs individual record names, the messages stay attached until the result is safely written.

Keeping Kafka moving

One slow build can delay unrelated customers even when workers sit idle. Kafka divides messages into ordered partitions and stores one committed offset per partition, marking the point before which everything is complete. Advancing that offset past unfinished work could lose a customer change after a restart.

The Zone Builder uses soft committing, an approach Cloudflare first described in its secondary DNS work. A build can finish out of order and be marked complete in memory, but the committed position only advances once every earlier message has also finished. If a restart happens, in-memory completions vanish and Kafka replays the uncommitted messages. The team deliberately accepts repeated work over losing a change.

Targeted plans shorten how long the oldest unfinished message stays open. The scheduler also stops one busy zone from monopolizing workers. After a zone builds, its next build waits for a per-zone rebuild interval that varies by plan, and changes arriving in the meantime fold into that next build. The scheduler normally enforces the interval but lets long-queued work run rather than block a partition for too long. Zone locks and retry delays still apply, so a failing build can't spin in a tight loop.

Verifying before and during rollout

A smaller build only helps if it produces identical DNS data. The team added equivalence tests for nameserver, wildcard, and apex paths. Each test runs a full build and a targeted build from the same database state, pins time-dependent values, and compares every Quicksilver key the targeted plan owns, byte for byte. Planner and scheduler tests cover classification, same-name zones, merging, purges, and escalation.

During rollout, a full build also ran beside every targeted build, and the outputs were compared. That was expensive by design, so it ran only until the mirror comparison gave equivalent confidence.

Tests can't cover every customer combination, so each release also runs through a mirror Zone Builder. It consumes the production Kafka stream and reads production database state, but writes to a separate Quicksilver environment that serves no customer traffic. A comparer checks a 20% sample of zones for missing keys, extra keys, and different values.

Temporary mismatches aren't automatically regressions, because the two environments don't always build a zone at the same instant. The team looks for new kinds of differences that persist after both pipelines catch up. To group and inspect those differences, they built a DNS Mirror Dashboard on the Cloudflare Developer Platform, using a Worker to receive results, Durable Objects for state, and Cloudflare Access for protection. The diff rate, meaning the share of compared zones where production and mirror disagreed on at least one key at comparison time, is watched two ways: it has to stay low, and no new category of difference can appear.

After the mirror results held up, the scheduler rolled out by a hash of the zone name rather than the zone ID. Hashing the name keeps every zone sharing a name on the same scheduler, so two schedulers never write the same metadata key. The rollout percentage rose for free zones first, then the same process repeated for paid zones.

What the numbers show

Over 24 hours, 86.81% of builds were resources only, 2.54% were resources and metadata, 2.30% were apex and metadata, and 1.30% were purges. Together, 92.95% avoided rebuilding the complete zone. The remaining 7.04% were full builds.

The two metadata build types used to be full builds, even for large zones. Together they account for 4.84% of builds. Add that to the 7.04% still running as full builds, and the pre-change full-build share comes to 11.88%. Full builds therefore fell by about 40% by count. That understates the savings, since the avoided builds skew toward the largest zones. The full builds that remain average 0.35 seconds, mostly because the events still triggering them, like a zone's first publication, affect new zones with few records.

The unified scheduler also made the 2022 batching plan unnecessary. It doesn't wait to form batches. It merges events only when work for the same zone is already queued. Previously, long full builds held back their partition's committed offset. With fewer of them, Kafka consumption now tracks the arrival rate instead of falling behind and catching up in bursts of up to 1,550 messages per second.

Build completion delay, measured from a change until the Zone Builder finished writing to Quicksilver, tells a more nuanced story. It excludes Quicksilver's distribution across the network, which varies by location and wasn't touched by this work. Production ran the old schedulers while the mirror ran the unified scheduler on the same events over the same 24 hours. Both consumed the same Kafka events, read the same database, and wrote equivalent data, which makes this a controlled comparison. The numbers mix every build type and plan, so they aren't a per-zone latency guarantee.

Typical delays barely moved, because most builds were already small. The gain is in the tail. The largest p99 spike fell from 14.6 minutes to 1.21 minutes, roughly a 12-fold reduction. The chart uses a logarithmic scale.

Memory moved more. Aggregate working-set memory, the memory the deployment actively uses, dropped from an average of 49 GiB to 7 GiB, an 86% reduction. Peak memory fell from 79 GiB to 12 GiB, down 85%.

Revisiting two safeguards

Cheaper builds let the team revisit two conservative settings. The first is the rebuild interval, which stops one busy zone from rebuilding constantly and starving everyone else. With builds now cheaper and far fewer reading whole zones, the interval could be shortened on most plans. The second is how long the DNS Kafka Aggregator holds each event so replicas can catch up. Once the databases began rejecting reads from lagging replicas, that buffer could shrink safely.

Together these halved the median build completion delay, from about 7.7 seconds to about 3.3 seconds. The rebuild interval change rolled out gradually across servers, which is why the delay fell in steps rather than all at once.

The percentile lines in that chart need a caveat. The histogram used buckets at 1, 5, 10, and 30 seconds. A percentile falling between two bucket edges gets estimated as if delays were spread evenly across the range. Most delays cluster near the bottom of their bucket, so those estimates run high. The old p99 reads close to 30 seconds, and p90 above 7 seconds. The exact average, which doesn't depend on buckets, confirms the halving. The team has since moved to a native histogram with automatically chosen fine-grained buckets, which puts p90 at about 5.5 seconds and p99 at about 12 seconds.

Why full builds still exist

In 2022, the team said it planned to eliminate full builds entirely. That's no longer the goal. A zone's first publication needs every record written. A change that can affect records anywhere in the zone needs every record checked. An event that doesn't say what changed still gets a full build, because uncertainty never buys less work. And for a small zone, a full build is cheap anyway.

The team will keep moving event types to targeted builds as it learns more about what they change. But the biggest gain has already landed. Events that used to rebuild the largest zones no longer do, and the full builds left are mostly tied to new, small zones.

The more transferable lesson may be the planning discipline. Making each build as small as possible is easy to state and hard to prove. Cloudflare's answer was to make the planner default to safety, verify output byte for byte, and let a mirror pipeline catch what tests miss. Teams facing similar scaling pressure in their own config pipelines will likely find that proving equivalence takes longer than writing the optimization. Here, that proof is what made the 85% memory drop and the halved delay safe to ship.