Cloudflare Wraps Up ‘Fail Small’ Initiative, Unveils Snapstone to Prevent Global Outages

Cloudflare has completed a major engineering overhaul, code-named Code Orange: Fail Small, aimed at eliminating the root causes of two catastrophic global outages that occurred in late 2025. The project, finished earlier this month, introduces a new configuration deployment system called Snapstone that the company says will prevent similar failures from affecting customer traffic.

“This work goes beyond fixing past issues — it fundamentally changes how we roll out changes to our network,” said John Graham-Cumming, Cloudflare’s CTO. “Snapstone ensures that no single configuration change can take down the global network again.”

Background

In November and December 2025, Cloudflare experienced two separate global outages that disrupted services for millions of users. The November 18 outage was triggered by a problematic data file; the December 5 outage stemmed from a misconfigured control flag in the company’s global configuration system. Both incidents exposed weaknesses in how configuration changes were deployed across Cloudflare’s vast network.

In response, the engineering team initiated Code Orange: Fail Small, a multi-quarter effort to harden infrastructure. The project focused on four key areas: safer configuration changes, reducing the blast radius of failures, revising emergency “break glass” procedures, and improving incident communication with customers.

What This Means

For Cloudflare customers, the most visible change is that internal configuration no longer reaches the network instantly. Instead, changes are rolled out progressively with real-time health monitoring. The company’s observability tools can now automatically detect problems and revert changes before traffic is impacted.

“In the past, a single bad config could cascade across our entire network in seconds,” said Manuela Reinhart, a senior reliability engineer at Cloudflare. “Now, with Snapstone, even high-risk changes are staged and monitored — we catch issues early, often before anyone outside knows.”

Snapstone: A New Layer of Safety

Snapstone is an internal system that bundles configuration changes into packages and releases them gradually, applying the same health-mediated deployment methods used for software updates. Previously, each team had to build its own checks; now Snapstone provides a unified, automated rollback mechanism for all configuration changes.

The system is flexible: teams can define any configuration unit — whether a data file, a control flag, or a routing table entry — that needs health mediation. This means the learnings from both the November and December outages are directly baked into the deployment pipeline.

Revised Procedures and Communication

Alongside Snapstone, Cloudflare has overhauled its break glass procedures and incident management playbooks. The new protocols require more rigorous peer review and automated sanity checks before any emergency access is granted to production systems. Customer communication during outages has also been strengthened, with clearer timelines and status updates.

“We know that trust is earned in the quiet moments, not just during incidents,” said Margaret Lee, Cloudflare’s VP of Customer Trust. “Our goal is to reduce both the frequency and impact of failures — and when they do happen, we’ll be more transparent about what went wrong and what we’re doing to fix it.”

What This Means for Your Business

The immediate takeaway: Cloudflare’s network is now more resilient against the kind of configuration errors that caused hours-long outages last year. Customers should expect fewer unplanned downtime events and faster recoveries when issues do arise.

For enterprises relying on Cloudflare for critical traffic, the Snapstone deployment model adds a layer of predictability. Changes that once posed a risk of cascading failure are now contained and reversible. As a result, businesses can feel more confident in using Cloudflare for mission-critical applications, including those requiring high availability SLAs.

“Fail Small isn’t just a project name — it’s a philosophy,” Graham-Cumming added. “We want every failure to be contained, understood, and fixed fast. This is how we get there.”