Cloudflare faces issues and outages,many websites affected including the forums and LTTstore.com,Cloudflare implement a fix & explains what went wrong

Vishera · June 21, 2022

Summary

Cloudflare faces issues and outages,many websites affected including the forums and LTTstore.com,Cloudflare released a fix & explains what went wrong

Quotes

Quote

A brief outage at Cloudflare-a popular CDN choice for many companies, took down multiple websites across the globe earlier today, many of which were left completely inaccessible to the general public. These included websites like Discord, Canva, Streamyard and even the official website for London-based startup Nothing.

Cloudflare quickly acknowledged the outage via a tweet, and announced soon after that it was working on a fix. The issue now seems to have been resolved for most websites affected by the issue.

What happened:

Quote

What had happened was a change to the company's prefix advertisement policies, resulting in the withdrawal of a critical subset of prefixes.

Cloudflare makes use of BGP (Border Gateway Protocol).

As part of this protocol, operators define which policies (adjacent IP addresses) are advertised to or accepted from networks (or peers).

Changing a policy can result in IP addresses no longer being reachable on the Internet.

One would therefore hope that extreme caution would be taken before doing a such a thing...

Cloudflare's mistakes actually began at 0356 UTC (2056 Pacific), when the change was made at the first location.

There was no problem - the location used an older architecture rather than Cloudflare's new "more flexible and resilient" version,

known internally as MCP (Multi-Colo Pop.) MCP differed from what had gone before by adding a layer of routing to create a mesh of connections.

The theory went that bits and pieces of the internal network could be disabled for maintenance.

Cloudflare has already rolled out MCP to 19 of its datacenters.

Moving forward to 0617 UTC (2317 Pacific) and the change was deployed to one of the company's busiest locations, but not an MCP-enabled one.

Things still seemed OK... However, by 0627 UTC (2327 Pacific),

the change hit the MCP-enabled locations, rattled through the mesh layer and... took out all 19 locations.