Cloudflare's CTO apologizes after error takes huge chunk of the internet offline — 'we failed our customers and the broader internet'

cloiudflare
(Image credit: Getty / Smith Collection/Gado)

Cloudflare has confirmed that a bug in one of its core services caused a major outage on Tuesday, taking large portions of the internet offline and affecting traffic to services including X, ChatGPT, and, ironically, Downdetector. The company’s CTO, Dane Knecht, posted a public apology shortly after services were restored, calling the incident "unacceptable" and attributing the disruption to a routine configuration change that triggered a crash in its bot mitigation layer.

The incident began at approximately 11:48 UTC on November 18, with Cloudflare's official status site acknowledging “internal service degradation”. As the issue spread, users across several regions reported failures to access not only Cloudflare-backed websites but also its Access and WARP services. The company later identified a specific dependency in its bot defense tooling as the source of the problem.

Google Preferred Source

Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.

Luke James
Contributor

Luke James is a freelance writer and journalist.  Although his background is in legal, he has a personal interest in all things tech, especially hardware and microelectronics, and anything regulatory. 

  • S58_is_the_goat
    Anybody thought about doing config changes on the weekend maybe? 😂
    Reply
  • thisisaname
    They just demonstrated that relying on just them is a very bad idea, they just dos'ed their customers.
    Reply
  • bit_user
    Didn't we just learn this lesson in the massive CrowdStrike outage that hit their customers, last year?
    https://www.tomshardware.com/software/windows/global-it-issue-strikes-windows-machines-cause-of-issue-allegedly-linked-to-crowdstrike-software-update
    Always test your changes in a sandbox, before deploying on the production network.
    Reply
  • derekullo
    bit_user said:
    Didn't we just learn this lesson in the massive CrowdStrike outage that hit their customers, last year?
    https://www.tomshardware.com/software/windows/global-it-issue-strikes-windows-machines-cause-of-issue-allegedly-linked-to-crowdstrike-software-update
    Always test your changes in a sandbox, before deploying on the production network.
    https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcR2CN0r--Ly3V0W40xRVkyZLje6irddck36zA&s
    Reply
  • tamalero
    S58_is_the_goat said:
    Anybody thought about doing config changes on the weekend maybe? 😂
    My bet is they are really trying to sweep code changes programmed by AI.. by using QA powered by AI..
    Hence why everything fails XD
    Reply
  • bill001g
    Th
    bit_user said:
    Didn't we just learn this lesson in the massive CrowdStrike outage that hit their customers, last year?
    Nope they learn nothing. When the worst that happens is the company AI bot says " i am so sorry" . This is not like a regular company that the end users could boycott.
    Even if they had to pay some fine to the government that would also make no difference because the employees who are not doing their job do not have to pay the fine it comes out of any profit that the shareholders have.

    There also is not likely a way to find a single person who is responsible. They have their massive committees and they can all point fingers at each other say it wasn't me.

    I bet pretty much everyone has gotten a letter saying your information was access and we will give you $.05 and a year of credit reporting. People seem numb to the incompetence in these large companies and the lack of any responsibility by employees to do what they are getting paid for.
    Reply
  • JRStern
    Sounds like they identified and mitigated it pretty quick.
    Reply
  • thisisaname
    bill001g said:
    Th

    Nope they learn nothing. When the worst that happens is the company AI bot says " i am so sorry" . This is not like a regular company that the end users could boycott.
    Even if they had to pay some fine to the government that would also make no difference because the employees who are not doing their job do not have to pay the fine it comes out of any profit that the shareholders have.

    There also is not likely a way to find a single person who is responsible. They have their massive committees and they can all point fingers at each other say it wasn't me.

    I bet pretty much everyone has gotten a letter saying your information was access and we will give you $.05 and a year of credit reporting. People seem numb to the incompetence in these large companies and the lack of any responsibility by employees to do what they are getting paid for.
    Yes but the. will say "next time lessons will be learnt" except they will not be.
    Reply