Insufficient mock data in the staging environment? Like no BYOIP prefixes at all? Since even one prefix should have shown that it would be deleted by that subtask...
From all the recent outages, it sounds like Cloudflare is barely tested at all. Maybe they have lots of unit tests etc, but they do not seem to test their whole system... I get that their whole setup is vast, but even testing that subtask manually would have surfaced the bug
dabinat•about 2 hours ago
I think Cloudflare does not sufficiently test lesser-used options. I lurk in the R2 Discord and a lot of users seem to have problems with custom domains.
martinald•about 1 hour ago
Just crazy. Why does a staging environment matter? They should be running some integration tests against eg an in memory database for these kinds of tasks surely?
asciii•about 2 hours ago
It was also merged 15 days prior to production release...however, you're spot on with the empty test. That's a basic scenario that if it returned all...is like oh no.
otar•about 1 hour ago
Reliability was/is CF's label.
It's alarming already. Too many outages in the past months. CF should fix it, or it becomes unacceptable and people will leave the platform.
I really hope they will figure things out.
argestes•about 1 hour ago
I have many things dependent on Cloudflare. That makes me root for Cloudflare and I think I'm not the only one. Instead of finding better options we're getting stuck on an already failing HA solution. I wonder what caused this.
slothsarecool•44 minutes ago
There are no alternatives, and those alternatives that did exist back in the day, had to shut down due to either going out of business or not being able to keep a paygo model.
Not everybody needs cloudflare, but those that need it and aren't major enterprises, have no other option.
pocksuppet•36 minutes ago
Lots of people who think they need Cloudflare don't. What are you using it for?
Sanzig•36 minutes ago
Bunny.net? Doesn't have near the same feature set as Cloudflare, but the essentials are there and you can easily pay as you go with a credit card.
arcatech•about 1 hour ago
Do you not feel concern about you and everybody else deciding to put ALL of their eggs into one basket like this?
alansaber•about 1 hour ago
Not sure why everyone is complaining, new MCP features are more important than uptime
atty•about 2 hours ago
I do not work in the space at all, but it seems like Cloudflare has been having more network disruptions lately than they used to. To anyone who deals with this sort of thing, is that just recency bias?
Icathian•about 2 hours ago
It is not. They went about 5 years without one of these, and had a handful over the last 6 months. They're really going to need to figure out what's going wrong and clean up shop.
NinjaTrance•about 2 hours ago
Engineers have been vibe coding a lot recently...
jsheard•about 2 hours ago
The featured blog post where one of their senior engineering PMs presented an allegedly "production grade" Matrix implementation, in which authentication was stubbed out as a TODO, says it all really. I'm glad a quarter of the internet is in such responsible hands.
dakiol•about 2 hours ago
No joke. In my company we "sabotaged" the AI initiative led by the CTO. We used LLMs to deliver features as requested by the CTO, but we introduced a couple of bugs here and there (intentionally). As a result, the quarter ended up with more time allocated to fix bugs and tons of customer claims. The CTO is now undoing his initiative. We all have now some time more to keep our jobs.
Ylpertnodi•about 2 hours ago
Typo: "shop", should have been with an 'el'.
(: phonetically, because 'l's are hard to read.
dazc•about 2 hours ago
Launching a new service every 5 minutes is obviously stretching their resources.
lysace•about 2 hours ago
It has been roughly speaking five and a half years since the IPO. The original CTO (John Graham-Cumming) left about a year ago.
jacquesm•about 2 hours ago
They coasted on momentum for half a year. I don't even think it says anything negative about the current CTO, but more of what an exception JGC is relative to what is normal. A CTO leaving would never show up the next day in the stats, the position is strategic after all. But you'd expect to see the effect after a while, 6 months is longer than I would have expected, but short enough that cause and effect are undeniable.
Even so, it is a strong reminder not to rely on any one vendor for critical stuff, in case that wasn't clear enough yet.
dazc•about 2 hours ago
I wondered what happened to him?
jgrahamc•about 1 hour ago
I am reading HN.
brcmthrowaway•about 2 hours ago
He's on a yacht somewhere
slophater•about 2 hours ago
been at cf for 7 yrs but thinking of gtfo soon. the ceo is a manchild, new cto is an idiot, rest of leadership was replaced by yes-men, and the push for AI-first is being a disaster. c levels pretend they care about reliability but pressure teams to constantly ship, cto vibe codes terraform changes without warning anyone, and it's overall a bigger and bigger mess
even the blog, that used to be a respected source of technical content, has morphed into a garbage fire of slop and vaporware announcements since jgc left.
goalieca•about 1 hour ago
I’ve had a lot of problems lately. Basic things are failing and it’s like product isn’t involved at all in the dash. What’s worse? The support.. the chat is the buggiest thing I’ve ever seen.
__turbobrew__•about 1 hour ago
You know what they say, shit rolls downhill. I don't personally know the CEO, but the feeling I have got from their public fits on social media doesn't instill confidence.
If I was a CF customer I would be migrating off now.
a24446ff87•about 1 hour ago
GSD! GSD!! ship! ship! ship!
**everything breaks**
...
**everything breaks again**
oh fuck! Code Orange! I repeat, Code Orange! we need to rebuild trust(R)(TM)! we've let our customers down!
...
**everything breaks again**
Code Orangier! I repeat, Code Orangier!
slophater•about 1 hour ago
amazing how my comment was flagged in 30 seconds... keep bootlicking
Betelbuddy•about 2 hours ago
Cloudflare Outages are as predictable, as the Sun coming up tomorrow. Its their engineering culture.
> Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed. The system interpreted this as all returned prefixes being queued for deletion.
if v := req.URL.Query().Get("pending_delete"); v != "" {
// ignore other behavior and fetch pending objects from the ip_prefixes_deleted table
prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx)
if err != nil {
api.RenderError(ctx, w, ErrInternalError)
return
}
api.Render(ctx, w, http.StatusOK, renderIPPrefixAPIResponse(prefixes, nil))
return
}
even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that block
bretthoerner•about 2 hours ago
> even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that block
If they passed in any value, they would have entered the block and returned early with the results of FetchPrefixesPendingDeletion.
From the post:
> this was implemented as part of a regularly running sub-task that checks for BYOIP prefixes that should be removed, and then removes them.
They expected to drop into the block of code above, but since they didn't, they returned all routes.
blibble•about 2 hours ago
okay so the code which returned everything isn't there
actual explanation: the API server by default returns everything. the client attempted to make a request to return "pending_deletes", but as the request was malformed, the API instead went down the default path, which returned everything. then the client deleted everything.
makes sense now
but is that explanation is even worse
because that means the code path was never tested?
jbxntuehineoh•about 1 hour ago
or they tested it, but not with a dataset that contained prefixes not pending deletion
bstsb•about 2 hours ago
doesn't look AI-generated. even if they have made a mistake, it's probably just from the rush of getting a postmortem out prior to root cause analysis
himata4113•about 2 hours ago
yep, no mention that re-advertised prefixes would be withdrawn again as well during the entire impact even after they shut it down.
dilyevsky•about 1 hour ago
> Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed.
Lmao, iirc long time ago Google's internal system had the same exact bug (treating empty as "all" in the delete call) that took down all their edges. Surprisingly there was little impact as traffic just routed through the next set of proxies.
jaboostin•about 2 hours ago
Hindsight is 20/20 but why not dry run this change in production and monitor the logs/metrics before enabling it? Seems prudent for any new “delete something in prod” change.
boarush•about 2 hours ago
While neither am I nor the company I work for directly impacted by this outage, I wonder how long can Cloudflare take these hits and keep apologizing for it. Truly appreciate them being transparent about it, but businesses care more about SLAs and uptime than the incident report.
llama052•about 2 hours ago
I’ll take clarity and actual RCAs than Microsoft’s approach of not notifying customers and keeping their status page green until enough people notice.
One thing I do appreciate about cloudflare is their actual use of their status page. That’s not to say these outages are okay. They aren’t. However I’m pretty confident in saying that a lot of providers would have a big paper trail of outages if they were more honest to the same degree or more so than cloudflare. At least from what I’ve noticed, especially this year.
boarush•about 2 hours ago
Azure straight up refuses to show me if there's even an incident even if I can literally not access shit.
But last few months has been quite rough for Cloudflare, and a few outages on their Workers platform that didn't quite make the headlines too. Can't wait for Code Orange to get to production.
jacquesm•about 2 hours ago
Bluntly: they expended that credit a while ago. Those that can will move on. Those that can't have a real problem.
As for your last sentence:
Businesses really do care about the incident reports because they give good insight into whether they can trust the company going forward. Full transparency and a clear path to non-repetition due to process or software changes are called for. You be the judge of whether or not you think that standard has been met.
boarush•about 2 hours ago
I might be looking at it differently, but aren't decisions over a certain provider of service being made by the management. Incident reports don't ever reach there in my experience.
samrus•about 1 hour ago
In my experience, the gist of it does reach management when its an existing vendor. Especially if management is tech literate
Becuase management wants to know why the graphs all went to zero, and the engineers have nothing else to do but relay the incident report.
This builds a perception for management of the vendor, and if the perception is that the vendor doesnt tell them shit or doesnt even seem to know theres an outage, then management can decide to shift vendors
anurag•about 2 hours ago
The one redeeming feature of this failure is staged rollouts. As someone advertising routes through CF, we were quite happy to be spared from the initial 25%.
NinjaTrance•about 2 hours ago
The irony is that the outage was caused by a change from the "Code Orange: Fail Small initiative".
They definitely failed big this time.
himata4113•about 2 hours ago
This blog post is inaccurate, the prefixes were being revoked over and over - to keep your prefixes advertised you had to have a script that would readd them or else it would be withdrawn again. The way they seemed to word it is really dishonest.
Advertisement
ssiddharth•about 2 hours ago
The eternal tech outage aphorism: It's always DNS, except for when it's BGP.
tokyobreakfast•about 2 hours ago
Is this trend of oversharing code snippets and TMI postmortems done purposely to distract their customers from raging over the outage and the next impending fuckup?
samrus•about 1 hour ago
Just seems like transparency. I agree that we should also judge them based on the frequency of these incidents and amwhether they provide a path to non-repeatability, but i wouldnt criticize them for the transparency per se
alansaber•about 2 hours ago
Well I still appreciate a good postmortem even if I have no doubt it'll happen again imminently
bdangubic•about 2 hours ago
and if they didn’t we’d posting about lack of transparency. damned if you do, damned if you don’t
NooneAtAll3•about 2 hours ago
again?
VirusNewbie•about 2 hours ago
If you track large SaaS and Cloud uptime, it seem to correlate pretty highly with compensation for big companies. Is cloudflare getting top talent?
bombcar•about 2 hours ago
Based on IPO date and lockups, I suspect top talent is moving on.
henning•about 2 hours ago
Sure vibe-coded slop that has not been properly peer reviewed or tested prior to deployment is leading to major outages, but the point is they are producing lots of code. More code is good, that means you are a good programmer. Reading code would just slow things down.
sp00chy•about 1 hour ago
that’s my feeling also. We will get this more and more in future.
Discussion (53 Comments)
From all the recent outages, it sounds like Cloudflare is barely tested at all. Maybe they have lots of unit tests etc, but they do not seem to test their whole system... I get that their whole setup is vast, but even testing that subtask manually would have surfaced the bug
It's alarming already. Too many outages in the past months. CF should fix it, or it becomes unacceptable and people will leave the platform.
I really hope they will figure things out.
Not everybody needs cloudflare, but those that need it and aren't major enterprises, have no other option.
(: phonetically, because 'l's are hard to read.
Even so, it is a strong reminder not to rely on any one vendor for critical stuff, in case that wasn't clear enough yet.
even the blog, that used to be a respected source of technical content, has morphed into a garbage fire of slop and vaporware announcements since jgc left.
If I was a CF customer I would be migrating off now.
**everything breaks**
...
**everything breaks again**
oh fuck! Code Orange! I repeat, Code Orange! we need to rebuild trust(R)(TM)! we've let our customers down!
...
**everything breaks again**
Code Orangier! I repeat, Code Orangier!
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
https://github.com/cloudflare/terraform-provider-cloudflare/...
the explanation makes no sense:
> Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed. The system interpreted this as all returned prefixes being queued for deletion.
client:
server: even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that blockIf they passed in any value, they would have entered the block and returned early with the results of FetchPrefixesPendingDeletion.
From the post:
> this was implemented as part of a regularly running sub-task that checks for BYOIP prefixes that should be removed, and then removes them.
They expected to drop into the block of code above, but since they didn't, they returned all routes.
actual explanation: the API server by default returns everything. the client attempted to make a request to return "pending_deletes", but as the request was malformed, the API instead went down the default path, which returned everything. then the client deleted everything.
makes sense now
but is that explanation is even worse
because that means the code path was never tested?
Lmao, iirc long time ago Google's internal system had the same exact bug (treating empty as "all" in the delete call) that took down all their edges. Surprisingly there was little impact as traffic just routed through the next set of proxies.
One thing I do appreciate about cloudflare is their actual use of their status page. That’s not to say these outages are okay. They aren’t. However I’m pretty confident in saying that a lot of providers would have a big paper trail of outages if they were more honest to the same degree or more so than cloudflare. At least from what I’ve noticed, especially this year.
But last few months has been quite rough for Cloudflare, and a few outages on their Workers platform that didn't quite make the headlines too. Can't wait for Code Orange to get to production.
As for your last sentence:
Businesses really do care about the incident reports because they give good insight into whether they can trust the company going forward. Full transparency and a clear path to non-repetition due to process or software changes are called for. You be the judge of whether or not you think that standard has been met.
Becuase management wants to know why the graphs all went to zero, and the engineers have nothing else to do but relay the incident report.
This builds a perception for management of the vendor, and if the perception is that the vendor doesnt tell them shit or doesnt even seem to know theres an outage, then management can decide to shift vendors
They definitely failed big this time.
Just joking, no offence :)