GitHub Outage Tracker: Is GitHub Cooked?

What we are seeing here is a trillion-dollar company screwing over its existing well-established customer base in a desperate attempt to get some crumbs of the current hype cycle. GitHub isn't "dealing with" scaling issues, they are intentionally causing it by shoveling giant piles of coal onto the AI fire.

GitHub's issues would be over tomorrow if they were to put reasonable rate limits on their services. Stop the bots and GitHub is no longer permanently overloaded. It is an intentional business decision not to do this, and instead welcome bots with open arms - to the detriment of their existing paying customers.

I do have some sympathy for the low-level GitHub staff who have to deal with this - especially those who aren't being compensated with insane Bay Area salaries.

I don't think this is commendable at all. I give GitHub a lot of money and I'm tired of it being wasted with downtime.

1. Github has enterprise users who paid for the service, their day job requires Github to be available and working

2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code.

It is a complexity in itself but the traffic should have been separated, the free users should not be allowed to bring down Github for enterprise customers (Just to clarify, I am free user myself). And if they do not have capacity it would have been perfectly fine to push back or throttle new users/repositories.

They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft.

And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.

I would have liked if they had left the open source projects alone and didn't create that FOMO for not using them.

I have no sympathy.

I can have sympathy for the humans caught in the crossfire but only managing one nine of availability on a commercial service is not acceptable.

No.

Their leadership went all in on AI and in the last blog post essentially admitted some missing test coverage for a critical path. Time to learn lessons and fix your vibe coded shitslop and stop using "user graph go up" as some kind of excuse.

That is an active choice they are making, over and instead of any reliability for the rest of their users.

It is a problem they are embracing, and actively encouraging, for themselves.

But we pay enterprise license and GitHub is a big dependency in our software flow.

If this continues to be a problem as an enterprise product they need to do something. Otherwise theyre are going to to start losing business

They can simply throttle access for everyone to a push/merge frequency which was normal for human-maintained projects.

They should look at how MMOs battle cheating/botting and apply similar countermeasures (it's actually quite absurd that MMOs which are actively attacked every minute have a better uptimes than Github, and on Github the problems are even self-inflicted, not caused by external attacks).

But of course throttling bots to human-level push/merge frequency wouldn't be compatible with the Github leadership's 'AI bullishness'.

Yes, that is scale. And yes, that's not actual requests per second. But it's the sort of scale that big (and even mid-sized) tech has known how to deal with for decades. Microsoft doesn't have an excuse.

How about no? They are owned by a company that is neck deep in foisting AI into everything, AI that a lot of us don't want in everything, they are reaping what they seem determined to sow.

So it goes.

It's probably not fair to the people who are left but alas, this is apparently how the world works.

As I said before (and it was not a popular comment), it's easy to be an "armchair quarterback," with these services, as I think the brittleness was already baked in, and just waiting for the right time to crack. The only true way to have a robust platform, is to design something that will scale, from the start, and many startups don't do that, because they are feverishly trying to get out an MVP; even if it is a mess of bubblegum and baling wire.

They always say "We'll get it done right, once we get funding," but that never happens.

But my sympathy is limited by the fact that MS paid a lot of money for this, quite a while ago, and that is one company that knows all about issues of scale. They should have seen this coming.

A restaurant makes pizzas. They suddenly get 100x more popular. They can't make 100x more pizzas. But they are still taking orders from 100x more people. Not only are they not getting enough pizzas delivered that they took orders for, but in their rush to make and deliver more pizzas, they set the kitchen on fire, which makes an even longer wait for pizzas.

When the pizza you ordered doesn't get delivered, do you have sympathy for the restaurant? Or do you tell them to stop taking orders they can't fill and try not to set the kitchen on fire?

Now consider the pizza restaurant has 21 billion dollars in cash, is taking your money, and not giving you pizza.

We certainly do not.

I don't have sympathy for them. They have disrupted my work so much in the past month that it's ridiculous, and my company likely pays them millions of dollars. And you can see from this site's plotting that there is a trend towards more frequent and more critical outages. It has been extremely bad the past month.

This is a multi billion dollar corporation. History of robbing and stealing from others of their labor or IP. History of enshittifying once great services.

I used to be on-call in a high-traffic environment where single customers pushed more bits than entire nations. I chose the role. I didn't want people's sympathy, if anything, I wanted them to complain to management.

If it gets too bad they can quit. Maybe that would be for the best, just wear the thing down until it outright fails and no one wants to touch it. One less bullshit service sucking all of the oxygen out.

I've started suggesting to my CTO that we move to something else because we do pay the Github bills and yet we are also affected by other people's activity. Avoiding that is not a hard engineering problem for an alternative.

A world where a record amount of code is pushed to GH but a large amount of paying customers leave would be "cooked"

    const incidents = e.detail.incidents;
    
    // ...snip...
    
    const now = new Date();
    const threeMonthsAgo = new Date(now);
    threeMonthsAgo.setMonth(threeMonthsAgo.getMonth() - 3);
    
    // ...snip...
    
    var currentFreq = recentIncidents.length / 3; // <- We out here, smoking these guns with our homeboy Claude
    
    // ...snip...
    
    var earliest = null;
    for (var j = 0; j < incidents.length; j++) {
      var dd = new Date(incidents[j].started_at); // <- eventually incidents[j].started_at is "2016-03-01T07:07:37.000Z"
      if (!earliest || dd < earliest) earliest = dd;
    }
    
    // ...snip...
    
    document.getElementById('n-since').textContent = earliest
      ? earliest.toLocaleDateString('en-US', { month: 'long', year: 'numeric' })
      : '?';
    document.getElementById('n-rate').textContent = Math.round(currentFreq * 10) / 10;

#n-rate has nothing to do with the #n-since month, it's just the last trailing 3 months. And even then, it's sort of underbaked? It's moving the date back by 3 calendar months not taking into account differing numbers of days, so it'll under-report short months.

I wouldn't trust the stats here.

Edit: whoops, author updated the template while I was writing this! It now says "Over the last 3 months", though that's still calendar months.

    GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month

Looks like they fixed it already

1. Hates the status quo

2. Only uses a handful of the 10,000 features the status quo has

3. Uses a different handful than everyone else

Because that model kept Excel as the center of buisnesses around the world for 30 years and counting

GitHub Actions is god-awful. Have you ever used any other CI tools?

you are the problem

And it may be even worse in EDT, which is currently in effect!

The contribution graph as an outage calendar idea is a commonly recurring one :). I definitely saw it somewhere else as a static asset first before I made this site.