Back to News
Advertisement
Advertisement

⚡ Community Insights

Discussion Sentiment

57% Positive

Analyzed from 2994 words in the discussion.

Trending Topics

#github#com#https#enterprise#status#why#more#self#down#move

Discussion (135 Comments)Read Original on HackerNews

frenchie4111•about 6 hours ago
I noticed earlier this week that my URL bar now pre-fills githubstatus.com instead of github.com when I type "gith"
sandermvanvliet•about 1 hour ago
As a joke I built an extension to the GitHub CLI called “omens” so you can run “gh omens” before you’re planning to use GitHub and it’ll tell you if it’s likely to work in idiomatic Aussie.

As an example:

$> gh omens

reckon github's about to go full bin chicken

—-

You can find it here: https://github.com/sandermvanvliet-stack/gh-omens

ncr100•about 4 hours ago
A browser extension to indicate githubstatus being yellow/red:

https://chromewebstore.google.com/detail/is-github-down/lcfo...

A VSCode extension:

https://marketplace.visualstudio.com/items?itemName=RuslanRy...

Firefox:

https://github.com/matagus/github-status-checker

Caveat: https://news.ycombinator.com/item?id=49450924

(disclaimer: none of these are recommended by me to use - merely sharing to build on for this discussion)

embedding-shape•about 3 hours ago
Another useful disclaimer: all of these lets the developers push updates to your computer as they wish by default in most setups, these days you might want to decrease that kind of attack surface and just check the status in the official website when "git push" suddenly stop working. Especially when extensions and stuff sometimes changes hand and it's kind of hard to keep track of it, except for when something bad happens and gets news attention.

I'm also not sure why you'd share links to software you don't even recommend yourself? Isn't it better to just not share that then? I think others could use search engine/LLM too if they need whatever, but most generally expect things shared in the comments here to actually at least have been looked at by the person sharing them.

jmb99•about 2 hours ago
Well, the alternatives are:

> It would be really cool if someone built an extension to show GitHub status live

"These already exist, why wouldn't you search before posting?"

> There exist quite a few extensions to show live GitHub status

"Why would you not post them if you know they exist?"

Or, what actually happened:

> Here are some extensions that might work for you to show live GitHub status

"Why would you recommend extensions to do this?"

Seems to me like, if your goal is to pick apart someone suggesting something, you'll find a way to.

LPisGood•about 4 hours ago
I can tell it’s getting bad because I went to close some Safari tabs and got confused because the tab next to this was [a dupe of] https://news.ycombinator.com/item?id=49330597.
parthdesai•about 3 hours ago
You've a tab from 9 days ago open? Maybe you should be closing your tabs more often :p
anvuong•about 3 hours ago
I have tabs from like 6 months ago. Modern browsers have gotten pretty good at hibernating unused tabs and restore when needed. And this website is extremely light, it gets restored in milliseconds.
vinnymac•about 5 hours ago
For me this started happening when I type "stat" too.

Which surprised me because of the many hundreds of services I rely on regularly that also have status pages.

b112•about 4 hours ago
Good lord, hundreds?!

Uptime reliability is nkt additive, it's multiplicative, sometimes even logarithmic.

Downtime of 99% and 99% is 98.1%. I hope almost all those services are non-prod related.

Melatonic•about 2 hours ago
I notice I sometimes wake up before my alarm because I get an SMS about GitHub being down.

It could be a new Microsoft feature !

tom1337•about 2 hours ago
i noticed it today as well - typing "g" is enough to go to the statuspage. and surprisingly often when i go there by accident, they have an incident…
shepardrtc•about 5 hours ago
Fun read about Azure and having 173 agents running a node: https://isolveproblems.substack.com/p/how-microsoft-vaporize...

Probably just a coincidence that Github started to have issues after beginning their move to Azure at the end of last year.

caust1c•about 4 hours ago
Azure's going to suffocate github. I'm curious to see what's next. Will self-hosting the code repository come back in vogue or will another social-coding platform take off?
anvuong•about 3 hours ago
I used to work in a small independent team of 30 people within a large corp, half of which was dev. We used to run our own gitlab on-prem, our CI/CD was also on-prem. It worked perfectly, never had down time, devops guy could configure them on-demand to our needs. Me (and some other guys) also jumped in times to times to help (mostly just ssh into the servers for health check, disk partition, etc.). Then we grew (the biz team, dev was the same) and some new PMs with fancy Ivy League degrees came in and pushed for on-cloud Bitbucket. Things went to shit pretty fast after that ... Our codebase was only a few hundred thousands lines, there was only like hundreds of commits per day, the servers our git + CI/CD lived on never saturated ...
pocksuppet•about 4 hours ago
We should worry less about what everyone else uses, and more about what we use. I'm self-hosting Gitea and thinking of upgrading to Forgejo. What are you using?
jscott817•about 4 hours ago
My small org has definitely had internal discussions around self-hosting gitlab. We'll see what happens.
vinnymac•about 4 hours ago
If you're actively considering self hosting, I recommend giving forgejo a try. The experience is much more similar to what GitHub offers, actions API is nearly identical to name one similarity of which there are many. Moving essentially becomes one prompt and a coffee later for a small team.
brazukadev•about 4 hours ago
probably not. The people that have done it before or willing to do it now is probably a very % of the commit volume, they leaving wouldn't change much, probably not gonna even move the exponential growth needle.
Syntaf•about 5 hours ago
One of my favorite articles posted here for this year, it's a great read and really gives you an idea of just how dysfunctional azure is.
slices•about 1 hour ago
"fun"

I thought I was cynical enough about goings on at Microsoft / Azure, but apparently not.

whoamii•about 2 hours ago
Keep in mind the seniority of the author. This was not written by a staff+ engineer.
outworlder•about 3 hours ago
> Probably just a coincidence that Github started to have issues after beginning their move to Azure at the end of last year.

Right.

I'd like to think that Azure has improved in a meaningful way in the past half a decade or so... but it has not. Maybe less inexplicable 400 errors in random API calls, I guess.

And yet Microsoft keeps posting record growth. Unbelievable.

tomw1808•about 5 hours ago
Things can go wrong, but really, its been a lot and we're normalizing that to an unhealthy degree...

I wonder if it was down that much, if users would get credits the way we pay when we use the services - its kind of ridiculous for a critical service to be down that much and all we do is "ah okay, its just github". Like, as if that was normal to be down that much...

zaik•about 5 hours ago
I think the authors of SMTP had a healthy attitude towards server uptimes:

   Retries continue until the message is transmitted or the sender gives
   up; the give-up time generally needs to be at least 4-5 days. 
https://datatracker.ietf.org/doc/html/rfc5321#section-4.5.4....
pocksuppet•about 4 hours ago
At the time, most email was either local to the host (big mainframe in the basement) or transferred once a day through scheduled dial-up connections during off-peak phone hours
creshal•about 1 hour ago
Which is a good fit to a distributed, offline first version control system, if you use it like that.
thinkingtoilet•about 4 hours ago
I don't think it's been "normalized". Github uptime is literally a joke in the tech community. They have first mover advantage and a behemoth behind them so they're not going away, but everyone knows how shit it's uptime is. It takes time for organizations to move away from services like this but I would bet anything that many are starting to try to move away, as well as new companies knowing they shouldn't use the service.
mportela•about 2 hours ago
I agree. I think it is the kind of situation that goes “gradually and then suddenly”, to quote Hemingway.
nr378•about 6 hours ago
GitHub needs to completely bifurcate their enterprise/paid services from their free services at the infra level.
saxonww•about 5 hours ago
They have that-ish as an option: https://docs.github.com/en/enterprise-cloud@latest/admin/dat...

I'm told that GitHub has asserted to us that moving to this model means we would not be exposed to github.com outages. It's not at feature parity with github.com though.

nr378•about 4 hours ago
Thanks, this option is good to know.

We're currently on "GitHub Enterprise Cloud" on github.com and are affected by this outage (even though we use self-hosted runners!), but we're not on "GitHub Enterprise Cloud with data residency" on *.ghe.com, which I understand is/may not be affected by this outage?

saxonww•about 1 hour ago
This is what they've told us. It's represented as basically a separate deployment of the entire GHEC stack, so you're not exposed to the load/scaling issues they believe are the underlying cause of all the github.com outages today.
herpdyderp•about 4 hours ago
Do you have any meaningful level of faith in GitHub's ability to deliver on stability? At this point, I have none.
saxonww•about 1 hour ago
Meaningful is subjective, but yes I do. It was very stable for many years, and I do believe the recent issues are mostly or all because they were caught flat-footed by the rapid AI-driven load increases.

This has been a bad time, though. I'm ready to move back to self-hosting if they can't get it together or we move to GHDR and it's still bad.

creshal•about 1 hour ago
A lot of the stability problems come from trying to scale a free product on a still WIP cloud solution without costing too much; the same software running on separate, paid for infra has a lot better odds.
sdetheridge•about 5 hours ago
According to their status pages (e.g. https://eu.githubstatus.com/, https://us.githubstatus.com/), their Enterprise Cloud uptime for Actions is significantly higher.
roastedfunction•about 5 hours ago
“GitHub Enterprise Cloud with data residency” is hosted on separate infrastructure and dedicated subdomains under *.ghe.com. It’s been around since November 2024z

It’s not the same thing as GitHub Enterprise Cloud hosted on the shared global network on github.com.

https://docs.github.com/en/enterprise-cloud@latest/admin/dat...

Melatonic•about 2 hours ago
So confusing and so Microsoft. They love to have licensing so complicated their own sales people aren't up to date and have to rely on third party spreadsheets.

Edit:

Read that document - why do more people not do the self hosted option with GitHub Enterprise Server ?

everfrustrated•about 5 hours ago
That is a different and later product with a confusingly similar name.
vinnymac•about 5 hours ago
Just to be clear, I am on Github Enterprise, and am also experiencing this disruption both privately and publicly on every org and project I have access to.
VCFundedGenYer•about 5 hours ago
That's what Azure DevOps is supposed to do, but for some reason GitHub has a redundant enterprise division.
weli•about 5 hours ago
That's what I don't understand. They could mitigate their name so much if they just split free/paid/enterprise. It's already shown that enterprise is much more estable and is largely unaffected from service disruptions. Why don't they go one more layer? For sure it's worth the extra complexity.
lbriner•about 5 hours ago
There is no such thing as "just split" there is 20+ years of legacy decisions and even if the split is relatively clean it is still probably 1 years work for 200 people for maybe a marginal improvement.

The real money is going to go towards, "make this all more reliable".

gaigalas•about 5 hours ago
Depending on the cause of the current issues, that move would likely cause more harm to paid services than good.

Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up.

Doubling the operation doubles the operational challenges.

rethab•about 5 hours ago
surely if they did that everybody would complain how github "lost its touch with open source since they now prioritize paid services"
john_strinlai•about 5 hours ago
enterprise is mostly separate, is it not? uptimes are significantly more reasonable on the enterprise status pages
saxonww•about 5 hours ago
We are in GHEC right now and GitHub Actions is not working. It's been down every time githubstatus.com says it's down.
Melatonic•about 2 hours ago
What country did you choose to host your data in ? Could be region based
mh-•about 5 hours ago
Same for us, I'm not even sure what product that other "Enterprise" status page refers to..
flohofwoe•about 4 hours ago
It would probably be better to run projects with extremely high commit/merge frequency on a separate "slop infrastructure", basically like MMOs move cheaters to their own servers ;)
inigyou•about 5 hours ago
I should make a business selling git hosting. Apparently it's really easy because it doesn't have to actually work.
brazukadev•about 4 hours ago
Sure but first you gotta be Microsoft.
pydry•about 3 hours ago
Microsoft and Oracle have the market cornered on selling stuff to corporations who are relaxed about whether it works.
SoMomentary•27 minutes ago
At this point it feels like it would be more honest for this to be titled "Disruption with Some GitHub Service".
everfrustrated•about 5 hours ago
>Update - primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues

And now they're blaming their upstream vendor! Embarrassing stuff to be writing on a public page.

ecshafer•about 4 hours ago
Vitess is a distributed mysql database. Github could very well be managing it entirely on their own. I have only seen people managing their own vitess, its entirely open source afaik.
AdrienPoupa•about 5 hours ago
I read that as an upstream service they own, but I agree the wording a bit weird.
heaney-555•about 5 hours ago
Why is that a problem if it _is_ an upstream vendor problem? (assuming it is)
CerebralCoding•about 6 hours ago
Must be a day ending in Y
nosioptar•about 6 hours ago
At this point, maybe it'd be more appropriate for people to post about github to HN when githubs actually working.
brian626•about 6 hours ago
98codes•about 5 hours ago
Certificate expired over 100 days ago
isaacdl•about 5 hours ago
Where do you see that?

Common Name (CN) www.dayswithoutgithubincident.com

Organization (O) <Not Part Of Certificate>

Common Name (CN) YR1

Organization (O) Let's Encrypt

Issued On Monday, August 10, 2026 at 10:01:51 AM

Expires On Sunday, November 8, 2026 at 9:01:50 AM

mrshu•about 3 hours ago
The historical uptime has been getting better but it is on a downwards trajectory in August:

https://mrshu.github.io/github-statuses/?view=all

ad_fontes•about 6 hours ago
> Update - We've identified an issue with a database primary and are failing over to a replica immediately

Seems like a weird thing to post on a status page. Shouldn't this have happened automatically and therefore precluded the need to inform users of it?

thecosmicfrog•about 5 hours ago
> Update - primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
mportela•about 5 hours ago
GitHub Actions upkeep is down to one 9. I miss the days big tech aimed for four or five 9s of reliability :(
Advertisement
xbryanx•about 5 hours ago
I spent a bunch of time during the outage last week setting up forgejo and some custom action runners. At the time, I was worried I was wasting time and getting distracted from my real work...alas, I guess not. Gonna finish up that work and complete the move today.
NewJazz•about 1 hour ago
Good for you. These kinds of migrations never feel productive at the time, but the right tools can make your life and work so much better.
rglover•about 1 hour ago
Good. Now everyone is getting to see what over-investment in complexity nets you. This bad of reliability screams "tangled spaghetti shit show" behind the scenes.

It's a shame that this is happening and hopefully Microsoft do the right thing and spin it back off into a private entity with its own leadership and resources.

Also, self-serving: this is why the whole "big company means better" thing is bs. We need to aggressively go back to a culture that rewards and supports independent teams who actually give a damn about what they're building and responsible for. Otherwise, this is just a preview of everything everywhere in a few years.

guhcampos•about 4 hours ago
Honestly?

If the problem is scalability, just rate limit git commands on free accounts already. Nobody realistically need to push multiple times per minute, and that alone is bound to trickle down to anything that triggers on commits and pushes.

brazukadev•about 4 hours ago
Sure. The problem is just an easy fix that nobody there thought about yet.
guhcampos•about 4 hours ago
My point is: they blame it on scale. If that's the story they want us to believe, then they have to explain WHY such a simple fix isn't possible.

There may be non technical reasons for it, even: legal, or PR, but if they want me to believe the "scalability excuse" they need to convince me they're trying.

materielle•about 3 hours ago
I don’t know, but typically when orgs don’t make logical decisions, it’s because internal politics are incentivizing something else.

I suspect that rate-limits would fly in the face of a lot of narratives like “Azure is ready for hyperscale”. And getting eyes on Github is probably a part of their Copilot product strategy.

pydry•about 3 hours ago
realistically the problem is vibe coding.
roncesvalles•about 1 hour ago
The problem is when you stuff your codebase with so much unreviewed vibecoded slop that engineers lose their grasp on how it all works.
NewJazz•about 1 hour ago
Keep pulling the lever, uptime has to improve eventually!
nickwanninger•about 5 hours ago
Same time next week?
qkwrv•about 5 hours ago
We can't keep living like this.
pocksuppet•about 4 hours ago
Obviously we can because we are choosing to. Because servers are scary.
pajamasam•about 5 hours ago
Apparently we can because a lot (most?) of us are still using GitHub even after all their outages recently.
nubinetwork•about 5 hours ago
Except nobody moves to a privately hosted "gitweb"...
JsonDemWitOster•about 3 hours ago
Unbelievable! Just this morning I was thinking to myself, hey Github's due for a downtime soon or they are finally getting their act together.
1matin•about 3 hours ago
Can't wait for Anthropic's downtime tomorrow
xray42•about 6 hours ago
So a normal Wednesday
theanonymousone•about 5 hours ago
The joke was a good one the first, second, or third time. It's not even funny anymore...
acedTrex•about 6 hours ago
Oh thank god my pink unicorn site is back online, its had great uptime lately so thats nice.
Advertisement
Elfener•about 5 hours ago
Ah so that's why I got a random "github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks"
sevenseacat•about 5 hours ago
Was wondering why all my Actions just stopped running
kelvinjps10•about 5 hours ago
I have switched off from github to my own server besides two websites that depend on gitbub integration to deploy to clpudfare pages
igleria•about 5 hours ago
must feel bad that at this point every dev checks github status before going to work like it was the weather app.
time0ut•about 6 hours ago
Notice odd behavior on GitHub. Get gaslit by a green status page. Notice more odd behavior on GitHub. Think it must be me this time. See unusual action queuing. Ah, an incident on the status page. Go for a walk and check HN on my phone. The AI SDLC.
serial_dev•about 6 hours ago
The status page is the last one to get the update. Reddit, HN, X, company chat are all always reporting it sooner.
everfrustrated•about 6 hours ago
> We've identified an issue with a database primary and are failing over to a replica immediately

This is why it's hard to take GitHub seriously. How can a single database cause an outage for everyone? This is amateur stuff. Have they no sharding or partitioning internally? Paying customers should not be impacted in the same way as free ones are.

ZiiS•about 5 hours ago
2.9B commits per month; 100M action runs per day; I think they probably have some sharding.
ferguess_k•about 6 hours ago
I wonder what is this database, and why it is hard to fall-over automatically.
inigyou•about 5 hours ago
RDBMS replication and failover is way more difficult and manual than anyone would like. You can't just set up two postgres, tell them they're clustered and have it basically work; at a minimum you have to design the client to somehow know which one is currently the master, or use some sort of proxy (which becomes its own SPOF).

RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.

ferguess_k•about 4 hours ago
Thanks! I didn't get the chance to manage RDBMs but that's good to know.
croemer•about 5 hours ago
Possibly vitess from the latest update:

> primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues

ferguess_k•about 4 hours ago
Thanks!
rkozik1989•about 5 hours ago
Did you not read it? Just because there's a database primary doesn't mean there is 1 primary database. There's likely man redundancies and they have issue with how they're allocating traffic to them which is in turn causing an issue with how much traffic redundancies are receiving.
inigyou•about 5 hours ago
Why shouldn't it? Most companies run on a single database server. If they can immediately fail over to a replica, that's doing it right.

Maybe you expect that part of GitHub to have a scale where a single database can't handle it, but evidently that isn't true.

We can criticise them for not splitting up free and paid customers but again, most companies don't do that.

firatsarlar•about 3 hours ago
Scaling is a dream or theory we think we solved. Distance between ... to practice. So ... should be reconsidered again.
rglover•about 1 hour ago
This level of scale is extreme but it can be handled with proper time investment and careful engineering.

But that doesn't fit into the "spend gobs of money as fast as possible so you can be a hypothetical feudal lord in a sci-fi dystopia novel you read when you were twelve" line of thinking.

firemelt•about 2 hours ago
at this rate this is a FEATURE!
lossolo•about 4 hours ago
How can you do so badly with Git when its architecture is basically so friendly to partitioning and horizontal scaling?
jp_sc•about 3 hours ago
Must be Wednesday
Advertisement
Traubenfuchs•about 3 hours ago
This is unacceptable for paying customers.

Either 1) infrastructure for paying and non paying customers must be separated, 2) excessive load must be ended by not offering the free tier anymore or 3) they must fix their broken monolith but they are seemingly incapable of doing so.

watermelon0•about 3 hours ago
It very well may be that the paying customers are actually the problem. Those can actually afford to throw enough money at AI providers to significantly increase the number of commits & other actions.
esafak•about 5 hours ago
Cursor had better not miss this opportunity.
stalfosknight•about 5 hours ago
So what’s stopping you (or your org) from leaving GitHub?
lucky_cloud•about 3 hours ago
I started the campaign to move us from Bitbucket to Github before the Microsoft acquisition. If I knew MS would be involved, I'd have campaigned for something else. I would never recommend any Microsoft product.

It took ages for us to get permission and licenses. We only recently finished moving the last repos over and shut down Bitbucket.

If AWS offered a product like GH, I could probably unilaterally start moving to that. Any other alternative would take 3-4 years of meetings, budgets, lawyers, and other such nonsense before we could declare we'd left Github.

esafak•about 3 hours ago
I don't like any open source solution, Codeberg and Gitlab are more of the same and Cursor isn't ready yet.
iso1631•about 5 hours ago
Must be a weekday
zackify•about 6 hours ago
Can't even run self hosted github actions lol
thatwasunusual•about 4 hours ago
I have been very happy with github for _years_. Lately, not so much. Now I want to try a self-hosting alternative again. I tried gitlab years ago (pre-pandemic, IIRC), and it wasn't for me (they had _serious_ security issues as well, which of course didn't affect my self-hosting thing, but still...).

So - where do we stand in August 2026?

*EDIT:* No, I don't want anything related to Felon Musk.

rvz•about 4 hours ago
Another outage, this time with GitHub Actions. Last time that happened was 5 days ago [0] and another outage happened on the postmortem announcement as well. [1]

While GitHub is imploding itself, maybe you should think about self-hosting.

[0] https://news.ycombinator.com/item?id=49379172

[1] https://news.ycombinator.com/item?id=49379225

fosterfriends•about 5 hours ago