Hey folks!

Unfortunately, roughly 2 hours ago, lemm.ee went offline. The cause was our load balancer: it suddenly decided that all of our servers had become unhealthy, despite all health checks responding successfully when I requested them directly. In such cases, the load balancer stops serving all requests, effectively meaning that lemm.ee is unreachable for all users. I am still not sure what exactly caused the issue, but I will try to investigate more over the weekend.

For now, we have partially recovered, and I am continuing to work on remaining issues. Hopefully we will be back to 100% very soon. Sorry for the inconvenience!

you are viewing a single comment's thread
view the rest of the comments

[-] ramble81@lemm.ee 9 points 3 months ago

Nginx? I had an nginx LB shit itself yesterday. Luckily it auto-recovered and I had HA but just weird it happened.

[-] sunaurus@lemm.ee 9 points 3 months ago

Actually, we're using Hetzner's cloud load balancer for lemm.ee. But if this issue repeats in the near future, then I will definitely consider setting up something else.

[-] db0@lemmy.dbzer0.com 4 points 3 months ago

haproxy is where it's at!

[-] eleitl@lemm.ee 2 points 3 months ago

It's probably a managed haproxy in Hetzner's case.

this post was submitted on 18 Jul 2024

199 points (97.2% liked)

Meta (lemm.ee)

3473 readers

4 users here now

lemm.ee Meta

This is a community for discussion about this particular Lemmy instance.

News and updates about lemm.ee will be posted here, so if that's something that interests you, make sure to subscribe!

Rules:

Support requests belong in !support
Only posts about topics directly related to lemm.ee are allowed
If you don't have anything constructive to add, then do not post/comment here. Low effort memes, trolling, etc is not allowed.
If you are from another instance, you may participate in discussions, but remain respectful. Realize that your comments will inevitably be associated with your instance by many lemm.ee users.

If you're a Discord user, you can also join our Discord server: https://discord.gg/XM9nZwUn9K

Discord is only a back-up channel, !meta@lemm.ee will always be the main place for lemm.ee communications.

If you need help with anything, please post in !support instead.

founded 1 year ago

MODERATORS

sunaurus@lemm.ee