Kris

joined 2 days ago
[–] Kris@feddit.org 4 points 1 day ago

Sollte es zum Anschluss von Kanada kommen, gibt es wohl ein rechtliches Schlupfloch da Musk's Mutter gebürtige Kanadierin ist oder so.

[–] Kris@feddit.org 6 points 1 day ago (1 children)

Not necessarily, no. We aim to preserve users, communities and posts/comments. Image uploads might get lost though.

Such an in-place migration will need extensive database operations and likely some support by the Piefed developer (to add support for bcrypt hashed passwords), but we are hopeful to make it happen and maybe this will result in a database migration script other Lemmy instances could also use.

If this turns out to be infeasible, we will stay with Lemmy rather than reset everything.

[–] Kris@feddit.org 4 points 1 day ago* (last edited 1 day ago)

As email in profile is an optional feature on our instance about 50% of the userbase opted to not share one so we would only reach some people. But yes, getting spam filtered would be another problem of mass-mailing.

But I found a way to put up a simple html error page and will add this later to inform our members and link to this thread.

[–] Kris@feddit.org 6 points 1 day ago

Lemmy has a lot of individual parts that don't interact very well with each other, especially the image host part. Futhermore the main UI is quite a mess and we were thinking of switching to an alternative already, but this would further increase the "too many moving parts" issue. Piefed on the otherhand has an integrated and very lightweight UI, which also has some nice additional filtering and moderation features Lemmy currently lacks.

And I personally feel more at home with the Python codebase, as it allows better troubleshooting and more standartized (Flask) tooling. The Rust codebase of Lemmy has a lot of obscure custom stuff and the error messages are extremely obstruse from a sysadmin perspective.

And looking at the performance metrics of Lemmy, the main limiting factor seems to be the Postgres database anyways, so the theoretically slower Python codebase of Piefed should not have much impact.

[–] Kris@feddit.org 9 points 1 day ago* (last edited 1 day ago)

We have a small write up about the hardware on our wiki, but it is also down right now.

I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.

One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.

[–] Kris@feddit.org 8 points 2 days ago* (last edited 1 day ago) (7 children)

Already thinking about how to make the best of it 🤷 Maybe we can use this opportunity to try and migrate to Piefed? I had this in the back of my mind for some time already and Rimu seems optimistic that it is possible.

[–] Kris@feddit.org 14 points 2 days ago

Glad that at least someone sees a positive side of it, because I sure don't... well at least it gives me a bunch of ideas how to avoid such situations in the future 🤷

[–] Kris@feddit.org 19 points 2 days ago

Thanks, I am trying, but it really is quite annoying to lose access like that.

[–] Kris@feddit.org 24 points 2 days ago (1 children)

Sort of yes. It is a homelab a few people have access to, but currently they are all either on holiday or on extended work related travel. It seemed like an acceptable risk as everything worked smoothly for years.

[–] Kris@feddit.org 6 points 2 days ago* (last edited 2 days ago) (2 children)

Yeah I had plans to set up something like that, but always other priorities and in this specific case I could maybe access other internal servers but i would need KVM access to reboot the firewall or some other way to cut physical power. And exfiltrating hundreds of GBs of lemmy database wouldn't work over such a small pipe either.

[–] Kris@feddit.org 27 points 2 days ago* (last edited 2 days ago) (1 children)

The feddit.de admin disappeared on a work trip to Japan. At least that is the official story.

[–] Kris@feddit.org 14 points 2 days ago (6 children)

Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.

333
Slrpnk.net outage (feddit.org)
submitted 2 days ago* (last edited 2 days ago) by Kris@feddit.org to c/fediverse@lemmy.world
 

Alt account of poVoq@slrpnk.net here.

Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

view more: next ›