Work-around for the federation bug found in 0.19.0 and 0.19.1 (sub.wetshaving.social)

submitted 10 months ago* (last edited 10 months ago) by walden@sub.wetshaving.social to c/lemmy@lemmy.ml

15 comments fedilink hide all child comments

I've found the following work-around works pretty well. If you host an instance that's currently on 0.19.0 or 0.19.1, consider implementing this.

There are two bugs that this helps with:

Work-around:
Create cronjobs that restart the Lemmy container every 6 hours (but not at midnight). The following example is used for a Debian system running Lemmy in Docker.

Type crontab -e into the terminal Add something like the following:

~~0 1 * * * docker container restart lemmy-lemmy-1
0 7 * * * docker container restart lemmy-lemmy-1
0 13 * * * docker container restart lemmy-lemmy-1
0 19 * * * docker container restart lemmy-lemmy-1~~

3 1-23/6 * * * docker container restart lemmy-postgres-1 && sleep 60 && docker container restart lemmy-lemmy-1

By restarting the container every 6 hours, outbound federation continues to work. There may still be some delays, but everything gets cleared up regularly.

By telling it what time to restart (0100, 0700, 1300, and 1900 as opposed to "every 6 hours"), it avoids restarting at midnight. This avoids the second bug.

My instance has been doing this for enough days where I'm confident that it's working. You can check your federation status here. Note that it's normal for there to be 0 up-to-date instances and a lot of lagging instances. As long as they sometimes turn "up to date", then everything is getting caught up.

all 18 comments

sorted by: hot top controversial new old

[-] RedKrieg@lemmy.redkrieg.com 16 points 10 months ago* (last edited 10 months ago)

I like to do things just off the top of the hour, since top of the hour is when many maintenance crons run. If you're running a modern cron daemon, you can rewrite that as:

3 1-23/6 * * * docker container restart lemmy-lemmy-1

https://crontab.guru/#3_1/6___*

[-] walden@sub.wetshaving.social 6 points 10 months ago* (last edited 10 months ago)

I figured someone more familiar with crontab would chime in. Thanks for the recommendation!

Edit: I've updated my cronjob to https://crontab.guru/#7_1-23/6___* and I'll keep an eye out to make sure it works on Debian 12.

[-] BlueEther@no.lastname.nz 4 points 10 months ago

The only thing that doesn't seem to federate on my 0.19.1 instance is new subscriptions, but in saying that it looks like the BE restarted an hour ago

[-] rikudou 2 points 10 months ago

Is "testing" a swear word to Lemmy "devs"? Such bugs after so many RCs? (Though it seems that for Lemmy "devs" RC actually means "random crap").

[-] walden@sub.wetshaving.social 2 points 10 months ago

Extra frustrating for you, I imagine, since you write a lot of API stuff for Lemmy.

[-] rikudou 2 points 10 months ago

Yeah, still have to fix some stuff, though I've taken a break during the holidays.

[-] walden@sub.wetshaving.social 2 points 10 months ago

That's great, hope you're having some fun with family and friends. Happy New Year.

[-] rikudou 2 points 10 months ago

Happy New Year as well!

[-] SeeJayEmm@lemmy.procrastinati.org 2 points 10 months ago

I didn't know how this tool is supposed to work but it always fails for me but my instance appears to be working ok.

https://phiresky.github.io/lemmy-federation-state

[-] walden@sub.wetshaving.social 2 points 10 months ago

Interesting. Yeah I tried too and it doesn't work for lemmy.procrastinati.org

[-] bjoern_tantau@swg-empire.de 1 points 10 months ago* (last edited 10 months ago)

Funnily enough, since I've started to regularly check if my comments made it out I haven't had any problems. Schrödinger's bug.

Edit: Then again, after checking with this post, although my comment made it out just fine, some comments haven't arrived on my instance. I'll try to restart.

[-] walden@sub.wetshaving.social 2 points 10 months ago

Comments from other instances should arrive to yours no problem, as the bug only seems to affect outgoing stuff. If the other instance is running 0.19.0+, then of course those comments might be stuck.

[+] Bungiefan_ak@sh.itjust.works -9 points 10 months ago* (last edited 2 months ago)

[removed by mod]

[-] InEnduringGrowStrong@sh.itjust.works 7 points 10 months ago

All this could have been conveyed without slinging insults. Maybe chill a bit.

[-] CrypticCoffee@lemmy.ml 6 points 10 months ago* (last edited 10 months ago)

Because it offers new features to immature software still in development.

It gives snowflakes like yourself the ability to block woke instances so your limited brain capacity isn't overwhelmed. It also gives scaled sort, the ability to see niche sub's with less content higher up, so you can see the latest content from your favourite country music communities or about the latest NASCAR race in which people take the same turn constantly for hours on end rly fast.

Dat's some good shizz and worth updating for, surely?

Gotta love those communists that build Lemmy and give you this much joy. That do the hard stuff like running updates and testing so anyone of any (limited) ability can enjoy :).

[-] Omega_Haxors@lemmy.ml 0 points 10 months ago* (last edited 10 months ago)

Downvoted out of spite for that last line. I mean come on, it was 1 downvote.

EDIT: Oh you're the owner of sh.itholefor.nazis? Cool, please instance ban me. I am sick of your shitty users constantly leaving harassing comments and downvote barraging anything left of hitler and would rather your users just not see my comments. Thanks, and GFYS.

this post was submitted on 30 Dec 2023

56 points (100.0% liked)

Lemmy

11947 readers

42 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.

founded 4 years ago

MODERATORS

nutomic@lemmy.ml