It all started with a report about federation breaking between Lemmy and NodeBB. I was subconsciously aware that something was going on, but had chalked it up to network issues.
Observed behaviour showed that some remote categories would be receiving content in spurts, with long gaps in between.
I spent the next 3-4 days looking into it, but came up empty. Whatever was happening wasn't throwing any obvious errors, and along the way, I found what I thought was related (it was), but I wasn't sure why: against some Lemmy servers, the "follow"/"unfollow" mechanic would simply stop working, and this would often coincide with gaps in content. In some egregious cases, the flow of content stopped completely!
Unable to make headway, I had to reach out to the folks at Lemmy to figure out what the issue was. NodeBB occasionally sends non-200 level responses depending on the activity. Specifically, the following scenarios:
- A remote user upvoting more than 20 posts in a single day (a spam prevention tactic) causing NodeBB to throw an error, which was caught and returned an
HTTP 500 Internal Server Error. - A
Dislikeactivity, which is not currently handled by NodeBB. In these cases, NodeBB would send anHTTP 501 Not Implemented
When encountering either of these responses, Lemmy would return the activity back to the queue for later delivery and mark a delivery failure. If enough of these (~40) happened within 24 hours, Lemmy would give the instance a time-out and pause delivery completely.
That was it — a quick pair of code updates later, and we started working through Lemmy's backlog of 4.1M activities.
As of 4am this morning, community.nodebb.org is no longer behind lemmy.world.
Fun week. Let's not do that again LOL.
v4.4.2 of NodeBB contains the updated logic for smoother Lemmy federation.