this post was submitted on 08 Jun 2025
2 points (100.0% liked)

ActivityPub

0 readers
4 users here now

Focused discussion related to ActivityPub integration in NodeBB


This is a forum category containing topical discussion. You can start new discussions by mentioning this category.

founded 5 months ago
 

In February 2025, I presented a topic at FOSDEM in Brussels entitled The Fediverse is Quiet — Let's Fix That! In it, I outlined several "hard problems" endemic to the fediverse, focusing on one particular complaint that is often voiced by newcomers and oldtimers alike; that the fediverse is quiet because you don't ever see the full conversation due to some design considerations made at the protocol level.

Since then there have been a number of approaches toward solving this problem, and it is worth spending the time to review the two main approaches and their pros and cons.

N.B. I have a conflict of interest in this subject as I am a proponent of one of the approaches (FEP 7888/f228) outlined below. This article should be considered an opinion piece.


Crawling of the reply tree

First discussed 15 April 2024 and merged into Mastodon core on 12 Mar 2025, jonny@neuromatch.social pioneered this approach to "fetch all replies" by crawling the entirety of the reply tree. When presented with an object, the Mastodon service would make a call to the context endpoint, and if supported(?) would start to crawl the reply tree via the replies collection, generating a list of statuses to ingest.

This approach is advantageous for a number of reasons, most notably that inReplyTo and replies are properties that are ubiquitous among nearly all implementations and their usage tends not to differ markedly from one another.

N.B. I am not certain whether the service would crawl up the inReplyTo chain first, before expanding downwards, or whether context is set in intermediate and leaf nodes that point to the root-level object.

One disadvantage is this approach's susceptibility to network fragility. If a single node in the reply tree is temporarily or permanently inaccessible, then every branch of the reply tree emanating from that node is inaccessible as well.

Another disadvantage is the reliance on intermediate nodes for indexing the reply tree. The amount of work (CPU time, network requests, etc.) scales linearly with the size of the reply tree, and more importantly discoverability of new branches of the reply tree necessitate a re-crawl of the entire reply tree. For fast-growing trees, this may not net you a complete tree depending on when you begin crawling.

Lastly, in the ideal case, a full tree crawl would net you a complete tree with all branches and leaves. Great!

Mastodon is the sole implementor of this approach, although it is not proprietary or special to Mastodon by any means.

FEP 7888/f228, or FEP 171b/f228

Summarized by silverpill@mitra.social in FEP f228 (as an extension of FEPs 7888 by trwnh@mastodon.social and 171b by mikedev@fediversity.site), this conversational backfill approach defines the concept of a "context owner" as referenced by compatible nodes in the tree. This context owner returns an OrderedCollection containing all members of the context.

A major advantage of this approach centers around the pseudo-centralization provided by the context owner. This "single source of truth" maintains the index of objects (or activities) and supplies their IDs (or signed full activities) on request. Individual implementations then retrieve the objects (or activities). It is important to note that should the context owner become inaccessible, then backfill is no longer possible to achieve. On the other hand, a dead or unresponsive intermediate node will not affect the ability of the downstream nodes to be processed.

The context owner is only able to respond with a list of objects/activities that it knows about. This does mean that downstream branches that do not propagate upwards back to the root will not be known to the context owner.

Additionally, consumers are also able to query the context owner for an index without needing to crawl the entire reply tree. The ability to de-duplicate objects at this level reduces the overall number of network requests (and CPU time from parsing retrieved objects) required, making this approach relatively more efficient.

Additional synchronization methods (via id hashsums) could be leveraged to reduce the number of network calls further.

A number of implementors follow this approach to backfill, including NodeBB, Discourse, WordPress, Frequency, Mitra, and Streams. Additional implementors like Lemmy and Piefed have expressed interest.

One technical hurdle with this approach is technical buy-in from implementors themselves. Unlike crawling a reply tree, this approach only works when the context owner supports it, and thus should be combined with various other backfill strategies as part of an overall conversational backfill solution.

Conclusion

2025 is shaping up to be an exciting year for resolving some of the harder technical and social problems endemic to the open social web/fediverse. It is this author's opinion that we may be able to make good headway towards resolving the "quiet fedi" problem with these two approaches.

It is important to note that neither approach conflicts with the other. Implementations are free to utilise multiple approaches to backfill a conversation. Both methods presented here have pros and cons, and a combination of both (or more) could be key.

Feel free to use this as a starting point for discussions regarding either approach. Does one speak to you more than the other? Are the cons of either approach significant enough for you to disregard it? What other approaches or changes could you recommend?

top 16 comments
sorted by: hot top controversial new old
[–] mikedev@fediversity.site 2 points 3 weeks ago (1 children)

We're using both here. There's an icon to let you know that you're looking at an actual conversation -- vs. a collection of microblog posts that once had a common ancestor.

The differences in signal/noise ratios between the two styles are quite dramatic. Neither is better or worse than the other. They are different. And they can both co-exist.

Also, conversation containers has the ability to "reply to all" as well as "reply to sender". Microblogs don't have this concept, and instead "reply to all" means "send to all your followers, instead of a reply directed to the actual conversation audience.

Additionally, consumers are also able to query the context owner for an index without needing to crawl the entire reply tree.

While this is certainly true, when conversation containers are working correctly, you never need to backfill a conversation. It is all delivered to you.

[–] silverpill@mitra.social 1 points 3 weeks ago (2 children)

@mikedev @julian

hen conversation containers are working correctly, you never need to backfill a conversation. It is all delivered to you.

I think there is one case where backfill is necessary: public conversations that are not discovered through following (e.g. by retrieving an object by its ID).

[–] mikedev@fediversity.site 1 points 3 weeks ago

True, but fetch one collection and you've got it all. Might be paged, and with Mastodon that means another fetch for every ten activities (seriously?), but those are just implementation details.

[–] julian@community.nodebb.org 1 points 3 weeks ago

silverpill@mitra.social mikedev@fediversity.site correct. Backfill is important even when you have good synchronization systems in place.

One example I use is Lemmy's use of 1b12. It is exceedingly good at keeping subscribers in sync, but if you discover a new node or leaf, then backfill is required to get you the conversation up to that point.

[–] silverpill@mitra.social 2 points 3 weeks ago (1 children)

@julian @trwnh @mikedev

neither approach conflicts with the other

I don't fully agree with this statement, because these "threading paradigms" suggest two different solutions to the problem of moderation. If the OP is the single source of truth, they can moderate the entire conversation (represented by context collection: Streams). If not, then each reply is independent and authors moderate only the direct replies (represented by replies collections: GoToSocial).

In theory two solutions can be combined, but at the cost of significantly increased complexity.

[–] julian@community.nodebb.org 1 points 3 weeks ago

silverpill@mitra.social said: > If the OP is the single source of truth, they can moderate the entire conversation (represented by context collection: Streams). If not, then each reply is independent and authors moderate only the direct replies (represented by replies collections: GoToSocial).

That is a good point. The approaches are broadly compatible when top-down moderation by the context owner is not assumed.

In a moderated scenario, crawling the reply tree would not be useful unless paired with some sort of "is member of" validation with the context owner... at which point the served collection would be more performant.

It could be useful for discovery by the context owner itself though.

[–] jonny@neuromatch.social 2 points 3 weeks ago

@julian

N.B. I am not certain whether the service would crawl up the inReplyTo chain first, before expanding downwards, or whether context is set in intermediate and leaf nodes that point to the root-level object.

Current impl starts at the expanded post and goes down - one can start a crawl at any point in a tree. If one starts at a lower point in the tree and then triggers a crawl higher up in the tree, lower part only gets crawled once within a configurable cooldown period to avoid double crawling.

[–] mikedev@fediversity.site 1 points 3 weeks ago (1 children)

I think a couple of folks have mentioned trying to consolidate both of these approaches into one. I once used something that resembled 1b12 (long before there was a "threadiverse"), but as I recall it didn't really work well with private groups and aspects/circles - where you're often dealing with third-party permissions. You can only relay public activities to third parties via an Announce, and so conversations with restricted audiences don't work out very well for viewers on Mastodon. The third party does not have permission to access the activity from its author, only from the conversation owner. Once you've run into this issue, you are likely to more fully understand the advantages and disadvantages of these two approaches. Container operations are pure relays and work correctly with third-party access control, assuming you're using signed objects (which everybody should be using, but that's a hill to die on another day).

[–] silverpill@mitra.social 1 points 3 weeks ago (1 children)

@mikedev @julian

but as I recall it didn't really work well with private groups and aspects/circles

Last time I heard about 1b12 private groups, the proposed solution was to use a "collection inclusion endpoint" to verify that actor is a member of a group

[–] julian@community.nodebb.org 1 points 3 weeks ago (1 children)

silverpill@mitra.social do you still need to if you're not using a shared inbox?

[–] silverpill@mitra.social 1 points 3 weeks ago

@julian @mikedev Yes, if you receive an Announce(Create), and Create is not signed, then you need to retrieve this Create from its origin. When that origin server receives your signed GET request, it needs to verify that you belong to the group, but it might not have that information.

[–] projectmoon@forum.agnos.is 1 points 2 weeks ago (1 children)

julian@community.nodebb.org said in Backfilling Conversations: Two Major Approaches: > A number of implementors follow this approach to backfill, including NodeBB, Discourse, WordPress, Frequency, Mitra, and Streams. Additional implementors like Lemmy and Piefed have expressed interest.

Is this implemented currently? One weakness I have noticed in NodeBB's current federation is that posts which are in reply to a topic (e.g. a Lemmy comment) show up as individual threads until (or if) the root post of that topic shows up in the local NodeBB. It's a bit confusing of a UX, I think. Because you think:

  1. (Before root post) Why is this a post that seems to be just a random comment?
  2. (After root post) Why did that other thread disappear and why am I seeing the same comment again?
[–] julian@community.nodebb.org 1 points 2 weeks ago

> One weakness I have noticed in NodeBB's current federation is that posts which are in reply to a topic (e.g. a Lemmy comment) show up as individual threads until (or if) the root post of that topic shows up in the local NodeBB.

No, Lemmy does not implement either strategy, they rely on 1b12 only.

If NodeBB is receiving parts of a topic that don't resolve up to the root-level post that might be something we can fix. I'll try to take a look at it.

[–] robz@toot.robzazueta.com 1 points 3 weeks ago (1 children)

@julian Quick, somewhat unrelated note - I follow you on Mastodon and see your posts with the HTML tags showing. Is NodeBB escaping those tags prior to sending out the AP message?

[–] julian@community.nodebb.org 1 points 3 weeks ago (1 children)

Hi robz@toot.robzazueta.com! This could be related to some better support for non-Note types introduced by Mastodon in later versions. Your instance is running v4.1.18 which is 11 months behind the latest version.

That isn't necessarily cause for concern, but I think that might be why you're seeing the HTML tags?

[–] robz@toot.robzazueta.com 1 points 3 weeks ago

@julian Ah... that actually may make more sense - thanks.

I'm working on my own AP implementation and hadn't yet run into this issue, so assumed.

Time to upgrade!