As somebody that mods other communities on ani.social, lemmy has a sizeable portion of its userbase that browses by All rather than by Subscribed. Lemmy also has a sizeable portion of users that are extremely hostile to anything anime or anime-adjacent (just see the lemmy.ml defederation drama from last year for example). So, random downvotes on active posts happen a lot since each new comment pushes that post to the top of the feed when sorted by Active. The important part is the engagement from your community members. One person commenting is worth >100 random up/down votes for keeping a community going.