They do not matter at all.

I beg to disagree. If "useless internet points" don't matter, why is there a billion dollar marketing industry surrounding them? I mean all kinds of data mining conducted on all forms of internet reactions. People are paid good money to crunch these types of numbers, including who is casting the votes (man, woman, white, black, American, not-American, liberal, conservative, etc, etc). Then there is the troll/astroturfing angle. There are different types of campaigns that pay drones to upvote or downvote stuff, for marketing purpose or state-actor agendas.

Sure basing your self-esteem on internet points is harmful and useless, but seeing internet reactions as a narcissist fuel only is also naive and misleading. Given the OP wants to get genuine feedback to his opinions to use as a political or moral compass, the question of the feedback quality is not moot at all.

It should have no bearing on your life what so ever.

The feedback quality is also indeterminate. We can't know the proportion of astroturf, spooks/trolls, and genuine users in any upvote/downvote score and/or reaction. This can lead to a situation where the feedback to your opinions is always muddy, and vague. Do my opinions suck or is this their problem? In real life you won't get honest feedback to your opinions anyway, for reasons of politeness. I read once this is why conspiracy theories thrive in Facebook more than Twitter (old study), because a network of acquaintances will not challenge your BS, but a crowd of strangers will.

For all these reasons I think the OP's question is a valid problem we don't yet have good answers to. And it is relevant to any platform, Lemmy included.