I agree. And I have already seen it. People going to inspect the vote history on all their own comments, then complaining that another person is going around downvoting them. Because I am exceptionally nosy, on 2 occasions I actually went to check to see if it was true. In both cases it did seem to be the case although neither downvoter was really vigorous. Like half the commenter's recent posts were downvoted by the same person.

OTOH my understanding is that it is required in order for votes/boosts to propagate in federation. If there was a technical workaround that people felt confident in, I think it would be better hidden. For that % of people where it really just makes them crazy and they can't resist checking.