I knew I would get a reaction like this. You're attacking the strawman you created that downvotes are hurting people's feelings and this is why people would want them disabled / hidden. This is not what I said.

People will consciously or subconsciously let these numbers influence them. Whether it's positive or negative doesn't matter.

You're right that they hold no real weight as the range of motivators is extremely diverse, but we will still try to infer value from the patterns and act accordingly.