I actually think up- and downvotes are inherently asymmetrical in this respect.

Upvoting things you agree with is fine and a main use of the function. Why then is downvoting things you disagree with wrong? Because the purpose of voting is visibility, you upvote things you want people to see, like arguments you agree with, and downvote those you don't think people should see.

Now if you believe in having an open discussion you don't want to suppress posts just because you disagree with them. Disagreeing is fine, so downvoting is reserved for posts that detract from the conversation.