But if a troll blocks everyone that down votes them, eventually no one will be left to see their stuff.
By letting people prevent others from downvoting bad content, you make more people see it. This problem is only "self-solving" ifː

1. everyone who sees heavily downvoted posts blocks the user. This probably isn't ideal behavior given how many people treat downvotes as an "I disagree" button.
2. if the person's posts aren't being exposed to new people. It doesn't matter if everyone who's already seen the user has blocked them if more people are getting exposed to them. And by preventing downvotes, you're making it more likely that people will see these posts.

On the other hand, a troll that you block can interfere with your stuff being shown to others depending on the sorting algorithm.

The number of trolls going around downvoting everything you do is minimal, and you can often deal with them by calling them out as was discussed in a previous thread about downvotes. Additionally, you have to consider the other side of the coin, which is content that should be getting downvoted getting more attention than they should.