The downside of free speech is having to endure the opinion of idiots, be it nazis or religous asshats or whatever bullshit people come up with. It's not that you solve a problem by shutting it up either. It just eludes your direct attention.

But it still is a very important thing that always slowly dies in favour of the majority's political flavour-of-the-month. Noone cares as long as they're part of the majority up until they suddenly aren't. Then the flaws of censorship emerge.