I'm just accurately recharacterizing your assertion here.

Clearly freedom can't exist unless we tolerate its worst excesses.

That's what you're saying, though I don't think allowing fascist and Nazi speech is analogous to the argument the free speech needs to exist for the assholes. Being a Nazi and a fascist is a terrorist act, not "being an asshole". An asshole pisses in the river. A Nazi is the American corporation dumping waste in the river en masse.

Never mind that we can look across the pond at Germany and see quite clearly that freedom of expression can exist alongside a ban on fascist and Nazi speech.