I've thought about this (and taboo and norms in general) for a bit, so I'll take an unresearched guess that can be summarized as "swears are bad because people agree they are". Words have an associated context; which ones you use give some indication about the kind of person you are. In the case of swears, the context is that most people think that it is wrong to say them (though exactly how wrong varies), and (this is important) that most people think that everyone knows how wrong it is to say them. If you say a swear, you are (in others' eyes) demonstrating that you are the kind of person willing to knowingly violate these norms. The implication continues, then, that you are uncaring about what they might think or believe, what everyone in the community thinks or believes, and are willing to demonstrate that to their faces. Obviously, that may not match how you intended the word, but I think that this perceived hostility lies at the core of the reactions of those who freak out.

Either that or trauma from their parents or teachers freaking out it, or fear of divine punishment or something similar.