It's important to remember it this way:

If you're in a club with 99 amazing men, and one totally perverted creep who flips to aggression on a dime, guys like that won't stop after harassing the first woman they meet in the club. By the end of the night, 4 out of 5 women who attended that club have had an experience of a guy hitting on them and getting aggressive when they reject him, and there's a good chance it's all the same guy. It doesn't matter that proportionally there are waaaay more good men at the club, the reality is that almost every woman in that club had a scary experience that night because of a man.

Now that happens every night you go out, if it's not you getting harassed, it's your bestie and you need to stick together. Sure, you and your girls are meeting 2 or 3 great guys who you have lovely interactions without. But in the uber home you're not talking about how nice that bloke and his mates you met on the dance floor was, you're checking in to make sure your friend is okay after that one guy tried to slip his hand up her skirt while she was ordering at the bar, only to get threatened with rape when she said "please stop that".

So, yes, it is a regular occurrence, not because the men who do this are regular in the population (though in some areas due to the local subcultures, they are) it's a regular occurrence because the few men who do this are serial harassers, and for every woman you politely and respectfully flirt with, the assholes are out there harassing 10 or 20 women.

Now I do understand how frustrating it is when we say "Urgh, men" and not "Urgh, specific men who like this", but when that one creep is a new creep every time you go out, and you're creeped on every time you go out by the one guy there who is a creep, the other 99 men fade into the background because they aren't a threat, you don't need to be vigilant around them, so you aren't thinking about them. He's one guy out of 100, but his level of threat and danger dominates the women's lasting perception of the safety of the space, and why is it unsafe? Because of a man, which man? It could be any man, you won't know until you interact with them, so until you know, the danger isn't a man, it's men. I know that while there are a thousand species of snake only about 30 have a truly fatal bite, but I'm still going to say "I'm afraid of snakes", even while I'm giving a chill pet carpet python a happy little cuddle.

And it sucks, I'm sorry you get lumped in with assholes due to the way women use language to describe their fears and concerns over some men.

What you're experiencing is how these bad men effect all people, not just the women they harass. And it's a great reason to join the social movements working to reduce behaviours of concern among these groups of aggressive men.

But while it's frustrating that this social issue causes you to feel prejudged as dangerous, at least this social issue isn't a risk to your physical safety the same way it is for women.