You're not the asshole, you're playing with the wrong group. Find people that want to play the same style of game as you, don't play with people just because they're who you're familiar with. Recipe for a bad time.