The issue with patriarchy is ultimately culture's views on gender and making artificial distinctions on who you should be, how you should act, etc. based on it – that's what feminism is at its core, gender issues and how human psychology interacts with the social construct of gender (which is why it's so closely tied in with the LGBT movement).

It's hard for society to even acknowledge – let alone overcome – unfair differences in treatment based on gender, when our culture raises us to have subconscious biases on what a man or a woman should be, that men and women are two different groups with certain behavioural archetypes that they surely follow, that they must have certain behaviours based on their gender. almost everyone, despite thinking otherwise, has a deep division between their understanding of different genders and behaviours associated with gender – men can or can't do X thing, women can or can't do Y thing. A man who lacks trait A is weak and pathetic, a woman who has that same trait is normal, or the other way around. Women telling others not to talk over her in a meeting is bitchy, a man crying or being "feminine" (physically or otherwise) is weird (as is a woman being "masculine"); a woman who works in a trade is assumed to be unskilled and is constantly demeaned by both customers and coworkers (applies to most "male-centric" jobs), a man who works a job with children is seen as an alien and might be seen as creepy by a lot of people. Single parents experience sexism a lot in different ways, in fact the sexism can be one of the most mental health eroding things some parents face from society.

Whether you're a man or a woman (or don't fit either of those norms) and which gender norms you follow (or go against) is one of the most important factors in determining how others treat you. You will face a completely different treatment from the same people based on your gender alone, and people will react to the same behaviours in radically different ways based on your gender. It's why a lot of feminists are gender abolitionists – "gender" and "sex" are ultimately dumb cultural concepts, yet they are some of the most important aspects of a person in our society and basically control how you can live your life, so we should work to get rid of the manmade concept of gender altogether (that's the thinking, anyway).

Ultimately feminism is only one part of the "social justice" movement as a whole; where feminism mainly focuses on the gender issues (and possibly sexuality), other movements may focus more on society's perception of race/ethnicity, class, etc. and a lot of the times these are very intertwined (a lot of research in feminism is centered on how race affects peoples' perceptions on gender, such as doctors tending to have a strong bias against minorities based on both their race and gender, for example).

Class does play a part in it though, so feminism and leftist movements e.g. socialism often overlap. The philosophical understanding is that gender equality can't happen under capitalism, as right-wing systems require hierarchies based on identity (including immutable traits) in order to function, so discrimination based on sex and gender is inevitable.

Explicitly calling it "patriarchy" has caused some problems, with men thinking it paints them to be the problem rather than the whole culture/society/government (and of course the ruling class), but as always the general populace misconstrues academic/movement terminology and there's not much that can be done to help that, especially when the public has adopted a preconceived idea of what "feminism" and "patriarchy" means that they really refuse to budge on.

That's like saying we shouldn't deal with neo-nazis who aren't in power. They're choosing to side with evil and thus deserve equal punishment. "Oh no they might feel bad about it" good they should feel bad. They should genuinely feel bad.