I'm not opposed to listening to people I disagree with, but that's not really the concern in this case. As others have pointed out, he doesn't really present a balanced perspective on anything, he just invites guests with extreme views, skewing heavily to the right, and then allows them to present their views with zero critical analysis. He just gets high and nods along like "wow bro so true" while lunatics like Jordan Peterson calmly explain some bullshit that can be summarized as "I don't like trans people existing, and I think we should put a stop to it."

You will also not find him entertaining any extreme ideas from the left, which if he were truly the unbiased "just asking questions" guy he purports to be, he would do at least occasionally.

So really it's a combination of the demonstrably false premise of him being an "unbiased centrist," his overt ignorance of the subjects he discusses, and his outsized influence on the minds of his listeners.

He's not just presenting ideas that I and others of the left disagree with, he is tacitly endorsing harmful and toxic ideas to his legions of fans, who then rabidly defend him in every venue online.

Edit: to clarify, I don't think trans rights is an implicitly political subject, but rather one that has been politicized and turned into fuel for the culture war. It's a minor thing but I felt I should make it clear since I used that as an example