Communication is a 2-way street. By definition you can't blame it on one person.

Having said that, there are definitely issues caused by monolingual English-only speakers. Someone who speaks English as a second language is, by definition, multi-lingual, and has the experience of knowing nuances between different languages and dialects. Often there are no exact 1:1 translations of concepts between two languages. So, they will have experienced that.

A monolingual English speaker probably doesn't have that experience. If they've traveled in the Anglosphere, they may have run across some of it, as different English dialects are different, and interpret things in different ways. But it's not the same as struggling with "to be" vs. "ser" and "estar".

It also happens that people speak too quickly, use slang, etc. OTOH, that's not just a problem with native English speakers. Indian English speakers (virtually always "English as a second language") have some strange expressions like "do the needful" or "what is your good name?". Singapore English (Singlish) has expressions like "Can" for "Yes I can" or "using Lah" to emphasize something.

It seems to me that the problem is more with people who spend too much time in an English monoculture who then try to communicate with people of a different culture. If you're Californian born and raised and have never left the state, you're probably going to have a hell of a time communicating with someone from Glasgow, let alone a French or Japanese person speaking English as a second language. At the same time, if you're Indian and speak English as a second language and the only English you've ever used or heard is Indian English, you're going to have the same problems.