Anytime I feel uncertain about whether I'm talking to relatives "enough", I remind myself that the phone works both ways.

I can relate to your situation. I have two older brothers and I don't speak to them much. I only see them on a couple of occasions. And it's my choice that I don't see them that much. I found that every time I visit them or they visit me they just sap my energy. I don't get anything good from the relationship, only frustration and the feeling I can't be myself. When I speak to them it's either about how good their life is or how they don't agree with my choices. So I decided it would be better if they aren't a large part of my life. And my life is better and more relaxed because of it. Don't let a 'blood bond' fool you into a one-sided relationship.

Relationships with families are two way streets.

If he chooses not to communicate or to visit with you then that's on him. You have no obligation to keep something going if you are the only one holding up your end.

Do what feels right for you. Families are complicated, and sometimes the only bond we end up sharing is genetic. Maybe it's his turn to put in the effort, should he choose to.

That road goes both ways. Do you invite your brother over? Relationships, even family ones (sometimes especially family ones), need to be worked on. They aren't just some magical thing like back in school as a kid where you are forced to be around your family and/or friends pretty much daily.

You talk about not having time for him, but that's bullcrap. The work will still be there regardless. And even if not, it takes like half an hour to drive 30kms to visit. If you wanted to, you could make the trip after work, be there for an hour and go back. So could he. It seems like you don't want him to visit, and you don't want to visit him.

Am I in “the wrong” by not visiting my brother regularly

Why do you say “the” wrong? What is the problem here?

I mean, there's two sides to every story, but he sounds like a douche. Especially when he can drive over to visit friends but not you.

I wouldn't waste much energy on him.

I have a strained relationship with the brother (2 years younger) who lives a 6 minute walk away and a great relationship with the brother (8 years younger) that lives 1000 miles away. I was a shitty teenager back in the day, and he had a really manipulative girlfriend 6 years ago that isolated him from the family as much as she could. It took years to repair the damage after she was out of the picture, but it's still a functional relationship.

You don't actually owe your family anything, but the flip side is that they don't owe you either. If you see him every couple months to chill with your nephew, that's just fine. Some people are best friends with their siblings, others hate their guts and don't talk to them for decades.