I've run my own mailserver for about 20 years. I don't know if I'd recommend others do the same, but I wouldn't recommend against it either.

Once it's up and running, it's surprisingly low-friction. I have a VPS with a provider I trust, and it's running nothing else. Other than keeping everything updated, it requires very little ongoing maintenance. Mostly making sure you keep up with dmarc, TLS, etc best practices before the big providers call them requirements, instead of after.

I think the real difficulty is starting fresh, greenfield. Not only can one misconfiguration ruin your day, but if it's an issue that other providers notice, the smell hangs around for a long time. Most the big providers (gmail, microsoft, yahoo) will do absolutely nothing to work with you, so if they take a dislike to you - well you're screwed. There's no way to get in touch with them, no way to ask them to look again, etc. The juggernauts will usually give the impression they don't actually have anyone working for them at all.

You'll also learn a lot more about DNS. Whether you like it or not :)

Things that aren't so fun .. OS updates are always the terrifying one. My provider is really good about letting you spin up a new instance while keeping the old one around for a month so you can switchover when you're ready. I use that for most things - but for my mailserver, I don't want to because I don't want a new IP. I like that it's my ball and I can pick it up and go play somewhere else if I want, but the amount of reputation that the big providers pin to IP, makes this a lot more difficult than it sounds.

The other fun sticking point is monitoring. I get emails if my mailserver (or DNS) go down .. but because my mailserver is down, I don't receive them until it's back. That's not ideal, but I never seem to get around to doing anything about it. (because when it's working, I want to leave it alone. When it's not working, it's too late.)

I think the main thing to keep in mind is that it's difficult to "lab" outbound mail. There's very little "just trying something", very little experimentation, etc. Getting things wrong has too many long-term effects. You wanted to try a new MTA and now Google think you're a spammer? Putting the old one back does not fix your reputation. Putting the old config back does not fix your reputation. Doesn't matter how much you clean, that smell is going to take a long time to go away.