Sadly, there will never be a truly self-hosted solution given how the devices in question rely on Google, Samsung, Microsoft or Apple servers to be active and available on initial enrollment. The control plane can be on-prem, but the actual enforcement is done through built-in management APIs that depend on external services.

That said, I created my own zero-cost MDM solution by leveraging Android Enterprise APIs along with Samsung Knox. There's no pretty UI though - everything is done through API calls using Postman. Enrollment is achieved by scanning a QR code on the device's first boot. I'm managing ~450 Samsung tablets and a dozen mobile phones using this approach.