Tailscale. I had some of the same concerns when I noticed someone else stumble across my stable diffusion external IP/port and generate a few random images.. yikes. I've since used Tailscale and completely removed all my router forwarding rules. It works seamlessly across devices, and is super easy on the setup. If it's for personal use only then this option works really well but I can't comment on if it could work as well for multi-user needs. There are some config options for sharing services I think.