Nothing. I have all devices using tailscale DNS and I refer to things in my network by their host name directly.

Is there a reason why you want prowlarr and sab behind a VPN in the first place? If you just turn on TLS in sabnzbd you're not going to run into any issues and it's probably a better idea to search with prowlarr through your home address depending on what trackers you're using. I don't have either of these behind a VPN for my setup personally.

Gluetun just needs to share a docker network with traefik and not use network_mode. Do you have a default network in this compose file? If not you should add the same network name to every container and test to see if you can reach prowlarr with docker exec traefik wget -O - http://gluetun:9696.

Gluetun with protonvpn disconnects constantly for me and requires dependent containers to restart when it reconnects (https://github.com/qdm12/gluetun/issues/641) so make sure if you're testing stuff they're all restarted together

Not offering the perfect solution you were looking for, but if you're still using the ISP router it might be worth it to upgrade that to something you can install custom firmware on that's also relatively cheap like an asuswrt-merlin compatible device (I personally use an ASUS RT-AX86S).

That would allow you to have a guest wifi using YazFi that routes everything through a VPN of your choice and it would leave your regular wifi untouched. You can even use the DNS director to intercept all traffic on port 53 and reroute it to your pihole, including queries going out to 1.1.1.1 or other public DNS servers from apps trying to circumvent DNS blocks.

I personally run tailscale on my host and a single traefik instance on docker. Anything that doesn't need to be accessed by other people listen to the hostname and get a 100.64.0.0/10 ip whitelist middleware to only allow connections through tailscale. That way a request sent through the public IP with a custom Host header can't be smuggled in and I don't have to deal with custom ports for a 2nd reverse proxy