I've solved this on my side with socket activation, which besides giving out the real IP, also has native network performance since it fully skips slirp4netns. You could even set nginx's network to none, but since I also use named networks for internal container DNS, so I kept network set.

I've built my own Nginx image and I'm using Quadlets instead of Compose, so my config is as easy as it gets, the socket file is something like this:

[Unit]
Description=container-nginx

[Socket]
BindIPv6Only=both
ListenStream=443

[Install]
WantedBy=sockets.target

And the quadlet file for NGINX goes like this for me:

[Unit]
Description=Web serving, reverse proxying, caching, load balancing, media streaming, and more.
Requires=nginx.socket
After=nginx.socket

[Container]
Image=localhost/nginx:latest
AutoUpdate=local
Volume=/data/containers/nginx/conf.d:/etc/nginx/conf.d:Z
Volume=/data/containers/nginx/certs:/certs:Z
Network=services.network
# Socket for systemd
Environment="NGINX=3;"

[Service]
Restart=always

If you check the socket activation link, there are a few other examples, but IMO that's the easiest setup out of the 5 examples. You could move NGINX out of the compose setup for easiness or adapt examples 3 to 6 (which invoke podman manually). That said, I wanted to use Caddy for easier certificate management, but it doesn't support socket activation, so this setup kinda hardlocked me to NGINX.