So like my title states I am having slow download and upload speeds while connecting back to my servers on WAN.

For a little context I have a netgate 4100 with HA Proxy installed. If I do a speedtest from speedtest.net I get the full 1g up/down. Same thing goes for the computer at site B that I am using to test this with along with my phone.

But if I run a speedtest(using librespeed) from site B to my servers at site A I get 20ish Mb/s not MB. This is running through HA proxy with a certificate. If I directly port forward and bypass HA Proxy I get 50ish Mb/s. Iperf3 gets 100mb/s. So there is overhead but not much.

However if I go to the domain name of my hosted speed test (from site A) I will get the full 1g up/down. If I go to the IP directly I'll get 10gb/s since it's on the same VLAN.

This tells me it's not a bottleneck of the test server, firewall, or HA proxy.

So I ask what could possibly be wrong?