You've missed a key detail in how asymmetric encryption works:
- For asymmetric encryption algorithms, you essentially have two keys - a "private" key, and a "public" key
- If you know the private key it is trivial to calculate the public key, but the reverse isn't true - just given the public key, it is essentially impossible to calculate the private key in a reasonable amount of time
- If you encrypt something with the public key you must use the private key to decrypt it, and if you encrypt with the private key you can only use the public key for decryption
- This means that my server can advertise a public key, and you can use that to encrypt the traffic so that only the server that knows the private key can decrypt it