You state you want to configure things so that nobody on the remote side "can access them"; I interpret this to mean that they should be firewalled off entirely. In which case, unless you have some way to physically lock anyone on the remote side away from your stuff, there's no way to ensure this. You'd essentially need to set up your own secure network on the remote end, one that nobody there can tamper with, and you'd want a separate firewall, probably one on which you'd run Tailscale with a local subnet.

And I think you'd want this anyway if you are looking to do remote management with iLO4. At least in my experience, remote OOB management comes in handy mostly when I screw something up on the server and it won't boot anymore. If Tailscale is on the server I can't boot, then I can't use it to access the OOB management either.

Now, if you can't lock things down physically, then things get even simpler. You should assume that someone there can do things like screw with the local console or with the hardware configuration, so you should not put things there in a form that they would be able to do anything with. This implies that your backup there should be encrypted (and it should be anyway; if a drive dies, you can discard it with no worry about data exfiltration). This also implies that it should have no ability to connect back to your home network, and that you should not run any services on it that you would be concerned about anyone breaking into or messing with. If all that is the case, then I think you probably are fine just plugging iLO4 into the same insecure remote network (again subject to the same caveat that it'll be useless if your machine ever goes down and you have no other way to connect in); if anyone manages to break in, you'll have controlled what they will be able to do with what they find.

The bottom line is very simple: don't trust untrusted environments!