I don't think vibes is good enough. There's not enough certainty there. You could ask the other instance admin, but who's to say that admin is cooperative? They could be actively malicious and hiding behind this mechanism. You would ask them what's going on and they'd just say "nothing bad to see here" and you would have no way to disprove that.

As for activitypub, there isn’t a generic server to server message that could contain a VoteTotals field while updates to the standard are proposed for addition?

There is no such thing at the moment, though Lemmy could in theory implement it among its own instances, though even that would be hard. But it would be very non-standard and wouldn't work with other ActivityPub implementations.