I think Kbin just doesn't support downvote federation yet. Dunno if it's just a Kbin thing or if Lemmy also hasn't gotten around to it, but either way, it's something that needs to be fixed.
If this is deliberate, it makes no sense. There are reasons to want or not want downvotes, but you need to commit to one instead of having this confusing jank where you only see downvotes from your instance.