Probably because they’ve proven that they won’t follow the oath they swore. So if they get reelected and swear the same oath (that they’ve already broken once) again, we already know they can’t be trusted to uphold it. So we don’t even give them the opportunity to be sworn in a second time.

But since an unsworn person never violated an oath of office, they’re still an unknown and could potentially be trusted. It’s a sort of “innocent until proven guilty” situation, where the person hasn’t broken any oath so by default they’re assumed to be trustworthy. But as soon as you break that oath, you’re not going to be trusted again.