As long as password auth is disabled you’re fine. No one is cracking your RSA key. You can add Fail2Ban to reduce the log noise, but security wise it’s fine.