The majority of the default fail2ban installations only bans an IP for 10 minutes and uses a 10 minute findtime, e.g. slow brute forcing is not at all banned.
Before I switched to crowdsec (which I really recommend you do, its quite easy) I changed my bantime and findtime in /etc/fail2ban/jail.conf (I think I made a local file... read the file it should say) to something like 8 hours (e.g. change 10m to 640m for both those variables).
I have an open ssh port and I use key auth with password as well as crowdsec. Even if people get my ssh key they would still need to know the password.