I host my own for mspencer dot net, used this 15-ish step walkthrough from linuxbabe dot com. Only maybe three instances of spam in two years, gmail and outlook receive my messages just fine, etc. (Successful spammers were using legitimate services, and those services took action when notified. Greylist delays emails by a few minutes but it’s extremely effective against most spammers because they never come back to retry messages after a few minutes, while legitimate senders will.) I don’t know if I would accept blanket advice against self hosting.

Fundamentally if your mail server can see the addressee, it can see the content. SMTPS encrypts both in the same channel. So at the point where you accept messages and store them in a mailbox, the messages have to be readable.

Encrypting them at rest isn’t something I currently do, but if you’re going to later serve those messages to an email client that expects to receive clear text, your server needs both the keys and the messages. They can be stored in different places.

Most of your needs could be met with full disk encryption on the box hosting Dovecot. If you’re worried about being compelled to decrypt, there’s always the deck of cards trick: The pass phrase for full disk encryption consists of a memorized portion plus the letters and numbers of the top N cards in this deck of cards you keep by the server. If someone were to shuffle that deck of cards, and the server were powered down, the encrypted volume would be impossible to recover.

I’m eager to learn what other Dovecot tricks people can recommend to improve security.