That can easily be achieved with dovecot and a sieve script.

I host my own for mspencer dot net, used this 15-ish step walkthrough from linuxbabe dot com. Only maybe three instances of spam in two years, gmail and outlook receive my messages just fine, etc. (Successful spammers were using legitimate services, and those services took action when notified. Greylist delays emails by a few minutes but it’s extremely effective against most spammers because they never come back to retry messages after a few minutes, while legitimate senders will.) I don’t know if I would accept blanket advice against self hosting.

Fundamentally if your mail server can see the addressee, it can see the content. SMTPS encrypts both in the same channel. So at the point where you accept messages and store them in a mailbox, the messages have to be readable.

Encrypting them at rest isn’t something I currently do, but if you’re going to later serve those messages to an email client that expects to receive clear text, your server needs both the keys and the messages. They can be stored in different places.

Most of your needs could be met with full disk encryption on the box hosting Dovecot. If you’re worried about being compelled to decrypt, there’s always the deck of cards trick: The pass phrase for full disk encryption consists of a memorized portion plus the letters and numbers of the top N cards in this deck of cards you keep by the server. If someone were to shuffle that deck of cards, and the server were powered down, the encrypted volume would be impossible to recover.

I’m eager to learn what other Dovecot tricks people can recommend to improve security.

in case you're not already aware; mail servers are favorite target of malware & intrusion enthusiasts so be sure to approach your build with security at the forefront of all your actions.

i found out; well after the fact; that my build got pwned at step 2 after spending money and weeks worth of time to do the same thing you're trying to do and i wish someone had clued me into this little bit of common knowledge back then; good luck.

This is a thing that folks have done in the past:

• http://kacangbawang.com/encrypting-stored-email-with-postfix/
• https://www.grepular.com/Automatically_Encrypting_all_Incoming_Email and https://www.grepular.com/Automatically_Encrypting_all_Incoming_Email_Part_2
• https://jnphilipp.org/posts/auto-encrypt-all-incoming-email-with-postfix/
• https://www.mydreams.cz/en/hosting-wiki/9663-configuring-automatic-email-encryption-in-postfix-on-centos-7-using-gpg.html

You want the mail-crypt-plugin in dovecot.

If the mail is sent unencrypted the admin can read it. What I have is a script that encrypt incoming e-mail with the users key, so that they are stored encrypted on the harddrive. That at least protect against an intruder reading past e-mails. I use a Perl script written by Mike Cardwell for that.

Another service you might like to have for your users is WKD/WKS, so that senders clients can automatically fetch the public key for your users.

As someone who used to maintain an email setup for a small company, I would stay far away from self-hosting Email.

This is a misconception. If the sender is outside proton mail the emails arrive in plain text. So it is possible for proton to read those emails. It is just that they pinky promise not read them and immediately encrypted them with your key. But if they wanted to they can read and moreover SMTP means your email has already traveled through multiple MX which could read your email but This is mostly not an issue since most email provider do encrypt with SSL of receiving MX but you might want to check few services use very very outdated softwares. But keep in mind SSL encryption is with Proton's keys and by necessity they have to first decrypt the SSL encrypted email and then encrypt with your key

I don't like using "encrypted" email because in fact email is really not a secure protocol by default, you can send secure email to each other but if you're communicating to gmail, yahoo, outlook... You will lost all your privacy. Hosting an email service is good but do not use encryption when talking randomly to gmail accounts.