downvotes come at a “cost”, whereby if you want to downvote someone you have to reply directly to them with some justification, say minimum number of characters, words, etc.
I think it's the complete opposite. Platforms with downvotes tend to be less toxic because you don't have to reply to insane people to tell them they're wrong, whereas platforms like Twitter get really toxic because you only see the likes, so people tend to get into fights and "ratio" them which actually increases the attention they get and spreads their message to other people.
In general, platforms without upvotes/downvotes tend to be the most toxic imo. Platforms like old-school forums and 4chan are a complete mess because low-effort troll content is as loud as high effort thoughtful ones. It takes one person to de-rail a conversation and get people to fight about something else, but with downvotes included you just lower their visibility. It's basically crowdsourced moderation, and it works relatively well.
As for ways to reduce toxicity, shrug. Moderation is the only thing that really stops it but if you moderate too much then you'll be called out for censoring people too much, and telling them not to get mad is just not going to happen.
My idea for less toxicity is having better filtering options for things people want to see. Upon joining a platform it would give easy options to filter out communities that are political or controversial. That's what I'm doing on Lemmy, I'm here for entertainment, not arguing.