Shit, you're absolutely right, I missed an (in hindsight very obvious) optimization - bit depth. It's been so long since I've actually needed to worry about it that I forgot that the setting existed! What makes it even worse is that I did already play with quantizing the colors dwon to a more limited space, I just never baked that in as the bit depth haha.