Slavery benefitted both white and non-white slave owners.

Edit: at least in the short-term, obviously you could argue differently for long-term effects.