Niger is the Latin word for black (not to be confused with the unrelated niger, as in the nation, which is not Latin in origin). We had stopped using it in English for anything but to (negatively) refer to black people. It didn't have any use except as a negative one.

The same thing is true for the r-word, outside of specific contexts like music where to retard is to slow, as from Latin. Using them in the context of music or medicine is fine (which retard isn't used for a medical condition anymore, but when things slow I belive the term is still used, because Latin). Using it to refer to a person or thing a person does is not. That term is only used negatively, which implies neuro-divergent people are bad.

Yellow can be bad also. If you're referring to someone in a negative racist way as yellow, that's not accepted in society, and I don't think you'd disagree. You can still refer to the color, but to use it as an insult it's not good.

We aren't losing words. If they had any use outside of an insult, they still almost always still have those uses. We gain more words to better refer to people of these groups without insulting them though. Language evolves. Some words leave fashion and new ones come into fashion. I don't think we need to be worried about "losing" words where we have perfectly good alternatives to use instead.