But magic doesn't have any grounding or association with the world as we know it. Neither does aliens. We can only use the world as we know it as a frame of reference for a story.

I'd argue in lbaudia's example that it is confusing if in 12th century Mongolia, there was some Nordic guy walking around, I'd imagine there to be a backstory of some kind. If there wasn't, then that would definitely be an example where I'd be annoyed at white people in fiction.

I thought a great example of casting was the TV show "The Expanse". To be able to cast someone as specific as Bobbie Draper so well - these studios have no excuse to whitewash as they do except laziness.