If Ruby's guess near the end is right, the whole point of her being there, haunting her, making people run away, was to make specifically this one man run away in terror, and there's no way whatever that was had anything to do with anyone else abandoning Ruby. She simply says something so terrifying that people run away, that's it, it doesn't matter what exactly, and it doesn't even have to be the same thing for everyone.

Which makes me think that perhaps it's like psychic paper: she could say anything (or even nothing), and the experience would be the most horrifying, disgusting, or terrifying thing to the listener, enough to drive them away from Ruby forever.