Next time someone says that, respond with "wow, so fuck anyone with brain issues that keep them from forming memories the same way you do, right? Guess grandpa Rick doesn't care about my family since his accident, huh."

you having a grandpa Rick, or family to speak of, is irrelevant. The point is, it forces them to confront the fact that there are many reasons someone might forget a thing. They'll probably still be shift about it, but at least when they've unbunched their loads, they might think twice before immediately concluding the other person sucks for forgetting a thing.

Maybe I'm being optimistic with that, but I don't appreciate when people try to assign my motives to me, instead of understanding my own, and get confrontational when that happens.