It might be another food allergy as well. I have skin reactions to dairy, and it took me years to figure it out because a) it's a slow reaction (6ish hours after I eat it), and b) I also assumed it was something my skin was coming into contact with
And in keeping with the OP, I also still eat dairy from time to time. Not a severe reaction, it won't become a severe reaction, and there's still no good replacement for actual cheese, dammit. (Daiya is close enough...sometimes...)
So basically, trying to figure out whether someone is actually being altruistic is just the poisoned chalice bit from The Princess Bride, forever