I think part of our brains are devoted to very simplistic pattern learning, and it does not respond well to nuance or change. So while you are not spending it is learning concepts like "bank balance go up = good, bank balance go down = bad". It completely misses the entire point of why you put money aside, so when you spend it that simple brain becomes a bit alarmed. It makes a useful warning system to double check with your rational brain if you are doing the right thing, but it can be tricky to disarm when it is giving a false alarm.