Using this logic, you would have to accept that people that are very good at a given game from the start have a fundamentally different experience to people that are very bad at a given game. And people that are average have another experience again.

So who's having the "true" experience? Is the good player having a degraded experience because they feel like they're playing on easy mode? Is the bad player having a degraded experience because they give up half way in? Is the average player having the "intended" experience of each part of the game feeling earned and hard won?

The reality is it's impossible to give the "intended" experience to everyone regardless. And if the average player experience is the intended experience, having difficulty settings will actually let the other players experience that, not take away from it. If the very good player and the very bad player can fine tune it so their relative experiences are the same as the average player, hard but not impossible, haven't you actually given the intended experience to more people rather than degrade it?