I've been above average fit with an above average activity level most of my life. Since 42, I've had less time walking around without aches and pains than with.
I may be an outlier, but lots of people I know that were sporty 10 years ago aren't sporty anymore, and it's not because of not wanting to