My head-canon is that they have a shorter window of fertility. I mean humans have menopause as well.

Though it would make more sense if that happens later in their lives, otherwise you're saying every adventurer has given up the possibility of raising kids. Also, i suppose if that's the case they haven't really matured in the sexual sense, so much as they've just grown enough to be independent.

I seem to remember some bit of lore saying dwarves only settle down to have kids near the end of their lives, but i can't find that anywhere ( Might've just imagined it, lol).

Maybe adventuring across ages with different heirs of humans who go from weak to strong extremely fast while elves start strong and grow slowly?

That sounds cool but you'd need all your quests to span generations and leveling in general would be very slow. Probably harder to make it balanced as well.