I think you're spot on.

Part of the reason I think it's a little easier to render women is that they typically have smoother less textured skin than men (video game women look like they are wearing makeup even if it makes no sense), and we're only just now getting the graphical fidelity to render realistic looking facial hair or fine stubble in realtime.

Not that long ago making stubble on a male character usually just involved putting an irregular brown shadow on their cheeks which can be pretty indistinguishable from dirt.