Also to add to what I said in my other comment, sex as a working definition affects many areas of our lives. You may define sex as the production of gametes, but being male or female affects gigantic areas of our social lives and comes with a massive number of tacked on traits. Far from merely being a definition of biology it affects our experience of every single aspect of society. Sex is all of those things too. We can argue that it shouldn't be, that it should be an entirely unrelated and inconsequential trait (which would also mean that we can easily recognize that people outside of the binary categories exist), but the reality is it doesn't mean that.

Society requires you to have a sex. If you are an intersex person you are functionally incapable of interacting meaningfully with a society that does not recognize people who are neither male or female. For instance, when bathroom bans are passed, where should intersex people go? Where can you go if society has adopted a rigid binary view of sex and you are not male or female? What social services are you entitled to? What prisons should you be put in? Sex in terms of a rigid binary category dichotomy functionally erases the existence of intersex people and adds a huge amount of barriers to their lives.