At some level, I think it's just pragmatic.
That is, I think it's because no single person or organization has enough political and cultural power to dictate language. And therefore, language is always a democratic (read: messy) process, and a negotiation. And, at least at this point in history, the words, 'sex' and 'gender' are both under contention, and are therefore ambiguous, unless you add qualifiers like "at birth".
I wrote an article a few weeks ago, attempting to provide a trans-inclusive definition of 'woman' and 'man' (because it bothered me how nonsensical some people's definitions were), and one of the commentors, a trans woman, was very insistent that I mention the biological difference of trans people's brains as a core part of the definition. (https://lockebesse.medium.com/a-for-effort-feb3c6cdce07)
In your conversation with Rex, you suggested that we could distinguish sex based on gametes. I think that's true - we could theoritecally define sex that way. But it's also not how we usually do it - at least in humans. We do it by looking at the genitals in utero, or, if not then, at birth.
You could argue that looking at the genitals is really just a convenient heuristic for predicting which gametes a person produces.
On the other hand, you could also argue that looking at the genitals is really just a convenient heuristic for predicting which brain a person has - and whether that person will see themselves as a man or a woman.
And by the same token, a person's gametes could be thought of as just another heuristic for a person's brain sex - a heuristic that isn't 100% accurate - like in the case of trans people.
When it comes to non-human organisms, we don't have any way of knowing if their brain "matches" their gametes or not. But with humans, we do. And since a person's brain sex is such an important part of identity, perhaps, for humans, the way we have to negotiate the language is by calling trans women "females" and trans men "males".
To be clear, I do recognize that this would constitute a migration from how we used to use language. And I do, as I assume you do, find it frustrating when people try to migrate our language without acknowledging that they are doing so.
But, as I said, language is a democratic process, and sometimes, you have to go with the flow a bit, and accept that ultimately, the goal of language is just to communicate, and the most important thing is just that you know how to communicate in a way that best facilitates understanding - even if it's not exactly the terminology that you would have chosen.
