That's a classic problem in these subject matters - if you say that even just PART of a disparity in behavior or preferences between men and women is due to biological differences (and often on a spectrum), people WILL say that you've argue it's ALL due to biology. https://twitter.com/jessesingal/status/1360626887035338752
The Google Memo wasn't a "screed" or a "rant" - it was actually well source and cautious in its claims. This doesn't necessarily mean that it was right - but making cautious, well-evidenced claims is how you discuss complex subject matters like differences between men and women.
If someone thinks that psychological traits are clear-cut bimodal between the sexes they're obviously wrong. The height distribution between the sexes isn't clear cut bimodal either -there are plenty of short men and tall women. But the distributions aren't the same!
It's not IRRATIONAL to believe that when you have differences in distribution between the sexes (which, again, for psychological traits, like other traits, are NOT bimodal!) at least PART of those differences are of a biological origin. On a surface level everyone agrees to this.
But in practice quite often if you suggest - even with evidence! - that this is the case you're often misrepresented as having said that ALL the difference is ENTIRELY biological, and even allegedly bimodal.
This isn't productive to a healthy discussion - you get tied up in stereotypes, culture wars, strawmen vs strawmen, and demonization of each side - first of those who extremize other people's claims, then of those who lionize other people's claims. It leads us nowhere.
People aren't just charicatures! The world isn't divided in chuds who think that women should stay in the kitchen and people who think that biological sexes don't exist and are just colonialism - SOME people like this exist but they're NOT everyone of us, not even the majority.
Even IF there are biological components to differences in distribution between the sexes, they're NOT strict Platonic categories! A short man is a man. A tall woman is a woman. Men on average are taller than women. ALL those sentences are true - it's a matter of distributions!
A BIG problem is the idea that there is some sort of psychological "essence" or "authenticity" to "maleness" and "femaleness" - that's NOT true. There are also PLENTY of cultural variations in terms of social expressions of psychological traits - denying it is stupid.
There is, however, SOME evidence that SOME distributions in traits seem to be, to SOME extent, cultural universals - one example is a greater disposition towards violence in men: https://twitter.com/Evolving_Moloch/status/979528091910287360
So it's not irrational to gather evidence that SOME degree of psychological differences between the sexes IS independent of culture - although it might be expressed in different ways in different environments. It's absurd to say that this is equivalent to Platonic categories.