The fact that it's only one way (banning men from competition in women's sports) is evidence against your point, not for it. If it was strictly anti-trans, then it would be an applied to both. The fact that no one cares that if a woman wants to participate in a men's event is pretty telling.