Sparring experiences with aggressive men as a woman

I am a 120lb female and I often spar with men and I genuinely do not expect them to baby me and hold back IF we agreed to go hard. But I have been dealing with a handful of men who, despite agreeing to go light, still are purposefully hitting with power. I still keep the round light because I feel like me also hitting with power is only going to bait their ego into going harder. This is only with a handful of men though, so not the size difference factor clearly. Any women here experiencing something like this? Honestly not sure what is their problem because I do not do anything to provoke anyone.