Probably, part of what you’re seeing with yaoi being treated the way it is is that men and their sexuality are basically constantly under attack in the culture war as creepy, oppressive, objectifying, low-status, etc.
But of course, as the sex with higher average libido and a more visual sexuality (because hormones), they are far more vulnerable to these sorts of attacks. Someone who has no sex drive at all can basically dunk on someone who does have a sex drive all day long, portraying them as evil and objectifying.
Of course, women (including lesbians, as TUOC has posted about) are not actually pure angels, despite what the culture warriors say.
Yaoi then, under this framework, is an exploitable vulnerability to snap back at the attackers and make them shut up - a weakness in the social justice armor.
And for those who have a Correct sexuality under SJ terms, it can be exploited for social standing.
Then of course there are the people that are actually annoyed by it. Not all gay men, for instance, really enjoy straight women coming into gay bars.