I'd heard of the lower sperm count concern being associated with plastic about 10 years ago. Polymers of plastic are shed into the bottled drinking water that's being consumed in much greater volume as trust in tap water declines, and humans didn't evolve with plastics as a component of their diet. Not that plastic contamination is the foremost suspect, it's just one of a bunch.Pollution may be to blame for collapse in sperm counts in industrialized world… (full text here).
I also think it may be discussed as an issue affecting men's fertility more than women's because the sperm count aspect of fertility is so much easier to check for than to attempt to diagnose how a woman's fertility is impacted. Sperm count isn't the only variable effecting men's virility; there's also sperm motility, and it's down, too.
Peace.