There are two ways that a plant turns out hermaphrodite. One is a genetic or natural hermaphrodite. At least one of it's parents were hermaphrodite, and certainly any seeds produced by self-pollination will ALL be hermaphrodite. And then there's the most common cause, and that is, people who are growing, and don't know any better, provide the plant with a wildly varying day length. In this case, a hermaphrodite is produced by an otherwise female plant by providing a short day until it begins to bud. Then turn around and go back to a 24 hour day for a few weeks, then perhaps back to 12 hours. This confuses the plant, stresses it, and viola! male flowers are produced on a female plant. This can be a good thing. It ensures that all seeds produced by self-pollination on that plant, will ALL be female for the next generation. Also known as "feminized" seeds.
That may be more info than you needed but to answer your question, the sex of a plant has very little to do with it's potency. It's the yield that is affected by sex but you're going to want to harvest a male or hermaphrodite BEFORE the pollen glands open.