Men have a tougher time ultimately I think simply because men tend to have much stronger desires for regular sex while women in general can more easily get by without having sex for longer periods of time before being distressed by it.
I don't think you can make this generalization. Sex drive is highly variable across the gender spectrum. It used to be thought that women were the ones with the high sex drive - it's a social construct that has changed in the last couple of centuries. http://www.alternet.org/sex-amp-relationships/when-women-wanted-sex-much-more-men