I think it's largely due to two factors. First and most significantly, being a "holy man" means you are a gatekeeper to the most profound form of social control available, which is a person's very immortal soul. That position, like all other positions of power (police, politician, etc), attracts a disproportionate percentage of people who seek it specifically in order to have that power over others. One of the main areas in which people purposely abusing positions of power will exert their influence is to exert sexual control over people. These are the kind of people who created these systems of control.
The other big reason, due to the sickness of the people who devised the system of control, involves adhering to, in many cases, an unhealthy position on sex. See, for example, catholic priests, who are expected to completely forego sex, and are left in a frustrated, unnatural position, coupled with remaining in a juvenile, childlike mindset about what sex is. This sort of repression can easily lead to someone's unfulfilled sexual desires manifesting in sick, deviant ways. The people who succumb to this are themselves victims of the system they are helping to uphold, and likely did not intentionally become dangerous creeps.