I think you could make a strong argument that it's not their choice to have those desires. I mean think about your own sexual desires: did you "choose" those desires, or did they seemingly appear more just as a natural development of your sexuality? This is especially apparent in the case of pedophiles because they're basically the most hated people in our society and no one would "choose" to have such sexual desires.
However, while it isn't your fault if you have a specific sexual desire, even a deviant or morally transgressive sexual desire, it IS your fault, and you SHOULD be held accountable, if you act upon those desires and hurt another person! I don't think that it's difficult at all to hold such a position.
And just to be clear, I'm well aware that just because I can't see something in "real life", that doesn't mean that it doesn't exist. Just because I don't see any pedos, that doesn't mean that they don't exist. Just because I can't see any twerking, gyrating trannies or whatever, that doesn't mean that they don't exist. Or (more relevant to left-wing anxieties, probably) just because I can't see any Neo-Nazis or avowed white supremacists, that doesn't mean that they don't exist. However, if you're taking on the position that pedophilia, of all things, is being NORMALIZED, you should be able to see SOMETHING to indicate that in the wider society I would think.