How young? As for me, I don't care if you're a racist, a pedo, wanna kill people, etc. — so long as you dont act on your fantasies that are bound to hurt others, it's fine. There is nothing intrinsically wrong with you simply for being attracted to children. Having thoughts is not wrong; there actions can be. I did read a study a while ago in which researchers gathered men to anonymously find out if they were attracted to children at any point in their lives. The results was around 22% of men.
You must also consider the historical aspect of pedophelia — that is, around 12 years old or younger. In past culture, not even that long ago, it was commonplace to have sex with young people. I recall reading that one culture had a norm of a father taking his own daughters virginity at a young age. Thankfully, our moral structure has strengthened given our progress in psychology and recognizing proper human development.