Hey SJB, I relate to the other parts of your post too, but this part I was wondering: is it because you let yourself be vulnerable with them and they accepted you moreso than others?
I think it's mainly because I've never been comfortable spending time one-on-one with another person, and drugs or romantic interest are the things that can to some extent overcome that. I suppose it's essentially derived from a discomfort with allowing myself to be vulnerable. I'm a deeply self-reliant person and I find it difficult both to rely on others and to be a person another relies on (not that I'm unreliable -- quite the opposite -- but it's a position I'm instinctively uncomfortable with).