I can relate, this is in fact the core reason that i travel alone when i do.
With that said i often attempt to create situations where i have control or at least maintain control in some form, i'm the kind of person who always has an exit strategy. In social circles i always maintain a sense of independent self.. i can cooperate with people and hang out but my sense of self is not defined by the social circle.. i can leave or stop contact and it has zero impact on me as i'm not invested. This has been paramount for me in my process of learning detachment.
I think my need to control is very much tied up with my desire for freedom, i hate feeling trapped mentally, emotionally or physically. Relationships have been a monumental struggle for me because of this very reason.. i want it right up until the very moment i have it then i feel trapped. I seek out unavailable women because of this.
Virtually any person, place, environment or situation where i feel trapped.. the need for control arise's. I think this could be applied to everyone to varying levels.. i do remember times when i was more carefree especially when the crowd i was hanging out with was very hippe-esque, however i found myself in too many undesirable situations that seemed to have resulted from a distinct lack of planning and structure and i became fed-up with it.
I've pushed my boundaries though and i find if i can get into the momentum of allowing everything around me to simply happen without attempting to control it i can fall into a more relaxed place.. last year i hitch-hiked for the first time which was so far out of my comfort zone and probably the worst position to be in for someone who wants 'control' hahah.. but it turned out to be an enriching experience and i suppose i learnt to trust in the motion of things a little more..