It might simply be that you feel like people will look at you differently depending on who you're with. May I ask how old you are? I somehow remember you as being pretty young but might be getting you mixed up with someone else. Just because if you are, I remember until I was about 16 I felt that if I wasn't hanging out with the 'cool kids' then people would think I was, well, uncool, or not worthy of attention/to talk to etc. So I would feel much more confident if I was hanging out with the popular bunch. I also think that has to do with people just being so judgmental in the like 12-16 age range IME. Once we got to the end of high school people were much more mature and it really didn't matter as much how you looked or who you hung out with; and now that I'm at uni I've learned to seriously not give a damn if the people I'm friends with are 'cool' or not.
So just to say you might grow out of it.