I know EXACTLY what you're going through
I was in your situation a few years ago... doing a lot of the drugs you listed but mainly opiates/synthetic opiates like oxycodone, suboxone, etc. I live in a semi-small town and a lot of my high school friends and their families knew I was involved in selling/buying the aforementioned drugs. Before this time I wasn't generally anxious, and I was using drugs in the years leading up to this point, but one day the anxiety started to grow. It seemed overnight I was unable to smoke marijuana because I would have panic attacks, and I had general anxiety issues while not smoking pot as well. I was in the ER a few times due to panic attacks mainly because I thought something might be wrong with me. I would tell the doctors "I've been smoking pot and using opiates for the last 5 years.... why all of a sudden I cant smoke even a TINY bit of marijuana without having a panic attack?" All they would do is say "well you need to quit doing drugs", give me a xanax and send me home. I quit smoking pot but I don't think this has very much to do with the anxiety. What I noticed caused a lot of anxiety were my own thoughts. Constantly thinking about how others perceived me (like you mentioned) as well as physically walking around with drugs on me, worrying about police while im driving and what-not. After about 5 years of worrying like this it just continues to grow on you I guess. So I started to stay away from as much outside pressure as possible. I played a LOT of video games lol, and mainly just tried to stay away from people I knew, family (wherever possible), etc.....
Today I'm still on suboxone, but nothing else. I went from daily anxiety/depression and occasional panic attacks, being extremely nervous and anxious in class and around other people, to no panic at all and actually enjoying daily life, looking forward to each day.... I've learned to think a different way, a way that doesn't involve giving the smallest SHIT about what other people think. No other single person (except for maybe family members) are as important as your own well being. If they have an issue with you then just know you're capable of telling them to fuck off and not let it bother you at all. The main thing I tell myself when facing a potential anxiety-inducing event (issues at work, issues with school work, problems with your grades, etc) is that none of this is as important as your own mental health and well-being. If you fail a class so what, if you lose your job it wouldn't even be the end of the world, you can go out the next day and apply for 20 different jobs. Nothing besides deaths of family/friends, issues with your own physical health, and other serious events should be the only thing you need to focus on or even feel the slightest bit anxious about. 99% of your mental state is controlled by you... and with some prolonged positive thinking you can fix your current negative state.
I don't know if you read all of this but this is my understanding of how I was able to overcome the anxiety I was experiencing, and it was pretty intense anxiety in relation to what others have experienced (from what I've read on here at least).