I understand exactly what OP is going through! I haven't rolled since 2000, and I STILL have major anxiety issues (I was diagnosed with Panic disorder with agoraphobia about a year after I ceased using). I too had some anxiety issues before my MDMA use, but I noticed it getting gradually worse as my drug use continued. Fast forward a couple years and I was a total mess. I would be woken up nightly by panic attacks, I was in a constant state of depression and I totally closed myself off from any social activities as anytime I would even attempt to leave the house I would have panic attacks to the point I was experiencing depersonalization. I quit my job, lost A LOT of friends, and I alienated my family. My relationship suffered greatly too as my partner continued doing drugs and just didn't understand why I had issues and he did not.
I stopped all drugs except for weed, started doing CBT, tried about every SSRI/SNRI you could think of and nothing helped; I only seemed to get worse. Benzos were the only thing that did help, but we all know where that leads! So here it is, over 13 years later, and I still have horrible anxiety. I talk with quite a few people from our crew from back then and the "funny" thing is, they ALL have anxiety issues of some type now. Some are very minor (mostly the people who had no history of anxiety or mental problems before use) where I and a few others seem to be totally consumed with anxiety issues. Do I think MDMA caused these anxiety issues for me? No, I do not, but I do think it made my anxiety worse, maybe triggering underlying conditions I had that had been suppressed from my drug use (self medication). I really didn't realize how bad my anxiety disorder was until I stopped doing MDMA. While on it, I felt amazing, it was always the comedown, which would make me want to do more so I wouldn't feel that way, it was just a vicious cycle that didn't end until I did a bad combo at a party one night and well that's a LONG story but it made me realize what all these drugs were doing to me and how I was actually just medicating myself, trying to escape the constant misery that I was feeling. I think most of us do drugs for a reason, there is an underlying issue (depression, anxiety, PTSD, etc) that causes us to seek out an escape from the reality of our condition and/or problems. While sober, those problems come back, usually tenfold, and that's when recreation turns to abuse. And then you have a whole other set of problems to deal with.
I guess my advice would be is to firstly, realize that your drug fueled partying days are now over. You have already convinced yourself that drugs is causing you anxiety, so now every time you try to do them you are going to likely have anxiety. I know how this works all too well, I used to call it pre-drug anxiety, as I would get horrible panic until the drugs would kick in, then I'd be great, and then on the comedown I would have horrible depression, I would be very hard on myself for using again and the whole downward spiral would start up again. If you haven't already, talk to your family doctor about your anxiety and see if he/she thinks you need a referral for a psychiatrist and/or therapist. CBT did help me but it did not cure me of my anxiety issues but it does help, it just doesn't happen over night. I now manage my anxiety with a cocktail of prescribed drugs along with monthly therapy and a GREAT support system of friends who understand and are very compassionate of my disorders. Having REAL friends who truly understand and care will help you more than any drugs or doctor will for your anxiety problems IME. Sometimes it can feel like it has to get worse before it gets better, but you WILL feel better eventually if you take care of yourself and get the help you need. I believe I have done permanent damage and will now always have to live with my anxiety disorders but at least now I have the skills and tools to manage them and have a fairly normal life most the time. Don't be afraid to seek out and ask for help, and don't put it off!