I've had a very similar experience. I rolled maybe 9 or 10 times, but in a shorter time frame (about a year, with one isolated use 10 months later than the rest). Now, I'm not gonna say I was anxiety-free before MDMA -- even before ecstasy, my brain NEVER stopped thinking -- but I never had the constant state of worry that you're describing until I did X. I was a daily weed smoker before I started rolling, but once I had rolled 2 or 3 times I started to get extremely anxious on weed. I ended up quitting weed because it felt worse than being sober.
It's been 2 years since I rolled or smoked weed, and over a year since I have drank. I did have an addiction to suboxone for a year (taking it every single day), and that definitely brought its own complications and slowed down my recovery, but I'm 6 months removed from that and I still believe the drug that did the most damage to me is MDMA. The anxiety from coming off of opiates is completely different from the anxiety I got after MDMA, and while I can feel that opiate-induced anxiety all but disappearing, I still have the panic attacks that I associated with molly. I'm learning how to deal with them, but it is a slow fucking process. The best thing to do is to stay busy and when you start to feel anxious, just DO something; ie: go to the gym, play keyboard, anything but sit still. A lot of times, that works. But sometimes it's so crippling you can't do anything but sit in your head. If you've found a way to deal with that, let me know, because I've just been trying to suck it up.
I can also relate to being super sensitive to stimulants after stopping MDMA. I used to smoke cigarettes and down red bulls like nothing. Now, if I want caffeine I have to drink the smallest coffee available and I still don't finish it 90% of the time because it puts me in a state of constant panic attacks. I also quit cigarettes 6 months ago because the nicotine just made my anxiety 10x worse.
The only drug I take currently is propranolol (a beta blocker), which is supposed to help with anxiety. It worked at first, but after you start taking it every day it just becomes something you need, not something that really helps.