I would lay off the psychoactives for a while, especially psychedelics, mdma/rave & party drugs, and cannabis as well. Spend some time sober to process what happened and gain a little perspective on your life, while allowing time for you body/mind to heal. Exercise and getting proper nutrition can really help with this process. Things like meditation, CBT, and group therapy can help with the psychological aspect.
I know it's already been 2 years, but if you've been constantly dosing since then and it induces extreme anxiety to the extent of panic attacks, it's only normal that you'd feel anxious even at the idea of dosing again. Then factor in the nature of psychedelics combined with a nervous or panicky mind, and it's probably not going to end well the majority of the time. If nothing else, I would suggest waiting until you feel confident with the idea that you'll have a good, panic-free trip. Maybe reflect on all the good times you had before the bad trip- and if that doesn't work, maybe consider listening to your body/mind and step away from the psychs for a while if not for good.
Personally, after abusing psychs myself, I have a similar reaction. After having experienced a psychotic break, tripping feels eerily similar, and it's something I find less and less interesting as of lately. Every now and again boredom spurs my curiosity and I'll try some again, but I won't touch them without benzos in arms reach for the same reason as you. Only thing that has really helped me get comfortable with them again is microdosing after a long break(and being in a healthy mindset). Even then, the good experiences I had aren't something I'm eager to repeat. I feel now it's not worth playing with my psyche for the sake of a good time, at least not on any sort of regular basis. I'm still open to the idea of tripping again at the "perfect opportunity", but I have no idea what that would really look like, I just now it'd have to start with me being 100% comfortable with the idea of tripping, and in a safe environment but also a somewhat special occasion.