I'd consider panic attacks and certain psychotic episodes to be some of the worst feelings in life. But for some reason I keep smoking weed every day, and 90% of the time I don't enjoy it (totally different from when I was more of a newbie). Especially if I'm on a stimulating drug, I sometimes feel panicky like you do just thinking about it.
Having zopiclone (fast-acting benzo-like), lorazepam (benzo), and seroquel (antipsychotic) near me can comfort me, even if I don't take them. Although the idea of taking one of these is very unpleasant and seems risky if I'm on drugs with psychedelic properties (can be any, from Sativa to 5-HTP), but once the benzos kick in, I just accept whatever the benzos do and it almost never goes bad. I don't use Seroquel atm. Hope I helped a little...