Like many others, I had to give up my daily MJ habit as my anxiety became worse and worse. I only smoke it once a month now. If you must smoke cannabis, I'd try pure Indica strains, these strains offer more of a body high, rather than the more "heady" Sativa strains. When I do smoke, I only smoke Indica strains, they're less anxiety inducing. Even smoking Indica, I have to be in the most relaxing set/setting, otherwise I get full blown panic attacks as well.
That said, it sounds like you're better off giving pot a long break. Try replacing pot with an exercise routine. It sounds lame, but exercise now gives me the euphoric relaxation that pot used to have for me.