Yes, what you describe is exactly how I felt abusing pot a long time ago, especially when the weed started being made really strong. I had a friend who's dad had a bunch of plants, he thought we didn't know but we would just pick the leaves and a joint would get us nice and happy for a half hour.
By the early 90's there was stuff so strong that you might be high hours, and that's when it becomes a problem. That and a few strong trips made me psychologicaly scattered and shattered and the panic attacks would come more often.
It took only a few weeks of quitting to feel better but I had the attacks until one day after having been thoroughly checked and blood tested by the dr. I had one and decided that I was going to go jog it off, and if I had a heart attack then so be it. I was fine, and the only thing that gives them to me now is when I overuse meds or otherwise do things to precipitate them. Live clean and you will be fine in a few weeks, indulge infrequently or not at all and you will stay fine. Get bloodwork done to put your m7nd at ease. And don't let the dr. talk you into ANY benzos or antidepressants, they will only make the problem worse. Talk to a trained psychologist if things don't change in a month after stopping but again do your best to steer clear of drugs.