Pot tends to enhance however I'm feeling...so if I'm already anxious, I'm probably just going to feel more anxious. My only full-blown-OMG-I-wish-I-wasn't-alive-and-feeling-this panic attack was due to me already being extremely anxious and smoking pot in an attempt to calm me down. I only get paranoid when I'm in a tricky situation, like hot-boxing a car and seein the NYPD roll by.
OP, you say you're an occasional smoker, so maybe you've just gotten lucky enough to never have blazed while already anxious?