Here's some advice, YMMV.
Start out VERY small like just a single hit from a one hitter/small bowl, and see how that is for you, and don't smoke anymore. Then keep smoking more often little by little, and try to get into a set pattern like at first on weekends or every few days and keep smoking more and more, just enough to get high and that's it. Eventually the anxiety you get from smoking pot will subside if you do it right. You may become a pothead/daily smoker eventually but if you do it right, you will stop getting anxiety from smoking since it will become normalized for you and not a big deal.
Pick and choose when you want to get high, like for example if after you smoke you get anxiety/panic attacks when you have to appear sober around strangers at a party or in public, don't do it then.
Alcohol can help but some people don't like this combo since it can make things worse, make them feel sick/flushed or like they're going to puke.
If you get anxiety/a panic attack just keep breathing and tell yourself it's just pot anxiety/a panic attack, and that it will be over soon eventually. Like someone else wrote in their reply learn how to talk yourself down from one.
I know it sounds corny as hell but meditation and daily cardio exercise really helped me with anxiety and some people have found that drinking tea like Chamomile and Rooibos/red tea can help with anxiety. Some people get anxiety from the food that they eat or for other reasons but some people are just wired for anxiety/panic attacks and a lot more people are than you'd think.
If pot makes you get REALLY bad panic attacks like to the point where you can't stand it, and it's pretty debilitating, perhaps stop smoking?
Avoid strains that are pure sativa or sativa dominant, as these can give people bad anxiety/panic attacks. Perhaps get a lot of one strain you like and just smoke that? Or start out with just a hit from a new strain until you get used to it and can tell what the high is like?
If you have a low tolerance don't try to keep up with your smoking buds/friends who you smoke with who have a huge tolerance, as sometimes when you smoke too much it's not fun with the body load, and panic attacks happen then.
Don't smoke when you're all stressed out/angry or in an anxious mood.
Avoid stimulants like caffeine/coke/meth/prescription amphetamines/MDMA/MDA, cut back, or don't smoke when you use them. You know yourself and your body best.