It could make it way worse or way better.
I'd say take a hit, just one hit, even if it's not very good weed (people who don't smoke can often get a decent buzz from a hit of even shwag), then wait 15 minutes and go from there. There's no way one hit will send you into a panic attack unless it's very good weed.
I have some anxiety too, not necessarily "anxiety disorder" but I can be jumpy and neurotic sometimes for sure, especially in social situations. Weed was hit and miss with it at first, it would either make me wrapped up in my own head and thus relaxed or extremely anxious, and as my tolerance for it went up it became excellent for it. When you smoke for a while the anxiety stuff goes away, but obv. that's not a real option for you.
If you do go into panic mode, keep a level head. You can rationalize your way out of a panic attack if you're good. If this happens your heart will appear to be beating dangerously fast. Just remind yourself that the THC makes it *seem* like your heart is beating faster than it is, and also that a healthy person can sustain a fairly heightened heart rate. Nobody's ever died from cardiac problems while smoking (I have seen one or two examples in the literature but meh that could very well be coincidental and both people had pre-existing cardiac disorders).
You also might feel very paranoid about the people you're with. Be honest with them before you smoke, just tell them that you have anxiety disorder and that if you start acting odd or withdrawn or scared that they should talk nice to you and "talk you down". That can help with social paranoia from weed, once you see the people you're with being very nice to you that's usually (usually . . .) enough to get rid of most of the paranoia.