Another issue I've encountered, and some of these things have only come with years and years of usage, is that groups of people have a major impact on how much of a given drug you're actually going to consume, especially with something like Cannabis. There's not a high level of titration involved when smoking with groups of people. Often, a joint is rolled or a pipe is packed and everyone smokes cylically until it's done/empty. Being that it takes over several minutes to reach peak effects from inhalation and only 1-2 minutes to pass a pipe around a circle, you can easily lose track of your intake and become too high.
I guess my point is that maybe you are medicating symptoms of anxiety with your Cannabis use, but the fact that you're with other people and trying to keep pace with them is causing you to consume too much, leading to your anxiety problem.