If his anxiety true revolves around large groups of people, maybe not get stoned with him in that situation idk? I feel a little like this kid, so I sort of cringe at all the situations of "weed is not for him" and stuff. If that is true, it is something he would need to decide on his own. If he is anxious around groups, maybe smoke with him alone next time? It seems like you should talk to him more than us really, I would ask him what made him nervous, what setting does he enjoy being high in. Maybe he will have a good answer, or maybe he will realize there are too few.