Would you be just as unhappy if he was spending all this time, for example, playing video games, rather than using drugs? I feel like some of your worry over the drugs comes simply because they're not something you're very familiar with. I can imagine why you'd be really concerned over the effects when most of the readily available information about drugs demonises them and hypes up their danger. In a way, it's a big unknown for you - how he reacts to them, what his behaviour is like on them, whether it's risky, and so on - and it's so easy to fill in the blanks with worrying and fear. I also think him having this whole world which you're not privy to could lead to feelings of insecurity. I think it can be hard when you feel left out, and that his friends get to share a part of him that you don't. I do think a lot of this could be dealt with by you becoming more informed about drugs, and familiar with the reality of them - so that they lose that mytique. Though, this may depend on him being willing to share some of those experiences with you.
It's hard to know whether your boyfriend really is prioritising time with his friends/drug use over you, or whether he's trying to keep the peace. Given that you are happy with him otherwise, I think it's worth trying to show him you're interested in learning more about drugs and his use of them. I think that even though you say you're not stopping him from doing them, you're not morally against them and so forth, he's easily picking up that you don't really approve either, and rather than face this, he's choosing to not tell you. Perhaps, though, if you do some research of your own accord, keep an open mind and let him know that you're learning more about it because it's something that's a part of his life, he might become more willing to open up about this. I think approaching it from a position of learning more about it (rather than finding out information to justify why he should or shouldn't do anything), can only be beneficial for you. I probably wouldn't ask to come along to one of his sessions until you're both a bit more comfortable with this issue, otherwise I can see a negative dynamic occuring during it. Then again, he may just not want to spend as much time with you as you do with him, but I think it's worth trying out some other approaches before coming to that conclusion.
