I think it comes down to how and why you do it. Your motivations for performing an action and the results of performing the action are what determines the morality. For example, I think it could be argued that a dealer who sells ecstasy at raves, tests his product for purity and potency, only selling clean MDMA and always informing his customers of the potency so they can gauge they dos,e is doing a moral thing, because the people who are buying from them would otherwise end up buying more dangerous products from less scrupulous dealers.
Honestly, I sometimes think that the most immoral thing I've done as a drug user is simply be a drug user. Drug dealers will always have customers, but drug users influence the people around them on a deep level. I count all the people who never touched drugs until I told them how great they were or invited them out to a club with me, who saw me as an example of a "recreational user" until I fell apart and so underestimated the addictiveness of certain drugs, who had never had the slightest interest in IV drug use until they saw me do it, and the tally is pretty large. Like most people I wrote it off with the old "it's their decision, I didn't hold a gun to my head," but the hard truth of the matter is that if I hadn't been using certain drugs in certain ways around certain people, those people would be better off in their life or would never have taken risks which could have potential lead them down a dark and dangerous road. I didn't hold a gun to their head or control their behavior, I did make my own choice to expose them to chemicals and ROA's they likely otherwise would not have been exposed to.