What drugs are you talking about specifically?
I can tell you almost any drug will likely cause a decrease in workout quality, motivation, recovery, strength, etc.
If one cannot moderate substances properly the gym often gets left behind.
Things like stimulants cut into much needed appetite for eating proper nutrition to fuel gains.
If you can moderate and maintain things like sleep, eating and working out, drugs won't really get in the way.
If you can't maintain these three things: sleep, eating and working out frequently, than ultimately your fitness will suffer greatly.
Edit: But like I said, plenty of serious bodybuilders / high level professional athletes use drugs for fun. If you can hit the bar friday night and blow down an eight ball of Cocaine, but come Sunday you're eating a 3 egg omelet, running a 5K and hitting a 2 hour weights work out at the gym, followed by chicken and rice dinner and in bed by 11 PM, you can be looking as good as Arnold Schwarzenegger in his prime.