Marijuana has not been linked to lower testosterone as far as I know, I believe that is a myth. It can cause a decrease in sperm count. Alcohol on the other hand, has been shown to cause a drop in testosterone levels, as well as other anabolic hormones.
Still, alcohol will probably not have much of an effect, unless you are getting trashed a couple times a week or more. Weed has a much more pronounced mental effect, in my opinion. After a night of drinking, I can go in to the gym and workout just fine. If I've been spending a lot of time smoking pot though, I'm just generally more lazy and unmotivated to do anything, and I definitely noticed these problems in the gym that could be attributed to smoking pot more often.
That said, I actually lifted a few times after smoking, and had great workouts...but, the one piece of evidence that has convinced me that weed and exercise do not mix is just that since I quit smoking (about a year ago now), my strength has been at an all-time high, and I have been making gains much more consistently. I still drink sometimes, but this never has any real noticeable effect on me like smoking did. Of course this may differ with other people, but this is just what I noticed for myself.
Also, just look at the people you know...most good athletes who I have known personally do not smoke pot on any regular basis. They might drink, but most of them seem to know that smoking is bad for their conditioning. Most of the people I know who are real potheads have more feminine physiques, are skinny as hell, seem to have no natural athletic ability...not to say that pot CAUSES this, but I have noticed a correllation.