It really depends what you mean by toxic I suppose. From a dependence point of view, I'd put benzos and opiates up there, with benzos taking the top spot; these things can linger for ages, with withdrawals not only lasting for up to 6 months, but they can also be potentially fatal-I'd call that pretty toxic. We all know how addictive opiates are, but in terms of toxicity, these drugs are very mild on the body; they have no long-term negative effects on any major organs.
I really doubt SSRI's are toxic; I've been taking them for eight years now, and recent liver and kidney function tests came back in the green, with very healthy results. I've heard horror stories of people coming off them, but I decided to stop taking them for six months, and felt perfectly normal. I really couldn't comment on the toxicity of amphetamines-I've used them a lot, but its not a drug group I know a lot about. For what it's worth, I was a heavy speed user for 2 years, and I used a hell of a lot of MDMA, until it lost it's "magic"; this was 8 years ago, and I've experienced zero long-term health effects, be it mental or physical.
One drug I've found extremely toxic is alcohol; it just destroys your liver in the long-run, among many other negative effects on health. It's a caustic drug, not to mention DT; in extreme cases these can also prove fatal.