Why exclude alcohol? It's one of the most dangerous downers (at least when you become dependant on it, the detox can be lethal.. So quite dangerous).
I guess tobacco is very bad for your health too, but it's not 'acutely dangerous even if you are addicted to smoking. Quitting is hard and will make you go nuts with cravings, but other than that it's not immediately dangerous.
Though I think for your psyche, stimulants are the most dangerous. As in, they can do a real number on your psychological health with excessive use. Downers are more 'dangerous' in the sense that detoxing from quite a few of them (barbs, benzos, alcohol, ..) cold turkey can kill you. But disregarding that (a proper taper is always available normally, even if you have to go to hospital for it), I think stimulants are more dangerous to play around with (at least when used at high doses & chronically). Especially stuff like MDMA, 4-FA, meth and other drugs (empathogens) that release lots of serotonin can be very neurotoxic with excessive use, and then we're talking irreversible damage.