Alcohol: This used to be no problem for me, when I drank regularly. The first few times I drank and drove, it was very difficult, but after a while I did fine. This, of course, is horribly stupid.
Acid: The biggest issue here is the peripheral visuals, it often seems like someone is about to cross the street in front of you or something. Parallel parking is also a bitch. Lights are pretty. Getting lost is a problem, too, as is being CONVINCED that you missed your exit several miles before it comes up.
Weed: Paranoia keeps stoners in line. I sincerely believe that most people who smoke weed fairly regularly are safer drivers while stoned. Even occasional smokers will probably do pretty well, but they will CERTAINLY think it's not an enjoyable experience.
Cocaine/meth/ritalin: Sweaty steering wheel, fidgeting with the seat belt. On the meth and coke I really wanted someone in the passenger seat to talk to. (Or, in one case, talked a lot to the person in the passenger seat.) Drove well on all three, though.
E: My worst driving experience was on an MDA pill (well, dammit, it was supposed to be MDMA and I was supposed to be down.) I had to slam on the breaks on the interstate to avoid hitting the car in front of me, which was going a reasonable speed. It just didn't seem to be getting closer and then I was right on its ass. It was still kinda fun though, but then again, I bet robbing a bank on acid would be really fun, too.
Benzos: I drove once on a few ativans. It was uneventful, and I think the anxiety about driving fucked up counteracted the ativans.
Ketamine: I have no idea why or how I did this (it was only a couple of keybumps, mind you. I had to borrow a key, too, cause mine was in the ignition.) It definitely did get from point A to point B, though. I don't remember many other details. This seems a good indication that it was a horrible idea.
PS I won't do some of this shit ever again, cept I will drive post-plateau on acid (8 or 10 hours in,) and don't see much of a problem with driving on weed or stimulants, as long as they aren't in the car with you. Driving on downers, dissociatives, or MDxx is just idiotic, though.