For me, Xanax, horrible, I ended up having a seizure while driving, totaled my car and someone else's and ended up in the hospital. I don't even remember the accident, there's about 7 minutes that I can't remember at all from that night. Luckily I only broke my wrist and sprained my ankle and the other person wasn't hurt at all. But before the seizure ever happened, the confusion, the constant panic and restlessness drove me crazy, then I started vomiting, stomach pains, just feeling like I was going to lose my mind completely. All this happened after coming off of 1-2mg of Xanax daily for 2 years, I went 3 days without it cold turkey all because of a mix up at the pharmacy (some kind of miscommunication between the doctor's office and the pharmacy). I didn't think I was at risk for a seizure, I was driving to pick up my refill when all this happened. Had I known I could have had a seizure I wouldn't have drove.
I've withdrawn from various opiates, mainly poppies, hydrocodone, oxycodone, suboxone, and they were hell to come off of too, but I'd never been on any of them for longer than 6-9 months at a time. There is just an element of psychosis that came along with the Xanax withdrawal that puts it at the top of my list.
So, Xanax and then opiates. I'm on Klonopin now. I can't seem to get much lower than 1.5mg Klonopin a day without feeling extremely on edge and uncomfortable.