Hardest withdrawal by far: benzos & barbs. I had a family member go through it. I'd rather go cold turkey off opioids for two months than that hell.
Hardest on social life: crack.
Hardest on body overall: probably Alcohol (long term overall damage)
Easiest to die off: barbs - If you go for ones used today, I'd go high potency opioids such as fentanyl.
Hardest drug overall: crack, closely followed by meth.
I do not rate opioids such as heroin as "hard" drugs in the same category because though a lot of problems come through its use, most problems are external (law, acceptance), whereas crack transforms you into a monster. Note that I rate them as hard drugs in context of withdrawal, use, and danger overall.
If I'd to include all drugs, also legal ones which are not abusable, its without a doubt neuroleptics and similars.