Generally, I think it really is that way. Maybe It is because Xanax and the likes have rapid onset and leave your system quickly, so you feel the need to take more after some time and hasn't it always been this way, that drugs that hit you the fastest and wear off quickly are the most addictive (3-methylfentanyl and Xanax in my case).
When I was taking Xanax, 3x1mg a day because the half life was 6 hours to cope with my anxiety and let's say I missed a morning dose. I could already feel it after 2 hours of the time I usually took it, the slight withdrawals/uneasy feeling creeping on and I had HELL coming off Xanax. Its meant for acute panic attacks but not for maintaining ur anxiety daily IMO.
Now I am on Clonazepam which many people regard much stronger than Xanax, it has a half life of 18-50 hours.
I take 2mg every morning (sometimes at night too, I am currently tapering Suboxone) and I can skip days without feeling any withdrawal, and coming off Clonazepam is basically a walk in the park for me. I have done it like 3 times after taking it for like a year every time and then having a few months pause.
When I am coming off Clonazepam I first start to skip days, take Monday, Wednesday, Friday, Sunday... then second week I already reduced my dose to 1/2 (1mg), still taking every other day and next week or even earlier I take a quarter 1/4 every other day and then I just stop. And all these times I have never felt withdrawal, maybe just a little bit higher blood pressure and some anxiety and thats it. I have even done it not even reducing dose after a week, but cutting it half every 3 days and results are the same. Walk in the park really.