^??? Adderall is a stimulant medication, completely different from the tranquilizer xanax. If you're experiencing fatigue the day after using xanax, it's not withdrawal.
OP wasn't clear whether or not they habitually use xanax, or whether it indeed took one use for them to feel this way.
I experienced rebound anxiety from xanax just taking it as prescribed 3-5 days per week, and it certainly took more than a couple days to feel normal again--I'd say about two weeks.
You basically have two options: continue using the substance (with the possibility of becoming entirely physically and mentally dependent on it), or stop using it. Xanax withdrawal, even from a small dose, is quite severe. I'd recommend you check out early.