its because upper is slang for CNS Stimulant, or central nervous system stimulant. Even though you might feel "chill" on cocaine, Your heart rate elevates, etc.
Downers is slang for CNS Depressants. They slow down your heart rate, breathing, etc.
Uppers sometimes "chill" people with ADD out. I would not call pot an "upper" or "downer".
Perhaps some benzos, barbitiuates, alcohol, opiates, ghb etc would be "downs"
Uppers would be "caffiene, meth, adderall, coke, etc."
I hope this helps clairfy.