There are different forms of toxicity, ways of receiving a toxic dose, speed of effect, how fully one might recover, how and what it damages, and so on.
You wouldn't believe it but there is some kind of ethical concern with testing toxicity on humans?
Acute toxicity has to do with a single lethal dose for 50% of test subjects (LD50); as opposed to chronic toxicity which requires multiple doses. It is not tested for every substance or on every species.
Intravenous exposure to rats in mg/kg:
Strychnine - 1
Glutaraldehyde - 13
Caffeine - 105
Psilocybin - 280
Alcohol - 1440
Fructose - 15000
Just to give a sense of scale I suppose. If you look at a standard Material Safety Data (MSDS) you see toxicity listed as the LD50.
"It is incorrect to say that chemicals with small LD50s are more dangerous than chemicals with large LD50s, they are simply more toxic. The danger, or risk of adverse effect of chemicals, is mostly determined by how they are used, not by the inherent toxicity of the chemical itself." - 1993
Dose-Response Relationships In Toxicology
This is talking about the difference between something being toxic and hazardous.
People claiming certain recreational drugs are not toxic are wrong.
I would not assume psilocybin is innocent of neurotoxicity until proven guilty; lasting effects can mimic brain damage.
There is a famous saying that everything is a poison depending on the dose. I disagree. Check out the last column of the periodic table for examples.