Everything's toxic if you put enough of it in your body. Table salt is toxic, even water is toxic.
The ratio between the recreational dose and lethal dose of LSD is one of the highest, maybe *the* highest of any recreational drug. No one knows the LD50 of LSD in humans, but in rats it's supposed to be around 16.5mg/kg, which is a tremendous dose of LSD. There are humans who have consumed hundreds, even thousands of doses in a single setting and have lived to tell the tale. Essentially, LSD has extremely low toxicity. This is very well studied and accepted by the scientific community.
However, just because it is for all practical purposes of low toxicity, doesn't mean it's without risks. I think Xorkoth summed it up nicely above.