Ridiculously high doses (the equivalent of eating close to 2 kilos of dried shrooms) could theoretically be toxic - that's the LD50 point for rats relative to body weight. There are many, many drugs for which such a comparison is useless.
As others here have pretty much explained, no human being is known to have died because of any toxic or otherwise physically harmful effect of psilocybin or psilocin, the active ingredients in shrooms. There are certainly other varieties of mushroom that are poisonous, but if you have one of the various edible strains of psilocybe mushroom, aside from a few obscure medical contraindications (wikipedia says don't combine it with therapeutic doses of lithium as this can apparently cause seizures... lithium is weird stuff from what I've read), there's no reason to believe they would be likely to cause any lasting physical damage, or any acute negative symptoms beyond perhaps a subjectively difficult trip and some vomiting or other short lived symptoms like that. I've had some rough times on mushrooms, but a bad trip is pretty much as bad as it can go aside from doing something stupid or getting arrested or something while tripping.
So, short answer: no, psilocybe mushrooms don't poison your body in any way unless you consider the trip itself to be a "poisonous" effect. They can, however, be difficult experiences and you may regret the experience in retrospect, though I am grateful for most of my mushroom experiences, including the most intense of the 'bad', difficult ones.
