Zacamo, people have had a stroke and then gotten honors degrees. Are you gonna argue that strokes aren't neurotoxic?
Very reasonable doses are toxic to neurons in culture, and no, not everything is toxic to neurons in culture, although you are right to a degree, they are definitely more sensitive.
Of course the doses of THC that are needed to damage a mouse or a rat are higher than those given to humans, but that goes for pretty much all toxic chemicals.
I mean, the doses of THC they usually give to rats (10-20mg/kg) doesn't usually make them act in a noticeably different way, but that same dose would knock you out.
The real point is, we don't know, one way or another. The statistics are very inconclusive as to whether cannabis use effects in a (semi)-permanent way.