Ketamine is not physically addictive and even if you consume huge amounts over a long period of time, the only thing that it can do physically to your body is damage your bladder - and the jury is still out on that as it seems some people are predisposed to it or it's only certain 'types' of K. I'm sure there are reports of people (John Lilly?) using it daily for a year without any long-term side-effects or withdrawals.
It's considered relatively 'safe' because it has such as wide dose margin. It's virtually impossible to overdose on. You could, for example, take 10x the required dose of ketamine and be physically fine - could you say the same about paracetamol (acetaminophen)? Not only that, it is also given to children and the elderly as well as accident victims.
Finally, regarding neurotoxicity:
http://www.drugscope.org.uk/resources/mediaguide/neuroscientific: "Cigarettes, which have only weak mental effects, are amongst the most physically harmful (they damage the brain by dramatically increasing the risk of stroke in the long term) while LSD and ketamine, which produce profoundly altered states of being, have perhaps the least lasting effects upon the human brain."