Dissociatives definitely cause brain damage if used heavily.
One sub-anaesthetic "line dose" of ketamine, an equivalent dose of PCP, or a third plateau DXM dose, is probably at least as damaging to your brain as a few day "bender" on hard liquor, and possibly more so because it affects specific areas of the brain.
The risk of brain damage is worse the longer you stay high at any given time; constant moderate-dose use is probably just as damaging as a brief, high-dose use.
Reaching the anaesthetic level is exceedingly hard on your brain.
Ketamine is probably the least harmful, PCP the most, and DXM somewhere in the middle, but this is a rough guesstimate. Nitrous oxide is brief acting, but it too may be dangerous; it is also known to damage both central and peripheral nerves by depleting vitamin B12
Some people may be more susceptible to Olney's lesions than others. There is, to my knowledge, NO way of knowing how susceptible you are.
In addition to brain damage, these drugs can also trigger psychosis, limbic seizures, temporal lability, depression, and other neurological and psychological diseases much more frequently than other types of drugs. The dissociatives can be highly addictive to a minority of users. In comparison, the marijuana and the serotonergic psychedelics (LSD, psilocybin mushrooms, peyote, DMT) are many times safer.
People who have used dissociatives heavily have shown clear evidence of brain damage. This is not necessarily conclusive, since the people who become addicted to them might have underlying conditions (specifically, temporal lobe complex partial seizures) which could be responsible for some of the damage. Nonetheless, I can't ignore the fact that most everyone who uses dissociatives both frequently and heavily ends up with some sort of neurological or psychological problem, ranging from impaired memory to a schizophrenia-like syndrome. Many of the impairments correspond exactly to the areas of the brain damaged in lab animals.
http://www.erowid.org/chemicals/dxm/dxm_health1.shtml
