Olney's lesions don't occur in humans as far as anyone can tell.
And the unfortunate part about this debate is that there just haven't been many comparative MRI studies of a varied cross section of dissociative users. I mean, if you think about it, even a study on the brains of PCP abusers would help here. Instead all we have is one study about ketamine, and that could mean a few things.
1a. All NMDA antagonists cause this stuff at any dose.
1b. All NMDA antagonists cause this stuff at high enough dosages.
1c. 1b, except "only in some people, sometimes".
1d. 1c, except "only in some people, sometimes".
2. Only high trapping NMDA antagonists cause this.
3. Only ketamine-type keto aryl cyclohexylamines cause this.
4. Only ketamine causes this.
5. There is an additional, unknown, exacerbating factor that everyone has missed.
For now, the best I can say is that regular moderate to high dose usage of NMDA antagonists at the very least isn't good for one's mental state (been there, done that, see also FastNBulbous). Partly because the person taking them often isn't keeping track of their dosages as they should be, partly because they tend to be self-reinforcing, etc., but mostly because eventually people just... snap. And do ridiculous shit, and then they end up hospitalized. Or jailed, institutionalized. Worst case, shot by police. Sometimes they undertake gratuitous acts of disfiguring self-harm (up to removing one's own eye with a spoon.) Or they just end up dead through their own ridiculous actions.
Most of the "urban legends" about PCP do have some basis of truth to them. And I'm sure this extends to stuff like MXE, diphenidine, and the like.
So take it easy and if you're going to use these drugs, keep track of your dosages. Keep it to once a week or even less.
Do these drugs negatively effect the structure of the brain? If a sample size of one means anything (it doesn't), I had been a really high dose, high frequency user of a bunch of dissociatives (DXM, Ketamine, MXE, 3-MeO-PCP) for a long time (years) up until I got hospitalized (which was to be expected), got a pair of CT scans, and my brain looks fine. No holes, nothing different from baseline.
But I do keep my mind active, and lead a pretty good lifestyle. (aerobic exercise, good diet). So maybe that saved me. I don't know.
So yeah, some words of wisdom and insight into sekio's personal life for y'all. I'm off the NMDA antagonists now.