Yes, it does cause permanent brain damage - Olney's Lesions (basically, holes in your brain).
Doing a reasonable (let's say under 100 whippets) amount over a lifetimes (NOT all at once) is unlikely to cause any substantial damage.
For some reason, nitrous oxide depletes vitamin B12, and thus, as B12 is essential to nerve health and function, supplementing nitrous use with extra B12 is a good idea, as some of you mentioned.
------------------
"All the bad boys/are standing in the shadows/And all the good girls/are home with broken hearts."
-Tom Petty, "Free Falling"