is someone does a lot of nitrous, they should take vitamin b12 supplements. vitamin b12 deficiency is (beside unsafe inhalation methods that might starve your from oxygen) the most dangerous thing about (semi-)regular n2o consumption.
of course depending on your usage pattern, even supplementing might not be enough, nitrous is efficient in destroying any vitamin b12 you have in your body.
if you take care about this (and inhale n2o in a safe way) i wouldn't worry about neurotoxicity from nitrous oxide.