I can say with total honesty that I have not experienced any mental or cognitive issues with any true psychedelics, and I have used them very heavily especially during several periods. I HAVE experienced significant mental impairment and instability following use of MDMA (NOT abuse, but infrequent use), which came back to me after I decided to not use it ever again. I experienced similar things when I abused AMT heavily (once taking it daily for a week), but that stopped within a few days of discontinuing AMT.
I have pretty consistently used some sort of psychedelic at least once a week for about 2 years now, sometimes 3-4 times a week and for short periods of time, even more than that. I certainly do not recommend using them this often to anymore. I have noticed changes in my mind (willpower issues mostly, and changes in thinking patterns), but I am performing very well at my job and my other life pursuits... in fact I value my heavily psychedelicized (new word alert) mindstate as helping me in my field, which is computer programming and writing.
It's worth noting that tripping is very rarely stressful for me anymore. I think that if I were tripping like I used to (ball-to-the-wall, blasted out of this world every time with visions of horror and grandeur), it would take a toll. But after you trip a lot, this stops happening quite so much. Which is both a good and a bad thing.