Long term psychological impacts are possible.
But hear me out on this one, in a fundamental sense our psychologies are based around our experiences as we go through this life.
Experiences need not be drug related to act as something that causes a long term impact on ones psychology. Look at post traumatic stress disorder, some of the time it is from a long term experience but it can be a single traumatic experience that leaves a mark on your psychology.
I believe many of the long term impacts, be they positive or negative stem from a similar direction. Our brains filter out a lot, you hopefully have a good amount of control over your emotions, but rolling around on the floor giggling is not productive for your greater "being", so our brains put limits up on things like your mood, and general sensory imput. People with bipolar disorder or those who experience massive anxiety attacks are good examples of what happens when there isn't stability.
Post traumatic stress is what happens when those filters that allow us to handle stressful situations are not enough to cope with them.
Psychedelics and entactogens remove the filters while evoking powerful experiences, emotions are magnified to a huge degree because we don't have anything to filter them. To a certain degree, traumatic seems to harsh of a word for the psychedelic expereince, but intense psychological experiences good or bad can leave a long term impact.
My first acid trip ended 3 years of depression in a night. It was like a switch turning on in my brain that allowed me to see the fundamental beauty of life instead of living day by day wishing that I wasnt. Could hardly speak for the next few days, the experience had been a profound one, profoundly beautiful in a way that I can't even begin to describe. Its been about 3 years, and I would say It left an impact on my psyche.