I think I may understand what you mean.. For me, it hasn't been psychedelic use that had changed my outlook either. Like most here have said, I've had my influential events impact my perspective and personality over the years. However, I actually can remember a specific instance in which hallucinogens changed me drastically and permanently.
I'd already been experimenting with entheogens for years when I discovered the research chemical scene. So, after fucking around with the 2c- phenethylamines for a few months, I sourced my first batch of MXE (as well as assorted substituted tryptamines to play with). When the post arrived, I decided to go all out and bender immediately.
Over the course of the next few days I ingested around three quarters of my MXE gram, along with what totalled about 160mg of 4-HO-MET and 140mg of 4-HO-MiPT. The entire thing was unreal.
There were old lizards in swamps, and cowboy scenes at dusty railyards, and Heaven's Gate's Marshall Applewhite was everywhere, aliens and occult throughout, chaos magic rituals, SRA, and way too much else, it was weird.
At the end of it all, my local news broadcasters were directing me on my holy agenda towards mass destruction.
Afterwards, I couldn't understand anything anyone said properly for weeks, maybe longer. It was as if the ambiguity of language and meaning was all out of whack. I know now it's called ideas of reference, but at the time all I knew was that I had broken reality. Everything said to me was laced with hidden malice or accusation, like the people in my life have been replaced by clay caricatures only to be animated and used by the Pattern.
It's still like that at times, though now I can decipher everyday chit chat. But yeah, I was nearly institutionalized, I became almost entirely socially isolated, I got into much legal trouble, and nothing has been the same since. And that was three years ago!
Anyway, I just thought I'd share because, the way you phrased the thread title, I explain it like that sometimes. It's like, nothing is physically different. I'm not hallucinating or even having symptoms of HPPD or derealization or whatever. But everything has been drastically altered nonetheless.