I feel like there has to be a connection with stress, because for me it seems to amplify the psychotic-like effects of so many drugs (e.g. stimulants, cannabis, psychedelics, dissociatives). Amy Arnsten has done a lot of fascinating work showing that high levels of norepinephrine essentially take the PFC offline, although she hasn't looked at psychedelics in particular. The PFC mediates executive function and working memory, both of which are compromised in psychosis, so I think her work still validates this idea.
It's odd because usually when I take psychedelics in a relaxed setting they feel almost anti-psychotic in a way, although I don't generally push the doses and with stronger psychedelics such as LSD it does seem like that would be more of a possibility (although LSD is a dopamine agonist and Arnsten has also shown that high levels of dopamine take the PFC offline as well). Dissociatives definitely have more of these psychotic patterns of thought, but then again they have a greater capacity to release neurotransmitters like norepinephrine and dopamine, so that could explain some of it. I don't think it's the whole story because abusing dissociatives gave me residual issues (alogia, flattened affect, delusions, anhedonia, etc) that I've never really experienced from psychedelics.