I think there can be differences because the rational vs. irrational mental processes are different on mushrooms than on LSD and this has a certain match with a certain personality.
That can lead to one drug being able to mindfuck you into a stranglehold while the other one does not tend to do that IMO.
On top of that: even if we believe that every psychedelic trip is unique, experiences in the past always gives us a sense of trust or distrust with a substance or even with psychedelics in general. And distrust is critical because like trust it is a self-reinforcing thing giving with a self-fulfilling effect.