Any drug can be abused, even psychedelics. Psychedelics are just stereotyped like every other drug and it's either "It's caffeine so it's harmless" or "It's crack so it's 100% dirty and dangerous and IDC what people say". It's nearly always black and white and that's sadly because many people have little to no knowledge about the reality of drug harms outside of their own flawed reasoning. You can blame much of that on the current drug policies and decades of demonization, propaganda and hysteria from the sham that was (and still is) the war on drugs. You're either a crack head who lives in a cardboard box on the street or you're drinking energy drinks but you're acceptable and it's definetly not an addiction because energy drinks are legal so they aren't classed as drugs.
The biggest harms with psychedelics I believe is from their immense power to alter the psyche of an individual, particularly at the level of personality and in massive amounts. This can lead on to all sorts of consequences, both positive and negative, depending on the responses from the individual and their psychological makeup, perspective, personality structure etc. And whereas many people will have successful experiences with psychedelics and their experiences be profound and alter them in positive and beneficial ways, sometimes letting go of content that has been there up to an entire lifetime, sadly others cling on to the ability for these drugs to distance themselves from their everyday waking reality. The culture around psychedelics doesn't help either. You're told it's a good thing to do getting high on acid or mescaline for example, and it is in many respects if done correctly, yet people develop a psychedelic narcissism whereby they believe they have somehow accrued universal wisdom and accessed other realms that fellow mere mortals haven't accessed and therefore they are superior to those who haven't tripped. And so it becomes encouraged in many ways for an individual to have these experiences under the false pretense they are somehow improving themselves when in reality they are actually developing further dysfunctional coping mechansims and escaping from who they are, what they believe, their pasts, their present and their futures etc.
How many people can actually handle the substances they are using? That is the question. How many are really aware of their immense abilities to alter reality and to understand they are using drugs which are far more complex and powerful than they care to think? How many are prepared to look at psychedelics like any other drug and assess their usage on the reality of how they are affecting them and their potential for harm? And yes, psychedelics have harm potential and they do cause harms in society. That is a fact. While the harms might be lower this does not mean the consequences of abuse is less severe. The consequences could actually be worse because LSD will take you out to deep waters and if you are not prepared nor able to handle it, that could set the ball rolling for a whole other thing altogether, and not something particularly attractive and fashionable like taking psychedelics tend to be looked at as.
So I think it's really down to people being able to be honest with themselves. And just like narcissism itself, it's full of contradictions and flaws (hence what narcissism stands for in the context of an overblown ego and defense mechanisms that are set about refusing to acknowledge the reality behind certain delusions/belief systems, pertaining to the identity etc) and being able to stop these contradictions and the gaping flaws that often are present can really help (I believe, anyway) in giving people the much needed insight into what they are doing, whether they want to be doing it and whether they are doing it for beneficial purposes. Or quite simply, whether it's just another addiction and despite all the self improvement in the world they cannot begin to accept the very drugs they are using are essentially a crutch to enable them to actually prevent themselves from gaining the self improvement they may so often boast about pursuing, while they at the same time, cannot see beyond their own incongruences.
I've met many people who have taken lots of psychedelics and yet very few are actually really improved. Perhaps at a micro-level, sure, they are to some degree but it's nothing substantial. Much of it is the grandiose thinking that comes from disconnection from reality which then acts as a reinforcement, albeit superficial and based on delusions of them somehow being better. They've been able to see beyond reality and now they manipulate it to meet their ends, but this is fruitless because it's essentially a delusion. The few that actually have improved didn't abuse them. They never formed addictions and they knew the limits. When you hear of the likes of psychedelic pioneers who so often get bundled into the category of frequent psychedelic users their usage compared to todays society is/was nothing. Terence McKenna for example took them more often when he was younger but then gradually reduced his experiences to the point where he wasn't taking them at all. Ram Dass stopped taking them after the 70s and spent his life finding new highs, which was from being around 30 until his death in his 80s (I believe he was in his eighties, someone might correct me on that one). It's easy to picture all these people dropping acid all the time but many of them were extremely rigorous in their usage and their regimens surrounding use. Many were academics and many were actually more qualified than most in administering and evaluating their effects. Today people have taken the words of these pioneers as Gospel and over-indulged to the point where perhaps the real value of taking psychedelics has diminished. This was partly the issue in the first place that got psychedelics made illegal as therapeutic use and research turned to mass usage and consequently abuse.
The way I see it, psychedelics are medicine. Medicine that at one point in history would be handed around the local community potentially gathered around a big tree or a fire or something like that. And it was used for the purpose of transforming consciousness, seeking answers, improving oneself and others, knocking down barriers (perceived and otherwise). It literally was what the definition of medicine is today (before medicine was bought out by big pharma). Today psychedelics tend to verge on the line between quackery and fringe 'psychology' and actual evidence based applied understanding and knowledge about their benefits for all the things we tend to relate to them. While you don't want to make everything too scientific and remove the unexplainable, the mystical, spiritual, the metaphysical, what cannot easily be quantified etc you also don't want them turning into drugs that inevitably carry a similiar label to other drugs that have far more negative connotations and just ruling them out as dangerous drugs without therapeutic benefits. It's a fine line and that is something people must understand, and what society must understand. I guess you could take this further and address all drugs on this basis and then perhaps assess their usage, merits and negatives and implications thereafter. At the core though? The individual is responsible and to simply believe psychedelics are harmless drugs without abuse potential is naive, immature and also the lack of understanding and knowledge we do not need associated with their therapeutic and beneficial use.