Set and setting is one of the most important things. It doesn't seem like it because in everday reality you don't look at things the same way you do when you are tripping and what seems manageable and okay will unfold into how things really are when the restrictive filter of everyday waking consciousness is taken away. This means that relationships you perceive as being bearable when you're not tripping but have nuances to them, such as a slightly controlling parent, passive aggressiveness, contrasting personalities, toxic relationships dynamics, trauma, past history of disagreements and general tension, the mental and emotional state of the other, their attitude towards you, personal boundaries etc this will all be out in the open when you are tripping. So if the atmosphere you are in is experienced how it really is when we're not filtering out as much as possible to be able to cope and live and let live, you will be affected by this.
It's not unusual for people to see this when they are tripping and it can be really uncomfortable sometimes potentially a cause for them to find themselves in negative thought loops and for the trip to turn dark. I once made the mistake of tripping with a 'friend' who most of the time was basically just an as*hole. I was in a bad place at the time and felt lonely so I had people around me so I didn't feel lonely but who were the least suitable candidates for healthy friendships. I wanted to believe I had friends but I knew I had settled for these people because that's all I believed I was worth at the time. Many of them quite frankly didn't give a f*ck about me but I grinned and beared it and normalized it all. Most were drug buddies who kept up the friend act in order to continue getting high with me, and likewise I was guilty of often being the same with them. Anyway, I brought hand picked locally sourced mushrooms to this guys house and we set about tripping. Originally I thought maybe we could breakthrough together and he would sort of change his ways and maybe we could connect on a deeper level. About an hour or so in he started fighting the trip, denying he could feel it, started getting really defensive and essentially layed on the hate towards me. His girlfriend was there (who he later tried to blackmail through a failed suicide attempt that he did mainly for attention) and he began the typical narcissistic triangulation techniques of making someone else the outsider of the group. That outsider turned out to be me. I was pretty high on these really good mushrooms and now I was essentially trying to preseve the state of my mind so my trip didn't lose control. The guy wasn't completely abusive nor was he a really spiteful and truly cold calculated guy. We were just not really friends and whatever point in our lives we were at, we as friends never really existed. It's in these moments you see a lot more than what appears on the surface, which is my point. Parts of him was good, genuinely. But he had a lot of demons and like a lot of my dysfunctional friends at the time, they all projected it outwardly and take it out on others. I went into life doing the same yet deep down I was searching for a real best friend. I saw a side to him that while vulnerable and open to suggestion in an exposed state of mind that while sober made me want to sock the guy in the eye. We actually had several fights but never laid a finger on each other because we somehow had just enough respect for one another to not take it that far. He once stole my new bike that I spent almost 5 months paying for in installments and we took it out into the street where we rolled around for a bit but we never threw one single punch. You cannot force these things. Psychedelics WILL reveal whatever act you are putting on, whatever illusions are present.
It's not that you will experience something like I did while tripping but what I'm again trying to emphasise is set and setting. This is something that since the very first breakthroughs in psychedelic psychotherapy and research occured has remained a PIVOTAL aspect of supporting an individual (or individuals) through psychedelic experience. The short fuse of a family member becomes a ticking time bomb in a trip. The passive aggressiveness becomes a potential trigger for endless anxiety and negative thought loops. Even the reality of people having completely different personalities and this while sober makes them clash sometimes will be emphasised when you're tripping and it may seem like you are worlds apart, which is not good when what you need during these times is grounding and to feel safe, secure and able to be present in your mind and body.
I would avoid tripping with family. Some people might be able to do it but those people are fortunate to have already a solid genuine relationship with their family that allows them to facilitate such experiences. Not every family understands one another beyond the superficial veil that many often hide behind in order to simply assume the family identity. Even close knit families while left open to the reality of their relationships with themselves and others will often experience difficulties. I would avoid tripping with friends who while sober raise concerns, if only because you genuinely care about them but you can see and feel things are not ideal. Listen to yourself and trust yourself.
Tripping on your own is fine. I've had several heavy mushroom trips on my own and they were singlehandedly the best experiences I've ever had. Difficult? Absolutely. But worth it? 100%.
That's where you'll meet yourself and whatever exists there.
Make sure you really take the time to really care about yourself before you trip, get everything in order, have the atmosphere prepared as best as possible, plan everything, be your own trip sitter in the way that everyday waking consciousness you made sure that tripping you could have the best possible experience

You'll be surprised how much impact all the positive influence you can have before you trip offers somewhat of a safety net, a blanket, while you are not your everyday waking consciousness self with the filter turned to full and attempting to live up to the illusion of fitting into the wider part of our social realities.