In the late 90's and early 2000's I destroyed my mental health with partying every weekend with copious amounts of speed (sometimes rarely meth), MDMA and cocaine. I used a lot of GHB and cannabis too, but I don't think they had that much impact on my mental health as the other "party drugs". Back then I also used LSD and shrooms (plus some RC's, mostly 2C-I and 5-MeO-DiPT) ocassionaly and it was definitely something very different from the others. When my impending mental breakdown came, I got 4 diagnoses at the same time (panic disorder, generalized anxiety disorder, obsessive-compulsive disorder and moderate depression). While there certainly were some genetic factors that triggered the mental illnesses, my horrendeously irresponsible use of MDMA was the worst factor. After becoming sick, I did no drugs for 7 years, just heavy medication and therapy.
It wasn't until my psychiatrist recommended me mindfulness and meditation I began to have an interest in exploring the mind and drugs again. But not just any drug. I remembered from my psychedelic trips in the past that the they were different. There wasn't any nasty comedown from LSD or shrooms. The experiences were deep and blissful. I was just too deep in the hedonistic partying to realize their potential. After many years of severe mental illness, I had got to a point where I could live a quite normal life. And as I was exploring the mind (through meditation, yoga, reading etc), I remembered the psychedelics. I knew they could be utilized as tools to explore your mind. So I started using psychedelics again, and IME they have hugely helped me with realizations that even years of psychotherapy couldn't do. Psychedelics opened my mind to the subconscious, to feelings and thoughts the heavy medication I took several years had suppressed. I have encountered many hard things on introspective trips, tripping has not always been fun, but the trips have (almost) always thought me something new about myself.
One could obviously wonder why start taking psychedelics when you're finally feeling mentally quite OK, since there always are risks with psychedelics. The worst case scenario could have been a return to the dark years of mental illness. But I always used them with a purpose. The purpose can be many different things. Personally I've been focusing much on tripping alone, i.e. introspective tripping. But I've also tripped with a good friend of mine and had shared experiences that has increased my life quality. Mental illness or a (family) history of mental illness is not necessarily a recipe for disaster when it comes to psychedelics. I know feel better mentally than ever since my mental breakdown. Psychedelics have played a part in this, although they are not some magic medicine that instantly will heal you. But they are tools that can be utilized for processing problems in life. A tool. There are many other tools that are equally good or better. But I stand as a living example that psychedelics in fact can cure and heal, while using them with the respect they deserve.
Edit: My bottom line is a history of mental illness in your family (my brother who does not use any drugs got a severe psychosis a few years ago) is not an automatic barrier to psychedelics. But thread lightly. Follow Erowid's golden rule. Know your Body - Know your Mind - Know your Substance - Know your Source.