Well that's pretty much a personal question. Currently I draw the line at anything "harder" than cannabinoids and my prescriptions for clonazepam, flurazepam, dextro-amphetamine, and buprenorphine- though I hardly abuse any of them.
As a former IV user of coke, heroin, and other opioids, as well as being a heavy drinker, I know I can't survive very long by crossing that line by snorting, shooting, or drinking alcohol, and as time passes I hope to not need to rely so much on the drugs I use today.
Like you said, you have to consider many factors, the big one being addiction. If you or your family has a history of mental illness and/or addiction, you're obviously at more risk and need to be careful. You might not become addicted but there's always the chance a drug can induce psychosis or make a mental disorder much worse. Your mind is essentially the most precious thing you have so treat it as such.
In my honest opinion, everyone has the right to experiment and try new things so long as they follow the right protocol in harm reduction. Obviously shooting up dope or even snorting or smoking it has its risks. At the same time, chronic insufflation of coke and especially prescription pills can be very harmful to your body. If you are going to snort something, make sure it's water soluble and not cut w/ APAP or fillers such as talc. Even taking something orally in high doses (like Vicodin, Percocet, etc.) is extremely harmful to the liver, so learn how to do a cold water extraction if you haven't already.
