I faced this dilemma when I started taking Methadone and continued for so many years. At some point I realized it was doing more harm than good. And that if it wasn't for the drug I was taking for so many years I was going straight back to addiction.
Of course it gave me quality life and that's why I got stuck with it for so many years. But the problem is that the longer you stay the more difficult it is to get out, to disconnect and live a free life. And when I did I felt I was numbed and had lost my life tapering instead of confronting the real issue. It's a very intense addiction because of long acting effects and because you are under the effects 24/7. These type of drugs take you off of the streets, 'solve' your addiction usual problems. I felt I was okay only because of the chemicals and that I was totally dependent, slaved of a routine I couldn't get way of.
On top of that, with time there are some side effects that was making my life quite difficult. Problems with hormones, metabolism. I was always constipated, with low energy, lethargic. Sweating like no one else, etc. etc. And it was so difficult to stop. The withdrawals are much harder, worse than expected from usual opiates. This did not happen in the beginning.
Of course there are people who adapt and live well with controlled replacement therapy for a while others just want freedom and a way out for good. You'll have to make this decision on your own considering your future on a long term. There are a lot of Blighters who stopped using subs and methadone and it's not about going through withdrawals but how to live without medication.