i agree with f33lg00d and blak kat, you should taper of the valium, but under medical supervision. that's going to the doctors, explaining what you are doing and coming up with an action plan that suits you and - i really can't stress this enough - is somatically sound. 20mg isn't an awfully large amount, but depending on your so many factors an abrupt withdrawal could still lead to complications. Happened to me a bunch of times.
just remember, valium is pretty much just as bad. but really happy to hear you are trying to get off it all. best of luck.