You won't get bad withdrawals. A long time ago I used to use valium for comedowns and after about 6 months of weekly use I started to notice a bad metallic taste and tingling nerves around Thursday. After a few weeks I worked out it was benzo withdrawal.
Just watch out with valium - if you are getting withdrawals they can often be a long time coming. That said .. you absolutely shouldn't worry given the amounts you've been taking - but with such a long acting downer you really need to be taking a week off between doses. If you have to dose more regulary (and not constantly) consider a shorter acting benzo. I always use xanax for comedowns and can take it two to three times a week without any problems - just have to take the odd week off to check it's not a problem. Be careful though - that is really not recommended as it's easy to slip into regular use and benzos have a proper bitch of a withdrawal. As a guy whose concept of withdrawal has been twisted to an unreal extent by hardcore fent withdrawal I'm still scared of benzo withdrawal cos of the seizure risk so please be careful if you're switching things up as that's often when it all goes wrong.