Yea just get down to 5mg a day and then jump off. 20mg a day of oxycodone is not much, but since you've been on it for 4 years your body will have to adjust to functioning without it, hence the withdrawal symptoms. I wouldn't recommend suboxone or anything since that will likely complicate things, being that suboxone is very potent (1mg is actually the same or stronger than the 20mg of oxycodone you take, and most suboxone comes in an 8mg strip) and your habit isn't up there. I wouldn't even use the loperamide since that too can complicate things, but if you insist on using it just take 2-4mg for the first few days, then just 1-2mg of the last few days. It should be over in 5 days so you would only need it for 4 days or so.
Once you get down to 5mg you have the option of skipping days, but if you are feeling too sick on the off days just quit since it's not worth prolonging it if you have to feel that sick before you are fully off of it.
An actual detox facility is not an option since I doubt that you would even be admitted with a habit that small. The methadone that they would give you would have you higher than ever before, so you would leave there worse off than you entered. The same goes for detox facilities that use suboxone, since they usually give you 16mg your first day or so and that would be like 16x your usual dose (with the ceiling effect of suboxone it would stop it at being around 6-8X your normal dose, but still overkill for that habit) so just taper on your own.
If you want the taper to be as short as possible wait until you are really sick and then take 5-10mg and that should be enough to eliminate your symptoms. I disagree with the person saying you shouldn't use the valium. With your small habit you can probably get by just going cold turkey and using the valium for sleep and to help with the anxiety that comes with withdrawal. Just wait until you are fully off of the oxycodone before starting the valium since you don't want to begin to develop a tolerance to it before you really need it.