Seizing from tramadol is more likely from taking a lot in one go (generally accepted safe level is 400mg/24 hours although some people have seized off less and some can take more and be okay) than from repeated dosing over several days - the amounts you have taken each day are within the daily limits so in theory you should be okay, but the fact that you can still feel it means it is probably wise to take a break! Although you shouldn't have problems with physical dependency after 4 days, psychological addiction is a different matter - the more days you take it, the harder you are going to find it to stop so I am glad you are taking a day off.
Tramadol lowers the seizure threshold which makes you more likely to seize; diazepam raises the threshold, making it less likely. So you are less likely to have a seizure on diazepam, although 5mg is a low dose and I wouldn't rely on it for that reason! The problem with diazepam and seizures comes when you are physically dependent on it then stop - that is when you can have seizures. Benzodiazepine withdrawal can be dangerous and even fatal..
Interesting reading:
list of medications which may lower the seizure threshold (making seizures more likely so try to avoid combining with tramadol!)