The problem with this combination is three-fold:
1. Metabolism
Both Buspirone and Tramadol are metabolized by CYP450 isozymes. Tramadol is metabolized via CYP2D6 (into M1, this metabolite is far more active as opioid-receptor agonist, so you want a lot of it) and CYP3A4 and CYP2B6 (those aren't as important for analgesia). Buspirone is primarily metabolized via CYP3A4 so there will be competetive inhibition. I'm not sure wether buspirone or tramadol has higher affinity. This can either lead to more M1 or a higher buspirone concentration, which can be dangerous.
2. Seizure treshold
Tramadol is known to induce seizures over 400 mg (and iirc there's at least one report of a woman having a seizure after 300 mg I think) due to the SNRI activity. While buspirone isn't a reuptake inhibitor it seems to increase norepinephrine (at least in the endothelium of blood vessels) and I found this
https://www.epilepsy.com/learn/prof...c-drugs-developmental-disabilities/comorbid-4
While a single case report isn't much I personally wouldn't risk it
Now to the main problem:
3. Serotonin syndrome (or better serotonin toxicity)
https://www.researchgate.net/public...void_the_Use_of_Tramadol_With_Antidepressants
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3865832/
Wikipedia
Now I think in most cases of SS it's due to the combo of drugs that increase serotonin, be it via reuptake-inhibition, increase in synthesis or increased release, leading to high activation of 5-HT receptors.
With buspirone you take an serotonin receptor agonist and with tramadol a SNRI.
This is only a guess and like you said, I'm not a doctor. With buspirone being a partial agonist of postsynaptic 5-HT1A receptors and only with some affinity for 5-HT2A I see it like this. At the 5-HT1A serotonin and buspirone compete for the binding site and buspirone is only a partial agonist, meaning the intrinsic activity is less than with serotonin, so less activation of the receptor. With 5-HT2A we have competition as well, so imo (as you said take it with a grain of salt) the potential of SS shouldn't be too high as they'll compete and not add up.
With this all being said I still see some problems with the combination and imo better be safe than sorry and not take it.