High testosterone does have an effect on the immune system. Same as it's related to higher rates of heart problems, high blood pressure etc.
For women, oestrogen does have a protective factor concerning general disease, but on the other hand it is linked to significantly greater susceptibility to developing autoimmune conditions.
It's kinda the evolutionary trade-off for greater physical strength in men (useful if you have to fight other men) and greater resistance to common infections in women (which is obviously good to have if you're the one carrying offspring).