I've tripped with a bad cold a few times before. All times it was absolutely fine, didn't ruin the experience at all. Only the first time I remember there was one slightly unpleasant moment, it was in the late comedown (I had barely noticed I had a nose at all for most of the trip actually) but in the late comedown my nose felt huge and I had a pressure on my head, which lasted about an hour. Might've felt it sober too, didn't ruin the experience at all, it was still a lot of fun.
The second time tripping with a cold was fine too, even finer because I just stayed indoors calmly all night, not a problem at all. That trip wasn't any less good then any others, and I didn't get the pressure on my head either that night either. I even attempted to snort some ketamine; one nostril was ok with it and the other one burned and stung terribly. Didn't get as much effect as usual either, did work though, especially with the non-hurting nostril. Either way, I don't advise snorting with a cold, but tripping with a cold is absolutely fine for me.
I've also tripped with a bladder infection; didn't affect the trip either. I also tripped with bronchitis twice, of which one of the times it was very cold out and it made my lungs hurt; a little distracting but as again, not any worse then sober, if not better then sober.
I think that were all the times I was unhealthy and tripped anyway. I also did shrooms with a cold once, other then getting annoyed by my tearing eyes that was not a problem either.
--
Now this is my personal experience; maybe someone else will find it much more annoying to trip while unhealthy. Personally its never been a problem for me, especially not with a common cold. I wouldn't trip with a fever though, but with a mild infection somewhere or a cold I really see no harm. If you feel its a good idea and are on a comfortable place you'll probably be fine.