Medically speaking benzos do very little physical damage to your body, even if you are on high doses long term. Cigarette smoking however causes lung and other cancers, emphysema, bronchitis, heart disease, peripheral vascular disease etc etc..
Benzos however can be fatal in overdose if taken with other CNS depressants (or more rarely alone). They can also be fatal if you stop taking them abruptly causing a seizure.
Just thinking about the physical effects doesn't take into account the psychological effects of having a benzo addiction or of long-term benzo use though, and benzo use can cause memory problems and other cognitive issues as well.. it is hard to compare the two I agree. Cigarette smoking does cause more physical harm than taking benzos however, especially in the long-term.
I wouldn't like to say what the risks of 4 cigarettes a day are compared to occasional benzo use, but if you can keep your use to that level (not always easy at all) then your use is pretty safe. The key is not to keep increasing your dose and taking them more often, it can be a slippery slope.. also not to combine them with other CNS depressants such as alcohol or opioids in case you overdose.