I think the reason people frequently say this is because they haven't properly experienced the effects of nitrous.
If my experiences with nitrous were like most people, i.e. breathing it in the form of balloons (or even worse, inhaling it from a balloon, then blowing it back into the balloon and inhaling it again, back and forth like so many people do), I too would also not be a fan of nitrous oxide. This form of use, which is by far the most common, is the reason why so many discount nitrous oxide, or view it as not a standalone drug, etc.
I only began to understand what nitrous was as a drug when I bought my own little whipping cream dispenser and started using chargers. At first, i used balloons because I didn't know any better. I even started doing that dumb thing people do, inhaling from the balloon, then blowing it back into the balloon and inhaling and exhaling over and over again (the worst, yet most common way, to inhale NOS).
But eventually I figured out that if i connected a rubber tube from the outlet of my little dispenser, and took hits directly from it, holding for as long as possible, exhaling, taking a few breaths of air, and then taking another hit, etc etc the effect was far different. After doing this with about 5 chargers you begin to enter the nitrous headspace. It is also when nitrous starts to become intensely euphoric. Like any drug, blood levels of nitrous oxide rise over time, so the effect is cumulative .
At this point nitrous becomes a standalone drug, not just an add on to trip. Some of the most intense and powerful trips i've had were on nitrous alone. And since I'm prone to anxiety, i have never been much of psychonaut and while ive taken many psychedelics in exteme doses, it was almost always in conjunction with benzos.
Nitrous oxide is unique however, being both a powerful dissociative and acting on the benzodiazepine binding site of the GABA-A complex. This makes the experience far more comfortable given the profound anxiolysis. Additionally, it seems to have opioidergic effects, seemingly from the release of opioid peptides, resulting in additional comfort and euphoria.
While I abused nitrous in the above described fashion for over a year (averaging 50 chargers a day, which is 400 grams -- almost a pound of gas), i didnt seem to be too negatively impacted. But i was in my early 20s and perhaps was lucky.
I hadn't used NOS for years, but when i started seeing the new tank versions of NOS being sold I made this (which ive only used a dozen time, despite spending maybe $500 building this contraption, which mixes filtered NOS with oxygen from an oxygen concentrator):
I had some mind blowing experiences with the above device, many which involve socratic discussions with some omniscient version of a very of yourself on "the other side". It is a very unique and powerful drug, and most people who say they don't like NOS have never experienced its true effects. Because who wouldn't like such a mindblowing and exceptionally euphoric and anxiolytic drug?
Regular abuse of course, is unsafe. I got away with it, but I was lucky.