I agree with OP's sentiment if we're talking MDMA for instance. You accept everything, love anything, and that includes music ordinarily trash to your sober ears.
This is different for substances with a psychedelic component, where appreciation isn't this automatic. There's a cumulative learning effect, which allows evolution so rapid that it becomes alienating for people not in tune with the mind state it builds on. In the same way many people just don't get Daoist art, and can't appreciate a Sesshu until they grock Eastern philosophy. But for starting to listen to the space accentuated by sound in minimalist electronic music, a single ketamine experience can be enough, for instance.
Another way to describe the synesthesia is that there are more brain parts involved. You listen with the mental function that usually sees, as well. Of course this makes whole genres inaccessible to teetotalers or other default mode network adherents.
