If you don't enjoy the experience most of the time that's on you, you chose the sex worker, your attitude and behaviour quite often set the scene. If there's shame to be felt then it points to an issue with you and not the sex worker, it's up to you to address it. Demeaning or denigrating anyone for an issue that you are partly or wholly responsible for is weak as piss and points to other deeper issues.
Agree with the danger of getting caught up in fantasies.
Also agree that chemistry is important between to two participants however, trying to politely decline after 5 minutes would understandably piss off the sex worker and in my case kill my mood and most likely any chance of enjoying the time spent at the establishment regardless, you are basically telling the sex worker thanks but no thanks and expecting a refund, when more often than not you have made the choice of sex worker and are a key contributor to the lack of chemistry. I would say this would make one out to look like a prick of a customer.
I'm pretty laidback and just go with the flow, I usually try to make the best out of any situation, there's a reason why its called punting, everyone is different and there are no sure things. If there's no chemistry there's no chemistry and I won't see that particular lady again, I made the decision to book or choose that particular lady and contributed to the lack of chemistry, so that's on me. Just because there's no chemistry doesn't necessarily mean you can't enjoy the time together, if you are willing to adjust your expectations, once again that's on you.
I think the bottom line is if you walk in with an open mind, no set expectations, respect the sex worker as a human/person (rather than say thinking you own chosen sex worker because you've handed over some cash for their time) and have a decent attitude then in my experience most of the time you'll have a great time.