I've had this happen, too. A couple of guys have declared me 'too fat to fuck' when I was barely a US size 10, and since I'm 5'7 like you, I'm nowhere hear the clinical definition of overweight at that point. Funnily enough, none of the guys who've said crap like that to me were exactly in great shape themselves. I think they get distorted ideas of what women's bodies actually look like from porn and stuff, and then they can't handle the slightest bit of softness on a woman's body; has to be all muscles and angles with two big spherical boobs instead. Even at my heaviest, when I've been as big as a 12, weight doesn't really carry around my middle, so I just get curvier; when I was 'too big' for the ex-husband to take any interest, I was getting hit on more than I ever have in my life, since a lot of guys seem to like the curves.
I get that different people are into different things, but bodies do change over time; if you want a 'partner' whose body stays exactly the same, year after year, you can buy one online and inflate her with a bike pump!