............ I feel like people look at me and see something different than what I see in the mirror.........
This is normal. Perception is, literally, in the eye of the beholder. What you, or anybody else, sees, is informed by personal experience and expectations. Though we may say we "agree" upon something, we are giving a general evaluation of the "thing" in question and saying that we concur about the general nature of it.
Now this may not be the case with abstract concepts such as mathematics, where we can feel fairly sure that 1+1 =2, but these are not real in the sense that you are real. We would never expect two of our friends to have exactly the same views of us - even though you might think you are "the same person" to both of them and they agree that you are indeed, Jim or Helen.
So what you see in the mirror is what you see in the mirror, and what someone else sees you as - either directly or in the same mirror - is what they see. And I wonder if the person who you thought had a "big confused look" would agree with you and say that they did in fact see themselves as having that sort of look?
Mirrors are incredible. Depending on the lighting I can appear to myself as unremarkable, but if the lighting is muted and I can't make out the fine detail I can see myself looking young and handsome (I find lifts are good for this). I know this is not true - but I "see" this and it is real for me at the time.
You is what you is, and that is you. If you think someone is looking strangely at you then you need to say to them and ask if they perceive the same - and hope they are truthful to themselves and you.