If I'm extraordinarily stressed I've noticed I talk to myself - usually when I'm taking a shower or doing household chores, but I catch myself doing it, and it's literally me just thinking out loud trying to problem solve. It usually just a sentence or two. I think I have trouble multitasking, so brooding on a complex problem while performing a routine task requires me to talk it out or hear it. What you're describing sounds different. What concerns me is that you are not certain whether you are actually talking out loud, and that it may happen frequently and in front of others. How long has this been going on? Did you by chance start a new medication recently, or have been on a substance binge? Are you getting adequate sleep?
As far as your coworker's comments, did you hear the entire conversation? Are you sure she was talking about you? If yes, have you complained about anything recently? People can exaggerate. You may have mentioned something in passing, and she understood it to be a major complaint, so she references it as you ranting. With your mom - she may know you very well. My mom often "knows" things about me because we're close, and think very similarly. She also knows my body language, so she is able to accurately infer information about something's whether I tell her or not. I can do the same with her, but not my dad even though he and I are close as well. Your mother may not have that type of relationship with your sister.
As far as confirming your suspicions that you're talking to yourself - get a voice activated audio recorder and wear it in your front pocket all day. It should pick up when you are speaking. If I were you I would talk to your mom about it and see if she has heard you talking to yourself. She has nothing to gain by lying about it, so she would probably give you a true answer. I however, would not approach any coworkers with that question, as they already have a tendency to gossip, and you posing that question would be added gossip fodder and would make you look paranoid. The only other thing I would suggest is seeing a psychologist, as I'm sure they've dealt with this before. Good luck!