Eye contact is incredibly intimate, which is something that I think is really downplayed in society. It's basically the closest that you can get to seeing into someone's brain, and (IMO) because of that they are truly windows into the mind/soul/whatever. For many years I had a huge problem maintaining eye contact with people for just that reason: it felt too personal, and made me feel vulnerable or naked. At the same time, I've been told that I have an 'intense gaze', which made me feel even more self-conscious.
That said, eye contact is vital for maintaining a connection to someone while speaking face-to-face, and the ability to hold, or even just avoid shying away from eye contact is a huge indicator of confidence and comfort in the situation in which one would find oneself. For me, it was a practise thing-- when I was walking around I would make a point of never looking to the ground. In time, I got comfortable enough doing this that I could make fleeting eye contact with strangers, which progressed to solid eye contact, which could even progress to the occasional random smile. Great fun!
While that progression was happening, I found that my eye contact with those with whom I was speaking became more relaxed, natural-feeling and longer-lasting. Now, I love eye contact, to the point where I have to remind myself to be careful to break it off in certain situations.
tl;dr-- it is normal to feel uncomfortable with eye contact, especially if you're in an uncomfortable situation. With time and practise, this can be overcome, and can help with overcoming the underlying anxiety as well.