Age is probably a big factor like many have said, I think it's also how you value yourself. Some people naturally have high self esteem (some are good at making it appear so), others have to work on it and your perception of yourself is important to that process. Many things can unconsciously color it, like comparing yourself to impossible (fictional) standards. Things get glorified in the media to unrealistic levels and that influences our view of the ideal human, or what we're supposed to make of ourselves. I can see how someone whose self-esteem is not built on anything real could come crashing down and have to rebuild.
I think the value of a human is better portrayed in philosophy than in the manuals of the monotheists or often in the media. The things we can observe about a person don't tell the whole story. I'm no psychologist but I think if you google 'superego' you might find some interesting reading. It's not exactly a hard science but related to this and great for raising thoughts.
Personally I have no idea why I don't care much. I'm conscious of what people seem to think of me but I'm not moved by insults, hearing rumors, being judged etc. I just rarely think the people doing it have the capacity to judge me. I guess I'm describing a narcissist. Over time I've started to value non-material things more than fulfilling the expectations laid out by the majority and my parents. Music, storytelling, nature (the universe), states of consciousness. In general, beauty and pleasure. I guess that shifts my ideals some. Money could buy a lot of experiences too and if I can ever make some without too much compromise, I will. But so many great things are available without being a millionaire, more than I will ever get tired of before it's time to exit.
edit: Talked about this with some friends and seems I was wrong. It's true for random strangers, acquaintances etc. but the people I have respect for..I do care what they think. Not that I'd watch myself or change my general behavior that much but I don't want to hurt them if that can be avoided and seem to value their expectations. The expectations of parents is another thing that many seem to carry around even in their later years. It would be nice to be able to break free from all that.