it's offensive because it brings out memories/images of a large history of systemic racism that occurred in the US specifically against one group. Yet we still make jokes about jews and stuff and the holocaust, i think it's kind of silly but you see enough comics willing to say 'the n word' and talk about it. It's more taboo around here but within circles of white friends, the term is used casually, often as a side kick to the word fuck or replacement for cunt (more popular in the UK i think).
It's a popular topic now though (racism) and the word itself is not often used by today's youth as a form of racism (IME). For example, before my grandfather died he would watch loads of baseball and get drunk everyday. He wasn't racist (well he hated natives) but still admitted many times that 'those niggers can play ball", he wasn't trying to be racist, just using the words he grew up with. Things are different now, i'd think being called a coloured person would be just as offensive but i don't know. You can't just go into a workplace or something and say it though, you will offend someone, the word just hits the ears wrong, likely a cultural and social thing and respect for what happened in the past.
personally i don't feel any word should be off limits as it just makes the word stronger in a social sense, once it's overused people become desnsitized to it and it's no longer as offensive. I still find it a double standard that everyone will throw around blatantly racist remarks against the Jewish people (watch that hard J sound lol) or especially immigrants and native people. I live in backwards ass racist as shit part of Canada though so it's not surprising. Older people (some, not all) will call anyone not white a 'new canadian' or some sort of slur against native people. It's really a good question though.