Maybe it's just anxiety. Since human beings define their reality through pain and misery, it is not uncommon for someone to feel guilty when enjoying something. Examples abound: sex, drinking, watching sports, porn, masturbation etc. Religion doesn't help either.
Here's the good news. Guilt is not real, it's a chemical reaction that occurs in your brain, designed to blind you from the truth. The truth is that, as long as your habit is healthy, it's not wrong. In fact, it may be healthier than everything else, you don't know. And if it's not healthy, make it healthy. You got a brain, use it. Read chemistry, read read read read. Understand. When you don't understand you start being afraid and feeling guilty for having fun.