i dont think your just being paranoid. i wish i would have been a little more careful when i was first getting into it. I started out pretty much at the same rate as you- just a few times a month. then it slowly became more and more of a habit. right at first i'd smoke about 1-2g's a week, and i'd always blaze with friends. it was purely a social thing at first, like drinking with friends. once it became the 'thing to do' as in whenever i kicked it with my friends it was understood that we'd be blazing, i started dropping all my friends that didnt blaze. felt like i had nothing in common with them anymore.
anyways, i got really lost in the plant and its culture. i started blazing every single day, alone, with friends, where ever and whenever. started carrying it on me wherever i went, and if i didnt have it my mind couldnt relax till i got a sack. went from smoking a dub a week to a dub a day to a dub in a single blaze session. at my peak, last summer, i was smoking between 3-7gs a day, and thats all i was doing. it became my main priority, didnt give a fuck bout anything else. and thats really the danger of this drug- it can remove your ability to care about anything besides getting high.
now im not saying i am like everyone, and that moderation is impossible, because that is not true at all. I know i was, and still am extremely irresponsible when it comes to blazing and its something i still struggle to deal with on a daily basis. I actually loose weight when i stop smoking cuz i have no sober appetite, cant sleep without it, and hate getting out of bed without atleast a few hits. not to mention im usually bitchy as all hell if i dont got it.
my story is about the same for most stoner i know, except i went a little further over the deep end. i guess part of me wanted to see how deep the rabit hole goes. part was trying to run from something. but honestly, i find it more addicting then coke. not a minute of my day goes by that i dont want to spark a bowl, but unless its at a party or im with a down ass chick, i'll pass on a line.
im sure you'll be fine if you keep it down to a few times a week/month. the only problem is whether or not you CAN keep it in moderation. just know your limits and watch yourself- if you see yourself starting to go down hill, STOP. do something that is actually going to make a change for the better. i think smoking only became a problem when i started using it to solve my problems.