I got my bachelor's degree in chemistry this May and have started on a PhD program this fall. If you're really interested in it, don't worry about whether you had chemistry classes in high school or if you did well in them. Everything in a chemistry bachelor's (or even associate's) degree program is going to start off at the ground floor anyway. You'll have to take general chemistry 1 and 2, and unless you are at a very, very high end school or in an honor's program, they will assume that many students have had no prior chemistry experience whatsoever, and teach accordingly. So you won't get left behind due to them expecting you to know things you don't.
However, the class generally does move pretty fast, and it is very easy to fall behind if you don't start working right away, and most of the material builds on itself so if you're lost at the start, you aren't going to find yourself later. All in all, general chemistry is not extremely hard, though, and if you've done well in other science and/or math courses, you'll probably do fine in general chemistry. Next up is organic chemistry, which most people either love or hate (its what my PhD is concentration is, so I'm one of the ones that love it). It is very different from general chemistry and has almost no math focus - it is entirely about logic, reasoning and spatial reasoning, with a fair amount of memorization thrown in for good measure.
You say you'd be interested in being a chemist - what sort of work do you imagine yourself doing once you finish your degree? And would you intend to stop at an associate's, bachelors, or go on past either of those? Those are important questions to have an idea of an answer to before you make such a major change to your degree plans.