I have always said that everything is psychoactive to some degree, in the sense that everything affects our brain chemistry in some way: Drinking a glass of wine, falling in love, going on a roller coaster, getting too much/not enough sleep... everything!
What I try to ask myself is, "Is it harmful?". Depending on the individual, I believe that there are things we need, things we can/should have in moderation, and things that should be outright avoided.
I know that I'm not saying anything new or profound, but our brain/body chemistry is a very complex thing, and nutrition can be one of the most confusing issues of all. Eating a piece of fruit, a bowl of pasta or a spoonful of sugar may all have the same calorie/carbohydrate content, but our bodies (and minds) can process them in completely different ways.