I tend to agree that setting is a big factor. There was one time a long time ago when we were all going to eat some mushrooms and head out to a big party out in the bush. I ate a few grams and away we went. We had all been doing this all summer, so no big deal, party on! The atmosphere around the bonfire was tense, there were a lot of people there that were just drunk and looking for a fight. This didn't particularly worry me as we were all young and badass heheh.
It seems though that the underlying tension did seep in, and created a state of anxiety in my subconscious that I was unaware of. Until, standing around the fire drinking a beer and chatting, everything suddenly went into multicolored checkerboard patterns. Now normally I would have said "cool" and enjoyed the show. This time however, the anxiety suddenly became a full on panic attack, and the only sensible thing to my mind, was to take off running, full speed, into the pitch black forest, which I was seeing as checker board land.
Of course running into a giant tree full blast in the dark tends to bring a different perspective. That of laying in front of the tree looking up and trying to figure out what the hell just happened. After laying there cursing, praying, and gibbering for a while I pulled myself back together and wandered back to the party to find people that I knew. Luckily all that happened was a small cut on my face, amazingly lucky thinking back.
Two things I learned from this, your setting is not to be taken for granted when tripping, and mushrooms are never to be taken for granted, give them due respect or they will kick your ass. Oh, and trees are really, really solid heheh.