I found LSD a little scary the first time I took it. Not frightening, but enough so that I realized that it was a powerful substance. No panic, just a little discomfort. I didn't know what was going on. I didn't really know what to expect. I took everything in, and got a little confused (thought people were sad, who weren't, thought people were mad who weren't, etc). There were moments when I was laughing and laughing, and laughing... but there were moments when everyone else was laughing and I threw in a couple of fake chuckles, to hide my discomfort.
You should be able to handle it. I just never found LSD 100% fun like most people seem to find it here. About 60% of the trip is fun (laughing, admiring the scenery) and about 40% is a little awkward and uncomfortable ("OK, can we be finished now, please?" type stuff). It always came and went. I would get a little uncomfortable and then it would pass and I would forget how I had felt a second ago. With higher doses, sometimes it was always awkward and uncomfortable. Sometimes it was a little scary. Never really frightening, or had me in panic though.
I find it a powerful drug. In fact, I find all psychedelics very powerful drugs. It's really unlike any other thing you can experience. Even though my trips were not the most fun, I always learned a lot about myself and my friends. When they were really uncomfortable, I always came out thinking I just won a race or something. Just glad to be back and loving life and sobriety.
So in short, I don't think it's really scary for most people. Most people tend to have a blast. I never really found psychedelics that fun, though. Some of the best days and nights of my life were on LSD. Not because they were the most fun though. Just because they were amazing.