A couple of factors from my experience:
1) Physics, especially at the undergraduate level, will require a lot of lab work. And not like high school physics labs (those were always cool, whereas looking at an osciliscope for 4 hours is not exactly dropping stuff off of the school roof). Confusing and expensive equipment, where things can go badly if you don't understand what's going on. For example, I took ~350 volts to the hand during a lab exam, simply because I didn't know how to use the apparatus. I had a friend who did something very foolish with magnets and ended up in the ER. However, a lot of people take physics because they like the lab element. Especially people who want to go into x-physics fields in further education (i.e. bio-physics). And some people like seeing things "in action" and playing with expensive machines.
2) You will have to be able to do math. You'll probably take about 1/2 as much math as a math student would. At minimum multivariable calculus, differential equations, linear algebra and possibly calculus of variations. You will also have to know how to code on a very basic level, and use functional programs such as Matlab, Origin or Mathematica. Furthermore, some physics courses will be essentially math courses, just in the physics department. I took a course on Fourier analysis and one in quantum mechanics, both which were completely math. But also, if you like math too much, especially in a sense that you are "wowed" by it, then physics might frustrate you becuase you won't necessarily prove things, and it can get a bit hand-wavy and such. I am a math person, and it always bothered me how the math presented in physics felt "incomplete."
3) If you are a heterosexual male, there will be little oppertunity for dating within the department.
4) Metaphysics is kind of frowned upon within most physics departments. The lectures get disrupted by people wanting to bring non-physical concepts into the classrooms (that aren't the non-physical math elements necessary for the degree). So it can teach you a lot about how the world works, but not why it works in this particular way (in a profound sense).
5) Is there an element of physics that particularly interests you? If so, I'd find a school with reputable staff that know it. If you want to do astrophysics, find a department with decent research in the topic. I first wanted to do astrophysics, but went to a school that only taught one course on it, and the faculty kind of joked around about that course.