It's the fear that the car waiting to turn left, across your lane of traffic at the intersection, will go ahead and do it (to beat his analogous yellow light) thinking you are going to stop, and then you WILL crash (and not a rear-ending). At least this is what I notice when I decide whether to go or stop...
If the assumption that all cars will NOT stop for yellow were common, "left-turner" would know not to turn. But it's not. It's a gray area. SOME PEOPLE REALLY WANT TO TURN LEFT ON THAT POINT IN THE TRAFFIC SEQUENCE!