In my book, there is no really "bad" food (unless you are talking about chemical stuff, which is a different story altogether).
Cows are important to some countries, where they are able to harness nutrients from foliage which are not edible to humans, or survive terrain that cannot be cultivated. In my country, we do not have a long history of cow dairy, as our varieties hardly produce milk.
However, I think that preserving the health given to us by milk depends on the diet of the cows and how the milk is processed (or not).
Take note that cows are ruminants but are often fed (aside from chemicals) grain instead of grass, this severely fucks up their digestion.
Raw or merely boiled milk does curdle and can later on be made into cultured butter with no addition of enzymes. If you are using store-bought, you need to add enzymes, because the thing is literally dead. No doubt this affects the good we receive from it-- something you cannot really glean from a nutritional chart.
Many of the food we encounter now cause allergies and negative effects because of the difference in how they are produced. For instance, eggs still look like they did 90 years ago, but the content is much different. While it used to be the first thing they would give babies in my country, now doctors advise against it because of all the ill effects. Studies show that because of poultry feeds, eggs today have twice as much cholesterol and are devoid of many past beneficial elements.
There are many things you can say about dairy, and no doubt countless people have developed reactions, but I would say that before drawing conclusions on a food group it would be good to have some fresh, wholesome versions of our supermarket friends, in the quantities dictated by natural production (milk and cheeses 5x a day everyday is not too feasible in this case).