^^ Affinity =/= agonism. Affinity is measured with a dissociation constant, so just like acids and bases, the formula will look like X = A*B / (A+B). In the case of protein ligand binding, this takes the form of K = P*M / C where P is protein, M is molecule, and C is protein/molecule complex. All of those variables, including K, are measured in terms of molar concentrations. So the dissociation constant is in this case determined to be the concentration of the drug/molecule required to reach a state where the concentration of protein bound to the drug equals the concentration of protein that has no bound substrate. Alternately you can think of this as a state where half the receptors in a given set are occupied by a drug and half are not.
What that means for potency and drug mechanics is as follows: We can't directly measure the interactions between a molecule of drug and a protein, the receptor in this case, to see how strong the attraction is / how likely a drug is to bind. So we measure the dissociation constant as proxy for that value, looking at a population of receptors instead of just one. So if a drug requires a concentration of X to reach a state where half the population of receptors are bound, that value X acts as a stand-in to determine the affinity of that drug for the receptor, because a drug that needs a higher concentration to bind half the receptors obviously has less affinity for those receptors than one that can bind half the receptor population at a lower concentration.
This is why Ki values are smaller for greater affinity. The smaller value indicates that the concentration necessary to bind half of a given set of receptors is low. If the concentration is low, then the dose must be accordingly small in comparison to something that requires a higher concentration to bind the same number of receptors, since Ki values examine only receptor/ligand interactions and don't take bioavailability or BBB-crossing ability into account.
Agonism level on the other hand zooms in further, exchanging a variable concentration of drug and a population of receptors for a single cell. Agonism is all about how strong the response is *within* that cell when a receptor on its surface (technically the receptor actually extends past the membrane in both directions) is stimulated by having bound a drug molecule.
Receptor affinity is a measure then of how much drug is necessary, in terms of molar concentration, to bind half of a population of receptors. In this manner it is a proxy for how well a drug binds and how attracted the drug and the receptor binding site are to one another. Agonism is a measure of what happens to a cell when binding does happen, regardless of external concentration of that drug. So you can have a drug that is massively potent, but only a partial agonist, like LSD, or a drug that is very impotent and requires a huge dose but is still a full agonist, for which I don't have an example on the top of my mind.
But the important point is that these quantities are completely independent of one another. It is true that a given functional group or moiety in a molecule can have an effect on both binding affinity and agonism, but the two values are nonetheless not the same thing at all. Efficacy, which you mention, actually is related (inversely) to agonism, and affinity is a separate quantity altogether.
A good primer on the subject is wikipedia, see the following links:
http://en.m.wikipedia.org/wiki/Receptor_affinity
http://en.m.wikipedia.org/wiki/Agonist